I'd be willing to bet your shooting your gun right handed? Typical hand torque for us right handers. (Thank God I'm ambedextrious) Why I don't see it with lefties I can't answer. Anyway, I see quite a few right handers squeezing the trigger toward the left side, instead of straight back. Just a thought, may not be your situation. (Crosshairs leaving the target to the right side?)
Another thought - are you shooting with your dominent eye? If not, you may be trying to compensate for that sub-consciencely(sp?). If you are right eye dominent and shooting with your left eye, you may be trying to aquire your sight picture with your right eye, moving you off target. I actually have to document which eye I use for load development, being "skillfull in the use of both hands" also means you are "skillfull with the use of both eyes". This can be troublesome when you work up a load with your left eye, and a few months later find your point of impact has changed because you're using the "wrong" eye. Trust me - it's happened.