Couple other things that could be a factor:
The different loading will have a different perceived recoil, and you may be "pushing" the gun to compensate for a flinch. I've seen shooters get fairly consistent results by self-compensating, but change the load, caliber or gun and they start getting different results. Dry fire practice will reveal this: put a coin or empty cartridge on the front sight. If it falls when you break the trigger, there's your culprit.
Also, if you shoot Isosceles, dropping the left elbow can send shots to the left.