This is trigger pull. If you don't pull exactly straight back, you are generating a lateral force with your trigger finger that tries to move the muzzle laterally, and you're grip/hand reacts with an opposing lateral force to compensate. You can't see this because the sights don't come out of alignment, but the tension is there, and when the trigger breaks, the lateral force from the trigger finger disappears for just a moment, creating a lateral wobble.
Heavier bullets move slower, leave the barrel later, and exaggerate the POI impact of any muzzle movement. With the vertical shift of POI with heavier bullets, that's normal. With horizontal shift, that's the shooter creating that movement.
Every shooting tutorial and every bullseye shooter emphasizes pulling the trigger straight back, straight back, STRAIGHT BACK, ad nauseum, because if you don't, the release of that lateral tension at the moment the trigger breaks moves the muzzle at exactly the worst moment.