Medium.
Which one should you be using?
For me it's the one that doesn't mess up my grip/sight alignment.
If I have the right grip panel in the pistol frame - or if the pistol fits my hand right (CZ85/P01/CZ75 Compact) as well as my XDM and M&P and 1911 handguns - also use the medium grip insert in the M&P FS pistols, can't remember about the XDMs, that was years ago) then the sights are aligned when I bring the handgun up to aim/shoot whether the target is in front at shoulder height - or down on the ground off to the left/right, or up high off to the left/right.
Never had such an issue till I decided to play with the inserts in the M&P's and see which ones felt best in my hand. It was the large insert. Pulled out the medium inserts in both pistols and installed the large. Felt good. I was happier. One day, while walking the dogs I decided to do some dry firing, so I unloaded the M&P 9MM and put it in the holster. Pulled it from the holster, aimed at the light on the big trailer down near the wood line and it was "good" - click, target hit, if I'd been firing.
Racked the slide put it back in the holster and then pulled it out and aimed off to the left at spot of yellow leaves in a brush pile.....WHERE's THE FRONT SIGHT?

I had to wiggle the pistol around with my wrist to find the front sight and line it up. Weird. Racked the slide and repeated it. Same thing. Had to "find" the front sight. Racked the slide and aimed off to the right. Same thing. Low, high, high left/right, low left/right. Anywhere but directly in front of me and the front sight did not come up lined up with the back sight.
Sure, i could have kept at it till my muscles/eyes/mind learned how to have my wrist angled correctly regardless of where the target was - but, then what would my other pistols be "doing" when I dry fired them high left/right or low left/right? I figured then the other pistols would be "off" so I put the medium inserts back in the M&P's and left them alone.
Try it. See if the insert makes a difference in your pull from the holster and aim off at an angle at a target sight picture/sight alignment. If it doesn't, you've go the right sight insert in. If it does, you make the choice. "Learn" the new angle/hold and chance things being off with your other handguns or work through the inserts till you find the one that works and doesn't force you to change.