Ammo matters quite a bit. Lots of popular .40 is pretty soft (Fiocchi, PMC, Magtech, Hornady come to mind), while lots of manufacturers including those are making hotter 9mm these days. The range ammo by me, the 40 and 9 feel pretty similar- Fiocchi's 40 and 9mm fmj are loaded almost identically. Full power .40 will have noticeably sharper recoil (Double Tap, Underwood, Cor bon, even Federal and Hornady's critical defense).
I carry a .40 rami for dispatching game, shooting Underwood or my handloads in the rami at the range gets quite a bit of attention...