9mm is just the bee's knees. A quality 9mm self-defense round, especially if you go ahead and make it +P, has terminal ballistics that are pretty much 'good enough' by any standard. Even in +P form, it's so easy to control that I don't see why anyone would go with .380 or .32...maybe those rounds were relevant when 9mm mostly dwelled in full-size duty pistols, but that's just not the case these days.
As for the 'superior' rounds, i.e. those with greater muzzle energy:
Standard .45 loads, especially those with the heavier bullets, are actually pretty close to 9mm+P in muzzle energy. It takes the higher velocity rounds to really give it any edge; whether that actually translates to a more effective round is debatable. Recoil is certainly more noticeable, and I don't find that the argument of a 'bigger hole' really holds much water; not a lotta difference in the actual size of expanded .45 hollow points versus 9mm. Still, I like .45...but I'd just rather have the extra 9mm rounds.
.40 is just terrible to me. In the only apples/apples comparison I've experienced (USP compacts), recoil is worse than .45. It's snappy, it's loud, and, yet again...am I really any more likely to incapacitate a target with it than with 9mm+P?