Why would someone take it out?
For fear that it will disintegrate and prevent the gun from operating properly. I have shot my shadow 2 in USPSA with the buffer since day one without any adverse effects. It’s the same thing as short stroking a rifle.
It’s a personal preference. Some like it, some don’t.
I use the buffer on my TSO also without any problems running either minor or major power factor loads.