That spring arm fitting down into the notch in the left side safety shaft does most of the work of keeping that safety in place.
You need to try to push the safety in farther and maybe see if there is enough room in the frame that the sear cage can be moved slightly towards the left side of the pistol frame. If you can't see that notch, you've got a problem that needs to be addressed.
And the safety won't fall out until you're up the $h!t creek and need the pistol to work - remember that safety is what holds the sear cage in place. If the sear cage can move the pistol may not fire when you need it to.
