Is it (safety and cam on the left side safety shaft) rotating fully into position under the sear?
I'm not sure if you can see the detent and spring in the left side of the frame or not (with the slide/barrel, etc off the frame) but if you can you may see the detent is not moving all the way forward to help hold the safety lever in the SAFE position.
Fitting a sear is a PITB process for me. I did it once, took just over twenty times of try it, remove it, stone a bit of metal off, install it, try it and repeat. Like you, I wanted it perfect. Too much metal removed and you need a new safety lever. Not enough and it may not work correctly. After that first one I started buying the adjustable sears from CGW and those are super easy to get right.