Dang, if I keep talking to myself and studying the gun I think I got it fixed.
I swapped safeties two or three times and sometimes one worked and sometimes it didn't and vice versa. The problem starting coming and going until I noticed the trigger pivot pin was sticking out about 1/8".
I pushed it back in and so far everything works fine with either safety installed.
I am going to the range tomorrow, that will be the true test.