The manual safety pin is staked in position. If you look on the underside of the beavertail you'll see a slight indentation where it's been staked. Depending on how hard it was struck determines how tightly it holds the pin.
Are you using a brass punch to drive the pin out? You may have to use a steel pin punch. I'd apply a couple of layers of tape over the pin to protect the safety from marring, mark the tape where the pin is, and drive it out. Reinserting the pin is just as firm as removing it.
Good luck!