I used to try to hold the roll pins (any roll pin going into a gun) with a pair of needlenosed pliers till I got it started.
Then I got an AR15 kit that had a roll pin starter punch and never looked back. The needlenosed pliers stay in the tool box when installing roll pins now.
I used to use regular punches for everything. Till I ran across a set of roll piin punches with that little button on the end that fits the hole in the center of the roll pin and helps keep the punch centered and helps hold it on the end of the roll pin. I even bought a set of roll pin starter punches to "save" me from bending the smaller pin diameter punches.
I used to use a small hammer to hit those punches. Sometimes it would skip off/miss the punch enough to smack a finger and I got tired of getting smacked fingers so I bought (what I've been told is an auto body repair hammer) a hammer with very large striking surfaces on both ends of the head that still isn't very heavy. No more sore fingers and no hitting of the frame/slide either. The striking surfaces on the hammer are many times larger than the back end of a punch.
Not real heavy but it works.