I started with "regular" punches and would often bend the smaller ones.
Then I bought a set of "starter" punches that have short tips on them. I get a pin moving with the short punch, keep an eye on it so I don't run the tapered/larger part down against the surface (or jam it into the hole) and switch to the longer punch.
A bigger hammer. Absolutely. I started off with a tack hammer. Small hammer for small pins, right? Small head and light. Smaller hammer face makes it harder to keep the face of the hammer on the punch and you have to move the hammer faster to get the same impact on the pin/punch.
I bought a 2 or 2.5 lb. shop hammer. Short handle, big head. Easier to hit the punch squarely. Even a slow speed it hits the punch harder.
Good luck.
Oh, if you're not in a hurry, start putting a drop of lubricant in the pin hole on top of the slide and letting it sit a bit (few hours). Then use the punch/hammer to shock the pin. If it doesn't move, put another drop of oll in the hole and let it sit again before repeating the attempt to remove it.
When I do a thorough cleaning of a gun (like after a range trip) I put a drop of oil on every pin and screw prior to wiping off the excess with the oily rag I wipe the slide/frame surfaces off with. Sweat/moisture can work it's way into a hole/pin/screw threads so I like to put a little oil in there when I get a chance.