Do yourself a favor and get a sight pusher.
Here's the one that I used when I replaced the sights on my SP01 and my TS. Best $40 I've spent on gun related tools yet
That's a nice looking unit. I always drift mine with a punch but when I have to make adjustments, I always end up chasing the rear sight back and forth. I try to measure it's location with a caliper to see if it moved or how much before I reassemble and head outside to give it a test run but I should invest in one of these.
Like others have said, if you're installing tritium sights, you probably don't want to bang on them.
To get the old rear out, I always put a drop of oil in the space between the sight and slide. Once things get moving, the oil helps keep the moving.