Are you trying to drift for windage or remove the sight completely?
Drift right to left as viewed from the rear. Use several layers of masking tape on the sight. Get a large brass punch and and a decent hammer. Put the slide in a vice (soft jaws or leather padding or plastic padding) and get it moving with the hammer. You may want to remove it completely to ensure there isn't loctite holding things up.
Once out, clean the dovetail and fit the bottom of the sight such that it drifts in by hand about half way. I use a flat surface (glass plate is ideal but anything that is flat will do) and a reasonable grit sand paper (300-400). Sand the bottom of the sight in a random motion checking frequently for fit in the slide. You can't add material back so go slow and check fit frequently.
Once it is fit you should be able to use the sight pusher.
Cheers,
Toby