Or, take the vise off the bench, mount it to a piece of 2X8 board, set the board/vise in the floor, stand on the board, bend down, use the punch and hammer to remove the sight. Between the flat piece of board on the floor and your weight on the board it should be pretty stable, if you can keep the punch on the sight and hit the punch with the hammer, something's gotta give.
I've (twice now) laid an AR15 upper receiver on the floor, put my foot on it and installed the barrel nut. Finally bought a barrel clamp fixture to go in my bench vise and do all that using a vise now, but my big foot and some of my (big) weight was enough to hold the receiver in place those two builds.