That's how it comes out, they can be real tough, the smith that reamed mine to .45-120 ruined the hood removing the sight and had to buy me a new one. Seems to me it will come out best to the left, but don't quote me on that, it's been a while since I replaced mine with a .404".
Wayne York had a heck of a time getting the sight out of my .38-55 Target barrel so he could rebore/rechamber it, and he's a pro!!
I bought a brass drift punch that's made for dovetail sights, it works pretty good.
Tim
Brass Sight Drift Punch