Have you tried to drift it out from left to right or right to left?
I think that the dovetail is tapered to keep the twist of recoil from changing the front sight's position. If so, it should be drifted out to the left, as seen from the rear (breech) end of the barrel.
Of course, I could be mistaken. I'm sure someone will be along to correct me if that's the case.
There is a tool called, I think, a sight press. It is basicly a tiny screw jack that hooks to the barrel and presses the sight whichever way it needs to go. works a lot finer a touch than a mallet and drift. You may need to hunt one up.