I've lapped them down with a green scouring pad before, both inside and out. Takes a while, but you don't have to worry about taking off too much metal. I stopped when I got it to the point I need a barrel bushing wrench to out on or take off. Take the inside off to the point the barrel slides smoothly back and forth dry. The barrel may take a pass or two to smooth it up, but not much. You're trying to polish it, not relieve it.