First of all, and especially with handguns, you need to wear hearing protection when shooting. I generally wear ear plugs AND ear muffs when target shooting and also wear hearing protection when hunting.
I get 1 to 1.5 inch 3 shot 100 yard groups with my 7mm/08 Encore handgun and my 460 S&W Magnum Encore handgun does even better (under 1 MOA). Neither gun has ever seen factory ammunition.
As for recoil, in a carbine setup, the 460 S&W Magnum shouldn't be too bad.
All of my Encore and Contender barrels except one are T/C barrels. For a custom barrel, I would go with SSK Industries, Match Grade Machine or On Target Technologies. If you get a custom barrel, make sure the contour will work with the forend you wish to use.