First of all you need to get the trigger pull down to something reasonable. No way to shoot POA with 12 pounds of trigger pull - 4" groups or not. I suggest you take that thing apart and clean and polish all the contact points, especially the sear.
Was the gun new when you bought it? If so, did you do a thorough cleaning internally and remove all (that's ALL) the grease and oil in the action parts? Actually, should do that anyway, new or not.
Second, what's wrong with adjusting the sights? But ok, you want to do that last. In my experience shooting with/without Wonder Wads will have no effect on the gun shooting left/right of POA. Changing ball caliber is also unlikely to have much effect, although it's not unheard of. I'd go up to .454 with wads first. Then if you still feel wads have an effect, try a few without wads.
Finally, adjust the sights.