Shooting off of a rest does three things. It changes your point of impact, versus shooting free hand, it obviously makes it easier to hold the pistol steady, and it tames down flinches. There's apt to be a radical difference between rested group size and free hand group size, especially for shooters that are early in the learning curve.
The best advice that I can give, is to put those heavy shooters in the safe for a while, and shoot a 22 lr until you have mastered it. When you go back to the 44 and the 454, shoot 44 special and 45 Colt cowboy action loads, or light reloads, until you master them. This should be a process that takes months, if you want to get really good. Gradually step up the power of the loads that you shoot, and when you're back up to full power loads, you'll be one heck of a pistol shot. Anytime your groups start to open up, put the heavies back in the safe again, and go back to the 22 for a while.
I hate to use the "f-word", but I suspect you're "flinching" from the blast and recoil, and throwing your shots high.
For rested groups, sit down with your back to a wall, and rest your arms on the insides of your knees. Your point of impact will be much closer to your free hand point of impact, compared to bench mounted rests and sandbags.