Revo, I feel for ya. This is a subject that I have been struggling with for a while also. Most of the good points have already been discussed.
If you are going to buy a new gun, the 454 will cost slightly more. Less than $100, but it adds up.
If you are going to buy a used gun, the 454s seem to be more common and plentiful. Some have been shot very little.
Reloading for the 44 will be cheaper, 454 brass is expensive, bullet selection is about even. Maybe a slight advantage to the 44.
You can reduce the power of the 454 to whatever level you desire. You can only increase the 44 so far. But for your needs, it sounds like the 44 would be plenty.
I have given up trying to decide which one would be more practical, and have decided to just get a second job so I can afford several. I think I will start with a short barreled model 97 in 45 Colt, then maybe a 7 1/2 inch 454 with a scope, then maybe a 9 inch 357 for punching paper, then maybe...
...you get the idea.
Picking the caliber is the best place to start, but then you will have to pick a grade, a barrel length, grips, sights, ports or no ports, trigger job...
...it just never ends. My best advice is start looking for that second job.
Or just flip a coin.