The main advantage of the .50, and it's a big one, is the wide variety of bullets and sabots available in that caliber. Lots of combinations to play around with to find that perfect load.
The .45 is a fine caliber and will certainly do the job on deer with the right bullet. There are just fewer options and they don't show up in almost every shop like the .50 does.
If you like shooting patched round balls, the .54 is the way to go. It's a fun load to shoot and effective out to 100 yards, 125 at a stretch.