I can see the value in starting with a .22LR handgun. The basic of handgunning are very effectively practiced with a .22, and if you desire to, you can do a little small game hunting with the .22 pistol/revolver. Shooting squirrels from various field positions is another aid in becoming a good handgun hunter.
With heavier bullets, 158 gr. and above, the .357 will give fine service as a deer hunting handgun. A Ruger Blackhawk in .45 LC is a very good choice for someone that handloads. I find the .45 to have a more pleasant sound than the .357, even with hearing protection.
Good luck, whatever you choose,
Doc