Metal and paper targets, squirrels, cottontails, prairie dogs: .22LR/ .22 Mag.
Jackrabbits, groundhogs, fox, PERFECTLY HIT javelina, etc.: .32 H&R Mag. Personally, I'd lose the .32-20 idea. A .357 is also a great choice for this category.
I'd also prefer a 5.5" barrel, for the extra sight radius. I use handguns as the primary hunting gun, not as a sidearm. Opinions and uses by others will vary, of course.