My suggestion, since you're going the Handi route, is get a 223, or 22-250 or 204, for the varmints and then add a 7-08 or similar barrel for deer. The 243 and 7-08 are very close to each other, but the 7-08 can use a lot heavier bullet, cause they are available.
The recoil of the 7-08 is very tolerable in the Handi, my 280 Rem barrel kicked a lot less than I expected and it has a bit more power then the 7-08 (same bullets, more speed). By the way, I was getting over 3100fps with 139gr bullets with loads well under max listed loads. Why limit yourself to the 7-08? Recoil of the 280 is very tolerable, greater power/range and better for bigger game like elk if you desire. Plus you get a 26" barrel. My .02 worth.
Yes, it take about one minute to exchange the barrels once you have them. remove the forearm screw, break open the action and lift off the barrel. Do it in reverse to install. It takes about that long to do!