Shooting deer with a 32 or a 40gr. pill of .2"caliber would, in my worthless opinion, be just be wrong. Eliminate the 204Ruger from your consideration...unless you want a dedicated vermin barrel. My opinion of the fast-stepping 22's used for deer, is only slightly less negative. I imagine a few posters will readily chastise me. I stand by it nonetheless.
The 243 is a really useful cartridge for dual-purpose usage. The ammo choices are plentiful, and can be game specific. The 25-06 is a wonderful, flat-shooting caliber, great for long-range varminting and or deer. I personally would give it the nod, if'n I was stuck with a choice of only one barrel.
Although the venerable .308 is a fantastic all-arounder, the extra bit of recoil, even though only moderate, could be a factor to some.
Just my $.02, based on actual field experience, not the latest magazine article.
Incidentally.....500 yards is much, much too far for them all. That is, of course, unless you're special or something.