Are you planning on shooting more varmints than deer, or more deer than varmints? If you are going to shoot it more for varmints, I'd go with the .243. It recoils a bit less than the 25-06, which adds up during a day of alot of shooting. With the premium bullets available the .243 is better than ever. With a Triple Shock X or a another deep penetrater, I'd take any reasonable shot at any deer, or bigger game too, for that matter. If you are just taking pot shots at whatever crosses your path while deer hunting, I like the 25-06. Be warned, both of them are hell on barrels for prolonged shooting.
I think that the .260 or the 6.5x55 are the most versatile of the list, but the lack of factory loads in both of these calibers is what's holding them down. Moderate recoil and awesome ballistics make them stellar rounds though.
Another round you might consider is the 6.8 SPC. I was actually looking for a rifle for the uses you are talking about It's mostly going to be used for predators and such, maybe a little moderate range deer hunting too. I was about a hair away from having a semi-custom .243 built, then decided, what the heck, I can get a bolt action 6.8SPC in a Remington 700 for about half the price (it's similar to the 700 LV-SF, but in 6.8 it's called a Tactical, and it's blued), and ammo is priced the same or less than .243. It should be better on barrels, has less recoil and blast than the .243, and ballistics very similar to the .257 Roberts.