the 204 ruger is the perfect varmint gun. i wasn't convinced, but after seeing coyotes shot with a .223, a .22-250, and a .204, the 204 wins hands down. better ballistics than the .223, very close to the .22-250, with less chance of hide damage as well.
there is no gun that makes a good varmint round and deer round. the .243 does *ok* on both, but excels with neither.
if you want a deer gun, buy a deer gun. if you want a varmint gun, buy a varmint gun.
if you bought the .25-06, you bought a deer gun. that is, unless you don't mind excessive hide damage on your varmints. if you're not concerned about that though, then buy any caliber you want.