You are absolutely positively correct.
If you are looking for one cartridge that is a happy medium, just a notch below the .30-06, that can handle anything from Elk to Varmints, then the .308 Winchester is the answer.
In standard factory ammo, the .308 has about 93% of the power of the .30-06, with 16% less recoil. (Recoil in ft pounds for .308 is generally 18 foot pounds, and for the .30-06, 21 or more foot pounds.)
All of the he-men tell me that they notice zero difference in recoil between these two rounds. However, I only weight 150 pounds, and I feel an enormous difference in the recoil.
And, if you ever want the .308 to have the equivalent amount of energy of a .30-06 for Elk, you can simply buy the 168 grain Hornaday light magnums in .308, which virtually duplicate the energy of a standard .30-06 in 180 grain. If you go back and look at hunting and shooting magazines for the past 3 years or so, you will find lots of articles where the .308, with premium factory ammo, was used on Moose and Elk, killing them stone dead in short order. And, Boddington took a Scout rifle to Africa in .308 and shot 20 head of the largest African plains animals, including greater kudu and wildebeast. Of these animals, 19 were one shot kills, and one required 2 shots.
Get the .308. You will be very happy. Yet, for a nice cheap and relatively quiet varmint rifle out to 125 yards, I highly recommend that you also add a .22 magnum bolt action. This would be instead of the .223.
Best, Mannyrock