I've never even shot an -06, so can't comment on that. When we moved up here, I had a 308, and figured it was all I needed. After living here a short while, I built a 35 whelen, which I subsequently had re-chambered as a 350 Rigby.
My personal thought is, if you shoot a medium bore as well as you can shoot a small bore, then why would you want to hunt with a small bore? Who cares that it holds more rounds in the mag, as your experience shows, you've only needed one shot each time. If you can't get it done with the first shot, and have a couple extra for insurance, more in the magazine certainly isn't the answer!
Nothing wrong with the -06 mind you, but I'd personally take a 338 over it any day.