To quote myself, “My favorite deer load in the 30-06 is a 165-grain Hornady, Nosler, Remington C-L, or Speer pushed by 56 grains of H414. This maybe a max load in some rifles.”
Some years back I spent a lot of time developing loads using 165-grain bullets for my 30-06. I obtained accurate loads with the 165 grain Remington C-L. The first was a case-stuffing load of H4831, and the other was the above H414 load. I prefer the H414 because it produced higher velocity with “slightly” less recoil. There is nothing wrong with the 165 grain Remington C-L in the accuracy department. Of course I would tell you different if you were my next-door neighbor. I would volunteer to dispose of them for you.
Experience tells me safe loads that have a Load Density over 90% produce the best accuracy. While IMR4064 is a good powder for the 30-06 at maximum the Load density is 85 or 86 percent. It is a better powder if you are concerned about functioning in an autoloader, otherwise you will obtain better results with a slower burning powder.
I do have a question; does your rifle produce acceptable accuracy with some other load? If so is it factory ammunition, or handloaded ammunition?
Is the problem the rifle, the scope, or some other factor? A build up of copper in the bore?
I have a 7MM Magnum that does terrible with factory ammunition from the big three manufactures. I can load the same cases with H4831/AA8700/H870 and obtain very good accuracy with bulk factory bullets. It is as true today as when I bought the rifle 25 years ago. The scope was okay, the bedding was sound, but it needed the right loads.