Longwinters, You're right. Factory ammo won't have filler material, nor is it needed when you have powder bulk at 75% or more of case capicity. Filler is a good choice for "problem" rifles that shoot loose groups or for the gun you want to really fine tune. It will give your loads a more consistant velocity and that usually means better accuracy.
A customer brought me a Winchester Mod 70 in 30-'06 that wouldn't shoot better than 5" groups with several factory loads. I floated the barrel and made sure all the screws were tight then started with a known good load of Win 748, mag primers and 168 gr bullets. My load shot 3.5". I annealed the cases and seated the bullet out to .010 from the lands and that got it under 2". Then I added filler and the groups printed less than an inch.
In your case, when you have a rifle that shoots 1" or better groups, why bother with extra steps. If you've read some of my other posts, you will see that I'm obsessed with accuracy and will do whatever it takes to make a rifle shoot straight.