You're using too much primer. I shoot 12.5 grs of Lil Gun with a Rem 6 1/2 primer. I saw an INSTANT improvement from just over an inch to shooting bug-hole groups by switching to the milder primer.
Also, seat them long. Seat them out to just off the rifling with the 40 gr's.
I gave a handful of my loads to a buddy to try in his new Savage 40 single shot, and they shot bug-holes in his gun too.
Hornady 40 gr Vmax, Rem brass Rem 6 1/2 primer, seated long (like 1.9" if I recall-I can look it up for you in my notebook if you want), and 12.5 grs of Lil-Gun.