Hello Lakota,
I certainly agree with KN, your barrel should at least be free-floated. You can use washers, pillar-bedding, a hangar-bar, or even PVC tape - but use something! Also, if you have a wooden forend that is a longer than normal (somewhere in excess of 8-9 inches), the wooden forend might be introducing some harmonics into the equation that you don't really want - you may also want to consider a laminated hangar-bar forend (preferably glassed-in) with a sling swivel stud for more rigidity but this is not mandatory.
I have seen the following be a problem (because I'm the one who did it!). When installing a sling swivel stud, don't remove any more wood than necessary - keep as much forend wood thickness as you can between the external portion of the stud and the nut holding it on from the inside, this can be critical I found out. I removed too much wood on one of my installations, and had to glass-in the nut itself and fill the hole with Marine-Tex to keep the stud from moving slightly (forward) during recoil. Once it was glassed-in, the rigidity and associated accuracy returned, but of course I probably won't get the stud out without some high-explosive of some sort!
I hope this information is helpful.
Good and safe shooting to you!
Javelina