What powders are you using and what velocities are you trying to achieve? My 44 mag is .002" over size on the groove diameter for a 44, yet it still shoots jacketed quite well. It is possible your barrel is out of spec, but I woud look to other suspects first. Have someone else fire the gun for groups. Try different ammo.
Have a friend load the rifle for you sometimes with a live round and sometimes with an empty case (don't just dry fire it, you'd be asking for a broke firing pin) You CANNOT know what is comming up, your friend must hide what he is loading from you. If on dropping the hammer on an empty case, the barrel drops forward or you react to a shots recoil that wasn't there; you're flinching. Having a flinch is not a bad thing, it is in fact quite treatable, but if you are in denial about it, then nothing will make the gun shoot well.
Have you tried any heavier bullets? Have you shot any soft lead bullets and I do mean soft, something like 20:1 alloy rather than WW or commercial cast which tends to run upward of 21 BHN or so. The softer didn't help in mine, but that could have been a velocity thing too. the quesation is how does it shoot it. Remington makes a plain lead 400 grain load that is perfect for this task. Every year I read reports of 300 grainers failing on deer, the complaints seem to be that the bullet is too soft for the velocity it is being driven. Might I suggest trying either the Remington 400 JSP or the Speer 400 JSP or an equivilent to those. Also don't drive them like the devil. Instead take a note from days past and try a more sedate load, say something about 1500 fps rather than going for 2000. See if such a load is accurate in your rifle, I have no doubt it will do the business on a hog.
Finally, how tight is the forearm screw? and how tight is the forearm fittedto the lug for the screw? I have found that I get much better results from rasping that hole out so it does not touch the lug, and only running that screw up finger tight, a tad more perhaps but the forearm should have a slight movement to it. I have not found the oring thing to work as well as this. I can get 3" groups from my 30-30 by doing nothing more than tightening that screw down till it's TIGHT!, no good, the forearm must float. If it rattles it is too loose, it should just wiggle a little, not so much as to make sound.
If you are using a scope, are you using the after market Weaver rail? If so, did you file a new slot in it to get the rings in a better position on the scope than the factory slots in the rail allowed for? If so, your slot may not be deep enough, this was the root cause of my scope coming loose on my 45-70 recently. Double check your scope mounting for sure, handirifle is dead on with this one.