If both factory and good handloads shoot that bad then the problem has to be the rifle, right. 6" from a bolt gun is horrible, I would be looking for something unusual, is the barrel loosening up or is there a crack in the stock. The little things we do to tighten a group don't apply when we are talking 6" groups. If I couldn't find the problem I guess I would start taking it to gunsmiths and challenging them to find the problem with a $50 dollar prize for the guy who can figure it out.
Another idea, have you had somebody else shoot it, can you still shoot good groups with other rifles, maybe you have some new glasses that are screwing you up.