The wear you describe is common. In fact, some shim the fore end to make sure there is good contact at the front of the frame. They say they do not want the barrel to drop open when the release button is pushed they want it to have a little resistance to open the barrel. It sounds to me like you are on the right track. Maybe another load/brand of 308 will tighten up your groups a little more. A Handi can be a little picky about what it likes to shoot. You said that you were getting 2 1/2 - 3" groups. Do you mean there was no vertical stringing any more - they were generally round in shape? There are lots more things to try. Bullet seating depth, powder charge change, bullet style/weight. then there is the Handi itself. Some have full floated the fore end, some have bedded the fore end. You might try to take the fore end off and see how it shoots. Better, worse? It could help you to decide if the fore end needs some work and what kind of work it will take. Then there is making sure the latch is clean and dry. You need to snap the action shut smartly each and every time. There is also trigger follow through, making sure you are pulling the trigger the same each and every time. Some get lucky and pull the rifle from the box clean it up good, mount their scope and off to the range to find they shoot nice tight groups. Most however do not find this. There are so many different things to try, some give up. Reread the FAQ. I would try one thing at a time. When you find some thing that works leave it alone then try some thing else, until you find that right combination. Remember shooting technique is very important to a Handi. Follow through on the trigger and make sure you are snapping the action shut with a little force. This makes sure the latch is mating with the barrel the same each time. That in itself can cause vertical strings. Quicktdoo (Quick for short) is a moderator here and has lots of Handis (35 or so). He shows them off in pictures now and then. A while back I noticed he was storing them in the rack "upside down". The older pictures he was storing them in the traditional way, muzzle up. He knows what he is doing when it comes to Handis and I suspect he is storing them that way to keep the oil in the bore from running down into the action and particularly the latch. Only he can answer this to be sure. Good luck to you.
