Did you shoot the rifle for group before adding the o-ring? I take it you were not satisfied with it's performance. A high percentage of these rifles shoot better without any o-rings, spacers, rubber bumpers or washers under their forearms. My 308, 243 and 25-06 shot better with a naked forearm, snugged tight. This is such a popular subject it could be that some shooters are jumping on the forarm before checking if its needed.
If you can routinely shoot 1 14/16 inches at 50yds, then this just about a 4 inch group at 100. (The distance most evaluations are make at.) Now, most of the Handi 308's will do better. But this is a good point to start trying other loads. Shoot 3 shot groups with the same crosshair hold. Forget how many inches you are up or left of anything. Just so the group is on paper. Now mesure your group. Record same. Let barrel cool. Repeat x 2. Average the three groups and that will tell you what you can expect from THAT load. Clean barrel. Foul barrel, repeat with different ammo.
Each rifle is a law onto itself. Some 308s will shoot better with 150, 165 or 180. I have found that most I have worked with did pretty well with 165. But you have to try each, unless the first load fills your expectations. What are you expecting from this rifle? Big game hunting? Target punching? Varmit shooting? The magical 1 inch rifle is desired by everyone, but in reality most big game could be taken with a 2 inch rifle. If big game is your goal, it is not far from your current under 4 inch to the needed 2 inch. You could very well be there with just finding the correct ammo. And remember that as you shoot/test/clean, you are breaking in that new barrel, and this could start shrinking your groups. Not to mention the benifit of additional experience.
Hope this helps.
Wally