I've loaded the 85 grain BTHP for years. There are several things you can try. I would think the savage would shoot a little better than that. Make sure the action is all the way back in the stock. Loosen the screws and lightly bounce the butt. The idea is to get the recoil lug to contact the stock squarely. This is where glass bedding the action can help improve accuracy. Also make sure the barrel is free floated and not touching the stock's forearm. A trigger job should help a little. Make sure your shooting off a steady rest.
First, haven't tried H Varget powder, but have had very good results using IMR4831. When you fire a rifle, the barrel vibrates like a tuning fork. The powder type and charge changes the rate the barrel vibrates. The idea is here is to find a powder and charge that make the barrel resonate, or cancels the vibrations.
Second, for a bolt action rifle, after the cases are fire formed, neck sizing instead of full length resizing should help. Cases, when fired, are very close to the true size of the chamber. By neck sizing, the case fits the chamber with the neck more aligned to the bore. When full length resizing, the case lays on the bottom of the chamber with the bullet out of alignment with the bore.
Third, OAL. Neck size a case with no powder or primer, just enough to hold the bullet. Seat the bullet way out. Smoke the bullet with a candle or such and chamber the round. You will see a mark on the bullet where the bullet contacts the lands. This will give the OAL of where the bullet touches the lands of the rife. Write this down and play with the seating depth using this number. While some say this increases pressure, I have found on my rifle, that the best accuracy comes when the bullet is just touching the lands of the rifle. Try adjusting the bullet depth just off the lands first. The idea is if the bullet enters the bore straight, when the seating depth is right, the "jump" before entering the bore won't create a "yaw" on the bullet.
Fourth, try a wilson bullet seating die. This is a hand die, but the bullet is seated with very little run out when using the wilson die. Again, the idea is to the have the case fitting properly in the chamber, the case neck aligned with the bore, and the bullet seated straight in the case with the depth adjusted so that everything is in as close to perfect alignment as possible to the bore.
There are lots of other hand loading technics such as inside neck reaming, outside neck turning, which again, affects the way the case fits in the chamber and the bullet in the case..........case head squaring, primer pocket uniformity etc........For hunting and varmit accuracy, as well as the time involved, the gains by doing these things are relatively small.
My rifle shoot terrible with factory loads........my hand loads makes it a totally different beast!