You have almost exactly the same situation I had with a friends .308. Those high points on the barrel face will pean down with shooting and a gap will open up and widen between the face and the standing breach. This will cause the barrel to bounce right and left when fired which is NOT good for accuracy. Acuracy will probably get worse as the situation is aggravated with continued shooting.
On my friends rifle, and I put a lot of bullets down range from it trying to figure out why it wouldn't shoot, the right side of the spacer was actually peaned down from it banging into the right side of the receiver.
When I finally realised what was happening, which was after emailing Tim about the problem about a thousand times, I squared up the barrel face and shimmed the action. She's flush and nice and tight now!
Load development for this season will start shortly.