I've learned this a while back especially after fitting several barrels to other frames, and mainly only two barrels that did me this way, was my .270 and my .243 standard contour barrel, and not that it makes any more of a difference than shooting a factory fitted one, it's just that now when I leave the range I want to make sure the first shot counts, and it's just a good practice to get into, I believe.
I'm not much of a bench shooter and I understand that you shouldn't let the barrel get hot, but when I first started the sight in process, I waited a few minutes and shot again until everything sort of came together and then stopped, only to find out as you did, that with a cold barrel, the first shot would be off, and at first attributed it to a clean barrel, and later finding out that even a dirty barrel would net the same result.
I've now gotten into the habit of when I sight in and I'm satisfied, I'll let the gun completely cool, and then fire one last shot, and if it's off, I'll adjust the sight and let it completely cool again, fire it and that shot will be right next to the last one, and that will pretty much make your gun a one shot, point of aim rifle, just as it's supposed to be.