I'm going to comment about the accuracy when you switch barrels, but I want to be frank with you, I think the last time I switched barrels was about a year ago. I've been trying to get dedicated receivers for each barrel.
The last time, I switched barrels, I took off a .223Rem barrel, and put on a .243. Both barrels were originally fitted to the same frame. Accuracy on the .243 was within 6" (IIRC) from the original POI. The POI was definitely different, but it would have still hit a deer sized target at 100 yards.
From what I understand, guys that regularly switch barrels back and forth on a frame usually use a FAT wrench to insure that the same amount of torque is applied when changing to a different barrel. This ensures that almost everything is the same as when the barrel was originally sighted in. Still I would guess there would be a little deviation in the POI.