I agree with both replies!
Simply switching barrels, or removing/reinstalling barrels on a frame, you shouldn't have any zero problems. The poi should fall within the mean average for that barrel.
Where you WILL get impact changes is in the forearm attachment/reattachment process. Unless you have a well-bedded forend, and can get screw tension the same, you will cause changes in impact. I have a 10" bull barrel that will shift about 3" depending on how the screw is tightened on the Pachmayr forend. When using the pillar-bedded wood forend, there is no poi shift.
Either bed forends for each barrel, use tape, washers, or whatever, but eliminate this variable. Your zero problems should go away.