If the barrel was very warm, it can affect groups big time, not usually so much with the big bores, but the best accuracy is usually with a cold barrel for the first few shots, then forend fit and how the forend is rested will affect group size more as it warms up. That's why it's recommended to shoot without the forend to diagnose what's causing the inaccuracy. Read all the tips in the Handi Basics sticky, all are important, not closing the action firmly, specially on a newly fitted barrel can cause problems too. But it may just need some more range time too. The good group you got at first may indicate a bad scope/mounting issue, or you were lax in making sure it was locked up good or not pulling the trigger all the way to the rear each time which can also cause problems with light pin strikes.
Tim