I have two Huntsman, and one has a slightly looser bore than the other. You might want to try to use a load, other than the 250 grain Shock Wave bullets (with yellow sabot), some sabots just fit tighter than others. If this gets to be too much of a problem, you might consider using Harvester crush rib sabots.
I've been using the T/C Cheap Shots 240gr. all lead hollow points, black sabots. They are less expensive than the Hornady 240grain XTP bullets, and the point of impact is close enough to not need to re-zero when you change bullets. Yes, they are a little snug, and yes, they get harder to load after a couple of shots. You might try to run a spit patch thru the barrel after the third shot, this will remove some of the gunk, and make it a little easier to load.
I recommend that you use a bullet starter, and a range rod. Using these make it easier to load. Save the OEM rod for when your hunting.