I'm also just your typical whitetail deer hunter, and the way it sounds is that you may have changed barrels, and expected them to shoot the same without readjusting the sights.
They could shoot where you had them sighted before, and then again they could just be close, but not close enough, or worse they may be finicky and need a little extra effort, which sounds like what you are experiencing, and the main reason mine are now dedicated to their own receivers.
I've got three Handi's I hunt with and have taken game with, and all have dedicated barrels that I no longer change out with other receivers.
This way, once they are sighted in they stay that way, and my latest rifle is my .357 mag. that I just modified to .357 maximum, and my first attempt at handloads, that I used today to take a nice mature doe at 110 yards, that dropped her in her tracks with a shoulder hit from a Hornady 180 grain XTP jacketed bullet.