Many variables there.
1) "first 50 rounds" - 50-75 is when mine starting coming in with no prior lapping.
2) "working up loads" - You already noted this but, if you think there is an issue stick with your best load to date for now until the issue is resolved.
3) Stick with the 50 yard range for now. We all want to shoot at 100 with the new toy. If your like the rest of us meaning a) not locking a gun in a vice b) don't have the best quality front rest,target scope,etc. c) shooting from wood benchs or "working up loads". Stay at 50 yards to start and take out some variables. I shoot from Caldwell bags front and rear. With out question I have a human factor of play espiecially the way a handi rides bags. You have even more as your just devolping a feel for the rifle, what grip it likes, the trigger, etc. For what your doing now, 50 yard shooting and development takes some of the variables out.
4) Did you get copper fouling? Cleaned for copper?
Here's what mine likes. Your millage may vary.
a) I get zero copper fouling in my Hornet. It prefers a fouled barrel. I simply pull a dry boresnake through every 20-25 rounds and deep clean with chemicals/rod/patches when it will sit for extended times or as needed. After and during cleaning I have to keep the solvents off the latch and below as best I can and make sure the chamber is completely dry.
b) Mine prefers a tight grip.
c) To achieve acceptable consistent first shot placement and non wondering groups with good loads I tried several things from upward pressure points to floating the end of the forearm. My first factory stock had good lock up fit. I replaced it with a factory laminate and my problems started. This is where I learned about lock up Fortunately I have a home 50 yard range for testing as I needed it for the lock up issues with the new stock. The replacement forearm was slightly longer which made the lock up tighter, effected accuracy in a negative way. If held horizontal and the action broke the barrel would not even drop to the ejector engagement point. I carefully cut down the forearm but took slightly to much off it. Now from the same position it would just flop open past the ejector engagement point with no restriction. Accuracy was negative. From there several times I would put the original forearm back on and test. Accuracy was improved. I then had to shim behind the plastic shoe on the new laminant forearm with alum. foil. When correct I expoxy bedded the shoe,shims to the forearm, allowed it to seep out the sides and wiped flush before dry. This also squared, filled gaps between the show and the forearm base and made a relatively perminate fix. The accuracy was back to original. The lock up was back to original and I learned something about break actions. Nothing I had done had effected accuracy more. Some here know the measurements of proper lock up etc. I did it by trial, error and comparison feel while continuesly fireing many groups on my home 50 yard range for testing (the hard way really, but I had no choice). Here is what I learned by feel. Originally my action from the horizontal position if eased on the barrel would slowly drop to the point the ejector wanted to engage and stop. That was the point of best accuracy and wear it is now. It shot much worst when it was overly tight and somewhat worst when it was overly loose. I notice from handleing Handis in gun shops the actions/lock up on some are what I now beleive is right. Others are IMO overly loose and just flop open. I don't feel many that are overly tight.
Last from there as noted having tried various pressure points,rings,etc. with the original stock still on. I opted to dowel sand the forearm to the lug and add the "O" ring. In field I shoot off sticks hunting varmints and couldn't be happier. My P.O.I. are consistent and predictable. Prior, I had added O rings with out doweling and tried several upward pressure points on the forearms last inch. Some of my best groups were with the upward pressure but, in my humid summer environment, when I hunt Woodchucks the most, and from shooting sticks placed at the end of the forearm, my P.O.I. was moving. Again, it sure helps to have a home range to keep on top of things. If I were a bench shooter and not a hunter and could have afforded a few sight in shots I may well have left the upward pressure point. Maybe the same if I was in a dry climent? But for me the dowel,float & O ring is the best and I have no regrets. I would set up another exactly the same. The time sounds like allot but was really minimal as it was a learning process for me. The doweling,floating and bedding of the lock up time in reality was less than a couple hours and less than 10 bucks. Again my original forearm was correct so one could eliminate that time and learning also.
Just some things to watch for on your Handi rifle adventure. Now that I better understand them and have a feel for them I would get another with out second thought as they are easy and good value.
Bill