I put together a half dozen of them and only had one I had to tweak with a shim on the pivot. That's my .357 and it is probably the most accurate of all of them. I also swapped my .45-70 ejector barrel for another member's extractor and both barrels fit their new frames well.
Then, I was going to give a buddy a spare '06 barrrel for his .280, and it wouldn't lock up at all on his frame. So, I guess you never know until you try it.
Buying a used Handi, you can tell if the barrel and frame are factory matched because the last 3 digits of the serial number are engraved in the barrel's underlug.
Pete