i'd say that one Possible problem is working with a scope that is improperly set for parallax for 100 yd' shooting -- such as a black powder scope set for 50 yd' parallax, supposedly, that is poor at 100 yds'. but, that's a stretch. are you shooting the rifle off the same rest/bench? i can see where a set up on a bench can be very problematic for someone who isn't paying attention to details....
2900 fps' out of a 22"-barreled Handi' might be a bit too much to expect in an '06. not saying it can't be done; just saying that it could be tough to achieve without paying for Federal High Energy or Hornady Light Magnums....or knowing what handloads your rifle will like.
in any event, good luck with your rifle,
ss'