Handguns, especially single actions, can be very sensitive to grip, and bench technique. Depending on how they are held and rested, groups can vary wildly from what you would get shooting off hand. The bench can do a great job in showing you your most accurate loads, just don't expect groups to be the same POI of hand held.
To minimize bench error, I usually hold the gun two handed, just as I would shooting off hand, and rest my wrists / lower forearms on sand bags, allowing for a steady hold, and for the gun to recoil as close to normal as possible. If you are resting the guns barrel on a hard object, recoil could certianly cause the gun to jump and group high.
Try some 12 shot groups offhand at 25 and 50 yards. They might not be tight, but once the sights are adjusted so the group is ballpark center of target, then the gun is where you want it to be for off hand shooting.
Larry