You are on the right track. Consistency is the primary requirement for accuracy. I think, you need to concentrate more on firing the gun with a clean barrel for hunting. Another thing I noticed as well, is barrel drift. As a barrel heats up, it hits a different point, this is often just from the natural stresses bending the steel ever so slightly as the barrel wamrs up. In my 1884 trapdoor, the first clean shot will be 5 inches high and to the right from where it settles down to be after the 4th shot. light barrel 22 rifles do the same thing.
Anyway, I would concentrate on giving the bore a good swabbing between shots and to not heat up the barrel. Some guns definitely shoot one way clean and another dirty. At target matches, folks often shoot one or two fouling shots before shooting for score, and they swab the barrel between every shot. Measure the powder the same each time, pour the powder the same every time, if using loose powder, rap the heel of your palm against the side of the barrel two or three times to settle the powder, the same every time, load the ball or bullet the same every time, use the ram rod to seat the ball/bullet, the same every time. If you fart before a shot, do it the same every time. It takes some practice. When I shoot, I swab after each shot. Two wet patches and two dry patches, then reload.
What load are you using. Most folks starting out have the mistaken impression that if a little powder is good, more is even better. Regardless of the Maximum loads for which guns are rated, the best acuracy generally comes from the same grains of powder as the caliber. ie, a 50 caliber gun, start with 50 grains of powder. The traditional wisdom was to that double the caliber was the maximum load for a gun, ie for a 50 cal, a max of 100 grains of powder. If you are shooting bullets for whitetail, you really don't need anything more than around 80 grains.
