You're right, a mold will cast only "close" to the designated size, actual diameter will depend on the actual mold, the temperature of the mold and of the lead and the purity of the lead, and on casting technique. That is why it is best to use a mold which casts a bit oversize, sizing them down to an exact consistent diameter. From your description it does seem like the mold you are using is a pretty good fit so maybe you should work with it for a while.
Although the "standard load" for military use was around 60 grains of powder, many serious competition shooters like much lighter powder charges, in the 40-50 grain range. Small changes in the powder charge weight and powder type can make a big difference in accuracy.
Keep us apprised of your results. I find it interesting that sources like the History Channel and gun writers who have never fired a minie all speak of how the Civil War rifled muskets were "accurate to 300 yards". Well yes, they can be accurate, with highly developed loads for the individual fine tuned rifle, but as you are learning, it can take some work to get there. For the ordinary issued musket with standard issue "one-size-fits-all" paper cartridges, I wouldn't bet money on hitting a man size target at 100 yards and after the bore is fouled I wouldn't bet much money at 50 yards.
I would however, bet you can get better accuracy instantly, from the very first load you try with a cloth patched .570" ball. They weren't designed to be round ball guns but they do shoot a patched ball quite nicely and without all of the fiddling and experimentation required to find an accurate load with minies. Try a .570" ball, spit lubed pillow ticking patch, and 80 grains of 2f Goex or Pyrodex RS.
Keep plugging away, keep us informed on your progress, good luck and have fun!
OH, and don't go back to Bagdad Joe!