"Jawa Tofu" ... dang ... he discovered my Star Wars name ...
Don't give up on that 1861 yet. I'd suggest you try smoothing the bore with JB Bore Cleaning Compound or Iosso Bore Cleaner (in a tube, with the consistency of toothpaste). Or you could just continue shooting it and the bore will smooth naturally from the slightly abrasive nature of black powder. This takes longer, though.
I've learned that some cap and ball revolvers take a few hundred rounds to "settle in." My most accurate C&B is an Uberti-made 1858 Remington in .44 caliber. It was also the most inaccurate when I first got it. At 25 yards, I could barely keep it on a 9" paper plate from a benchrest, when new, but now it will put six .454 balls or 200 gr. Lee conical bullets into 2" groups all day long, sometimes even better.
The Colts must be wedged together tightly, for best accuracy. If you can push the wedge out with thumb pressure, it's too loose. I take a clean gun, unfired, and gently tap the wedge in until the cylinder begins dragging slightly against the rear of the barrel. This reached, then I gently tap the wedge OUT just a hair, backing it away from the dragging.
New revolvers may not be loose enough for the above. As long as the wedge is in tighter than you can thumb-push it, it's all good.
You're on the right track, using a lubricated felt wad, FFFG black powder and .380" ball.
I'd suggest you measure the chamber mouths, and then the bore, for comparison. You may find that the chambers are smaller than the bore, a condition not conducive to best accuracy. This can remedied by having a gunsmith open the chamber mouths a bit, so the seated ball is slightly larger than the bore.
Remove the nipples from the cylinder, seat an oversized ball in each chamber, then gently drive out the ball with a scrap of brass welding rod or a brass drift. DO NOT use steel or you may mar the chamber. Brass is much softer and won't score the chamber wall. Measure the extracted balls with a micrometer.
Then, drive an oversized ball down the bore with a brass rod or hardwood 5/16" dowel and measure the ball.
Soft lead projectiles will upset a fair amount to fill out bores, so I wouldn't sweat chambers that are .0005smaller than the bore, but anything larger may require the services of a gunsmith.
But before you do all that, continue shooting and experimenting with loads. Like I said, it may take up to a few hundred shots for that revolver to settle in.
I purchased an Uberti-made 1861 from Cimmaron last fall. I've put less than 100 rounds through it, and it's delivering 3" groups at 25 yards. I smoothed the bore with Iosso Bore Cleaner and figure it will settle in to tighter groups with use. That's been my experience. Cimmaron selects Ubertis of high quality, then smooths them a bit before sale. The best price I found was from Buffalo Arms in Ponderay, Idaho. The fit and finish of this 1861 is outstanding. I'm waiting for warmer weather to try it in extended sessions.
Have patience with your 1861. Like my Uberti-made Remington, it may become one of your better-shooting guns.