What ammunition are you using?
Jacketed? What diameter of Lead? Gas-checked?
If you are shooting plain-based lead, you might get your groups by switching to jacketed. If the jackets work, great.
If you want to shoot lead but plain-based aren't accurate, have you tried gas-checked bullets?
For my .32-40, which shoots the same bullet as the .32 Special, I use a bit of filler in the case between the powder charge and the bullet--Original Shot Buffer. This rifle would not shoot lead accurately, but with the buffer in it, open sights at 100 yards, lemon-sized objects are in trouble. Before you alter the rifle, this might be the solution to your accuracy problems. The buffer does not dirty, nor does it harm your rifle.
If you would like to try shooting reloads that I would build for you, I'd send you some--no charge. I reload for the .32 Special anyway, so it would be easy, a pleasure, to help out.
I can send jacketed, gas-checked, buffered, and plain-based. All or some. Let me know.
And yes, my bench is uncluttered. Safety is foremost in my reloading. My three sons are precious to me. They shoot my loads.