Failure to polish is probably your problem, depending on where the leading is. If the whold bore is leaded, it is roughness. If only in the grooves, then groove diameter is larger than your bullets, and bullet diameter should be increased, as well as cylinder throats so they will pass through.
If softer bullets fix the problem, small bullets are definately the problem, but softer bullets will not give you the accuracy I like to see customers get. You need to get bullets up to groove diameter before they are fired.