All of my H&R/NEFs like to be shot from a dirty bore, so just foul it and then shoot for score. I know of many, many hunters that won't clean their bores once they sight in prior to hunting season unless they hunt in inclement weather, most guns shoot best fouled, up to a point, depending on bore condition.
Even tho you're running a dry patch down the bore, that's not enough to wipe all of the oil out of the bore, it takes a solvent or alcohol or something more to make sure there's no oil present. If the search was working, I'd point you to a link on shooting from a clean bore and the difference it made to make sure there was absolutely no oil or solvent in the bore, it was quite enlightening!!
Tim