Any media that finds it's way into the flash hole in the brass would cause inconsistant ignition and accuracy will suffer to some extent. I don't know as it would cause a misfire though, unless some media dust gets behind the anvil in the primer. (maybe)
I wouldn't bet my life on such ammo, but it sounds like you are not doing that.