I've seen the same with lower pressure loads using IMR 4198 and 3031. In fact, in a few isolated instances, the powder didn't ignite very well and I hardly got the bullet to make it out to the target, believe it or not. No recoil and no big "boom" either. When I looked in the chamber, the barrel was full of half burned powder. My guess was that temperature and the packing of the powder had something to do with this. I was getting a lot of hang fires all day which suggested the this was the case.
I went home and loaded some rounds and added a small amount of polyester fiber fill to the rounds in order to ensure that the powder would remain near the primer when tipped in the shooting position. Doing this typically should give a better ignition and burn. The results showed no hang fires, no unburned powder and and not a whole lot of "crumbs" in the barrel.
Please keep in mind, however, that anything additive to the overall case volume will increase pressure so don't cram the case full of polyester wad.