How much lower is the velocity for the errant shots? Degrading a propellant like 5744 in a powder measure should not effect the charge much if at all, and the difference in charge weight shouldn't be that much either - try weighing ten throws and see what the spread actually is.
Perhaps it is the powder's position in the case. If every now and then the charge is, say, at the front of the case then the velocity may be somewhat different than it is when the charge is centered or at the back of the case.
5744 is not noted as being very position-sensitive, but the charge weight you list produces relatively low pressures - under 14,000 psi. Do you note a difference in the amount of unburned powder in the barrel after the slow shot?
What primer are you using? Pistol primers are often used for CB loads, but they may not be hot enough for this load; Accurate recommends rifle primers only. Is it possible that the bullet lube you are using is contaminating a few powder charges? The above are a fw of the posibilities.