You have die adjusted wrong, wrong powder bushing in the die, or need to check your rythem. There should be no powder left in the die when you bring the press handle up. Are you allowing sufficient time for the powder to leave the charge bar and drop thru the powder funnel? 30-06 is not like a 45acp or similar pistol cartridge where the powder drops like a rock thru the powder funnel. It takes a couple of seconds for 50-60 grains of powder to drop through. You want to give yourself a "one thousand one one thousand two one thousand three" or similar count when press handle is down to allow the powder to get from charge bar through the powder funnel and into the case.
Make sure your charge bar has minimal movement after press handle is fully down. If powder die is seated down too far, your charge bar will hit maximum movement before you complete your down stroke. If powder die is too high, the charge bar will not dump all the powder and action from the return spring will shake it out. (Grip your powder measure with your left hand and try pressing the charge bar further when your press handle is all the way down. If you can move your charge bar, your die is too high. )
If you have the wrong powder funnel in the die, you could have grains of powder setting on the rim of the funnel that vibrate off as the press handle comes up. The powder measure would still work if the powder funnel is in the die upside down. Another ledge to hold powder and then allow it to trickle off as press handle comes up.
A little Tap Tap with the press handle at the bottom of the press stroke can also shake the last remnants of powder thru the funnel if a little powder "dams" up.
Good luck. Hope this helps.
Steve