Lone Yankee,
I'd always heard: "You can't miss fast enough to win." Guess that's not an absolute, especially if you're shooting against very slow accurate shooters.
Seriously, you shouldn't let that bother you. Concentrate on shooting against yourself - constantly try to better your time while maintaining or improving your accuracy. Sounds like you've got the accuracy thing down pat. Now you need to adopt a modified version of the "Don't work harder, work smarter" philosophy, and start shaving seconds off your draw, reholster, movement witihin the stage, shotgun reloads, etc. This can be practiced at home with dummy ammo without firing a shot. Speed will come with better familiarity with the various required actions. If you're serious about getting faster, PRACTICE, PRACTICE, PRACTICE.