If you put a bullet in somebody, the costs to you will be $20,000+ to stay out of prison, most likely, and ANOTHER $20,000+ to not lose your butt in a lawsuit. Every shot you fire, even those that miss and richochet, you are responsible-for.
since it's well proven that over 90% of attackers WILL back down at just the sight of your (ready) gun, (IF you get it out and "on" the attacker in time for him to notice it and flee), you definitely want to get the gun out "early", and let him back down, rather than wait until you HAVE to draw and fire.
This is where some hand to hand experience, and keeping your hand ON a pocket gun (in the pocket) in a Kydex pocket holster (secured in the pocket with velcro) is a very good idea. It lets any duffer be as fast on the ccw draw as the fastest men are with a ccw belt rig (still hidden under a garment). In bad weather, transfer the rig to the coat pocket. You can "palm" the rig and gun as you doff or don your coat, to the appropriate pocket.