If you use a Barnes all copper bullet, size is not an issue. My wife was punching holes completely thru deer last year with a 40 caliber 175grain bullet. Barnes DOES NOT come apart.
If you use a standard copper/lead bullet, I would not go smaller than 200 for sure, and if you are hunting big northern deer, maybe even 250. Thats just my opinion, smaller bullets will certainly kill deer, just preference.