Deer walk right past and sometimes in, my back garden and they never bother a thing.
I live in a food rich area, and also i DO have food plots planted for deer/turkeys and to take the hay off, they must like all that "other" food better and yes, i do have a LOT of deer here, even in season.
I do have some rabbit and woodchuck problems, i shoot the WC's, and fence the rabbits out as best i can. I only fence the front garden, so i plant the things that rabbits like, in it. I fence it because i just don't care to shoot all the rabbits, except in rare cases i will shoot one.
DM