The most unpleasant recoil I've ever experienced from a handgun was a 10" .44 Mag fired from a Contender. Second is a 10" .30-30 Win. from a Contender, and that was more "brisk" than "unpleasant".
I don't know how big your deer are, but the .357 Mag will give you plenty of versatility. Load 158 gr. or 180 grain JHPs or JSPs for deer and be sure of your shots. Load 110 or 125 gr. JHPs for varmints (coyote and smaller). Load any weight of any type of bullet you like for plinking, or even shoot .38 Specials.
If you have larger deer and need a tougher bullet to take them, the .44 Mag is a better choice. Load it with 240 grain JHPs or JSPs for deer (and be sure of your shots anyway). 180 grain or 200 grain bullets for varmints, but don't expect to get much in the way of hides. You can shoot .44 Specials for plinking if you want, but they're not as easy to find as full power .44 Mags (they're out there, don't get me wrong, just not all over like the mag ammo or the .38/.357 ammo).
Either one is going to be loud, and a brake is only going to make them louder. I'm not a fan of brakes, myself and try to avoid them. As far as the people complaining about muzzle blast and flash from the .357 Mag, it depends on your load. First of all, anything worthy of the title of "magnum" is going to have a lot of blast to it. As far as flash, it depends on your loads. I don't really notice a lot of flash from my 158 grain JHP loads pushed by 2400, but if you stuff the case full of W296 or H110 and start shooting 110 or 125 gr. JHPs, you'll get a very impressive fireball (which can be fun in its own right, IMO). You can get the same results with .44 Mag if you're inclined. Lots of slow burning powder under light bullets will result in a lot of flash.