I agree with the other guys, if you are hunting legally there is nothing to worry about. If they harass you call the DNR, we have hunter harassment laws here, I hope you do to.
Like rickyp stated, the human ear wont be able to detect any difference between a 50 grain charge and a 90 grain charge. Find the load your rifle likes and you will only have to shoot once anyways, the deer deserve to be killed as quickly as possible.
If you are only shooting 30 yards max and want to be quiet, why not use your bow if it is legal to use during your firearms seasons. I know that is not what you want to hear, but it don't get much quieter than that.