Mike, you are totally correct.
Absolutely positively the hardest thing about deer hunting is finding that 40 acres of private woods with mixed fields that you can lease for yourself alone. No clubs, no sharing hunting rights with the farmer and his "family" (sons, nephews, cousins.)
Once you have found that, and located the funnels where the deer always cross the fields, and set up in the woods 15 yards from the field, watching the funnel, and put out the right attractant scent in the middle of the field right where the funnel path crosses the field, then killing a deer is a virtual certainty. During the rut, the deer actually run through these funnels at break neck speed all day long. That's when you can take a big buck.
The second hardest thing about deer hunting is getting up in the winter before dawn. That's why I quit doing it. With private land, you can choose to just hunt in the afternoon, and you will be very successful.
The third hardest thing is being able to sit in the bitter cold for three hours at a time. Curing that is just a matter of buying exactly the right silk, wool, and down garments, with fabric that breathes.
And as you pointed out, only hunt where the deer are. Hunting marginal land is a total waste of time. Your time is far better spent traveling the back roads and finding a good piece of private land to lease.
Also, it is far easier to lease land if you tell the farmer that you only want to hunt during archery and muzzleloading season, and not the regular rifle season. Most farmers don't use bows or muzzleloaders, so the money you pay them for this is sheer gravy to them.
Best time to ask about a lease is the first week of February, when the farmer is dead broke and has just gotten all of his credit card bills from his Christmas spending, and needs cash to service his farming equipment. Show up with cash to pay him immediately and get his signature on a little 3 sentence lease at the same time. Post the land.
After you lease for archery and muzzleloading for a few years, it is pretty easy to "expand" that lease to include either the first or second half of rifle season. Just be sure to let him pick which half he wants to retain for personal hunting, and be generous with the extra money you offer him.
Mannyrock