I'm sure the 20-tac is a fine caliber and will outshoot the 204, but can you buy off the shelf ammo for it? I mean, I reload everything I shoot, of course other than rimfires, but there is the rare occasion that I run out of shells while dog hunting and have to run to a local sporting goods and buy more ammo. Of course you have to re-zero the gun, but at least I can still continue to shoot. Also there's not allot of info on the 20-tac as far as barrel life, chamber pressures and such. I,m kind of a stickler on the barrel life part because of a problem I had with a Rem 220 swift. My accuracy started to open up after about 3000 rounds and became unshootable, as far as accuracy goes, at about 6000. Took it to the gunsmith and was told that it had lost the first 2" of throat. Now I might have just had a bum barrel, but it kind of soured me on the 220 swift and I read other people saying similar things about the 22-250. Some of my 223s have had over 10,000 rounds with only slight changes in group size. So when the 204 came out and I started reading reviews on it I was happy to find that they were finding barrel life to be as good as the 223 and significantly better than the 220 swift and 22-250. That and the fact that I can get ammo almost anywhere swayed me to go with the 204.
As far as wind drift, your exactly right. Wind will have a greater effect on lighter bullets, but that becomes part of the game when my buddies and I are shooting. In fact, some of the ranches we shoot dogs on don't allow centerfire rifles at all so it becomes quite fun to figure out how much over and how much left or right to hold. We routinely kill dogs at or over 200 yards in high wind with 10/22s and bulk ammo. You just have to figure out where to hold.