I took many deer with a 20 ga as a kid and I never felt under gunned. The 20 gauge is ultra lethal for whitetails and with the yardage you shoot as stated, I would see no reason to part with what you have. I have never taken deer much over 40 yds with a 20ga as mine was an old smooth bore single shot with a bead, but it sounds like guys have had success out at 150yds and beyond with a good dialed in 20 ga slug gun.
I have a NEF Tracker II in 12 ga and it kicks like a mule. It doesn't bother me, but it does kick. I'm sure the 12 ga USH kicks less, as weighs about 100 lbs (just kidding) but that thing is heavy. If you do a lot of walking while hunting I wouldn't recommend it.
LAZER