The 30-06 cartridge is much better suited for rifles with barrels 22+" long, given its case capacity, bullet weights, and pressures. The 30-06 is not really designed to be a 100-yard deer cartridge, although it will easily kill deer within this range with plenty of power to spare, and has been used in this capacity by many for a long time. Its forte, though, is carrying medium-to-large game killing power downrange to 300+ yards. To this end, shortening the barrel of any 30-06 under 22" is in a way defeating the purpose of the cartridge.
Whether to shorten or not is purely a matter of personal choice, but if you ever decide to use the rifle at greater ranges or on larger game, or pass it on to someone else, you will definitely have lowered its value. My own recommendation would be to leave the 30-06 alone and either add or switch to a cartridge/rifle combo better designed for more compact handling and shorter range medium game hunting, if that is your goal. There are a whole handful of cartridges designed to deliver 100 to 150 grain bullets at modest velocities for medium game that would better fit a 20" barreled rifle. Some of the better ones IMO are: 243, 6mm, 250 Savage, 257 Roberts, 260, 6.5x55, 7-30 Waters, 7mm-08, 7x57, 7.62x39, 30-30, 308, and 35 Rem.