One of the old wives tales is that it breaks down the powder, creating a dangerous condition. This has been tested many times, there is one guy that tumbled loaded ammo for a week, then compared velocity to his reference sample that hadn't been tumbled at all, there was no difference in average velocity between the samples.
Some might be scared of a unintentional discharge inside the tumbler, but a cartridge that isn't contained within a chamber doesn't carry a large risk, if it discharges the greatest danger would be to your eyes, there will be some scrapnel, but if you aren't around it when it happens all it will do is make a mess, or if you are wearing safety glasses to protect your eyes the danger would be minimal.