I used to worry about it, I don't anymore. I do tumble them in corncob media, but frankly if I didn't have the tumbler already I don't think I'd bother cleaning them. Discoloration does not affect function, and they're too cheap for me to worry much if they last a few reloadings less.
With nothing more than a tumbling every 2-3 uses, I'm still on the same batch of cases I started with 3 seasons ago.