I would also add that a slug, because of it's much heavier weight, does not need as much powder, or velocity, to do it's job.
Very generally speaking, you want some velocity with a round ball because it is lighter, sheds velocity faster, and has little sectional density for penetration.
What makes a slug "work" is weight and sectional density, so velocity is less important.
Another thing about overloading a minnie, is that the skirt will flare out after it leaves the barrel, which will hurt accuracy. If you are going to shoot real heavy loads under a minnie, you need to turn the base plug down on the mould, so it will cast out with a thicker skirt.
So yes, for deer, and even elk, 120 grains under a 420 grain slug, IMHO, would be a bit of "over kill". 100 should be plenty.
Of course if accuracy improves as you go over 100 grains, then it might make sense to increase the charge until best accuracy is obtained, and that might happen with a flat based slug. With the minnie, as others have mentioned, best accuracy should be around 100 grains or less.