Yes, all in all, the heavier grain slugs are for heavier game, although plenty of guys shoot deer with the heavy slugs. If it should happen that your gun is more accurate with a heavy slug, then you should use it. But, I bet you can get accuracy with a 250 grain slug.
For whitetail at 100 yards or less, try the Hornady 250 SST.
I wish you would use the powder rather than pellets. Powder is cheaper. You are able to fine tune the gun with powder. It may be that you get 3 inch groups with 100 grains, but 1 1/2 inch groups with 90 grains.
If you have to use pellets, try 2 of the 50 grain 777 pellets. This is all the power you will need. I know a guy who killed a buck at 207 yards with 85 grains of 777 and a 245 grain slug.
Also, if you use 777, use the Winchester 777 primers.