I have done quite a bit of predator hunting with a shotgun. Mainly coyotes and fox either called or ran with dogs. I definately perfer a 12ga either 3 or 3 1/2 magnum shells. THe best ammo to get is lead 2s, #4 buck, and the dead coyote ammo. THe dead coyote ammo is really spendy like $30 for 10, but effective. Always pattern your gun with different chokes to get a good pattern. I really like the paternmaster choke tube, they are pretty spendy but worth every penny. With the dead coyote ammo I have shot coyotes out to 60-65 yards and with #4 buck I have rolled a few out closer to 100 yards. I did a test one time with 3" buffered #4 buck at 80 yards shooting at a 2 foot square of cardboard. There were 27 out of the 41 buckshot still on the cardboard. when hunting them with dogs it was more important to put some lead into them, they dogs would catch awounded coyote or fox fairly easily. FOr predator calling I have used lead 2s on fox, coon, and bobcat with very good results out to 35-40 yards, dead coyote and #4 buck out to 60-70.
You really have to pattern this stuff to find out what works best. Also avoid steel shot, they really don't kill well when compared to lead.