Naphtali,
I will attempt to give you some general input here but as youll see its not, for the most part, going to directly answer your questions. My experience with rifled choke tubes is limited mainly because the few my friends and I have tried to make shoot well have been big disappointments. After spending more on ammo experimenting than it would have cost me to buy a fully rifled barrel I gave up on the tubes.
With sabot slugs in a rifled choke, accuracy will not be much better than an open choke smooth bore shooting Foster slugs and sometimes much worse. Ive seen several combinations of rifled tubes and sabots where the bullets were tumbling and it was hard to even get them on the paper at 25 yards. Inspection of the sabot sleeves indicated that the rifling never fully grabbed the sabot (sleeves were tore up with no clear rifling marks).
It is nearly impossible to answer which inserts and ammo are better other than saying that I think the longer the tube you can get (within reason) the better. There are so many variables involved (sabot design, bullet weight, bullet shape, tube twist rate, velocity, the gun itself, etc.) in addition to the fact that even shotguns and tubes of the same make seldom perform exactly the same with the same ammo.
Considering only accuracy, I dont believe there is a significant difference among bore sizes when using factory ammunition with the same barrel type and slugs (although Ive never shot slugs through the 16). I do believe the 20 has a very slight edge over the 12 although not enough to make it a real determining factor.
As for making your own slug ammo there are a number of Foster type slugs/molds and recipes that work well but Im assuming your talking sabots here. Ive tried for many years to load sabots that performed as well as factory loads (including making my own bullet molds) and have yet to achieve equal performance to the factory offerings. The only one I found that came close was called Collet Cup Sabot (I used to get them through Ballistic Products) but the cost was so high that I just started using factory ammo again which gave me a little better accuracy.
Possibly someone here has found a rifled tube/ammo combination that shows marked improvement over a smooth bore and will give you some better direction but dont be disappointed if the same setup doesnt perform as well for you. Unfortunately the tolerances involved with shotguns and their ammo are considerably looser than than those of rifles and predicting there performance is problematic at best. Personally, I would either go with an open choke tube and the foster type slugs or byte the bullet and go with a fully rifled barrel and sabots.
Good Luck,
Bob.