If we were discussing modern guns, I would say rifling and (for legal purposes only) the type of cartridge used for ammunition (rifle cartridge vs shotgun shell)--even if rifled shotguns severely blur the line with this. But since we're discussing the much more interesting muzzleloading guns...
The way it has been explained to me, and makes sense based on what reading I have done, is that it isn't just the bore being rifled or smooth, but rather the whole of the gun in question and the various characteristics involved. For example, fowlers tended to have wider and flat buttplates, while rifles tended to have curved buttplates. Fowlers, or at least most of them, had just a front sight, while a rifle had a front and rear. So it isn't a matter of just the bore being rifled or not, but a combination of characteristics. The smooth rifle is a smoothbore built to handle, look, stocked, etc as a rifle would be, but with a smoothbore, and usually of a somewhat smaller caliber than the usual fowler. Now a musket (rifled or not) on the other hand, is just a weapon intended for military use in those days, and usually built somewhat heavy due to the intent of having a bayonet mounted on it. The architecture of the stock is much less obvious today with modern weapons due to almost everything, unless it is a replica of an older design, having a flat buttplate and most being designed for scoped use, but there's a lot of people who have a hard time understanding the difference because they think it terms of comparing everything to modern guns, or just what they are used to in terms of older ones. But it is rare nowadays to meet anyone who actually knows the old guns, and the history and reasons behind them, and even more rare to meet anyone who can speak about them from experience due to having hands on experience with them, weather it is a true antique or a replica. My smoothrifle is a Lyman GPR flinter with a Green Mountain .54 smoothbore barrel on it. It is a rifle in architecture, but has a smooth bore. It handles like a rifle in terms of balance, swing, etc, etc., due to the weight and balance of it, so is not a good design for wing shooting, but is a great little gun for stationary small game like rabbits and squirrels. Horrible for dove, but great for turkey. And then there is what it can do with a signle patched round ball when going after deer or hogs. Granted, I have to pass on the longer shots, where a rifled bore would be much better due to the longer accurate range, but with most of my shots being in the 50-60yd range or closer, the 4" or less groups I get at 50yds are perfectly adequate, and I trust this gun to about 75-80yds to keep all it's shots in the vitals of a deer or hog. If your shots exceed the ranges I deal with, the smoothrifle may not be for you. Same if you intend to use it for dove. But for the hunting I do and the ranges I usually deal with, it is no handicap at all, and has the advantage of being able to take one gun, hunt deer and hogs in the morning, then pull the ball and spend a few relaxing hours going after small game during the day while loaded with shot, and then go back to the ball towards the evening. Most fowlers aren't as accurate as my smoothrifle with a PRB, and no rifle is as effective with shot as a smoothbore. So the man armed with a smoothrifle gives up a little accuracy to the man armed with a rifled rifle, but this isn't an issue when you consider the extent or westward expansion in the 1700s til the early 1800s, due to the areas that were "settled". But the rifle armed man can not load up with shot and expect anything resembling decent patterns, so some of us will keep and carry our smoothrifles, and the rest of y'all can carry a big game rifle and then also lug around either a small caliber rifle or a smoothbore of your choice for those times when you want small game. This is much easier understood by those with experience with this style of gun, than those who stubbornly stick with having several specialized tools. Then again, we all need an excuse for a new toy, so maybe y'all who are opposed to these just may have an ulterior motive?