I would think that the various 40gr plastic-tipped bullets would be ideal for home defense scenarios. But I also think that you would be adequately served with any 52gr to 55gr pointed soft point or hollow point meant for varmints. After all, bad guys are just two-legged varmints.
These rounds will not overpenetrate, they won't exit a bad guy on a body shot unless your shot is truly glancing, they'll go through body armor, and any upper torso hit should be instantly incapacitating... and will be nonsurvivable.
A 16" AR-15 is shorter than an 18" shotgun, has almost no recoil, holds more rounds, is quicker to reload, is equally devastating against humans, is easier to use for precision shooting (the "hostage rescue" scenario as an earlier poster mentioned)... what's not to love?
In short, the same .223 factory load you'd pick as a coyote hunter concerned about pelt damage would be ideal for home self-defense.