Thats quite a difference in group sizes! I see your dillemma. Strange that it shoots the 168's so well and the others...not so well. Then again...I have killed a lot of game with guns that seldom grouped better than 2" at 100 yds FROM A FIELD POSITION.
But I agree that with a deers smaller vitals area one likes all the accuracy you can get. I know I do if the ranges run 300-400 yds.
I would check how far you are off the lands with the slug that shoots well...then load the other slugs you wish to try, at that same distance if possible. I would also try changing powders. Try something a bit slower as well as faster than the 4895. Reloader 15 and IMR3031 immediately come to mind.
I'd try different primers too, go to Win WLR, or RP's, Federals, etc.
I would also clean the barrel very well, before trying any loads. Fire 2 or 3 fouling shots too.
Nosler Accubonds and the Ballistic Tips seem to tend to be accurate. the BT is far more frangible, but on deer you should be OK. I would also try a 180 gr slug too. You may be shooting those 168's better as they are LONGER and match your bbl twist rate better.
I would surely not get hung up on 165 gr versus 168 grain. Your shooting the 168 slugs well because they are a high quality Match slug, and apparently that load is a good combination in your barrel. Now to duplicate that load with a hunting bullet.
This is an interesting situation.
FN in MT