Your Encore 308 should have no problem with deer. It comes down to shot placement. If you only had " Just to clarify, my first shot was taken when the deer was looking straight at me, so I didn't have much of a cross section to aim at", to me your better shot would have been to center punch him in the chest. The Hornady SST bullet (you never mentioned what grain weight) should be adequent.
Bottomline, the neck shot is a bad %shot. Use the right bullet and take the high % shots. Personally, I would use a Nosler Partitian 165 or 180 gr in your 308. I like big heavy bullets...a couple hundred fps is not going to matter, dead is dead. When I had my 309JDJ barrel, I used 180 Parts and they worked great on elk. A 30 yad shot would go thru both shoulders of an elk and I would usually find the bullet under the skin back on the haunch. The 180 is probably overkill on a deer, but the 165 should be the meal ticket and allow for a higher % of angled shots in the boiler room. Also, the meat damage should be at a min.