I don't personally own a Weatherby ultralite, but some of my friends do. I thought about getting one, due in part to the fact that they truly are quite light. However, like your experience, most of these guns apparently do not have out-of-the-box accuracy like the Remington M700s, which all of mine shoot sub inch at 100 yards. (Then again, I have heard that the Remington Titanium light weight rifles also do not have that great of accuracy.)
On the other hand, I guess you will be able to get better accuracy since you handload. Also, accuracy probably is not THAT important of a factor considering that you shot your buck at 60 yards.
It's all about priorities, and a lightweight rifle is more important to you than having a heavy tack-driver. No problem, it sounds like, based upon your needs, your Weatherby is a real winner. :grin: