Neck sizing only keeps the uniformity and case-to-chamber fit that is obtained in firing a cartridge in a specific chamber.
Almost all competive shooters (BPCR, benrest, etc) go for neck-sizing only. That should tell you how important case uniformity is.
Oh, by the way, did you change primer brands in making up your reloads? That can have a considerable effect too.
Best wishes and let us know how it works out!