The two largest mechanical contributors to accuracy are barrel quality and ammo quality. Factory 7mm Mag ammo is not loaded with the goal of achieving maximum accuracy - that isn't needed since few big game hunters really care or can shoot well enough to tell a difference. This is not the case with varmint calibers, and I know from personal experience that there is plenty of highly accurate factory ammo in .223 and .22-250. I assume the .204 is the same. You will still have to try different brands and bullet weights to find the combination that is best for your barrel. How many loadings did you try with the 7mmMag?
Barrel quality is tough. Some factory barrels will shoot very well, but another in the same chambering may not. The factory buys barrels in bulk and the QA/QC is not what custom barrel makers use....if it was the cost would be too high. If you want the best chance of very good accuracy, a custom barrel is your best bet. There are a number of good suppliers, but the cost is about double a factory barrel.
I too would suggest a .223 barrel regardless. The
real-world performance difference between it and the .204 and .22-250 is quite small, and there is the greatest number of factory ammo choices in the .223. It is a bit cheaper to shoot, easier to find ammo for, and is suitable for over 90% of the varmint shooting most folks ever do.
Do not be fooled by all the accuracy claims you see here on the forum. Most guys seem to calculate their accuracy by shooting ten groups, then choosing the smallest one as their rifle's "accuracy". Groups size will vary considerably even with identical loads in the same barrel shot the same day, so just because your barrel won't shoot ten 1/2" groups in a row doesn't mean that you have a bad barrel.
.