I've got the same rifle as you...204 ultra Varmint fluted with synth stock, and have been happy with it.
I've not heard of any problems peculiar to fluted barrels. However a recurring theme with NEF Handi's, is that accuracy varies between each gun and sometimes they need tweaking to optimize their performance. Some can drive a tack right out of the box, and others need to either float or bed the barrel.
To optimize your rifle, I would perform the following:
1. Optimise your sights. Unscrew the scope rail and thoroughly clean any oil or solvent between it and the barrel (mine had some machine oil). Then apply thread bond between the barrel and rail, and on the screws. Tighten everything firmly and wait to dry. I've heard you can also apply bedding compound under the front rail overhang, but I haven't tried it myself. Then use a scope mounting kit to ensure the rings are aligned and firmly mount the scope.
2. Optimize the barrel. Pass a dollar bill between the front stock and barrel. If it hangs, use sandpaper to sand the ridges in the synth stock until the dollar passes easily. NOTE: Try this configuration first. Then, if unsatisfied with the accuracy, you can always add a small spacer or pad between the barrel and stock for comparison.
3. Optimize the rifling. Make sure the barrel is clean and polished. Make sure there are no burrs from the machining. Some folks use a little Fitz to polish the bore.
4. Optimize your ammo. Some guns perform better with certain brands of ammo, than others. I have tried remington and Hornady ammo in mine, and have found greater accuracy and consistant performance with Hornady, than the other ammo.
5. Optimize sighting in your rifle. Bore sight your scope before going to the range. Fire a few rounds at a 50yd target and adjust the scope to fire consistantly near the bullseye. These shots will also help break in the bore. Then, clean the bore and let the barrel cool while you prepare 100yd targets. When the barrel is cool, fire from a rest. When the barrel is cool, fire again. Do this 3 times and adjust the scope as necessary. Clean the bore again, and repeat until you have consistant shot groups.
According to Remington's ballistic software, if you zero approximately 0.9" high at 100yds, you will be zeroed for about 235 yards. With that zero your bullet will hit within 1" of your aimpoint between 25 and 250 yards, and dead on at 55 and 235 yards.
Hope this helps.