I looked into it for .223 and decided not to do it because I was already getting 3/4" center-to-center groups and there was no need to improve on that. However, for those who want the extra accuracy, it certainly won't hurt. I don't think any serious rifle competitor uses brass that hasn't had the flash holes uniformed.
On the first loading I inspect each case, including the flash hole. With the Winchester brass I use for rifle cartridges, I've never seen anything worrisome. Out of a hundred cases, one case typically has a neck that's been deformed in handling, but that's quickly solved by sizing the case. If I saw a problem, I'd be uniforming my flash holes too. I don't clean primer pocket holes either, and it doesn't seem to be a problem. The pockets are never so encrusted with fouling that I can't seat the primer properly.