Those of us that take reloading serious don't mind doing a few extra steps if it will improve accuracy and maybe have a few more side benefits. Annealing is one of those things. Besides extended case life, the soft shoulder and mouth form to the chamber when fired and make a good seal. That means your cases don't soot up and you don't ever get collapsed shoulders.
Most shooters dont know and probably dont care what happens inside the chamber when a round is touched off but Ill tell you anyway. When the pressure inside the case reaches a certain point, the bullet is forced out and strikes the bore. This always causes the bullet to suffer some damage, especially if it strikes the bore ever so slightly off center. When the bullet has damage on one side, it will wobble as it is going down range. The more damage, the more it wobbles and the less accurate it will be.
The idea is to do things to the case so the bullet will enter the bore as straight as possible with minimal damage. Many bench rest shooters neck turn their cases so they are perfectly centered. This works quite well but is not necessary if you anneal. When the case mouth and neck are soft, the bullet is pushed from the case before the pressure builds too high. This makes the bullet strike the bore more gently and reduces bullet strike damage. At the same time, the case puffs up into the chamber and releases the bullet straight on with the bore. Combine annealing with bullets that are seated out to within .010 of the rifling and you have minimized bullet jump damage. Most guns will see a dramatic increase in accuracy with annealing and proper bullet seating depth.
Ever look at new factory brass, especially RP or Lapua brass? The case shoulder to mouth is discolored and has a pink hue. That's because they are annealed at the factory. Winchester, Starline and others do this too but they tumble their brass afterwards so it isn't as evident. I've found that annealing every 2nd firing keeps the brass soft enough. After 3 shots, and 3 sizings, it's probably as hard as it will get.
Many guns have excessive free bore. This is the area in the chamber that is an extension of the case neck. Free bore is slightly larger in diameter than the neck of a loaded round. You can measure the diameter of a spent case neck and will come close to your rifles actual free bore diameter. Most free bores are at least .025 larger in diameter than the bullet diameter. Additionally, the free bore is usually drilled too deep. In all, the free bore gives the bullet an opportunity to wander off center on its way to the rifled bore. If you trim your cases to SAAMI specs, they will almost certainly be too short. This also causes bullet strike damage. With annealed cases, they will grow a little and fill in the excessive free bore. I trim cases to square up the case mouth. This makes the bullet release more uniform, almost like the crown on a barrel.
I use a set of chamber tools that measure the precise chamber depth, then trim my cases to the optimum length. I also use a bullet seating depth tool to measure for optimum bullet seating depth. If your gun shoots good groups, dont bother. If you want to really improve accuracy, try annealing, bullet seating depth, and case length control. A few more steps, but well worth it to me. See:
http://www.cactustactical.com/reloading/reloading.html