Just about everything has already been said but I do want to add just a bit. Again, the OAL (COL) is not the problem. If a case neck is too long, and not trimmed, and you chamber the round, the case mouth will jam into the throat of the rifle, and can crimp or squeeze the bullet so hard as to cause considerable pressure problems.
My personal procedure is to trim, chamfer and de-burr after every firing. I use the Lee cutter / trimmers

which work very well and are EXTREMELY accurate, and usually only shave a very small amount of metal, and sometimes none at all, but sometimes there is considerable lengthening, and by doing this, I KNOW that I will not be running into problems, and that ALL of my loads are consistent...
IMHO... go ahead and trim each time to be safe, and to have the best possible loads...