There's a couple of things you can do to prevent peeling. You can bell the case mouth so the bullet gets started past the rim. You can also use a boat tail bullet. The tapered base will load without peeling. You might consider getting a new expander ball for your sizer die that is a bit larger. Personally, I usually go for a tight fit and like expander balls to be .008~.010 smaller than bullet diameter, just like yours.
Yes, peeling can affect accuracy. The bullet seating process may deform the case too. Any time you have a variation in the process, you risk loosing accuracy.