I always pull out before rifle season here, too. Just had too many bad experiences with it. Had coyotes blasted to nothing with high-power shells, had traps run over, stolen, shot to heck, etc. let alone thought of walking into an area and being nothing more than "movement" to someone watching from across the section. Seems to be a select local few here too that like to "shoot into the section" to scare them out.
I also always felt like that was their time to enjoy their sport, and I could always go back in later on and enjoy my sport...sort of a "share the woods" thing. Sad thing was though that in some sections I was the only one with permission so not sure who had the 4-wheel drive that ran over my trap 10-15 times because they couldn't pull the earth anchor out of the ground to steal it.