Fortunately, I live in Oklahoma, where we have a good strong castle law. I don't have to worry about handloads vs factory. That said I use factory in my 1911. But if it had handloads while I'm at my range, I wouldn't swap magazines before defending myself.
I've been known to have a loaded '51 Navy or '60 Army in my vehicle, while traveling to my shooting range. Since I make my own blackpowder and cast my own bullets, I wonder how I'd fare in a New Jersey court if I had to use one of them for self defense.
A similar discussion used to come up on the SASS Wire about using our cowboy guns for SD. A lawyer member advised not to, for the same reasons as handloads. Finally, he was challenged to come up with a single case where a person had a bad legal outcome due to using a single action "cowboy gun". He backed down when he couldn't find a case.