The biggest aspect of self defense is knowing your gun and being able to hit your target. If you are comfortable with, say, a 2" barrel .32 Mag, go for it. And PRACTICE, that's the important part.
As for the .32 Mag being loud...I'm not sure I agree with that. All guns are loud, but factory loads in .32 don't seem overly loud to me. I think people might be recalling the blast of the old .32-20, which was reportedly fierce in a revolver.
For self defense, I'd go with Federal's 85 gr. JHP. In my .32, it's terrifically accurate. And I'd get a speedloader or two.
Sincerely,
Michbob