It's uncomfortable stuff. The issue is more protection / more discomfort vs less protection / less discomfort. Everyone has to make his own decision, but bear a couple of things in mind: about 10x more people are shot with .38 Specials and below than are shot with .41 magnums and above. Also, if you don't wear it, it doesn't do you any good.
The temptation is to buy too much protection; then it just sits in your closet.