I'm in total agreement that he has the RIGHT to wear it, but given the environment, was it RIGHT to wear it? Apparently not. You have the RIGHT to wear a KKK t-shirt and cap, in New Orleans, or a Black Panther t-shirt and cap to a Klan rally, but would it be the RIGHT decision to wear it there?
He wanted a reaction and he got one. If you know you aren't welcome there, why go?