For the larger guys, I would say it is common practice. And also in those places, the salesmen usually have no control over price. If you go into your small LGS, you are most likely talking to the owner or a close friend or relative of him/her, and they have been given the power to haggle over price. If you went into Cabelas, thought a price was unfair, and made an offer.. they would tell you sorry.. can't do anything.
I think that especially with used guns it is important to shop around and see what others are selling a gun for, and also keep a copy of the blue book of gun values and gun traders guide. They cost about $25 each and can pay for themselves pretty quick. If you have a place you like, and see a gun cheaper elseware, make an offer.. if they don't accept the offer... just let them know you will take your business elsewhere.
But I would also say the same can be true of some gun shops, but for a different reason. When I went to college, I was in Flagstaff, AZ, and there was a shop there that we frequented which had a great selection of firearms. Anything you could imagine, and the guy also had his class III license, so you bet your @$$ we were in there quite often seeing what he had in stock. He had that draw because of his tactical and exotics stock, but if you looked at his regular and used gun prices, they were usually about $100-$150 higher, across the board, than the next shop, which was about 1/4 of a mile away. I heard him talking to a guy who had sold him a few used handguns... and when he was talking about the prices he would buy them off the guy... there is no way I would of accepted the money he was offereing, but it was because of the diversified inventory that he didn't need to worry about your "normal" everyday guns and their prices.