Krems,
The .500 Linebaugh, .500 JRH, and .500WE are all essentially clones of each other, from a performance standpoint. Most of my .50/.51 experience is with the .500WE, and I like it a lot. I also have a good deal of shooting and hunting experience with the .475 Linebaugh. I believe it can be made to shoot a little flatter, and penetrate a little deeper than those .50's because of the s.d. of similar weight bullets, but frankly, we're splitting hairs here, from a practical hunting perspective. As you point out, you're not using all the potential of your .500 Linebaugh as it is.
However, I think you've already made the most important statement: If you feel you're missing out on something, get a .475 Linebaugh...you won't be sorry.
You didn't ask, but I'd recommend a Field Grade FA with Micarta grips.