I've heard that before and I have to admit I don't really try to shot tight groups with my 454 and hunting loads, it has been more of survival tool against black bear here in Colorado, which for some reason what to break into our trailer during our fall hunting seasons. I wasn't that worried about it, most self defense shots would be pretty close.
Though I have to admit hunting with the 454 as my primary weapon I hadn't considered until this year during a Moose hunt and I found my shots wouldn't have been beyond 25 yards max so again didn't worry.
You make a good case for the longer barrel length for accuracy, it only makes sense and helps with hunting, it can't weight that much more than my current one in 4.75".
Bottom line I wouldn't trade off my 4.75" for a longer barrel, it wasn't my first choice but was cheaper than the 7.5" at the time and now really love the handling of the shorter barrel with mild loads. Instead I'd sell my first born for a 2nd FA in longer barrel.
I think both have there purposes!