I think there are several reasons that Remington's big 8 failed. I don't think it would have failed on any one of the given reasons, but with all at the same time, it was doomed.
I'll start by saying that I did, indeed, have a Remington Custom shop rifle in 8mm Mag several years ago. I also owned a 338 at the time. With factory loaded ammunition, there was really no difference in the ballistics between the two. The 338 had already been around for about 20 years when Remington decided to introduce their medium magnum. If someone was looking for a rifle to fill that niche, the 338 had more loadings available from the factory and better bullets, in more weights, available to the handloader. It was once said that the 8mm Mag wouldn't catch on until Nosler started making a Partition for it, and Nosler would make a Partition for it until it caught on. Nosler did finally relent and make an 8mm Partition bullet, 200 grains, but it was after Remington quit chambering the BDL for it and it was strictly a custom shop venture.
Another reason was that because Remington never had any great sales success with it, no one else would pick it up. If you didn't like Remington, you had to go the custom route. Or you could just buy a 338 from any of the several different manufacturers and get just as powerful a chambering.
For a long time, all the bullets made in the 8mm caliber were designed to perform around the 8mm Mauser velocities and reports kept coming in that, when pushed at the Mags velocity, they weren't penetrating well and coming apart. Remington only loaded 2 bullets weights for at the time, a 185 grain Core-Lokt (which was very explosive at close range) and the 220 grain Core-Lokt. The 185 grain was a deer bullet for an Elk, Moose, Bear rifle and not what most 8mm Mag purchasers were after. The 220 grain load didn't measure up to the 210, 225, or 250 grain loads offered for the 338. Why compromise?
In my opinion, though, Remington's biggest mistake was just slapping a regular BDL stock on it. The stock is not designed well for rifles in this recoil class, not for the shooter or for the rifle. Even the cross-bolted Custom shop stock I had was too thin in the wrist and the cross-bolts were in the wrong location. The recoil lug was insufficient as well.
I think if Remington had been really serious about wanting this to succeed, they should have redesigned an action and stock around it. A more substantial extractor, even brought back something like the M30.