I kinda think that the 8mm mag was just a ba$tard cartridge. I can't believe recoil was excessive, look at some of the stuff already being shot!
Bullets probally had something to do with it. But if what you wanted was a long range cartridge, theres many as good or better for no reason other than bullets; 264 win mag, 7mm rem mag, 300 mags (take your pick). If you want to crush bones, theres also better choices, again bullets may be a factor. 338 mags, 350 mags and 375 mags and up!
I think that most big magnums, even smaller ones, are special purpose cartridges. Few would sit on a praire dog town all day with a 338 mag. Most large mags are definatelly more then needed for most game hunting.
The 8mm mag didn't fill any nitch and did it with what I understand was a lot of recoil. With the cartridges avaliable today, I doubt even good bullets can breathe life back into it.