OK someone correct me if I'm wrong, but the powder compression isn't for more power, but rather a smoother, more consistent burn, correct?
My guess is that you have incomplete burning if the bullets are hitting the ground at 150 and 4 FEET HIGH at 90?
Sounds like a charge from a 30-30 trying to push a 400gr bullet. I have owned a 45-70 and shot a fair amount of BP through it. I've never, not even on uncompressed loads, had a bullet lob like that.
I'll bet you could have looked downrange and found big chunks of unburned powder on the ground.
I often heard it's not good to load BP cartridges too far in advance, maybe this is why? Also might have some damp powder.
Otherwise sounds like it was a fun day at the range, and yes, .4-.5 does seem like a LOT of compression. I could never get close to that without destroying a case. If you're trying to stuff more powder in the case that way, the best method is to seat the bullet out farther, but that too, has it's limit, as you mentioned.