Measuring the groves and lands of your barrel is imperative.
I'm sure that after you do, you will come to the realization that .458" might not be enough. For my rifle, .458" bullets are OK (now I'm doing 2" groups at 100yrds) but I already ordered a batch of .459" to see if there is going to be an improvement.
In any case, 405 gr bullets, according to most experienced opinions in this forum, are like the perfect recipe for this barrel.
Please check visually the inside of your barrel to make sure the rifling spirals evenly from end to end. (I got mine initially with a faulty rifling).
Casting good bullets is difficult and at the beginning frustrating because any little imperfection will reflect in the size of your groupings.
For now, I strongly recommend you to get a batch of 50 0.459" bullets from a good supplier such as Montana Bullet Works. That guy really knows what he is doing. That will eliminate doubts about the actual bullets and put the onus in the rifle and your load development.
Ah, by the way, how do you know if your scope is still good? Not too many scopes can handle repeated 45-70 impacts. In the cheap department, I only buy scopes designed for air rifles. Air rifles' internal recoil destroys normal scopes in no time. And air rifle's scopes are built to take it. Otherwise, stick to Leupold scopes.
I think you will have to accept the loses with the bullets you currently have.