Have you fired a group with the .454 balls?
Yes they would be shaved down to the same size as the .451, but, there would be more lead in contact with the rifling, there would be a wider flat spot on the side of the larger ball.
I would try those .454s since that is what your manual says to use.
My theory is, the bigger the better, my manual says use .454 balls and I use .457