I use nothing but .454 or .457 balls, and have for many years. It simplifies things.
I occasionally use .457 balls because I bought them on sale a few years back. On the main, however, I use .454 balls. I gave up on .451 balls years ago.
Using .454 ensures a tight seal in the chamber. This prevents movement of the ball from recoil. Using the .454 (or .457) ball also creates a wider bearing band for the rifling to grip. The difference in the amount of force required to seat a .454 ball, as opposed a .451 ball is minimal. Disregard dire warnings from shooters who claim it unduly strains their rammer; this is hokum.
Or, they have a cheap revolver made of inferior metal. Any decently well made revolver will certainly accomodate a ball .003 inches (three-thousandths) larger without strain. If the rammer bends or is damaged, the fault lies in shoddy workmanship and materials, not ball size.
I buy and cast .454 balls. Similarly, I began using .380 diameter balls years ago in my .36s and never looked back. Never had a problem with them in any cap and ball revolver.