You can kill a deer with 50 grains of powder, no doubt.
I would increase that load to 70 grains, and see how your accuracy does.
I have shot deer with 80, and 90 grains of black powder and the .490 round ball, and never had a problem. I couldn't tell much difference between the two loads.