For a .54, go 1.5 x 54 = 81 grains as a starting load, then go up or down to find the most accurate one. YOu don't need the 100 grain + loadings to kill deer or even elk & moose, unless that is where you get the best groups.
IIRC, that rifle has a 1:48" twist. It should shoot elongated bullets well. Try the REAL bullets from Lee. I found they worked very well in a .50 flintlock I had and give mroe oomph at 100 yards than the patched round ball.
Be aware that there can be a problem with bullets shifting off the powder charge when you are walking in the woods. Keep the muzzle tilted up as you carry the rifle, and very so often run the ramrod down to seat the bullet again. This doesn't happen with PRB, nor does it happen with every gun. It's something to remember, though.
-WH-