80grs in a .50 is certainly enough to take whitetails cleanly out to about 100 yards...If that's what your gun groups best with, stick with it...
A couple of suggestions...What do your patches look like after the shot??? It's important that they don't burn through as that is what imparts the spin on the ball...Once you get over 80grs your patches may be burning through...If you want to shoot higher charges you might need either, better parches, thicker patches, a better lube or something between the powder and patch to help keep the patch intact...Items used have been , hornets nesting, another lubed patch on top of the powder or a lubed felt wad...
I used to shoot as much as 120grs in my .54...Through the years I went down in charge and now have settled on 80grs of Goex FFF, it shoot flat out to 100 yards and delivers plenty of power for deer...At one time I went through all sorts of experiments with patching, lubes and wads between the powder and ball to help protect the patching...
I finally settled on a .018 patch made from pillow ticking from WalMart and a homemade lube of bees wax, caster oil and Murphy's Oil soap...My gun still burns through patches lubed with Bore Butter or commercial pre lubed patches...It is trial and error, like hand loading...
If you want more info on making your own lube go to
www.muzzleloadingforum.com Look in Members Resources and it tells you how to make the above lube...If I remember correctly, it's 2 parts bees wax, 8 parts Caster oil and one part Murphy's Oil soap...It was developed by one of the members and it is good stuff...For about 25 years I used SnoSeal, that works good as well...
The nice thing about making your own lube is that you can vary the ingredients for your climate...If you want it thicker and harder, reduce the caster oil, if you want it thinner, increase the caster oil...
What "fixed" my gun was getting away from store bought, pre lubed patches, they just were not as tough as buying your own cotton pillow ticking and lubing yourself...I believe that the lube in the store bought ones break down the cotton fibers after a while as when I use the bought pillow ticking and lube with SnoSeal, my patches still look good...
I guess what I'm saying is you are real close to having a good load, I would now work with different materials for patches and different lubes...Through the years I have heard of guys using olive oil, Balistal, crisco, SnoSeal, vasoline, deer and bear tallow and even axle grease to lube their patching...Some of the guys over there also use 40 drill cotton cloth from JoAnnes fabrics as their patching, you just need to experiment and find what works best...
Don't hesitate to try some of the thicker (0.18) patching because you are currently using a .015 patch...Once you start the ball with a short starter the lead ball conforms to the bore so it will load about as easily as the .015 patched ball....If you get good groups in a clean barrel with the thicker patch, you will probably be able to load the second ball with a .015 patch with a dirty bore and have a good accurate 2nd shot...That's how mine does so you might want to try that as well...Good Luck...