Those sights are probably not "worn down" but were built that way. Our notions of sights have changed a lot over the past couple of centuries, most original muzzleloading rifles had barely visible front sights and just a bit of a scratch for a rear notch. My own first muzzleloader was a heavy barreled half-stock .36 caliber Ohio rifle with a back action percussion lock. The front sight was like a grain of wheat lain on the barrel. At age 15 I could see them well enough but on a warm sunny day with sights so low to the barrel, the heat shimmer made aiming pretty much impossible. Still, that style of sight was the American standard for two centuries or more. I guess the reasoning was "you need fine sights to do fine shooting". We now know that first of all you need sights you can see.
Your 35 grain load should be giving you around 1600 fps, which is more than plenty for small game and target shooting, I'd be more likely to reduce it than increase it. I would suggest you try some thicker patch material, .010" is pretty flimsy stuff and if the bore has any roughness at all those patches will be shredded.
