Try a 95 to 100gr SWC (I like RCBS 98gr, Lee's 95gr is not bad) and find some 4756. Work up but stop at 5gr. Heavier loads have been listed, but that Army Special is 90 years old and steels and heat treatment were not as refined as now. I have used Unique, 2400 and a dozen other pistol powders for 45 years, but for some reason, 4756 seems to be the trick for consistant performance in the .32WCF. Loads heavier than this also begin to develop an excessive muzzle blast. Do not begin to think that your Army Special has anywhere near the strength of a Colt SAA and whatever you do, do not use loads listed for RIFLE ONLY.
Have you checked the barrel to see where your bulge is? A very high percentage of pre-war .32-20 revolvers have a barrel bulge. At Handloader magazine, Pearce and Scovill found that the inconsistant performance (low velocity) of high friction JACKETED bullet factory ammo would leave a bullet in the barrel often enough to be disconcerting. Lead bulleted ammo did not seem to be an issue due to its lower barrel friction.
Take care and enjoy that Army Special. They are hard to come by and mine's not for sale.