It could be a combination of the things you mention.
First, leaving lube on the cases or oil in the chamber is a big NO-NO with the single shots. But your load is "up there" and may well be over max for your particular gun regardless of the lube/oil.
I'd drop the charge about 5 grains, clean and dry the chamber (swab w/ acetone) and make sure the cases are free of lube.
Load up a couple rounds, try them -- and then if all is well, work up a half-grain at a time.