Pattersons came in 28 (265), 31,34 and 36 caliber. Buckshot sizes are .27 #2, .32 "0", .34"00" and .38 is "0000". The sizes may have come from that.
Why the Patterson evolved to the holster size 36 cal is a mystery, other than the "0000" buck size, but the conical bullet for a 36 Navy is indeed 140gr, the same as the 44 roundball, at 50 per pound.
Perhaps the 50 per pound just reaffirmed the 36 caliber as a choice. The .28 and .34 calibers were not used after the Pattersons except for the .28 (actually .265) in the Colt Root pocket pistols.
A note of trivia from James Serven's book on the history of Colt is that the .54 caliber single shot horse pistol roundball weighs the same as the 44 caliber conical at 1/2 oz. (approximately).