It is a combination of what you have mentioned, but most importantly, it is a matter of the construction of the bullet.
They softness of the lead, the type of jacket material (where a jacket is involved) and any special little "goodies" different manufacturers hype on (Gold Dot, JHC vs. JHP, Hydra-Shok and Hydro-Shok, XTP, etc.).