Hammer-
Like everyone else is saying, pretty much just recoil. Bullet placement, and penetration are what kill big game, especially at handgun velocities. Foot pounds of energy is not a good measurement of killing ability. Heck, guys wound deer with 300 magnums, and they still run off. A bad shot is a bad shot...period.
I have shot a fair amount of big game with .44 mag / .45 LC, and have found a hard cast 250 - 300 grain bullet at 1000 - 1200 fps are plenty to get good penetration, and quick kills. In my experience, a deer hit with this load in the chest will travel 25 to 50 yards, then go belly up. A lighter bullet, say 180 grain, at say 1500 fps, might drop a deer a little quicker on a perfect broadside shot, but how often do those perfect shots present themselves? Don't depend on a lightweight handgun bullet at high velocity to provide deep penetration on deer size game.
I have used the Speer 300 grn PSP, and the 300 grn XTP, loaded hot, out of a long barrel Ruger Redhawk, to take several bucks. They didn't drop any quicker then those shot with loads that were a little milder.
Larry