Not knowing what rifle you are looking at makes it difficult for me to give a really good recommendation. Some rifles are much better in one caliber than they are in another. Speed is not the most important criteria in an air rife, regardless of if you are punching paper or fur. Accuracy is what counts.
IMHO, windage is the biggest hurdle in hunting with a pellet rifle. If you are used to firearms, take a look at the BC's of typical domed pellets (don't even bother with anything but domed if you are shooting past 30 yards). I can range find much better than I can read wind. Therefore, I typically go with .22, but under calm, really calm conditions, I still use a .177 at times.