Alec,
I have the Compact in 308. It can shoot better than I can. If you weigh at least 100 lbs. you should be able to handle the 308. I believe can get "reduced" power loads. These will be adequate and you have much more versatility in bullet weights in the 308. At the bench be kind to yourself and wear hearing and eye protection. Also pad your shoulder with a beach towel and do not shoot more than 3 rounds without a break. When shooting at a deer you will probably only need 1 shot, and you'll be so excited you won't even notice the recoil.
If the above is not an option, I would move up to the 260 Remington.
MY $.02.
ScoutMan.