I toyed with lighter bullets in my 270, 110's, even installed a target trigger and higher mag scope on it just to get some use out of it on varmints since I no longer hunt big game. Ended up just using up the 130's I had left for deer, and later when they ran dry the 150's I had used for elk.
For vermin you're not going to eat or save fur from any bullet your rifle likes will do just fine, including your deer bullet, and it would be lots of very good practice for when you do go after Bambi.
FWIW