Yep, Try different bullets. My own 700 Rem 7mm, shot X bullets all over the place, not that there a bad bullet, it just takes to long and to many of them + primers + powder + fiddling with overall length etc, to find a load that groups. Can get rather frustrating, as I think you are finding out. Smallish deer out to long range, if you are set on 120's try Noslers BT, contrary to some of the opinions it isn't to fragile i.e. it does work and is very accurate. BUT, a 140 will serve your needs better.
A 139gr Hornady Interlock is by far the best 7mm deer bullet ever made. (Think I might have some experience with that one?) CRASH87