It all depends on the amount of recoil you are willing to tolerate. The way to make a light recoiling load is to shoot the lightest bullet that can do the job, say 85 or 100 grains at very modest velocity, but that is not necessarily the best use of your 25-06"s potential, or the best deer killer either. I think if the recoil of a 25-06 was bothering me I would start with 120 gr bullets and make a load that shot about 3000 fps. See if you can handle that. If it is still too much for you then go to the 100 gr and 3000 fps. I wouldn't go lighter than 100 gr for big game, and I would try to keep velocities at around 3000 unless I just couldn't handle it.
Remember that what you are experiencing at the bench is a lot different than field shooting. A load that will punish you firing 10 or 20 rounds on the bench , you would never notice shooting one round offhand at a deer. On the other hand I commend you for acknowledging that you don't like recoil and not shooting something that is going to make you flinch. A mild load shot straight is 1000 times better than a hard shooting one that is jerked.