i'd expect a definite difference in point of impact, with any given cartridge recipe/load to shoot lower on the target with the weight added. the further towards the frame the weight is put the lower i'd guess it would shoot.
by removing the weight i'm guessing that your rifle will shoot higher is what i'm saying. the closer the weight was to the recoil pad, however, the less i'd expect the point-of-aim to be impacted. if shooting in dense woods i'd definitely re-sight the rifle to help with shooting over/under branches where one needs to "thread the needle".
changing a scope on a rifle, naturally, requires re-sighting the firearm.
good shooting to you,
ss'