NEF/H&Rs and a lot of other rifles are notorious for needing a fouling round or two before shooting to the same POI, no secret there, just don't clean it before you hunt with it, I've been doing that for the better part of 50yrs with all my rifles, zero it, then leave it dirty until the season is over or it gets wet needs to be cleaned.
Tim