Your action may be moving around on your stock (it don't take much). Also, if you have 2 screws holding it in, try tightening the back one and just lightly snug the front one (just enough that it don't fall out). Black Diamonds were bad for this. Let us know how any of the previously mentioned suggestions (all sound) help out.
Denny