If you had damaged the cylinder, or bulged it, you would know it. You can sight down the outside, catch the reflections of the light and see a bulge, but the Remington cylinder is so close to the frame, a bulge would stop the rotation, and a detonating charge would have cracked something. Most likely, a charge as light as 20 gr would still put the ball on top of the powder within the stroke of the loading lever.
If you worry about that and want to load 20 gr or less, get a cylinder loading tool, load off the gun, and you can be more sure of contacting the powder charge. A Wonder Wad also fils up some chamber space.