Probably because they weren't accurate enough combined with the area nature of the targets.
Actually, the later ones did have sights although most were not located as would be typical for small arms. The front sights tended to be on the rimbase instead of the muzzle and, consequently, the rear sights were on the rear of the breech to one side instead of the top. In the photo of a Parrott rifle below, the front sight (with bent to the right top) is to the left of the cap square and the rear sight base is in the lower left corner of the image.
Below is a different picture of the front sight that shows its location better.