My Glock 23 has the 3.5 connector installed. If you use the gun primarily for carry, I would leave the 5lb connector in. I would worry a bit with the lighter pull if my adrenaline was pumping and finger on the trigger. If it's a target gun, it is much easier to target shoot well.
As a corollary, I did NOT find that the lighter trigger increased my accuracy much when going rapid fire (ie, if I was trying to put 3 center-of-mass on a bad guy).