The reason the Hogue grips feel funny to you guys is just what I stated. Normally with a set of factory grips, your finger is behind the trigger guard, the Hogue fills that area in. Your hand is then holding the gun lower on the frame, which puts the bore at a higher center above your hand, resulting in more pronounced recoil. That effect is, of course, the exact opposite of why you put them there in the first place in most circumstances. The other issue I have with them is that they look "funky", for lack of a better term with any more meaning than that one, because it makes the trim and packable single action look like it has frying pan handle hanging off of it.
On a S&W N frame you'd have to pry them away from me, wood or rubber, but on the Rugers it's different.