I just got an Xperia Ray a few months ago. Maybe it's just because it's my first smartphone, but I love it. I think the thing to keep in mind is that it's more like a phone that can do smartphone stuff. It's not like the S3 that's a pocket computer that can do everything (oh and you can phone people with it too).
Sometimes there is some lag when switching apps, loading through picture albums, etc. I'm pretty patient so it usually doesn't bother me. I rarely remember it stuttering when scrolling browser windows, but it does happen. On the other hand, the battery life is actually better than any of my old dumbphones; I have a screenshot of the battery lasting five days. During that time I used the phone exactly like I had my old phone, too, so it wasn't like I put it in airplane mode to turn off the radio and stuffed it in a sock drawer for the week.
It's expected that it'll be underpowered compared to flagship products, but I'd say one of the biggest things to watch out for is the small screen size. I actually love its small size, but it does strain my eyes a bit when I'm doing heavy reading (and I still have young eyes). I've compared its size to the S3 and its screen is half the size. I have small fingers, too, so I can type on the on-screen QWERTY keyboard while the phone's in upright orientation, but some people will need to either rotate the phone to type on it or set the option to use good old dialpad T9 when upright.
Wasn't the Motorola Defy's main claim to fame being ruggedized and waterproof? I think there were a few compromises that it made it order to do that.
Supposedly, the Defy can (eg) be dropped in a toilet and still work (which is actually a useful property, given that we have two small children, but hardly a deciding factor)
EDIT - how rugged is the Ray? A review I saw claimed the screen scratches easily)