That's what I did
Coming from the HTC Desire, there's indeed a big difference, in terms of speed. If you are used to using manufacturer's skin (Sense, Touchwiz, Blur, etc), I think in the beginning you will feel lack of some features like T9 dialer (I miss it on my Desire A LOT) and some widgets like Calendar, Tasks. Of course all of these can be downloaded as a 3rd party app from the Market.
The default calendar widget on ICS is crap, I think it displays only the current week's schedule. And if you like to link your friend's facebook profile into your contact, sorry but that will not work for now (You can't add Facebook in Accounts & Sync).
I have to mention another setback though, the screen is not that perfect. When brightness is set at low and you view a white or grey color page, you will notice the screen have some weird texture on it. It's like viewing an ISO 6400 photo, very very noisy. Some people said it's "normal" on an AMOLED screen, but I think it's bullcrap, even my HTC Desire AMOLED screen produces better color without this "normal" color texture. So, you got to live with that.
But overall, the high resolution screen, the contrast colors (sometimes too much though), the speed, the plain vanilla ICS, for me I think it is worth getting.