I've gone 5.1 as well, so far so good. Can't say what the difference in performance really is because I also did a factory reset. Having done both the phone is blazing fast now and battery life improved a lot, much less battery drain when on standby too.
Did you receive the OTA update or manual flash?