Though I'm a warrior and have no need of a mount (unless I want to pretend to be a Pally), here's what my caster friends have told me about horses.
The biggest advantage is lack of sit-aggro, while still getting mana regen as if sitting. One of my guildies made an exhaustive study of mana regen while bored one time, and found that No, you don't regen mana while moving, and faster than you would unmounted. Nor do you regen mana faster while attacking or casting. Only while standing still. But at least it means you don't have to worry about a mob pounding on you just because you sat down to med.
Drawbacks of horses:
They can't be SoW'd, so if you got a cheap horse, your movement speed could be lower than without them.
You can't use them indoors.
You can't use "ducK" while on a horse...so you need to actually move if you need to interrupt a spell for any reason.
Horses eat like...well....like horses. So upkeep can get rather pricey.
Now, whether you still think it's better to have one than not is for you to decide. I'm just a warrior, I don't know first hand =)