Is a horse actually faster than running?
Not only is the horse faster (not even considering their higher base stamina -- Horses can sprint much longer than a stamina-light Dohvakiin!), but they can be ridden while encumbered.
So if two Dovahkiin (Dovahki?) were to race across a flat plain, one on a horse and the other on foot, the horse-bound hero would win every time. Horses are marginally faster
So what about when you're not traveling in a straight line?
Turns out, the horse will still win, as, despite their cumbersome turning, horses flaunt a blatant disregard for physics that can quickly turn into an incredible timesaving path.
Consider: When you're on foot, you must run around any mountains in your path.
Horses, on the other hand, have no qualms with running straight up the mountainous slopes.
Image credit to Emperorr of Reddit.
Don't forget that you can sprint with a horse.
It might be that you know that you can sprint on your character, but not on your horse. Please note that your horse is also able to sprint using stamina. You can do this using the default Alt key.
Horses are indeed faster, transportation was simplified.
Since Oblivion, they took Acrobatics and Athletics skills away, thus you can't improve your speed in such way anymore that it would be faster than a horse. The speed improvements by not being Fatigued and not being overweight are not enough to be significant.
All horses in Skyrim have the same speed and hardiness, they are all draft types that resemble a Clydesdale, and all cost 1000 septims.
— Wikia - Elder Scrolls V: Skyrim - Horse
All horses in Skyrim are the same, seems that they wanted to simplify the whole transportation idea.
Restrictions apply on both your character and horse.
Without mods, a horse can't be used inside a city. If you do a lot of role-playing within cities having a horse might not be useful. At the same time, characters are limited in Skyrim as Raven Dreamer noted; while you can get over some mountains by jumping left and right trying to get over a rock, a horse will be more superior in performing this task as it is less restricted by physics. You can however be smart enough to choose your path right, rather than trying to work against the physics...
In a loading text they stated that it is not faster but the horse can run for a longer time, thus making it better.
As a bonus you can still move fast on the horse, even when you are overloaded by your loot!