Is there such a thing as a restore point in Ubuntu?

You can try Back in Time:

sudo add-apt-repository ppa:bit-team/stable
sudo apt-get update
sudo apt-get install backintime-qt4

Note: GNOME/KDE UI has been deprecated, hence -qt4 now. See here.

There are different backup-utilities available, but the functionality of the "Windows Restore System" does not exist in that form (at least not that I know of) (since it's not really necessary see Footnote).

If you have a small system partition you can easily create an image of it using dd, saving it to disk and if necessary restore it.

dd if=/dev/sdXX of=/media/backup/system.img bs=1M

Footnote: I don't see the need for such functionality because there's no way to wreck the system by installing something as long as you stick to apt. Updated kernels might not boot or work correct, that's why the old ones are remaining (and can still be booted) until they're removed by hand. Of course you can still screw everything, but that's a lot harder to do.

Time Shiftcan also do the same.

sudo apt-add-repository -y ppa:teejee2008/ppa
sudo apt-get update
sudo apt-get install timeshift