Apple - What is wrong with my iMac? (video corruption, unstable)

Backup first! Copy all your essential files, and make a Time Machine backup if possible.

It can be about software or hardware, it's hard to say.

I would first recommend creating a new user account and try there.

Then, reset NVRAM and SMC and try if that help.

The last good solution is Safe Mode and cleaning cache with apps like CCleaner or CleanMyMac.

The bad solution is just installing Mountain Lion on the same partition without cleaning your old system. If that doesn't work, upgrade to Mavericks and try. If that still doesn't work, you might need to do a clean install.

[EDIT: You can also just install to an external drive.]

Please remember to backup your Mountain Lion installation app if possible, or you might need to find is somewhere else later.

If all those above doesn't work, you might have to send it in for repair.

EDIT: Try BootCamp and run a benchmarking tool which pushes your CPU and/or GPU to the limit. That will show if it's a hardware problem.

UPDATE: Check out this Apple Discussion Thread, there's apparently people having problem like yours.

There's also a guide to testing your memory, something you can do if you want to.