Is Telegram secure?
TL;DR: No, Telegram is not secure.
I'd like to ignore the comparison to WhatsApp because WhatsApp does not advertise itself as a "secure" messaging option. I'd like to instead focus on whether Telegram is secure.
Telegram's security is built around their home spun MTProto protocol. We all know that the first rule of Cryptography is Don't Roll Your Own Crypto. Especially if you aren't trained cryptographers. Which the Telegram people most certainly aren't.
The team behind Telegram, led by Nikolai Durov, consists of six ACM champions, half of them Ph.Ds in math. It took them about two years to roll out the current version of MTProto. Names and degrees may indeed not mean as much in some fields as they do in others, but this protocol is the result of thougtful and prolonged work of professionals.
Source: https://news.ycombinator.com/item?id=6916860
Math Ph.Ds are not cryptographers. The protocol they invented is flawed. Here is a nice blog post explaining why. In addition to that, Telegram has issued a rather ridiculous challenge offering a reward to anyone who can break the protocol. Except that the terms they set makes even the most ridiculously weak protocol difficult to break. Moxie Marlinspike has a nice blog post explaining why the challenge is ridiculous.
So, no. Telegram is by no means secure. For commonly accepted definitions of secure, not the one Telegram made up.
If you want a real secure means of communication on your phone, look to more reputable projects such as Signal or WhatsApp (which, since this answer was first written, now uses the Signal Protocol for end-to-end message encryption).
UPDATE
- 09 January 2015: A new 2^64 attack On Telegram has been announced.
- 12 December 2015: A new paper demonstrating that MTProto is not IND-CCA secure.
- 22 December 2017: Replaced outdated recommendation for CryptoCat with a more up-to-date recommendation for Signal and WhatsApp.
As the Telegram FAQ mentions, there is a 'secret chat' option that does not store chats on their servers.
As for the underlying question of, "does storing chats lower their security?" then that is something to consider. Chats being stored on the server does mean that copies can be made on the server for decryption later. This increases the exposure of the messages. Encrypting the messages means that there is a high cost to decrypt the messages, but there is still some exposure.
Taking this added exposure into account, the real question becomes (as it always does), "what are you protecting from?" If you are worried about secure communications in transit, then Telegram 'appears' to be more secure. If you're worried about secure communications at rest, then WhatsApp 'appears' to have a better model, except that none of it is encrypted.
The answer, then, is 'it depends on your focus', and encryption is better than non-encryption, and there is the Telegram's 'secure chat' option.
November 2015:
New research shows deep problems with the crypto: https://medium.com/@thegrugq/operational-telegram-cbbaadb9013a#.gb7od1j6i
EFF's Secure Messaging Scorecard currently rates "Telegram (secret chats)" with a 100% security rating. However, the software of the servers Telegram uses is not open; cf. the FAQ "Why not open source everything?"
WhatsApp was docked on the "Is the code open to independent review?" metric. Telegram is now completely open; source code here. Being open, you can verify for yourself that there is no back-doors which could possibly be in a closed app. WhatsApp is closed-source now that it's gone proprietary (Facebook bought it).
A good alternative is Tox or Signal, which is open and peer-to-peer / end-to-end encrypted only and received a high EFF rating.