Double encryption with home brew algorithm

Kerckhoffs's principle states:

A cryptosystem should be secure even if everything about the system, except the key, is public knowledge.

Therefore if AES is broken, your homebrew algorithm is likely to be much easier and would have been broken already.

I realise part of your original question stated:

provided they do not have the algorithm

However, as this violates Kerckhoffs's principle your home brew approach is in itself flawed and insecure.

The main reason this is a problem is that it is unquantifiable how much more security it adds to your system. I refer you to Thomas Pornin's answer to a similar question here:

An additional twist is that algorithm obscurity can harm security. What I explain above is that obscurity cannot be trusted for security: it might increase security, but not by much (and you cannot really know "how much"). It turns out that it can also decrease security. The problem is the following: it is very hard to make a secure cryptographic algorithm. The only known method is to publish the algorithm and wait for the collective wisdom of cryptographers around the world to gnaw at it and reach a conclusion which can be expressed as either "can be broken that way" or "apparently robust". An algorithm is declared "good" only if it resisted the onslaught of dozens or hundreds of competent cryptographers for at least three or four years.

So therefore you should use another algorithm if you are worried that AES may soon be broken (that or don't use AES at all). I'm assuming the rest of your system isn't 100% secure - so instead of wasting time, effort, energy and resource on creating a home brew algorithm this effort should be spent elsewhere, where it can be quantifiable that it is actually increasing security. All code has a cost, and it is not just the salary of the junior developer that is creating it. It needs to be maintained and understood by everybody - and each extra person that learns how it works is an extra avenue for the algorithm to be leaked. Don't waste your time.

It could work, and would be security-in-depth instead of security-by-obscurity, but there are a few ways to mess this up catastrophically:

• Using a HomeBrew algorithm that is vulnerable to side-channel attacks. Now the attacker can do a simple timing or cache analysis, for example, and bypass the AES part entirely. Are you that secure in your implementation?

• Reusing material between the algorithms. If you reuse keys and nonces, for example AES(HomeBrew(openText, "secret"), "secret"), it might be vulnerable to cryptanalysis. This is a distant threat, but necessary if you want to keep the same security standard of just AES(openText).

• Using a brittle HomeBrew implementation. If it might break when the text contains nulls, or are above a certain size, or are below a certain size, or look like this, or contains valid unicode, or contains invalid encodings, etc. This is a programming concern, but opens your software to anything from denials of service to improper authentication or worse.

All in all, I find this a fun idea, but not a practical one. Maybe using something that is not homebrew would work better; there are safer ways to add depth to your security.

By itself, your home-brew algorithm is a form of security through obscurity. However, when applied in combination with a known good encryption, your home-brew algorithm may be considered a defense in depth strategy. Quoting directly from Wikipedia,

A system may use security through obscurity as a defense in depth measure; while all known security vulnerabilities would be mitigated through other measures, public disclosure of products and versions in use makes them early targets for newly discovered vulnerabilities in those products and versions. An attacker's first step is usually information gathering; this step may be delayed by security through obscurity.

If a vulnerability in AES is publicly disclosed today, it would still take some time for an attacker to figure out how to crack your home-brew algorithm, giving you time to switch to another algorithm. As the saying goes, "don't put all your eggs in one basket".

For the really paranoid, you can adopt two layers of different standard encryption such as AES+DES. Having multiple layers of different encryption is not cost-free though. The trade-off is that your ciphertext becomes chunkier.