Is there an encryption algorithm that allows for a single payload to have two different outputs based on the password?
The theoretical question here is a size/entropy question. The key rule is that the entropy of the encrypted message is the same as the entropy of the initial message. If the algorithm involves no compression, the size of the encrypted message is the size of the original message.
For the encrypted message to be decoded in two different ways depending on the key you need that it contains 2 different messages. A way would be to encrypt a real message with one key and a fake message with another key, and combine both encoded messages (after padding the shorter) for example with one byte from each. Provided you add a known id at the beginning of the messages before crypting them, at decode time, the algorithm could only give back the message that had the correct initial ID.
The problem is that there two different messages + some additional info. And having an encrypted message longer than the initial one is not that plausible...
That's the reason why I know no algorithm implementing plausible deniability, but only containers, that is a complete encrypted filesystem. This was implemented in the excellent TrueCrypt before its support was discontinued. Fortunately, a fork of the project still offers it, VeraCrypt.
VeraCrypt actually offers hidden volumes. In fact a part of the empty space of a normal volume is used to store a distinct volume with a distinct key. When you present a key, an attempt is made to find a hidden volume, if not to find a normal volume. When you open the normal volume in normal mode, nothing allows to guess that it also contains a hidden volume, simply it has some unused space which is no surprise for a file system. Of course, if you write a lot of data in that mode, you will overwrite and definitely destroy the hidden volume data. But a special mode exist to give both keys and use the external volume with no risk of overwriting the internal one.
Above was the technical part. But plausible deniability also requires the content of the external container to be worth it, because the capability of hidden containers is a well known feature. IMHO, there are two acceptable ways:
Sort your information in 3 classes, simple, sensitive and secret and store them accordingly: simple outside the container, sensitive in the external part (use the double password access) and secret in the hidden part. That way as the external container contains more than innocent files the existence of a hidden container is not evident. But you must actually use the external container, at least as much as the inner one and be prepared to disclose its content
Use the external container as your main storage. That is a common usage on corporate laptops, because it ensures that in case of theft or loss of the laptop, no confidential information are immediately accessible (addresses the confidentiality part of security...). The backside is that you should mount the container as a single one at boot, and in normal usage immediately unmount it to remount it manually in dual mode.
And depending of who the adversary is VeraCrypt pages about plausible deniability and security requirements for hidden volumes has additional advices, such as only using the hidden volume when booting from a read only CD/DVD...
As usual technical security is only a part of global security. If the information is sensitive enough that you can fear for your physical integrity, you should use physical security measures and not rely of plausible deniability alone: it is a nice tool that can help to hide secure information but not a magical silver bullet. Only Snake Oil Company can provide those...
With many common algorithms, by changing the key, it is quite possible to get any arbitrary data in the first block. Where you run into trouble is when the data is longer than they key block size. The rest of the data (beyond the first block) would look random after swapping keys, assuming you are using a secure chaining method. You could use an insecure chaining method, but I wouldn't recommend it.
You could use XOR OTP encryption. This is different from most encryption where the key is mathematically re-used. With OTP (One Time Pad), the key is the same length as the data. With XOR (one of the most common OTP) you have the flexibility to arbitrarily produce the desired message.
(Your adversary may be aware of this possibility?)
To start, create files of equal length:
- original message
- random data (the real key)
XOR together gives you
- encrypted message
then take
- fake message
- encrypted message
XOR together gives you
- fake key
To decrypt, take
- encrypted message
- real or fake key
XOR them to get
- real or fake message
As @Alpha3031 commented, you could actually encrypt one message (e.g. the fake one) with traditional AES, (more common, and requires a smaller key file) and then use XOR (less common) to produce the other. (e.g. the real message?)
XOR can be applied to any files that are sufficiently unpredictable/random. (e.g. encrypted data)
Cryptographic Camouflage
CA technologies has patented a technology known as Cryptographic Camouflage.
A sensitive point in public key cryptography is how to protect the private key. We outline a method of protecting private keys using cryptographic camouflage. Specifically, we do not encrypt the private key with a password that is too long for exhaustive attack. Instead, we encrypt it so that only one password will decrypt it correctly, but many passwords will decrypt it to produce a key that looks valid enough to fool an attacker. For certain applications, this method protects a private key against dictionary attack, as a smart card does, but entirely in software.
So yes, you can take a message (in this case, a key), and encrypt it in such a manner that there is one correct key and many "fake" keys that produce false plaintext.