Encryption/decryption doesn't work well between two different openssl versions

The default digest was changed from MD5 to SHA256 in Openssl 1.1

Try using -md md5

cgs@ubuntu:~$ echo "it-works!" > file.txt
cgs@ubuntu:~$ LD_LIBRARY_PATH=~/openssl-1.1.0/ openssl-1.1.0/apps/openssl aes-256-cbc -a -salt -in ~/file.txt -out ~/file.txt.enc -md md5
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
cgs@ubuntu:~$ LD_LIBRARY_PATH=~/openssl-1.0.1f/ openssl-1.0.1f/apps/openssl aes-256-cbc -a -in ~/file.txt.enc -d
enter aes-256-cbc decryption password:
it-works!

The ugly details:

The entered password is not used as is by aes (or other encryption) but the command implicitly derives a key from it. The key derivation uses message digest that was changed in openssl 1.1 Use SHA256 not MD5 as default digest.

In case you want to keep it simple password, and not start messing with the keying martial (-K,-iv) just force the same digest with -md

I tested the AES encryption and decryption with version 1.1.0a (downloaded from openssl.org) and the version 1.0.2g-fips (from my ubuntu 16.04)

When using the -p option on with 2 different versions of openssl, the IV and key are different:

$ LD_LIBRARY_PATH=~/openssl-1.1.0a/ ~/openssl-1.1.0a/apps/openssl aes-256-cbc -a -p -salt -in file -out file.enc
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
salt=6A80B2A3B4CFE048
key=637E17094DF7892A7AFC14957EAA13991DFFD3273A2459EDA613F3AD8A406C38
iv =6AC7CE5C9AADC6C46C633BF5124DAFBF

$ openssl aes-256-cbc -a -d -p -in file.enc -out file.dec
enter aes-256-cbc decryption password:
salt=6A80B2A3B4CFE048
key=6220AF2E25CB0B5D9994A0A1B05503D82AC5B0B4C9015E241CACBF8BF62DAC77
iv =2DC04EF29AA57478EBE606DF87277EA6
bad decrypt
140557073118872:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:592:

I suspect a different derivation of key and IV based on the salt with the 2 versions.

If you want to get rid of this decryption error, you may remove the -salt option and use the options -K for the key and -iv in your openssl command.

This issue can also occur between OpenSSL 1.1 and LibreSSL. In this case, and in other cases where more secure message digests are available, you should avoid using -md md5 to encrypt new files since the MD5 algorithm has extensive vulnerabilities.

You should instead use -md sha256 or some other more secure message digest supported by all versions. -md md5 should only be used for decrypting old files, and they should ideally be re-encrypted using sha256. This is also mentioned in the OpenSSL FAQ:

A message digest is used to create the encrypt/decrypt key from a human-entered passphrase. In OpenSSL 1.1.0 we changed from MD5 to SHA-256. We did this as part of an overall change to move away from the now-insecure and broken MD5 algorithm. If you have old files, use the "-md md5" flag to decrypt them.

---

To check which message digests are supported by the different versions you have in play, run openssl help:

LibreSSL 2.2.7 (included with macOS 10.13 High Sierra):

$ openssl help
…
Message Digest commands (see the `dgst' command for more details)
gost-mac          md4               md5               md_gost94
ripemd160         sha               sha1              sha224
sha256            sha384            sha512            streebog256
streebog512       whirlpool
…

OpenSSL 1.1f:

$ openssl help
…
Message Digest commands (see the `dgst' command for more details)
blake2b512        blake2s256        gost              md4
md5               rmd160            sha1              sha224
sha256            sha384            sha512
…