AES CTR 256 Encryption Mode of operation on OpenSSL

Usually, you will be intending to call AES_ctr128_encrypt() repeatedly to send several messages with the same key and IV, and an incrementing counter. This means you need to keep track of the 'ivec', 'num' and 'ecount' values between calls - so create a struct to hold these, and an initialisation function:

struct ctr_state {
    unsigned char ivec[16];  /* ivec[0..7] is the IV, ivec[8..15] is the big-endian counter */
    unsigned int num;
    unsigned char ecount[16];
};

int init_ctr(struct ctr_state *state, const unsigned char iv[8])
{
    /* aes_ctr128_encrypt requires 'num' and 'ecount' set to zero on the
     * first call. */
    state->num = 0;
    memset(state->ecount, 0, 16);

    /* Initialise counter in 'ivec' to 0 */
    memset(state->ivec + 8, 0, 8);

    /* Copy IV into 'ivec' */
    memcpy(state->ivec, iv, 8);
}

Now, when you start communicating with the destination, you'll need to generate an IV to use and initialise the counter:

unsigned char iv[8];
struct ctr_state state;

if (!RAND_bytes(iv, 8))
    /* Handle the error */;

init_ctr(&state, iv);

You will then need to send the 8 byte IV to the destination. You'll also need to initialise an AES_KEY from your raw key bytes:

AES_KEY aes_key;

if (AES_set_encrypt_key(key, 128, &aes_key))
    /* Handle the error */;

You can now start encrypting data and sending it to the destination, with repeated calls to AES_ctr128_encrypt() like this:

AES_ctr128_encrypt(msg_in, msg_out, msg_len, &aes_key, state->ivec, state->ecount, &state->num);

(msg_in is a pointer to a buffer containing the plaintext message, msg_out is a pointer to a buffer where the encrypted message should go, and msg_len is the message length).

Decryption is exactly the same, except that you do not generate the IV with RAND_bytes() - instead, you take the value given to you by the other side.

Important:

0. Do not call init_ctr() more than once during the encryption process. The counter and IV must be initialised once only prior to the start of encryption.

1. Under no circumstances be tempted to get the IV anywhere other than from RAND_bytes() on the encryption side. Don't set it to a fixed value; don't use a hash function; don't use the recipient's name; don't read it from disk. Generate it with RAND_bytes() and send it to the destination. Whenever you start with a zero counter, you must start with a completely fresh IV that you have never used before.

2. If it is at all possible that you will be sending 2**64 bytes without changing the IV and/or key, you will need to test for the counter overflowing.

3. Do not omit error-checking. If a function fails and you ignore it, it's quite possible (even likely) that your system will appear to be functioning normally, but will actually be operating completely insecurely.

It looks like the basic problem with your test program is that the mode values of the fopen calls is not correct. I think you need to change your fopen calls in encrypt to this:

fp=fopen("input.txt","rb");
op=fopen("output.txt","wb");

And the ones in decrypt to:

rp=fopen("recovered.txt","wb");
op=fopen("output.txt","rb");

One other thing worth pointing out is that ckey should probably be declared as a 32 byte (256 bit) buffer. It is true that the 128-bit encryption only uses 16 bytes of the data from the key. But the OpenSSL function AES_set_encrypt_key (at least in the version I am using) reads 32 bytes from that buffer. It only uses the appropriate number of bytes, but the read does occur. That means that if the buffer is only 16-bytes and happens end at the end of a page that is adjacent to a non-readable page in memory, it would result in an access violation.

Oh - and I just noticed that there is an extraneous call to free in there. The free(buffer); call is not valid since buffer was never allocated. I realize your code is just a simple test, but ... well, we are programmers and can't help ourselves.