How to handle improper key sizes (AES encryption)
This is an API question, not an cryptography question. The AES standard (FIPS 197) is silent on how nonstandard key sizes are handled. The AES implementations I've seen all reject nonstandard key sizes with an error; however a higher level crypto implementation (that does more than 'just AES') might choose to provide more services (depending on the job it is designed for).
On the other hand, the question typically doesn't come up; we most often generate an AES key with a key derivation process, where we specifically ask that process "give us 128/192/256 bits of key", and that's how much key it gives us.
As for your alternatives, IMHO, silently truncating the key value is Evil; what is the right thing would depend on what you're actually doing (and who's actually generating the key, and why might it not be a standard size).
Also, ECB mode? Unless you know exactly what you're doing (and if you have to ask this question, you don't), you should avoid it.
I'll answer as developer and designer of some additions to Crypto-API's.
First of all, Poncho is correct that no specification of AES or mode of operation (such as the insecure ECB) will indicate what to do in this case. They will almost certainly not propose #2 or #3, if just because that would severely upset any security proof of the algorithm.
1. returning an error ?
The only correct answer in my opinion. AES only accepts 128, 192 or 256 byte keys. This is called a pre-condition. Failing to comply with a pre-condition should result in a failure. In Java I would quickly upgrade it to a RuntimeException as providing fewer bits for a symmetric key is always a programming error (you don't expect a user to directly enter a key, and if a hexadecimal key does need to be entered, you need to verify it before offering it to the crypto API).
The only place where this might not be the best answer is very low level API's where just a pointer to a key with known size is returned. But users should never call or even have access to that low level API.
Returning an error (or exception, for higher level languages) is also fail-fast, which is a good security design practice.
2. truncating the key to 128-bit ?
No that's horrible. You should not perform a cryptographic function with less security relevant data than the user provided. You're would be deciding for the user that less security is acceptable, something that obviously is not the role of an API designer.
Extending the key with zero's is equally horrible. You're telling your user that they are using AES-256 while the actual security offered is lower than that (and possibly even next to zero).
Just like (3) this will also fail the principle of least surprise.
The PHP mcrypt based functions take this route and there is a never ending stream of questions on SO trying to replicate mcrypt functionality in well behaved crypto API's.
3. treat the key as password and use some KDF function?
No. If the user wants to use a KDF, then offer them a KDF. Performing a KDF in a mode of encryption for AES violates all kind of design principles, including the principle of least surprise. An AES API should just do that: offer a cipher interface to obtain confidentiality. An API could make it easy to use a KDF, but it should not itself choose a KDF and set of parameters and make that a default (which at that time seems secure, and can never be changed afterwards).
There are many users of the CryptoJS API that mistake the encryption routine that uses a password with the one that uses a key, just because they have the same name.
Creating an encryption function that accepts a password is OK for a well defined higher level API such as one implementing OpenPGP.
There is the PKCS#5 Password Based Encryption standard. Unfortunately, because the that standard leaves too many configuration open without specifying a specific binary container format, it can still be hard to make PKCS#5 compatible ciphertext compatible between runtimes. So it can only be used as a start to define a password protected container format.
Notes:
- Some Crypto API's like Java do not directly accept a binary key. The key must first be generated from the bytes. One reason to not directly use bytes as key is that hardware devices (HSM's, smart cards) may not even expose the bytes to the software in the first place. In that case the error may be thrown in advance, depending on the implementation.