Reading Public/Private Key from Memory with OpenSSL

SquareRootOfTwentyThree's method not work for me. Here is my solution.

BIO* bio = BIO_new(BIO_s_mem());
int len = BIO_write(bio, pem_key_buffer, pem_key_buffer_len);
EVP_PKEY* evp_key = PEM_read_bio_PUBKEY(bio, NULL, NULL, NULL);
RSA* rsa = EVP_PKEY_get1_RSA(evp_key);

You are on the right track. You must wrap the PEM key already in memory by means of a BIO buffer via BIO_new_mem_buf(). In other words, something like:

BIO *bufio;
RSA *rsa

bufio = BIO_new_mem_buf((void*)pem_key_buffer, pem_key_buffer_len);
PEM_read_bio_RSAPublicKey(bufio, &rsa, 0, NULL);

The same approach is valid for an RSA private key (via PEM_read_bio_RSAPrivateKey), but in that case you most certainly need to cater for the pass phrase. Check the man page for details.