Why I can't read openssl generated RSA pub key with PEM_read_RSAPublicKey?

You might try PEM_read_RSA_PUBKEY() instead of PEM_read_RSAPublicKey().

This is all about formats.

The default public key file format generated by openssl is the PEM format.

PEM_read_RSA_PUBKEY() reads the PEM format. PEM_read_RSAPublicKey() reads the PKCS#1 format.

So if you want to stick to PEM_read_RSAPublicKey() you could generate the public key file using the PKCS#1 format by specifying the -outform DER option when generating the public key.

it seems there are two format of rsa public key, with different encoding.

A. RSA_PUBKEY

RSA* rsaPubKey = PEM_read_bio_RSA_PUBKEY( bio, NULL, 0, pass ) ;

read PUBKEY with this format

-----BEGIN PUBLIC KEY-----
...
-----END PUBLIC KEY-----

generated by

$ openssl rsa -in key.pri -pubout -out key.pub1

B. RSAPublicKey

RSA* rsaPubKey = PEM_read_bio_RSAPublicKey( bio, NULL, 0, pass ) ;

read PublicKey with this format

-----BEGIN RSA PUBLIC KEY-----
...
-----END RSA PUBLIC KEY-----

generated by

$ openssl rsa -in key.pri -RSAPublicKey_out -out key.pub2

convert

A to B format

$ openssl rsa -in key.pub1 -pubin -RSAPublicKey_out -out key.pub2_

B to A format

$ openssl rsa -in key.pub2 -RSAPublicKey_in -pubout -out key.pub1_