Smart Card for RSA private key for SSL?

I've never seen configuration in Apache or nginx which would seem to indicate support for anything other than file-based SSL private keys.

You are using the Yubikey as a Hardware Security Module (HSM). They key is stored on and never leaves the HSM and all encryption and decryption is done on the HSM. The Yubikey seems to fulfill such the requirements.

You might need to write your own PKCS#11 driver since there isn't one out there. For the server side, you would have to use mod_nss instead of mod_ssl. mod_nss allows you to use a custom PKCS#11 driver whereas mod_ssl uses the default openssl engine.

Compared with other purpose built HSMs, I am unsure of the stability of using the Yubikey as a HSM. This is because the main use case of a yubikey is for applications which require one-time access, e.g. authentication, encrypting an email and not for applications which require continuous access. A server under load might overwhelm the Yubikey and result in sluggish performance.