strongSwan IKEv2 + Windows 7 Agile VPN: What is causing Error 13801

Figured this out. @ecdsa pointed me in the right direction, and I finally was able to solve the problem by following this guide.

ipsec pki --gen --type rsa --size 4096 --outform pem > vpnca.key.pem
ipsec pki --self --flag serverAuth --in vpnca.key.pem --type rsa --digest sha1 \
    --dn "C=US, O=Example Company, CN=Example VPN CA" --ca > vpnca.crt.der
ipsec pki --gen --type rsa --size 4096 --outform pem > vpn.example.com.key.pem
ipsec pki --pub --in vpn.example.com.key.pem --type rsa > vpn.example.com.csr
ipsec pki --issue --cacert vpnca.crt.der --cakey vpnca.key.pem --digest sha1 \
    --dn "C=US, O=Example Company, CN=vpn.example.com" \
    --san "vpn.example.com" --flag serverAuth --outform pem \
    < vpn.example.com.csr > vpn.example.com.crt.pem 
openssl rsa -in vpn.example.com.key.pem -out vpn.example.com.key.der -outform DER

cp vpnca.crt.der /etc/ipsec.d/cacerts
cp vpn.example.com.crt.pem /etc/ipsec.d/certs
cp vpn.example.com.key.der /etc/ipsec.d/private

About the error

The error message was "Error 13801: IKE authentication credentials are unacceptable", which sounded like my user credentials weren't working. However, this is a message about authenticating the server, which is done (per my configuration) by the server's SSL certificate. Microsoft has published documentation on Troubleshooting IKEv2 VPN Connections that lists possible causes for this error:

• The certificate is expired.
• The trusted root for the certificate is not present on the client.
• The subject name of the certificate does not match the remote computer.
• The certificate does not have the required Enhanced Key Usage (EKU) values assigned.

In my case, my problem had to do with the EKU values. Following the guide I linked at the top, I was able to generate a certificate with the correct EKU values, and it worked great.

To troubleshoot this, you can disable EKU checking on your Windows client (of course, this should only be done for testing):

• Launch regedit
• Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\RasMan\Parameters
• Add a DWORD called DisableIKENameEkuCheck, and set its value to 1
• Microsoft documentation instructs you to reboot after doing this, but I didn't need to in order for this to take effect.

I had an identical problem and solved it by ensuring I had the certificate chain in the certificate file (end-entity cert, intermediate CA, root CA - in that order). TLS is fun.

After restarting strongSwan, this stopped working, but started working again when I dropped the intermediate and root CA into /etc/ipsec.d/cacerts.