How to Load RSA Private Key From File

Two things. First, you must base64 decode the mykey.pem file yourself. Second, the openssl private key format is specified in PKCS#1 as the RSAPrivateKey ASN.1 structure. It is not compatible with java's PKCS8EncodedKeySpec, which is based on the SubjectPublicKeyInfo ASN.1 structure. If you are willing to use the bouncycastle library you can use a few classes in the bouncycastle provider and bouncycastle PKIX libraries to make quick work of this.

import java.io.BufferedReader;
import java.io.FileReader;
import java.security.KeyPair;
import java.security.Security;

import org.bouncycastle.jce.provider.BouncyCastleProvider;
import org.bouncycastle.openssl.PEMKeyPair;
import org.bouncycastle.openssl.PEMParser;
import org.bouncycastle.openssl.jcajce.JcaPEMKeyConverter;

// ...   

String keyPath = "mykey.pem";
BufferedReader br = new BufferedReader(new FileReader(keyPath));
Security.addProvider(new BouncyCastleProvider());
PEMParser pp = new PEMParser(br);
PEMKeyPair pemKeyPair = (PEMKeyPair) pp.readObject();
KeyPair kp = new JcaPEMKeyConverter().getKeyPair(pemKeyPair);
pp.close();
samlResponse.sign(Signature.getInstance("SHA1withRSA").toString(), kp.getPrivate(), certs);

You need to convert your private key to PKCS8 format using following command:

openssl pkcs8 -topk8 -inform PEM -outform DER -in private_key_file  -nocrypt > pkcs8_key

After this your java program can read it.