Create X509Certificate2 from Cert and Key, without making a PFX file

There are a couple of different things you're asking for, with different levels of ease.

Attaching a private key to a certificate

Starting in .NET Framework 4.7.2 or .NET Core 2.0 you can combine a cert and a key. It doesn't modify the certificate object, but rather produces a new cert object which knows about the key.

using (X509Certificate2 pubOnly = new X509Certificate2("myCert.crt"))
using (X509Certificate2 pubPrivEphemeral = pubOnly.CopyWithPrivateKey(privateKey))
{
    // Export as PFX and re-import if you want "normal PFX private key lifetime"
    // (this step is currently required for SslStream, but not for most other things
    // using certificates)
    return new X509Certificate2(pubPrivEphemeral.Export(X509ContentType.Pfx));
}

on .NET Framework (but not .NET Core) if your private key is RSACryptoServiceProvider or DSACryptoServiceProvider you can use cert.PrivateKey = key, but that has complex side-effects and is discouraged.

Loading the private key

This one is harder, unless you've already solved it.

For the most part the answer for this is in Digital signature in c# without using BouncyCastle, but if you can move to .NET Core 3.0 things get a lot easier.

PKCS#8 PrivateKeyInfo

Starting in .NET Core 3.0 you can do this relatively simply:

using (RSA rsa = RSA.Create())
{
    rsa.ImportPkcs8PrivateKey(binaryEncoding, out _);
    // do stuff with the key now
}

(of course, if you had a PEM you need to "de-PEM" it, by extracting the contents between the BEGIN and END delimiters and running it through Convert.FromBase64String in order to get binaryEncoding).

PKCS#8 EncryptedPrivateKeyInfo

Starting in .NET Core 3.0 you can do this relatively simply:

using (RSA rsa = RSA.Create())
{
    rsa.ImportEncryptedPkcs8PrivateKey(password, binaryEncoding, out _);
    // do stuff with the key now
}

(as above, you need to "de-PEM" it first, if it was PEM).

PKCS#1 RSAPrivateKey

Starting in .NET Core 3.0 you can do this relatively simply:

using (RSA rsa = RSA.Create())
{
    rsa.ImportRSAPrivateKey(binaryEncoding, out _);
    // do stuff with the key now
}

(same "de-PEM" if PEM).