Asp.Net Core SAML Response Signature Validation

The signature should be validated by a certificate provided by the Identity Provider. From your update, you are storing the metadata xml from the IdP and then retrieving the certificate from it.

Another alternative is to acquire the certificate from the IdP as a .cer file and install it in the Trusted Root Certificate Athorities store of the Local Machine. Once installed, this can be accessed in code and be used to verify the signature. This way you don't have to be concerned about spoofing of the certificate in the XML.

In this approach, the certificate in the response XML is used to acquire the Serial Number by which the saved certificate is found in the store.

Building on top of @Evk 's response

public static bool VerifyXml(XmlDocument Doc) 
{
    if (Doc == null)
        throw new ArgumentException("Doc");
    SignedXml signedXml = new SignedXml(Doc);
    var nsManager = new XmlNamespaceManager(Doc.NameTable);
    nsManager.AddNamespace("ds", "http://www.w3.org/2000/09/xmldsig#");
    var node = Doc.SelectSingleNode("//ds:Signature", nsManager);
    // find signature node
    var certElement = Doc.SelectSingleNode("//ds:X509Certificate", nsManager);
    // find certificate node
    var cert = new X509Certificate2(Convert.FromBase64String(certElement.InnerText));            
    signedXml.LoadXml((XmlElement)node);

    //Find installed certificate from store
    X509Store store = new X509Store(StoreName.Root, StoreLocation.LocalMachine);
    store.Open(OpenFlags.ReadOnly);
    X509Certificate2 storeCert = store.Certificates.Find(X509FindType.FindBySerialNumber, cert.SerialNumber, true)[0];

    return signedXml.CheckSignature(storeCert, true);
    //^^^ If certificate is installed in the Root location then 
    //this method returns true after validating it as well
    //In addition to validating the signature
}

Also

    return signedXml.CheckSignature(cert);
    //^^^^ This will not work. 
    //CheckSignature(X509Certificate2 certificate, bool verifySignatureOnly) 
    //needs a boolean flag as well

Try to verify signature like this (your does not verify for me, but that might be caused by changes made while posting it here):

public static bool VerifyXml(XmlDocument Doc) {
    if (Doc == null)
        throw new ArgumentException("Doc");
    SignedXml signedXml = new SignedXml(Doc);
    var nsManager = new XmlNamespaceManager(Doc.NameTable);
    nsManager.AddNamespace("ds", "http://www.w3.org/2000/09/xmldsig#");
    var node = Doc.SelectSingleNode("//ds:Signature", nsManager);
    // find signature node
    var certElement = Doc.SelectSingleNode("//ds:X509Certificate", nsManager);
    // find certificate node
    var cert = new X509Certificate2(Convert.FromBase64String(certElement.InnerText));            
    signedXml.LoadXml((XmlElement)node);
    return signedXml.CheckSignature(cert);
}

If that doesn't work, also try the same but call

return signedXml.CheckSignature();

instead of

return signedXml.CheckSignature(cert);

Note that just verifying this signature is not enough to ensure that response has not been tampered with. You verify signature using key provided in response itself (X509Data), which means attacker could have intercepted response, extracted information and resigned it with his own key, so signature will be valid, but key it was signed with will be not. So after extracting certificate (or you can use signedXml.CheckSignatureReturningKey method to get public key related to signature) you need to verify that it's valid and that it's certificate you are expecting (for example by comparing its hash with hash of certificate you expect).