Verify Digital Signature on Android

Digital signature is a process of computing digest (function H) of data (C) and encrypting it with asymmetric encryption algorithm (function E) to produce cypher text (S):

S = E(H(C))

Signature verification takes the signature decrypts the given signature (function D) - which results in H(C) only if the public key used in decryption is paired with private key used in encryption, and computes the digest of data to check if the two digests match:

H(C) == D(E(H(C)))

It's clear from this that the bytes given to the hash function (C) must be exactly the same in order for the signature to validate.

In your case they are not, because when you're computing the digest using openssl dgst the output (H(C) on the right) is literally something like:

SHA1(someHTMLDoc.html)= 22596363b3de40b06f981fb85d82312e8c0ed511

And this is the input to the RSA encryption.

And when you're verifying the signature, the output of the digest (H(C) on the left) are the raw bytes, for instance in hex:

22596363b3de40b06f981fb85d82312e8c0ed511

So you end up encrypting bytes to produce (H(C) on the right):

0000000: 5348 4131 2873 6f6d 6548 746d 6c44 6f63  SHA1(someHtmlDoc
0000010: 2e68 746d 6c29 3d20 3232 3539 3633 3633  .html)= 22596363
0000020: 6233 6465 3430 6230 3666 3938 3166 6238  b3de40b06f981fb8
0000030: 3564 3832 3331 3265 3863 3065 6435 3131  5d82312e8c0ed511
0000040: 0a                                       .

and comparing against bytes (H(C) on the left):

0000000: 2259 6363 b3de 40b0 6f98 1fb8 5d82 312e  "[email protected]...].1.
0000010: 8c0e d511                                ....

Also you need to use -sign with openssl dgst in order to have proper output format (see Difference between openSSL rsautl and dgst).

So on the OpenSSL side do:

openssl dgst -sha1 -sign privateKey.pem someHTMLDoc.html > signature.bin

On the Java side do:

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.security.KeyFactory;
import java.security.Signature;
import java.security.interfaces.RSAPublicKey;
import java.security.spec.X509EncodedKeySpec;

import org.spongycastle.util.io.pem.PemObject;
import org.spongycastle.util.io.pem.PemReader;

public class VerifySignature {
    public static void main(final String[] args) throws Exception {
        try (PemReader reader = publicKeyReader(); InputStream data = data(); InputStream signatureData = signature()) {
            final PemObject publicKeyPem = reader.readPemObject();
            final byte[] publicKeyBytes = publicKeyPem.getContent();
            final KeyFactory keyFactory = KeyFactory.getInstance("RSA");
            final X509EncodedKeySpec publicKeySpec = new X509EncodedKeySpec(publicKeyBytes);
            final RSAPublicKey publicKey = (RSAPublicKey) keyFactory.generatePublic(publicKeySpec);

            final Signature signature = Signature.getInstance("SHA1withRSA");
            signature.initVerify(publicKey);

            final byte[] buffy = new byte[16 * 1024];
            int read = -1;
            while ((read = data.read(buffy)) != -1) {
                signature.update(buffy, 0, read);
            }

            final byte[] signatureBytes = new byte[publicKey.getModulus().bitLength() / 8];
            signatureData.read(signatureBytes);

            System.out.println(signature.verify(signatureBytes));
        }
    }

    private static InputStream data() throws FileNotFoundException {
        return new FileInputStream("someHTMLDoc.html");
    }

    private static PemReader publicKeyReader() throws FileNotFoundException {
        return new PemReader(new InputStreamReader(new FileInputStream("publicKey.pem")));
    }

    private static InputStream signature() throws FileNotFoundException {
        return new FileInputStream("signature.bin");
    }
}

I've used Spongy Castle for PEM decoding of the public key to make things a bit more readable and easier to use.