This knowledge point is easy to understand, just Baidu Encyclopedia. In fact, it is also an algorithm, which is to obtain a summary after performing an algorithm on a data source, which is also called a data fingerprint. Different data sources have different data fingerprints. Just like people.

Message Digest Algorithm (Message Digest Algorithm) is an algorithm that can produce a special output format. Its principle is to extract some form of information from the original data according to certain operation rules. The extracted information is called the original data. Message digest.
Famous digest algorithms include RSA's MD5 algorithm and SHA-1 algorithm and a large number of variants.
The main features of the message digest are:
1) No matter how long the input message is, the length of the calculated message digest is always fixed. For example, the message digested by the MD5 algorithm has 128 bits, and the message digested by the SHA-1 algorithm finally has a 160-bit output.
2) Generally speaking (without considering the collision), as long as the input original data is different, the message digest generated after digesting it must also be different, even if the original data changes slightly, the output message digest will be completely different. However, the same input must produce the same output.
3) It is irreversible, that is, only forward information summarization can be performed, and no original message can be recovered from the summation.

2. Signature files and certificates

The signature file and the certificate appear in pairs, and the two are inseparable, and we can see from the source code later that the names of the two files are also the same, but the suffixes are different.

In fact, the concept of digital signature is very simple. As we all know, to ensure reliable communication, two problems must be solved: first, to ensure that the source of the message is indeed the person it claims; second, to ensure that the information is not tampered with by a third party during the transmission process, even if it is tampered , can also be detected.
The so-called digital signature is created to solve these two problems. It is a specific application of the aforementioned asymmetric encryption technology and digital abstract technology.
For the sender of the message, a pair of public and private key pairs must be generated first, and the public key should be given to the receiver of the message.
If the sender of the message wants to send a message to the message receiver one day, in addition to the original message, another message must be added to the sent message. This message is generated through the following two steps:
1) Extract the message digest from the original message to be sent;
2) Encrypt the extracted message digest with your own private key.
The message obtained through these two steps is the so-called digital signature of the original message.
For the recipient of the information, the information he receives will contain two parts, one is the original message content, and the other is the additional digital signature. He will verify the authenticity of the message through the following three steps:
1) Extract the message digest from the original message, and note that the message digest algorithm used here must be consistent with that used by the sender;
2) For the attached digital signature, use Decrypt with the pre-obtained public key;
3) Compare whether the two pieces of messages obtained in the first two steps are consistent. If they are consistent, it means that the message is indeed sent by the expected sender, and the content has not been tampered with; on the contrary, if they are inconsistent, it means that there must be a problem in the transmission process, and the message is not credible.
Through this so-called digital signature technology, the problem of reliable communication can indeed be effectively solved. If the original message is tampered with during transmission, then at the message receiver, the digest extracted from the tampered message must be different from the original one. Moreover, since the tamperer does not have the private key of the message sender, even if he can recalculate the digest of the tampered message, he cannot forge the digital signature.
Therefore, to sum up, a digital signature is actually a digital string that only the sender of the information can generate and cannot be forged by others. This digital string is also an effective proof of the authenticity of the information sent by the sender of the information.
I don’t know if you have noticed that the digital signature method mentioned above has a prerequisite, that is, the receiver of the message must obtain the correct public key in advance. If the public key is tampered with by others from the beginning, the bad guy will be regarded as a good person by you, and the message sent to you by the real message sender will be regarded as invalid by you. Moreover, in many cases, there is no information channel to communicate the public key in advance. So how to ensure the security and trustworthiness of the public key? This is solved by digital certificates.
The so-called digital certificate generally includes the following contents:
the issuer of the certificate (Issuer),
the validity period of the certificate (Validity),
the public key of the sender of the message, the algorithm used by the digital signature
of the certificate owner (Subject) , it can be seen from the digital signature that the digital certificate is actually Digital signature technology is also used. It's just that the content to be signed is the public key of the sender of the message, and some other information. But different from ordinary digital signatures, the signer in the digital certificate is not just any ordinary organization, but an organization with certain credibility. It's as if your college diploma is usually signed by a respected principal. Generally speaking, the root certificates of these trusted institutions have been pre-installed on your device before the device leaves the factory. Therefore, a digital certificate can ensure that the public key in the digital certificate is indeed the owner of the certificate, or the certificate can be used to confirm the identity of the other party. Digital certificates are mainly used to solve the problem of safe distribution of public keys. To sum up, in summary, the general process of digital signature and signature verification is shown in the following figure:

3. jarsign and signapk tools

After understanding the knowledge points of the three files in the signature, let's continue to look at the two signature tools in Android: jarsign and signapk

It is easy to confuse these two tools at the beginning. Is there any difference between them?

In fact, these two tools are easy to understand. jarsign is a tool that comes with Java itself, and it can sign jars. And signapk is a tool specially designed for signing Android application apk. There is no difference between the two signature algorithms. The main reason is that the files used for signing are different. This will lead to the third question.

4. The difference between keystore file and pk8, x509.pem file

We learned above that both jarsign and signapk tools can perform signatures in Android, so the difference between them is that the files used for signing are different.

The jarsign tool uses the keystore file when signing

The signapk tool uses pk8, x509.pem files when signing

Among them, when we use the Eclipse tool to write the program, when the Debug package is produced, the jarsign tool is used for signature by default, and there is a default signature file in Eclipse:

We can see this default signed keystore file, of course we can choose our own specified keystore file.

Here is another knowledge point:

We see the summary of MD5 and SHA1 above. This is the data summary of the private key in the keystore file. This information is also the information that we need to fill in when applying for many development platform accounts, such as applying for Baidu Maps, WeChat SDK, etc., will The MD5 or SHA1 information of the application needs to be filled in .

5. Manually signed Apk package

1"Use keytool and jarsigner to sign

Of course, when we release the package at the official signature, we need to create our own keystore file:

Here we can give the keystore file our own name, and the suffix is irrelevant. After the file is created, the MD5 and SHA1 values will also be generated. This value does not need to be recorded. You can view the MD5 and SHA1 values of the keystore file through commands.

keytool -list -keystore debug.keystore

Of course, we all know the importance of this keystore file. To put it bluntly, it is equivalent to your bank card password. you know.

Here we see that using Eclipse to automatically sign and generate a keystore file, we can also use the keytool tool to generate a keystore file. This method is available online, so I won’t introduce too much here. Then we can use jarsign to sign the apk package.

We can manually generate a keystore file:

keytool -genkeypair -v -keyalg DSA -keysize 1024 -sigalg SHA1withDSA -validity 20000 -keystore D:\jiangwei.keystore -alias jiangwei -keypass jiangwei -storepass jiangwei

This command is a bit long, and there are several important parameters to explain:

-alias is to define an alias, here is debug

-keyalg is the specified signature algorithm, here is DSA, the algorithm here is directly related to the suffix name of the signature file in the apk later, and will be explained in detail later

Signing with the jarsigner tool

jarsigner -verbose -sigalg SHA1withDSA -digestalg SHA1 -keystore D:\jiangwei.keystore -storepass jiangwei D:\123.apk jiangwei

In this way, we successfully signed the apk.

Problems encountered during the signing process:

1"The problem that the certificate chain cannot be found

This is because the last parameter, alias, is the alias of the keystore.

2》When generating the keystore file, it prompts that the password is wrong

The reason for this is that there is already a debug.ketystore in the current directory. If a debug.keystore is generated, an error will be reported

3"The problem of not finding the alias

The reason for this problem is that when we use keytool to generate the keystore, we use the alias of debug. This problem has troubled me for a long time. I finally found out after doing many examples. As long as the alias of our keystore file is debug, it will be Will report such an error. This should have something to do with the alias in the system's default signature debug.keystore being debug, right? I didn't find the source code of jarsigner, so I can only guess, but these three problems are marked here, in case they are encountered in the future.

Note: Android allows the use of multiple keystores to sign the apk . I will not paste the command here. I created several keystores to sign the apk:

After decompressing the signed apk here, I found that there are three signature files and certificates (.SF/.DSA)

Here I can also notice that we use the DSA algorithm when signing, and the file suffix here is DSA

And the file name is an alias of the keystore

Hey, here is a clear understanding of how we use keytool to generate keystore and use jarsigner to sign.

2"Use signapk to sign

Let's take a look at the signapk tool for signing:

java -jar signapk.jar .testkey.x509.pem testkey.pk8 debug.apk debug.sig.apk

Two files are required here: the two files .pk8 and .x509.pem

pk8 is the private key file

x509.pem is the file containing the public key

If you sign here, you will not be demonstrating, there is no problem here.

But what needs to be noted here is: the names of the three files in the META-INF folder in the apk after signapk signature are like this, because signapk is not like jarsigner in the front, it will automatically use an alias to name the file, here is to write Dead is the name of CERT, but the file name does not affect it. Later in the analysis of the Apk verification process in Android, it will be said that only the suffix name will be used to find the file.

3. What is the difference between the two signature methods?

Then the question comes, jarsigner uses the keystore file when signing, and signapk uses the pk8 and x509.pem files when signing, and they all sign the apk, so is the keystore file and pk8, x509.pem between them? What's the connection? The answer is yes. I searched the Internet and found that they can be converted. I won’t analyze how to convert them here. There seem to be many examples on the Internet. There are special tools for conversion:

So here we have figured out the difference and connection between these two signature tools.

3. Analyze the signature process mechanism in Android

Let's start to look at the signature mechanism and principle process in Android from the perspective of source code

Because the source code of jarsigner was not found on the Internet, but the source code of signapk was found, so let's take a look at the source code of signapk:

Source location: com/android/signapk/sign.java

Through the above signature, we can see that after the Android signature apk, there will be a META-INF folder, here are three files:

MANIFEST.MF

CERT.RSA

CERT.SF

Let's take a look at what these three files do?

1、MANIFEST.MF

Let's take a look at the source code:

public static void main(String[] args) {
    if (args.length != 4) {
        System.err.println("Usage: signapk " +
                "publickey.x509[.pem] privatekey.pk8 " +
                "input.jar output.jar");
        System.exit(2);
    }
 
    JarFile inputJar = null;
    JarOutputStream outputJar = null;
 
    try {
        X509Certificate publicKey = readPublicKey(new File(args[0]));
 
        // Assume the certificate is valid for at least an hour.
        long timestamp = publicKey.getNotBefore().getTime() + 3600L * 1000;
 
        PrivateKey privateKey = readPrivateKey(new File(args[1]));
        inputJar = new JarFile(new File(args[2]), false);  // Don't verify.
        outputJar = new JarOutputStream(new FileOutputStream(args[3]));
        outputJar.setLevel(9);
 
        JarEntry je;
 
        // MANIFEST.MF
        Manifest manifest = addDigestsToManifest(inputJar);
        je = new JarEntry(JarFile.MANIFEST_NAME);
        je.setTime(timestamp);
        outputJar.putNextEntry(je);
        manifest.write(outputJar);
 
        // CERT.SF
        Signature signature = Signature.getInstance("SHA1withRSA");
        signature.initSign(privateKey);
        je = new JarEntry(CERT_SF_NAME);
        je.setTime(timestamp);
        outputJar.putNextEntry(je);
        writeSignatureFile(manifest,
                new SignatureOutputStream(outputJar, signature));
 
        // CERT.RSA
        je = new JarEntry(CERT_RSA_NAME);
        je.setTime(timestamp);
        outputJar.putNextEntry(je);
        writeSignatureBlock(signature, publicKey, outputJar);
 
        // Everything else
        copyFiles(manifest, inputJar, outputJar, timestamp);
    } catch (Exception e) {
        e.printStackTrace();
        System.exit(1);
    } finally {
        try {
            if (inputJar != null) inputJar.close();
            if (outputJar != null) outputJar.close();
        } catch (IOException e) {
            e.printStackTrace();
            System.exit(1);
        }
    }
}

In the main function, we see that we need to input four parameters, and then we do three things:

Write MANIFEST.MF

//MANIFEST.MF
Manifest manifest = addDigestsToManifest(inputJar);
je = new JarEntry(JarFile.MANIFEST_NAME);
je.setTime(timestamp);
outputJar.putNextEntry(je);
manifest.write(outputJar);

Take a look at the entry method:

/** Add the SHA1 of every file to the manifest, creating it if necessary. */
private static Manifest addDigestsToManifest(JarFile jar)
        throws IOException, GeneralSecurityException {
    Manifest input = jar.getManifest();
    Manifest output = new Manifest();
    Attributes main = output.getMainAttributes();
    if (input != null) {
        main.putAll(input.getMainAttributes());
    } else {
        main.putValue("Manifest-Version", "1.0");
        main.putValue("Created-By", "1.0 (Android SignApk)");
    }
 
    BASE64Encoder base64 = new BASE64Encoder();
    MessageDigest md = MessageDigest.getInstance("SHA1");
    byte[] buffer = new byte[4096];
    int num;
 
    // We sort the input entries by name, and add them to the
    // output manifest in sorted order.  We expect that the output
    // map will be deterministic.
 
    TreeMap<String, JarEntry> byName = new TreeMap<String, JarEntry>();
 
    for (Enumeration<JarEntry> e = jar.entries(); e.hasMoreElements(); ) {
        JarEntry entry = e.nextElement();
        byName.put(entry.getName(), entry);
    }
 
    for (JarEntry entry: byName.values()) {
        String name = entry.getName();
        if (!entry.isDirectory() && !name.equals(JarFile.MANIFEST_NAME) &&
            !name.equals(CERT_SF_NAME) && !name.equals(CERT_RSA_NAME) &&
            (stripPattern == null ||
             !stripPattern.matcher(name).matches())) {
            InputStream data = jar.getInputStream(entry);
            while ((num = data.read(buffer)) > 0) {
                md.update(buffer, 0, num);
            }
 
            Attributes attr = null;
            if (input != null) attr = input.getAttributes(name);
            attr = attr != null ? new Attributes(attr) : new Attributes();
            attr.putValue("SHA1-Digest", base64.encode(md.digest()));
            output.getEntries().put(name, attr);
        }
    }
 
    return output;
}

The code logic is still very simple, mainly depends on the meaning of the loop:

Except for the three files (MANIFEST.MF, CERT.RSA, CERT.SF), other files will perform a SHA1 algorithm on the content of the file, which is to calculate the summary information of the file, and then encode it with Base64. Below we use the tool Let's take an example to see if this is the case:

First install the tool: HashTab

Download URL: hashtab_Baidu Search

Then there is another website that calculates Base64 online: Hex to base64 converter

Then let's start our verification work:

Let's verify the AndroidManifest.xml file, first find this entry in the MANIFEST.MF file, and record the value of SHA1

Then after we install HashTab, find the AndroidManifest.xml file, right-click, and select Hashtab:

Copy the value of SHA-1: 9C64812DE7373B201C294101473636A3697FD73C, go to the Base64 conversion website above, and convert it:

nGSBLec3OyAcKUEBRzY2o2l/1zw=

It is exactly the same as the entry content in MANIFEST.MF

Then we know from the above analysis, in fact, what is stored in MANIFEST.MF is:

Traverse all the entries in it one by one, if it is a directory, skip it, if it is a file, use the SHA1 (or SHA256) message digest algorithm to extract the digest of the file and encode it with BASE64, as the value of the "SHA1-Digest" attribute Write to a block in the MANIFEST.MF file. This block has a "Name" attribute whose value is the path of the file in the apk package.

2. Let's take a look at the contents of the CERT.SF file

The content here feels similar to the content of MANIFEST.MF, let’s take a look at the code:

//CERT.SF
Signature signature = Signature.getInstance("SHA1withRSA");
signature.initSign(privateKey);
je = new JarEntry(CERT_SF_NAME);
je.setTime(timestamp);
outputJar.putNextEntry(je);
writeSignatureFile(manifest,new SignatureOutputStream(outputJar, signature));

Enter the writeSignatureFile method:

/** Write a .SF file with a digest the specified manifest. */
private static void writeSignatureFile(Manifest manifest, OutputStream out)
        throws IOException, GeneralSecurityException {
    Manifest sf = new Manifest();
    Attributes main = sf.getMainAttributes();
    main.putValue("Signature-Version", "1.0");
    main.putValue("Created-By", "1.0 (Android SignApk)");
 
    BASE64Encoder base64 = new BASE64Encoder();
    MessageDigest md = MessageDigest.getInstance("SHA1");
    PrintStream print = new PrintStream(
            new DigestOutputStream(new ByteArrayOutputStream(), md),
            true, "UTF-8");
 
    // Digest of the entire manifest
    manifest.write(print);
    print.flush();
    main.putValue("SHA1-Digest-Manifest", base64.encode(md.digest()));
 
    Map<String, Attributes> entries = manifest.getEntries();
    for (Map.Entry<String, Attributes> entry : entries.entrySet()) {
        // Digest of the manifest stanza for this entry.
        print.print("Name: " + entry.getKey() + "\r\n");
        for (Map.Entry<Object, Object> att : entry.getValue().entrySet()) {
            print.print(att.getKey() + ": " + att.getValue() + "\r\n");
        }
        print.print("\r\n");
        print.flush();
 
        Attributes sfAttr = new Attributes();
        sfAttr.putValue("SHA1-Digest", base64.encode(md.digest()));
        sf.getEntries().put(entry.getKey(), sfAttr);
    }
 
    sf.write(out);
}

First of all, we can see that we need to make a SHA1 of the entire content of the previous MANIFEST.MF file and put it in the SHA1-Digest-Manifest field:

Let's take a look at the manifest variable that has just been written into the MANIFEST.MF file.

We can verify this:

and convert it

See it, it is the same as the value in the file.

Let's continue to look at the code below, there is a loop:

Map<String, Attributes> entries = manifest.getEntries();
for (Map.Entry<String, Attributes> entry : entries.entrySet()) {
    // Digest of the manifest stanza for this entry.
    print.print("Name: " + entry.getKey() + "\r\n");
    for (Map.Entry<Object, Object> att : entry.getValue().entrySet()) {
        print.print(att.getKey() + ": " + att.getValue() + "\r\n");
    }
    print.print("\r\n");
    print.flush();
 
    Attributes sfAttr = new Attributes();
    sfAttr.putValue("SHA1-Digest", base64.encode(md.digest()));
    sf.getEntries().put(entry.getKey(), sfAttr);
}
 
sf.write(out);

The mainfest variable just passed in is still used here, traverse its entry content, and then calculate the SHA algorithm in Base64:

In fact, it is to do a SHA for each entry in the MANIFEST.MF file, just save it, and make an example to verify:

Taking AndroidManifest.xml as an example, we copy and save the entries in the MANIFEST.MF file to a txt file:

It should be noted here that after we save, we need to add two newlines, we can see the logic in the code:

Then we calculate the SHA value of the txt document:

See, the value calculated here is the same.

Here we know what the CERT.SF file does:

1》Calculate the overall SHA1 value of this MANIFEST.MF file, and after BASE64 encoding, record it under the "SHA1-Digest-Manifest" attribute value of the CERT.SF main attribute block (on the file header)

2》Calculate the SHA1 of each block in the MANIFEST.MF file one by one, and after BASE64 encoding, record it in the block with the same name in CERT.SF, the attribute name is "SHA1-Digest

3. Finally, let's take a look at the CERT.RSA file

All we see here are binary files, because the RSA file is encrypted, so we need to use the openssl command to view its content

openssl pkcs7 -inform DER -in CERT.RSA -noout -print_certs –text

For this information, you can see the following picture:

Let's take a look at the code:

/** Write a .RSA file with a digital signature. */
private static void writeSignatureBlock(
        Signature signature, X509Certificate publicKey, OutputStream out)
        throws IOException, GeneralSecurityException {
    SignerInfo signerInfo = new SignerInfo(
            new X500Name(publicKey.getIssuerX500Principal().getName()),
            publicKey.getSerialNumber(),
            AlgorithmId.get("SHA1"),
            AlgorithmId.get("RSA"),
            signature.sign());
 
    PKCS7 pkcs7 = new PKCS7(
            new AlgorithmId[] { AlgorithmId.get("SHA1") },
            new ContentInfo(ContentInfo.DATA_OID, null),
            new X509Certificate[] { publicKey },
            new SignerInfo[] { signerInfo });
 
    pkcs7.encodeSignedData(out);
}

We see that the previously generated CERT.SF file will be used to calculate the signature with the private key, and then the signature and the digital certificate containing the public key information will be written into CERT.RSA for storage. CERT.RSA is a file that meets the PKCS7 format.

4. Why do you sign like this?

Above we introduced the details of the three files after signing the apk, so let’s summarize why Android uses this method for encrypted signatures. Is this encryption the safest? Let's analyze what happens if the apk file is tampered with.

First of all, if you change any file in the apk package, then when the apk installation is verified, the changed file summary information is different from the verification information of MANIFEST.MF, so the verification fails and the program cannot be successfully installed.
Secondly, if you calculate a new summary value for the modified file and then change the corresponding attribute value in the MANIFEST.MF file, it must be different from the summary value calculated in the CERT.SF file, and the verification will still fail.
Finally, if you still don’t give up, continue to calculate the digest value of MANIFEST.MF, and change the value in CERT.SF accordingly, then the digital signature value must be different from that recorded in the CERT.RSA file, and it still fails.
So can we continue to forge digital signatures? Impossible, because there is no private key corresponding to the digital certificate.
Therefore, if the repackaged application can be installed on an Android device, it must be re-signed.

From the above analysis, it can be concluded that as long as any content in the Apk is modified, it must be re-signed, otherwise it will prompt that the installation fails. Of course, it will not be analyzed here. The next article will focus on analyzing why the installation fails.

5. Combing knowledge points

1. The meaning of data fingerprint, signature file and certificate file

1 "Data fingerprint is to perform SHA/MD5 algorithm on a data source, and this value is unique

2 "Signature file technology is: data fingerprint + RSA algorithm

3"The certificate file contains public key information and other information

4"After the Android signature, where SF is the signature file and RSA is the certificate file, we can use openssl to view the certificate information and public key information in the RSA file

2. We learned that there are two ways to sign in Android: jarsigner and signapk. The difference between these two ways is:

1 "When jarsigner signs, it needs a keystore file, and when signapk signs, it needs a pk8, x509.pem file

2"The SF and RSA file names after jarsigner signature are aliases of keystore by default, while the file names after signapk signature are fixed: CERT

3 "In Eclipse, when we run the Debug program, the jarsigner signature is used by default, and the system default debug.keystore signature file is also used.

4 "keystore file and pk8, x509.pem file can be converted to each other

6. A quick way to re-sign apk

Requirements: fully compiled Android source code project;

1. Create xxxApk under /package/apps, such as TestApk

2. Put the apk file to be re-signed into the TestApk directory

3. Create Android.mk, such as

LOCAL_PATH:= $(call my-dir)

include $(CLEAR_VARS)
LOCAL_MODULE := TestApk
LOCAL_MODULE_TAGS := optional

LOCAL_SRC_FILES := TestApk_unsign.apk
LOCAL_PRIVILEGED_MODULE := true
LOCAL_MODULE_CLASS := APPS
LOCAL_BUILT_MODULE_STEM := package.apk
LOCAL_CERTIFICATE := platform

include $(BUILD_PREBUILT)

4. mmm package/apps/TestApk -j12 can regenerate the signed apk (the example here is signed with a platform certificate)

5. View the application signing certificate through the jarsigner command: jarsigner -verify -verbose -certs xxx.apk

Refer to the practice of aosp: https://android.googlesource.com/platform/frameworks/base/+/android-7.0.0_r1/packages/CtsShim/Android.mk

Android signature mechanism - detailed explanation of the signature process

I. Introduction

2. Prepare knowledge

1. Data Summary