From the Java perspective block chain practice Series 4: Principles handwriting implement SHA-256 algorithm and Merkle tree algorithm

A lesson, we talked about the principle of Merkle tree, its essence is a kind of hash binary tree, leaves borrowing your hash value, derived from the non-leaf nodes of the data value is based around its two child nodes double hash calculated.

Merkle tree algorithm is widely used in the chain block, which includes a zero-knowledge proof surface application, file integrity checking, and other fields.

Now, let's get into the theme of the actual handwriting SHA-256 algorithm and Merkle tree.

Encryption Algorithm

At present, the encryption algorithm is divided into symmetric encryption algorithms, asymmetric encryption algorithms as well as non-reversible encryption algorithm three categories.

Symmetric encryption algorithm

Symmetric encryption algorithm is an encryption algorithm of the earlier application, the same symmetric key to decrypt the encryption algorithm, the keys for the main stream cipher and block cipher implemented in two ways. The main features of symmetric encryption algorithm is disclosed algorithm, to calculate the amount of small, fast encryption speed, high efficiency, but uses the same encryption key, security can not be guaranteed. At present, the mainstream symmetric encryption algorithms AES DES, 3DES, IDEA and the US National Bureau of Standards.

Asymmetric encryption algorithm

Since different keys for encryption and decryption, it is called "asymmetric encryption," which is divided into public and private keys, with the use of public and private key pairs. Asymmetric encryption algorithm can be implemented digital signatures. Common are the RSA algorithm, the US National Bureau of Standards and Technology Research Institute (NIST) DSA algorithm is proposed. Encryption is characterized by slow and inefficient. (In the signature section will explain in detail the principles RSA)

Irreversible encryption algorithm

Non-reversible encryption algorithm need not use a key, the encrypted input plaintext algorithm processing directly into ciphertext by the system, a one-way digest encryption algorithm, the encryption algorithm difficult to crack, check old data is required to use the same kind of encryption algorithm, the results were compared. At present, the widely used RSA is mainly research and development of MD5 and American Standard Board recommended SHS (Secure Hash Standard, Secure Hash Standard) and other standards-based algorithms.

Hash hash function

Wikipedia defines it this way:

Hash function is a function to create fixed size data from data of any size. The hash function to the message digest, or data compression, so that the data becomes small, the fixed format of the data. This function will disrupt mixed data, recreate called hash value (hash values, hash codes, hash sums, or hashes) fingerprints.

Main features are strong compression, simple calculation, the result of a one-way encryption, hash collision probability problem by guarantee safety. It will compress the contents of the infinite into the finite number of digits of the range, hash collision is inevitable possibility of effective probability, the hash function must have a good collision resistance.

At present, the hashing algorithm is divided MD series (MD4, MD5, HAVAL) series and SHA (SHA-1, SHA256), presented here SHA series of SHA256.

1byte = 8bit byte bits, the length of a hexadecimal character is 4bit bits.

SHA-1 algorithm

Input SHA-1 algorithm is limited to a length 264, packet processing input message 512 in accordance with a group, the output is 160 bits message digest. High speed SHA-1 algorithm, simple.

SHA-2 algorithm

SHA-2 algorithm output preferably length 224, 256, 384, 512, respectively SHA-224, SHA-256, SAH-384, SHA-512, in addition to SHA-512/224, SHA-512 / 256, two algorithms have greater flexibility and security of the output length than the previous algorithm. These variants except for some minor differences in length, the number of times of the digest generation cycle operation, the basic structure is the same.

SHA-3 algorithm

American Standard Board opted Keccak as the encryption standard algorithm SHA-3's, Keccak algorithm has good performance and the anti-encryption decryption capabilities.

Mathematical foundations of SHA-256

Natural numbers : integer greater than zero.

Primes : a natural number greater than 1 refers to, in addition to 1 and no longer has its own natural number of other unexpected factors.

Modulo operation : also known as "modulo", that is to finding two natural numbers do remainder division operation. If natural numbers a, b, then a% b = c or a mod b = c, when a <b, c = a. Modulo operation is more, in addition to a simple four operations and our subsequent signature algorithm used in inverse mode.

High / Low : The so-called high / That is a binary digit, doing shift operation, the left side of the removed bit bit low on the contrary, after the displacement operation of insufficient bit bit.

Binary left (<<) : m << n and m represents an integer binary number of n bits to the left, the upper n bits are discarded out, low complement 0 (at this time may become positive there will be negative). m << n number under the premise that there is no overflow, for positive and negative numbers, n bits are equivalent to left multiplying m by 2 ^ n.

5, for example, left 4 ( since Java int type accounted 32bit bit, so here by 32-bit binary presentation ):

5 binary: 0,000,000,000,000,000 0,000,000,000,000,101

4 to the left: 0000 0000 0000 0000 0,000,000,001,010,000 (less than 0 up, the result is 80 decimal)

Binary right shift (>>) : m >> n and m represents the integer of n bits to the right of the binary number, m is a positive number, high fill all 0; m is negative, all the high S.1 . i.e., m >> n corresponding to m divided by 2 ^ n, an integer is obtained, that is the result. If the result is a decimal, then two things occur:

If m is a positive number, unconditional discard Chamber obtained decimal places;

If m is negative, discarding the fractional part, then the part is increased by +1 integer value obtained displacement.

Unsigned right shift (>>>) : n >>> m: integer represented by m binary right by n bits, regardless of the number of positive and negative, 0s are high.

Bit and ( & ) : comparison of two binary strings one by one, if only two are 1 to 1, the other cases are zero. 5 & e.g. 3:

5 binary: 0,000,000,000,000,000 0,000,000,000,000,101

3 binary: 0,000,000,000,000,000 0,000,000,000,000,011

Position and results: 0,000,000,000,000,000 0,000,000,000,000,001

Bit or ( | ) : comparison of two binary strings one by one, if only two are 0 to 0, 1 other circumstances are.

NON - (- ) : single binary string, the contents of bit negated. It is a changed 0,0 to 1.

Bitwise XOR ( ^ ) : Comparison of two binary strings individually, if the same bit value, and 0, otherwise it is one. E.g:

5 binary: 0,000,000,000,000,000 0,000,000,000,000,101

3 binary: 0,000,000,000,000,000 0,000,000,000,000,011

After bitwise XOR: 0000 0000 0000 0000 0,000,000,000,000,110

SHA-256 relates to the binary operator conversion table

bit represents a binary bit, byte = byte, word SHA256 algorithm is the smallest operation unit called a "word" (Word).

1 byte = 8 bit

1 int = 1 word = 4 byte = 4*8 = 32 bit

Equation Logical Operators	Java logical operators	meaning
∧	&	Bitwise "and"
¬	~	Bitwise "fill / no"
⊕	\|	Bitwise "exclusive / or."
$S^{n}$	Integer.rotateRight()	The n bit rotate right
$R^{n}$	>>>	N-bit unsigned shift right

Related to the logic function Java implementation table:

Logical Functions	Java implementation
$\sigma _0 (x) = S^7(x) \bigoplus S^{18}(x) \bigoplus R^3(x)$	`private static int smallSig0(int x) { return Integer.rotateRight(x, 7) ^ Integer.rotateRight(x, 18) ^ (x >>> 3); }`
$\sigma _1(x) = S^{17}(x) \bigoplus S^{19}(x) \bigoplus R^{10}(x)$	`private static int smallSig1(int x) { return Integer.rotateRight(x, 17) ^ Integer.rotateRight(x, 19) ^ (x >>> 10); }`
$\large Ch(x,y,z) = (x\wedge y)\oplus ( \sim x\wedge z)$	`private static int ch(int x, int y, int z) { return (x & y) \| ((~x) & z); }`
$Ma (x, y, z) = (x \ wedge y) \ bigoplus (x \ wedge z) \ bigoplus (s \ wedge z)$	`private static int maj(int x, int y, int z) { return (x & y) \| (x & z) \| (y & z); }`
$\sum _0(x) = S^2(x) \bigoplus S^{13}(x) \bigoplus S^{22}(x)$	`private static int bigSig0(int x) { return Integer.rotateRight(x, 2) ^ Integer.rotateRight(x, 13) ^ Integer.rotateRight(x, 22); }`
$\sum _1(x) = S^6(x) \bigoplus S^{11}(x) \bigoplus S^{25}(x)$	`private static int bigSig1(int x) { return Integer.rotateRight(x, 6) ^ Integer.rotateRight(x, 11) ^ Integer.rotateRight(x, 25); }`

SHA-256

SHA-256 is widely used bitcoin block chain, the basic summary of all places Bitcoin involving encryption have chosen SHA-256, but also can be seen everywhere in the public chain SHA-256 project applications.

Since the avalanche effect , even with a very small change in the message it will also lead to different results in the final Hash.

For a message of any length, SHA-256 will output a 256bit bit Hash summary, this summary is usually a hexadecimal string of length 64, corresponding to the length of all four bytes of 32-byte array. SHA-256 algorithm minimum operation unit is called "word" (Word), it is a 32-bit word.

The encryption process:

1, the initialization constant is provided;

2, fill bits of the input message, the length of the fill bits the multiple bit 512bit;

3, each block is divided into blocks of complementary bit 512bit message bits:

$\LARGE M^2,M^2,.....,M^N$

4, a series of complex logical operations for each block, the finally obtained hash digest:

$\LARGE H^{}i = H^{(i - 1)} + {{C_{M}}^{i}} (H^{(i - 1)})$
Then we realized in accordance with the SHA-256 algorithm.

Detailed principles SHA-256

Constant initialization

1, 8 initialization original hash H0, the former is a natural number 8 prime / prime (2,3,5,7,11,13,17,19) of the square root of the front 32bit bit fractional part.

such as: $\sqrt{2} \approx 0.414213562373095048\approx6*16-1+a*16-2+0*16-3+...$

Then, the square root of the fractional part of mass 2 from the former to 32bit hexadecimal notation as0x6a09e667。

    private static final int[] H0 = {0x6a09e667, 0xbb67ae85, 0x3c6ef372,
            0xa54ff53a, 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19};

2, the initialization constant of 64 keys, decimal 32bit front portion 64 from the front of the cube root of the prime natural numbers, the hexadecimal notation, the corresponding constant sequence is as follows:

We use $K_{t}$ represents t keys.

    private static final int[] K = {0x428a2f98, 0x71374491, 0xb5c0fbcf,
            0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
            0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, 0x72be5d74,
            0x80deb1fe, 0x9bdc06a7, 0xc19bf174, 0xe49b69c1, 0xefbe4786,
            0x0fc19dc6, 0x240ca1cc, 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc,
            0x76f988da, 0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7,
            0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967, 0x27b70a85,
            0x2e1b2138, 0x4d2c6dfc, 0x53380d13, 0x650a7354, 0x766a0abb,
            0x81c2c92e, 0x92722c85, 0xa2bfe8a1, 0xa81a664b, 0xc24b8b70,
            0xc76c51a3, 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070,
            0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5, 0x391c0cb3,
            0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3, 0x748f82ee, 0x78a5636f,
            0x84c87814, 0x8cc70208, 0x90befffa, 0xa4506ceb, 0xbef9a3f7,
            0xc67178f2};

Message pre - Complement bit

Bit fill operation follows the following equation:

$\large l + 1 + k \equiv 448 mod 512$

Why 448?

Because the pre-fill after the first bit of the second step will be the additional data on a 64bit, used to indicate length information of the original packet. And 448 + 64 = 512, just makes up a complete structure.

We SHA256 ( "abc") as an example:

Original character	ASCII code	Binary format
a	97	01100001
b	98	01100010
c	99	01100011

因此，其二进制形式为：01100001 01100010 01100011，长度 $\large l = 24$ 。

1、补1，即直接在二进制消息尾部添加 1；

例如：消息“abc”的二进制位，补1后 01100001 01100010 01100011 1。

2、补0，个数为 $\large k$ ，代入消息长度 $\large l$ 可求得；

例如：代入 $\large l$ 算得 $\large k = 448 - l -1 = 423$ ，因此补 423个0。由于448 < 512，因此，可忽略掉后面的mod 512（对整数取模时，若整数小于模数，则取模结果为本身）。因此补位后，01100001 01100010 01100011 10000000……00000000。

3、补入消息长度 $\large l$ 的64位二进制内容，若长度的二进制不足64位，则在长度前面添加0，知道二进制字符串长度为64。

例如：消息“abc”的二进制长度为24，则最后进行的补位操作就是将24的二进制形式 ‭00011000‬。

调用pad()方法可以获取到补位的结果，代码实现<单击一键，代码尽显>：

    /**
     * 填充给定消息的长度
     * 512位（64字节）的倍数，包括添加 1位，k 0位以及消息长度为64位整数。
     *
     * @param message 要填充的消息，String.getBytes()可获得消息的字节数组。
     * @return 一个带有填充消息字节的新数组。
     */
    public static byte[] pad(byte[] message) {
        final int blockBits = 512;
        final int blockBytes = blockBits / 8;

        // 新消息长度：原始长度 + (补位的)1位 + 填充8字节长度（也就是64bit）
        int newMessageLength = message.length + 1 + 8;
        int padBytes = blockBytes - (newMessageLength % blockBytes);
        newMessageLength += padBytes;

        // 将消息复制到扩展数组
        final byte[] paddedMessage = new byte[newMessageLength];
        System.arraycopy(message, 0, paddedMessage, 0, message.length);

        // 第一步，补位：在消息末尾补上一位"1"。0b代表二进制，10000000 是二进制的128
        paddedMessage[message.length] = (byte) 0b10000000;

        // 第二步，跳过，因为我们已经设置了padBytes数组的长度，所以内部所有元素已经是0了（默认值）

        // 第三步，补入消息长度l的64位二进制的8字节整数，（java使用的是byte，1 byte = 8 bit）
        int lenPos = message.length + 1 + padBytes;
        ByteBuffer.wrap(paddedMessage, lenPos, 8).putLong(message.length * 8);
        return paddedMessage;
    }

加密计算Hash摘要

1、将补位后的消息分解成n个块，每一个块512bit。

用java实现时，我们不需要按块“存储”，因此可以省略此步。

2、遍历所有区块，将单个区块从16 word重构成64 word。

对于每一块，将块分解为16个32-bit的big-endian的字，记为w[0], …, w[15]。当前区块的第t个“字”我们用 $W_{t}$ 表示。

2.1 前16个字直接由消息的第t个块分解得到。经过第一步进行区块分组后，每个区块都包含了16 word。

    /**
     * 块数组
     */
    private static final int[] W = new int[64];

    public  static byte[] hash(byte[] message) {
        // ……省略部分代码
            System.arraycopy(words, i * 16, W, 0, 16);
        // ……省略部分代码
    }

2.2 其余的字由如下迭代公式得到：

$\large W_{t} = \sigma _{1}(W_{t - 2}) + W_{t-7}+ \sigma _{0}(W_{t - 15}) + W_{t - 16}$

    /**
     * 块数组
     */
    private static final int[] W = new int[64];

    public  static byte[] hash(byte[] message) {
        // ……省略部分代码  smallSig1为 σ1 函数，smallSig0为 σ0 函数
            for (int t = 16; t < W.length; ++t) {
                // 根据加法交换律，修改先后顺序不印象最终结果
                W[t] = smallSig1(W[t - 2])
                        + W[t - 7]
                        + smallSig0(W[t - 15])
                        + W[t - 16];
            }
        // ……省略部分代码
    }

3、循环对每个“块”加密，也就是说需要循环64次。

经过步骤二后，W将成为具备64 word，加密的过程如下图：

SAH-256 algorithm function call graph — SAH-256算法加密函数调用图

代码实现<单击一键，代码尽显>：

    /**
     * 块数组
     */
    private static final int[] W = new int[64];

    public  static byte[] hash(byte[] message) {
        // ……省略部分代码  


        /*
               3、循环对“块”加密：也就是说循环64次。
               例如：对“abc”加密时，就相当于依次对 c、b、a进行加密，c的加密结果，保存到a位置
             */

            // 设 TEMP = H，H是初始变量，即8个初始hash值，也就是前8个质数的平方根的前32bit位。
            System.arraycopy(H, 0, TEMP, 0, H.length);

            // 在TEMP上操作
            for (int t = 0; t < W.length; ++t) {
                // t1 = H[7] + Ch(H[4],H[5],H[6]) + Σ1(H[4])
                int t1 = TEMP[7]
                        + ch(TEMP[4], TEMP[5], TEMP[6]) + K[t] + W[t]
                        + bigSig1(TEMP[4]);

                // t2 = Ma(H[0],H[1],H[2]) + Σ0(H[0])
                int t2 = maj(TEMP[0], TEMP[1], TEMP[2]) + bigSig0(TEMP[0]);
                System.arraycopy(TEMP, 0, TEMP, 1, TEMP.length - 1);

                // 设置中间散列
                TEMP[4] += t1;

                // 设置头部散列 t1 + t2
                TEMP[0] = t1 + t2;
            }

            // 将TEMP中的值添加到H中的值
            for (int t = 0; t < H.length; ++t) {
                H[t] += TEMP[t];
            }
        // ……省略部分代码
    }

经过64轮后，代码中的H就是最终SHA-256加密的结果。

Java实现Merkle树

Merkle树原理在上一节，我们已经详细阐述过，这里就直接贴实现的代码了<复习快速跳转>。

定义叶子节点<单击一键，代码尽显>

@Data
public class TreeNode {
    /**
     * 左子节点
     */
    private TreeNode left;
    /**
     * 右子节点
     */
    private TreeNode right;
    /**
     * （孩子）节点数据
     */
    private String data;
    /**
     * SHA-256的data
     */
    private String hash;

    public TreeNode(){}
    public TreeNode(String data) {
        this.data = data;
        this.hash = DigestUtil.sha256Hex(data);
    }
}

Merkle树计算<单击一键，代码尽显>

package org.lmx.common.merkle;

import cn.hutool.core.collection.CollectionUtil;
import cn.hutool.core.util.ObjectUtil;
import cn.hutool.crypto.digest.DigestUtil;
import lombok.extern.slf4j.Slf4j;

import java.util.ArrayList;
import java.util.List;

/**
 * 功能描述：Merkle算法实现
 *
 * @program: block-chain-j
 * @author: LM.X
 * @create: 2020-03-31 14:53
 **/
@Slf4j
public class MerkleTree {
    /**
     * 交易列表
     */
    private List<TreeNode> treeNodes;

    /**
     * 根节点
     */
    private TreeNode root;

    public MerkleTree(List<String> treeNodes) {
        createMerkleTree(treeNodes);
    }

    /**
     * 功能描述: 构建默克尔树
     *
     * @param transactions 内容列表
     * @return void
     * @author LM.X
     * @date 2020/3/31 14:58
     */
    private void createMerkleTree(List<String> transactions) {
        if (CollectionUtil.isEmpty(transactions)) {
            return;
        }

        // 初始化列表
        this.treeNodes = new ArrayList();

        // 格式化节点信息
        treeNodes.addAll(createLeafNode(transactions));

        // 合并叶子节点，获取默克尔根
        while (true) {
            treeNodes = createParentList(treeNodes);
            if (treeNodes.size() < 2) {
                root = treeNodes.get(0);
                return;
            }
        }
    }

    /**
     * 功能描述: 创建叶子节点
     *
     * @param transactions 内容列表
     * @return 返回叶子节点列表
     * @author LM.X
     * @date 2020/3/31 15:09
     */
    private List<TreeNode> createLeafNode(List<String> transactions) {
        List<TreeNode> leafs = new ArrayList();
        if (CollectionUtil.isEmpty(transactions)) {
            return leafs;
        }

        for (String transaction : transactions) {
            leafs.add(new TreeNode(transaction));
        }

        return leafs;
    }

    /**
     * 功能描述: 合并所以叶子节点
     *
     * @param nodes 节点列表
     * @return 返回合并后的节点集合
     * @author LM.X
     * @date 2020/3/31 15:41
     */
    private List<TreeNode> createParentList(List<TreeNode> nodes) {
        List parents = new ArrayList();
        if (CollectionUtil.isEmpty(nodes)) {
            return parents;
        }

        int len = nodes.size();

        for (int i = 0; i < len - 1; i += 2) {
            parents.add(createParentNode(nodes.get(i), nodes.get(i + 1)));
        }

        // 当奇数个叶子节点时，单独处理
        if (len % 2 != 0) {
            parents.add(createParentNode(nodes.get(len - 1), null));
        }

        log.info("本轮合并后，节点长度：{}", parents.size());
        return parents;
    }

    /**
     * 功能描述: 合并左右子节点
     *
     * @param left  左子节点
     * @param right 右子节点
     * @return 返回合并后的父节点
     * @author LM.X
     * @date 2020/3/31 15:35
     */
    private TreeNode createParentNode(TreeNode left, TreeNode right) {
        TreeNode parent = new TreeNode();
        parent.setLeft(left);
        parent.setRight(right);

        String lh = left.getHash();

        String hash = ObjectUtil.isEmpty(right) ? lh : doubleSHA256(lh, right.getHash());

        parent.setData(hash);
        parent.setHash(hash);
        log.info("合并【{}，{}】，创建父节点：{}", left.getData(), ObjectUtil.isEmpty(right) ?
                null : right.getData(), hash);
        return parent;
    }

    /**
     * 功能描述: 双哈希运算
     *
     * @param lh
	 * @param rh
     * @return SHA256 结果
     * @author LM.X
     * @date 2020/3/31 16:02
     */
    private String doubleSHA256(String lh, String rh) {
        return DigestUtil.sha256Hex(DigestUtil.sha256Hex(lh + rh));
    }

    public static void main(String[] args) {
        List<String> txs = new ArrayList() {{
            add("1");add("2");add("3");add("4");
            add("5");add("6");add("7");add("8");
            add("9");add("10");add("11");add("12");
            add("13");add("14");add("15");add("16");
        }};

        MerkleTree merkleTree = new MerkleTree(txs);
        log.info("获取到的默克尔根为：{}", merkleTree.root.getHash());
    }
}

总结

这一节，我们剖析了在区块链的底层核心所涉及的加密算法，深入了解如今被广泛使用的SHA-256算法的实现原理，其通过简单的位移取模操作生成摘要，SHA256的抗碰撞性保证了信息的安全。

之后我们实战了Merkle树算法，从实战中可以了解更多它的优缺点，也可以检验上篇我们讲解的Merkle原理，进而让我们在了解区块链的道路上更近一步。

好啦，今天我们到这里就结束啦！

近期由于搬家找房子后续文章可能会有所延后发布，文章有不足之处，欢迎指出，我将不断完善之~

最后，给大家奉上后续Java实战计划安排表：

目录	内容
P2P网络实战	这一篇将为大家带来Java实现P2P通讯网络的实战。
区块链共识系列	Learning series you will learn the consensus of the most complete history of the consensus algorithm at present, but also to learn Java to achieve real consensus algorithm. They include: RAFT consensus, classic PWD, PWD, DPOS consensus, the latest PoET, POB consensus, and consensus mixed.
Series Signature Algorithm: RSA Signature Algorithm	This one you will learn the RSA cryptography, RSA signature principle and practical
Signature Algorithm series: ECC Elliptic Curve signature	Learning ECC elliptic curve encryption algorithm you say knowledge harvesting elliptic curve cryptography, elliptic curve mathematics.
Block Chain Security	Explain the relevant safety and accident prevention measures.
Solidity written contract	The principle to explain the contract, the actual contract. It includes real ERC20, ERC-223, ERC-721, NEP series, TRC series written contract and so on.
Contract Audit	Explain the contract audit-related knowledge, so that you can skillfully master the knowledge of a contract auditor need. All the knowledge you impart to the highest single-day audit 10+ contract programmers.
Anonymity research	Explain the block chain anonymity associated implementation and practical.
Real exchange purse large flow design	Explain the relevant Exchange wallet design, covering security, analysis of user intent, transaction acceleration.
Exchange difficulty breakthrough	It includes exchange business and architecture selection, distributed combat match engine and so on.
Block Browser	Explain how to build a block browser, easily one hundred GB of data blocks, how fast retrieval and so on.
Block chain wallet	It includes design-related content such as wallets.
other	It includes history, major currency trends block chain analysis and so on.