Table of contents
2. Characteristics of Trie structure
3. MPT structure
1. What is a trie structure
Trie : The dictionary tree, also known as the prefix tree, is also a (key, value) tree. Generally speaking, strings are used for keys, such as arranging some words into a trie data structure. For example, the data structure of general, genesis, god, go, and good arranged into a trie is shown in Figure 3-1 below.
Figure 3-1 is the result of a trie. These words all start with G, and then the second letter begins to split. The left is E, the right is O, and the first two words on the left are both N and E. , and then separate below, R and S, and then the last three letters, the branch on the right, the branch O, Go is over, from this we can see that the word may end at the middle node of the trie, and then D on the left, O on the right, God on the left, Good down on the right.
2. Characteristics of Trie structure
(1) In the trie, the number of branches of each node depends on the value range of each element in the key value . In this example, each is an English word and is capitalized, so the number of branches of each node The maximum is 26, plus an end flag, indicating that the word ends at this place. So what is it like in Ethereum ? The address is expressed as 40 hexadecimal numbers, so the number of forks (sometimes called Branching Factor) is 17 , because it is 0~f in hexadecimal, plus the end flag, so it is 17.
(2) The lookup rate of the trie depends on the length of the key. The longer the key value, the more times the search needs to be visited. Different from the example in Figure 3-1 above, in Ethereum, all key values are 40, because the addresses are all 40-digit hexadecimal numbers , and the addresses of Bitcoin and Ethereum are not universal. The format lengths are all different (the address in Bitcoin is a 256-bit hash value obtained by hashing with the public key), but one thing is similar, the address in Ethereum is also obtained by converting the public key , In fact, it is to get the hash of the public key, and then don’t need the previous part, as long as the latter part is needed, you will get a 160bit address.
(3) As long as the two addresses are different, they must be mapped to different branches in the tree in the end, so the trie will not collide .
(4) The tree constructed by trie is not affected by the input sorting . As for the Merkle tree mentioned earlier, if it is not sorted, one problem is that the order in which accounts are inserted into the Merkle tree is different, and the resulting tree structure is also different. Compared with the trie, for example, these five words are inserted into the tree in a different order, and the result is actually the same. As long as the given input remains unchanged, no matter how the input is disturbed and rearranged, it is finally inserted into the trie to get The trees are the same. Different nodes, no matter how you insert these accounts in order, the tree constructed in the end is the same.
(5) Every time a block is released, the status of most accounts in the system remains unchanged, and only a few affected accounts will change, so the locality of the update operation is very important. In the example in Figure 3-1 above, suppose you want to update the value corresponding to the key of genesis. In this figure, only the key is drawn, and the value is not drawn. You only need to visit the branch of genesis, and you don’t need to visit other branches, and you don’t need to traverse the whole tree, the locality of updates is good .
(6) Disadvantage: waste of storage . Like the left branch in Figure 3-1, there is only one child node. In this case, if the nodes can be merged, the storage cost can be reduced, and the search efficiency can also be improved, instead of going down one by one. try to find. This introduces the Patricia tree , which is also written as the Patricia trie , which is the prefix tree compressed by the path , that is, the compressed prefix tree.
3.Patricia tree
Compress the path of the trie in Figure 3-1, and the result is shown in Figure 3-2. It can be seen that E and O are still bifurcated under G, and EN is followed by E, and E and S are further down. Fork, and then all back together, the right side is the same. After such compression, intuitively, the height of the tree is significantly shortened, the number of memory accesses will be greatly reduced, and the efficiency will be significantly improved . For Patricia tree, when a new word is inserted, the original compressed path may need to be expanded. For example, in this example, if geometry is added, the left branch cannot be compressed in this way.
Under what circumstances does path compression work better? If the distribution of inserted key values in the tree is relatively sparse, there is a big difference in the effect of path compression. For example, in this example, English words are used. For example, each word is very long, but there are not many in total. Words, for example, include misunderstanding, decentralization (decentralization), disintermediation (non-intermediation) (intermediaries: middlemen). These three words are inserted into an ordinary trie and it becomes as shown in Figure 3-3 below. It can be seen that the efficiency of such a structure is relatively low, and it is basically a line. If Patricia tree is used, refer to Figure 3-3 below.
The efficiency of the structure shown in Figure 3-3 is relatively low. The result of using the Patricia tree is shown in Figure 3-4, and the height of the tree is significantly improved. When the key value distribution is relatively sparse, the path compression effect is better. If it is applied in Ethereum, the key value is the address, and the address is 160 bits, so the address space has a very, very large number. If you design an algorithm for a computer program, the number of calculations that need to be performed is
, then this It is impossible to calculate it in our lifetime, and the total number of Ethereum accounts in the world is far from that large. Compared with this number, it is very small, so the accounts of Ethereum are very, very sparse. Why is the design so sparse, and the address length is not shortened, so that the access efficiency is faster, and there is no need to be so sparse? The creation method of ordinary accounts in Ethereum is the same as that of Bitcoin accounts. There is no central node, and each user creates an account independently, and generates a public-private key pair locally, which is an account. In order to prevent two people's accounts from colliding (that is, the generated accounts are the same, this possibility exists, but the actual probability of collision is very small), the address is designed to be long enough and the distribution is sparse enough . This may seem a bit wasteful, but this is the only way for a decentralized system to prevent account conflicts , so it is very sparse, which is why Patricia tree is used in the data structure.
4.MPT（Merkle Patricia tree）
(1) What is a Merkle Patricia tree
The difference between the blockchain and the ordinary linked list: the ordinary pointer is replaced by a hash pointer. The difference between Merkel tree and Binary tree: replace ordinary pointers with hash pointers. In the Ethereum system, organize all accounts into a Patricia tree (use path compression to improve efficiency), then replace ordinary pointers with hash pointers, calculate the root hash value, and write it in the Block Header. There is only one root hash value in the Block Header of Bitcoin, which is the root hash value composed of the Merkle tree composed of transactions contained in the block. There are three in Ethereum, a transaction tree composed of transactions, and a state tree composed of user states. , and receipt tree , the state tree is composed of the account state, and its root hash value is also written in the Block Header.
(2) The role of the root hash value
①Prevent tampering. The root hash does not change, and there is no way for any part of the entire tree to be tampered with. That is, the root hash can be used to verify that the state of the account has not been tampered with.
②Merkle Proof: Prove what the balance of the account is . The branch where the account is located makes a Merkle Proof and sends it to the light node, and the light node can verify the balance of the account.
③ Verify that a transaction does not exist . Before transferring money to an address, verify whether the account information exists in the full node, that is, to prove that a certain key value in the MPT does not exist. The proof method is similar to the Sorted Merkle tree. If it exists, what kind of branch is it? Send this branch as a Merkle Proof to prove that a certain account information does not exist.
(3) MPT in Ethereum - Modified MPT
What is used in Ethereum is not the original MPT, but the Modified MPT. Here, some modifications have been made to the MPT. These modifications are not essential modifications. As shown in Figure 3-5, the root node Root in the tree is obtained after hashing The root hash value will be written in the block header, as shown in the upper left corner of Figure 3-5. In addition, the locations where the addresses of each node in the figure originally store are all hash values. For details, refer to Bitcoin-Thinking (1) .
Every time a new block is released, the values of some nodes in the state tree will change. These changes are not modified in situ, but create a new branch, so the original state is actually preserved. As shown in Figure 3-6, there is actually a large MPT in Ethereum that contains many small MPTs, and each account has a small MPT.
What each full node in the system needs to maintain is not an MPT, but every time a block appears, a new MPT must be created, but in these state trees, most of the nodes are shared, and only a few of them change Node to create a new branch.
Why should the historical state be preserved, and the state tree cannot be directly modified in place? Sometimes there will be forks in the system. Temporary forks are very common. Ethereum reduces the block generation time to more than ten seconds, which makes temporary forks become the norm (because it may take ten seconds for the block to spread on the Internet). seconds). Assuming that there is a fork, the two nodes obtain the bookkeeping rights at the same time, the upper fork wins and becomes the longest legal connection, and the lower fork node will roll back. That is to say, the current state of this node (accepting the state of the following node) needs to be cancelled, return to the previous state, and then move down along the legal chain above, as shown in Figure 3-7, where the red block is The chain is what needs to be rolled back. To achieve rollback, it is necessary to maintain these historical records.
Unlike Bitcoin, in the Bitcoin system, the transaction types are relatively simple, and sometimes the previous state can be deduced through reverse operations. Suppose there is a transfer transaction, A→B (10BTC). The impact on the account balance is: A’s account is 10 BTC less, and B’s account is 10 BTC more. If this state needs to be rolled back, the B account will be reduced by 10 BTC, and the A account will be added 10 BTC. A simple transfer transaction rollback is relatively Easy. Why can't Ethereum roll back directly? Because there are smart contracts in Ethereum, which are Turing-complete, and the programming function is very strong. In theory, very complex functions can be realized, which is different from the simple script of Bitcoin, so if you do not save For the previous state, after the execution of the smart contract, it is impossible to calculate the previous state. Therefore, if you want to support rollback, you must save the historical state.
4. Data structure in Ethereum
1. Definition of Block Header
// Header represents a block header in the Ethereum blockchain.
type Header struct {
ParentHash common.Hash `json:"parentHash" gencodec:"required"`
UncleHash common.Hash `json:"sha3Uncles" gencodec:"required"`
coinbase common.Address `json:"miner" gencodec:"required"`
Root common.Hash `json:"stateRoot" gencodec:"required"`
TxHash common.Hash `json:"transactionsRoot" gencodec:"required"`
ReceiptHash common.Hash `json:"receiptsRoot" gencodec:"required"`
Bloom Bloom `json:"logsBloom" gencodec:"required"`
Difficulty *big.Int `json:"difficulty" gencodec:"required"`
Number *big.Int `json:"number" gencodec:"required"`
GasLimit uint64 `json:"gasLimit" gencodec:"required"`
Gasused uint64 `json:"gasused" gencodec:"required"`
Time *big.Int `json:"timestamp" gencodec:"required"`
Extra [ ]byte `json:"extraData" gencodec:"required"`
MixDigest common.Hash `json:"mixHash" gencodec:"required"`
Nonce BlockNonce `json:"nonce" gencodec:"required"`
}
2. Definition of Block
// Block represents an entire block in the Ethereum blockchain.
type Block struct {
header *Header
uncles []*Header
transactions Transactions
//caches
hash atomic.value
size atomic.value
//Td is used by package core to store the total difficulty
//of the chain up to and including the block.
td *big.Int
//These fields are used by package eth to trackl
//inter-peer block relay.
ReceivedAt time.Time
ReceivedFrom interface{}
}
3.External block
This information is released when a block is actually released online, as shown in Figure 4-3.
type extbllock struct {
header *Header
Txs []*Transactions
uncles []*Header
}What is saved in the state tree is (key, value), and the key is the address. At present, we mainly talk about the key value and the management method of this address. How is value, the state of the account, stored in the state tree? This requires a serialization (RLP: Recursive Length Prefix) process before storage. RLP is a serialization method characterized by simplicity, minimalism, and the simpler the better. Protocol buffer: Protobuf for short, is a well-known serialization library. Compared with these libraries, RLP's philosophy is that the simpler the better, it only supports one type, nested array bytes (byte array), that is, an array composed of one byte (can be nested). All other types in Ethereum, such as integers and hash tables, will eventually become nested array bytes. Therefore, implementing RLP is much simpler than implementing Protocol buffer, because the difficult things are not done, and all are left to the application layer.