Note 36-Ethereum transaction pool architecture design

The content of this document comes from the Internet.

The current Ethereum public chain can process 30 to 40 transactions per second on average. Therefore, once a hot DAPP appears in Ethereum, transaction congestion is extremely likely to occur.

The low transaction processing speed can never be compared with the existing centralized services. When a large number of transactions are queued in the network, how do miners select and manage these transactions?

1. Transactions

Transaction refers to transferring certain assets from an external account to an account, or sending a message instruction to a smart contract.

In the Ethereum network, transaction execution belongs to a transaction. It has the characteristics of atomicity, consistency, isolation, and durability.

Atomicity: It is the indivisible smallest unit of execution, either doing it or not doing it.
Consistency: The execution of the same transaction must change the Ethereum ledger from one consistent state to another consistent state.
Isolation: The transaction will not be interfered by other transactions during the execution of the transaction.
Persistence: Once the transaction is submitted, the changes to the Ethereum ledger are permanent. Subsequent operations will not have any effect on it.

Because it is transactional, we need to ensure that the transaction meets some design requirements before executing the transaction.

The transaction must be unique, different transactions can be distinguished, and the same transaction cannot be repeatedly submitted to the ledger.
The content of the transaction must not change, the transaction received by each node must be consistent, and the change of the ledger status when the transaction is executed is also consistent.
Transactions must be legally signed, and only correctly signed transactions can be executed.
The transaction should not occupy too much system resources and affect the execution of other transactions.

The design requirements for transactions involve all aspects of the software system, but the most basic part is the transaction data itself. Next, I will elaborate on the trading design in geth.

Transaction data structure:

The figure below is the Ethereum transaction data structure. According to the purpose, I divide it into four parts.

At the beginning is a uint64 type number, called a random number. Used to cancel transactions, prevent double spending, and modify the Nonce value of the Ethereum account.
The second part is about the setting of transaction execution limits, gas is the upper limit of fuel willing to run the Ethereum virtual machine. gasPrice It is the unit price of fuel that is willing to pay. gasPrcie * gas It is the highest commission that is willing to pay for this transaction.
From the logic of program execution, I can explain the third part in this way. It is the initial information that the transaction sender enters the Ethereum virtual machine to execute the transaction: the virtual machine operation object (receiver To), the asset transferred from the transaction sender to the operation object (Value), and the virtual machine runtime input (input) . When To is empty, it means that the virtual machine has no operable objects. At this time, the virtual machine will use the input content to deploy a new contract.
The fourth part is the signature result of the transaction by the transaction sender. The transaction content and signature result can be used to derive the signer, that is, the address of the transaction sender.

The combination of four parts solves the problem of transaction security, realizes the interactive mode of smart contracts, and provides flexible and adjustable transaction fees.

2. Transaction processing flow

When you send a transfer transaction to Zhang Sanshi through the Ethereum wallet. How did this transaction enter the network and finally be packaged into the block by the miners?

The following figure shows the key process of a transaction from birth to the transaction entering the block.

First, users can send transactions to a running Ethereum geth node through the Ethereum wallet or other calls to the Ethereum node API (eth_sendRawTransaction, etc.).

At this time, because the transaction is received through the node's API, the transaction is regarded as a local (represented by a red ball in the figure) after a series of checksum processing. The transaction successfully enters the transaction pool, and then the transaction is sent to the connected neighboring node.

When a neighboring node, such as a miner node, receives this transaction from a neighboring node, before entering the transaction pool. The transaction will be marked as a transaction from a remote (remote) (represented by a green ball in the figure). It also needs to go through the checksum processing, enter the transaction pool of the miner node, and wait for the miner to pack it into the block.

If the neighboring node is not a miner, it does not matter. Because any node will send the received legal transaction to neighboring nodes in time by default. Thanks to the P2P network, a transaction spreads to all nodes of the entire Ethereum public chain network within 6s on average.

The reason why transactions entering the Ethereum transaction pool are distinguished between local and remote is because the node treats local transactions and remote transactions differently. Simply put, local transactions have higher priority than remote transactions.

3. Ethereum transaction pool design

The transaction pool processing details are not mentioned above, and the complete process of the Ethereum transaction pool processing a transaction will be explained in detail here. Before explaining, you also have to understand the design model of the Ethereum trading pool. From 2014 to the present, Ethereum's transaction pool has been continuously optimized and never stopped. It also shows from this that the transaction pool is not only important, but also requires high performance.

The following figure shows the main design modules of the Ethereum transaction pool, which are transaction pool configuration, real-time blockchain status, transaction management container, local transaction storage, and new transaction events.

Each module affects each other, the most important of which is transaction management. This is also the part that we need to focus on.

3.1 Transaction pool configuration

There are not many transaction pool configurations, but each configuration directly affects the transaction processing behavior of the transaction pool. The configuration information is defined by TxPoolConfig, and the information is as follows:

// core/tx_pool.go:125
type TxPoolConfig struct {
   Locals    []common.Address
   NoLocals  bool
   Journal   string
   Rejournal time.Duration
   PriceLimit uint64
   PriceBump  uint64
   AccountSlots uint64
   GlobalSlots  uint64
   AccountQueue uint64
   GlobalQueue  uint64
   Lifetime time.Duration
}

Locals: defines a set of account addresses that are regarded as local transactions. Any transaction from this list is considered a local transaction.
NoLocals: Whether to prohibit local transaction processing. The default is fasle, allowing local transactions. If prohibited, all transactions from local will be treated as remote transactions.
Journal: The file name for storing local transaction records, the default is ./transactions.rlp.
Rejournal: The time interval at which local transactions are regularly stored in the file. The default is once every hour.
PriceLimit: The minimum Price requirement for remote transactions to enter the trading pool. This setting has no effect on local transactions. The default value is 1.
PriceBump: The minimum requirement for the percentage increase in the price increase required for the replacement transaction. Any replacement transactions below the requirements will be rejected.
AccountSlots: When the amount of executable transactions in the trading pool (transactions that are waiting for miners to pack) exceeds the limit, each account is allowed to keep the minimum number of transactions in the trading pool. The default value is 16 pens .
GlobalSlots: The upper limit of the executable transaction volume allowed in the trading pool. Part of the transaction will be released when the upper limit is exceeded. The default is 4096 transactions .
AccountQueue: The upper limit of non-executable transactions for a single account in the transaction pool, the default is 64 .
GlobalQueue: The upper limit of all non-executable transactions in the transaction pool, 1024 by default .
Lifetime: The maximum time that remote non-executable transactions can survive in the transaction pool. The transaction pool is checked every minute. Once an expired remote account is found, all non-executable transactions under the account will be removed. The default is 3 hours .

The above configuration contains two important concepts: executable transactions and non-executable transactions . Executable transaction means that a part of the transaction selected from the transaction pool can be executed and packaged into the block. The opposite is true for non-executable transactions. Any transaction that has just entered the trading pool is in a non-executable state and will only be promoted to an executable state at a certain moment.

How does a node customize the above transaction configuration? The Ethereum geth node allows to modify the configuration through parameters when starting the node. The trading pool configuration parameters that can be modified are as follows (by geth -h viewing):

TRANSACTION POOL OPTIONS:
  --txpool.locals value        Comma separated accounts to treat as locals (no flush, priority inclusion)
  --txpool.nolocals            Disables price exemptions for locally submitted transactions
  --txpool.journal value       Disk journal for local transaction to survive node restarts (default: "transactions.rlp")
  --txpool.rejournal value     Time interval to regenerate the local transaction journal (default: 1h0m0s)
  --txpool.pricelimit value    Minimum gas price limit to enforce for acceptance into the pool (default: 1)
  --txpool.pricebump value     Price bump percentage to replace an already existing transaction (default: 10)
  --txpool.accountslots value  Minimum number of executable transaction slots guaranteed per account (default: 16)
  --txpool.globalslots value   Maximum number of executable transaction slots for all accounts (default: 4096)
  --txpool.accountqueue value  Maximum number of non-executable transaction slots permitted per account (default: 64)
  --txpool.globalqueue value   Maximum number of non-executable transaction slots for all accounts (default: 1024)
  --txpool.lifetime value      Maximum amount of time non-executable transaction are queued (default: 3h0m0s)

3.2 Chain status

All transactions entering the transaction pool need to be verified. The most basic thing is to verify whether the account balance is sufficient to pay for the transaction execution. Or whether the transaction nonce is legal. The latest block StateDB maintained in the transaction pool. When the transaction pool receives a new block signal, the statedb will be reset immediately.

After the transaction pool is started, the block header event of the chain will be subscribed:

//core/tx_pool.go:274
pool.chainHeadSub = pool.chain.SubscribeChainHeadEvent(pool.chainHeadCh)

And start listening for new events:

//core/tx_pool.go:305
for {
   select {
   // Handle ChainHeadEvent
   case ev := <-pool.chainHeadCh:
      if ev.Block != nil {
         pool.mu.Lock()
         if pool.chainconfig.IsHomestead(ev.Block.Number()) {
            pool.homestead = true
         }
         pool.reset(head.Header(), ev.Block.Header())
         head = ev.Block

         pool.mu.Unlock()
      }
  //...
  }
}

After receiving the event, execute the func (pool *TxPool) reset(oldHead, newHead *types.Header)method to update the state and process the transaction. The core is to delete and update the transactions that do not meet the requirements in the transaction pool.

3.3 Local transactions

There are multiple purposes for marking transactions as local in the transaction pool:

Store the sent transactions on the local disk. In this way, local transactions will not be lost, and when the node is restarted, it can be reloaded into the transaction pool and broadcasted in real time.
It can be used as a channel for external programs to communicate with Ethereum. The external program only needs to monitor the changes in the file content to get the transaction list.
Local transactions can take precedence over remote transactions. Operations such as restrictions on transaction volume do not affect accounts and transactions under local.

Corresponding to the local transaction storage, the local transaction storage capability is enabled according to the configuration when starting the transaction pool:

//core/tx_pool.go:264
if !config.NoLocals && config.Journal != "" {
		pool.journal = newTxJournal(config.Journal)
		if err := pool.journal.load(pool.AddLocals); err != nil {
			log.Warn("Failed to load transaction journal", "err", err)
		}
    //...
}

And load existing transactions from the disk to the transaction pool. When a new local transaction enters the transaction pool, it will be written into the journal file in real time.

// core/tx_pool.go:757
func (pool *TxPool) journalTx(from common.Address, tx *types.Transaction) {
   if pool.journal == nil || !pool.locals.contains(from) {
      return
   }
   if err := pool.journal.insert(tx); err != nil {
      log.Warn("Failed to journal local transaction", "err", err)
   }
}

As can be seen from the above, only transactions belonging to the local account will be recorded. You haven't noticed that if this is the case, will the journal file grow indefinitely following local transactions? The answer is no, although the transaction cannot be removed from the journal in real time. But it supports periodic updates of journal files.

Journal does not store all local transactions and history, it only stores local transactions that exist in the current transaction pool. Therefore, the transaction pool will periodically execute the rotatejournal file, write the local transactions in the transaction pool into the journal file, and discard the old data.

journal := time.NewTicker(pool.config.Rejournal)
//...
//core/tx_pool.go:353
case <-journal.C:
			if pool.journal != nil {
				pool.mu.Lock()
				if err := pool.journal.rotate(pool.local()); err != nil {
					log.Warn("Failed to rotate local tx journal", "err", err)
				}
				pool.mu.Unlock()
			}
}

3.4 New trading signals

At the beginning of the article, it was mentioned that transactions entering the trading pool will be broadcast to the network. This is dependent on the trading pool to support external subscription of new trading event signals. Any sub-module that subscribes to this event can receive real-time notification of this event and obtain new transaction information when a new executable transaction appears in the transaction pool.

It should be noted that not all transactions that enter the trading pool are notified to the outside, but only after the transaction changes from a non-executable state to an executable state will the signal be sent.

//core/tx_pool.go:705
go pool.txFeed.Send(NewTxsEvent{types.Transactions{tx}})
//core/tx_pool.go:1022
go pool.txFeed.Send(NewTxsEvent{promoted})

In the trading pool, there are two places where the signal will be sent. One is when it is used to replace an existing executable transaction when trading. The second is after a new batch of transactions has been upgraded from a non-executable state to an executable state.

External parties only need to subscribe to SubscribeNewTxsEvent(ch chan<- NewTxsEvent)new executable transaction events to accept transactions in real time. In geth, the network layer will subscribe to transaction events for real-time broadcast.

//eth/handler.go:213
pm.txsCh = make(chan core.NewTxsEvent, txChanSize)
pm.txsSub = pm.txpool.SubscribeNewTxsEvent(pm.txsCh)
//eth/handler.go:781
func (pm *ProtocolManager) txBroadcastLoop() {
   for {
      select {
      case event := <-pm.txsCh:
         pm.BroadcastTxs(event.Txs)
      //...
   }
}

In addition, miners subscribe to transactions in real time so that they can be packaged into blocks.

//miner/worker.go:207
worker.txsSub = eth.TxPool().SubscribeNewTxsEvent(worker.txsCh)
//miner/worker.go:462
txs := make(map[common.Address]types.Transactions)
for _, tx := range ev.Txs {
		acc, _ := types.Sender(w.current.signer, tx)
   	txs[acc] = append(txs[acc], tx)
}
txset := types.NewTransactionsByPriceAndNonce(w.current.signer, txs)
w.commitTransactions(txset, coinbase, nil)

3.5 Transaction Management

The core part is the transaction management mechanism of the transaction pool. Ethereum divides transactions into two parts according to status: executable transactions and non-executable transactions. Recorded in the pending container and queue container respectively.

As shown in the figure above, the transaction pool first uses a txLookup (internal map) to track all transactions. At the same time, the transaction is divided into two parts, queue and pending, based on the principle of local priority and price priority. The two transactions are tracked separately by account.

So what are the details of the transaction when it enters the transaction pool for management? Wait for my next article to introduce in detail the transaction management of the Ethereum transaction pool.