In the previous article, we completed the operation of connecting go to the blockchain. In this chapter, we will complete the function development of obtaining data on the chain and persisting it to the database
This series of articles
1. [Using go to develop blockchain] to obtain on-chain data (01)
2. [Using go to develop blockchain] to obtain on-chain data (02)
3. [Using go to develop blockchain] Get on-chain data (03)
4. [Use go to develop blockchain] get on-chain data (04)
1. Obtain blockchain data
1.1. Obtain the corresponding block information through the block height
In the previous chapter, we finally obtained the latest height of the blockchain through the following code
blockNumber, err := global.EthRpcClient.BlockNumber(context.Background())
Next, we need to pass in the obtained block height blockNumber as an input parameter to obtain the block information corresponding to the block height:
lastBlock, err := global.EthRpcClient.BlockByNumber(context.Background(), big.NewInt(int64(blockNumber)))
Let's look at its definition
func (b *Block) Transactions() Transactions {
return b.transactions }
func (b *Block) Transaction(hash common.Hash) *Transaction {
for _, transaction := range b.transactions {
if transaction.Hash() == hash {
return transaction
}
}
return nil
}
func (b *Block) NumberU64() uint64 {
return b.header.Number.Uint64() }
func (b *Block) MixDigest() common.Hash {
return b.header.MixDigest }
func (b *Block) Nonce() uint64 {
return binary.BigEndian.Uint64(b.header.Nonce[:]) }
func (b *Block) Bloom() Bloom {
return b.header.Bloom }
func (b *Block) Coinbase() common.Address {
return b.header.Coinbase }
func (b *Block) Root() common.Hash {
return b.header.Root }
func (b *Block) ParentHash() common.Hash {
return b.header.ParentHash }
func (b *Block) TxHash() common.Hash {
return b.header.TxHash }
func (b *Block) ReceiptHash() common.Hash {
return b.header.ReceiptHash }
func (b *Block) UncleHash() common.Hash {
return b.header.UncleHash }
func (b *Block) Hash() common.Hash {
if hash := b.hash.Load(); hash != nil {
return hash.(common.Hash)
}
v := b.header.Hash()
b.hash.Store(v)
return v
}
It can be seen that it provides many methods, such as Hash() to obtain the block Hash, Transactions() to obtain the transactions contained in the block, etc. We can choose to store the required data according to our business
1.2. Parse the transaction data contained in the block
1.2.1. Analyzing transaction data
We can get the transaction lastBlock.Transactions() contained in the block through the block, which is actually an array of Transactions, and we can traverse it through range:
for _, tx := range block.Transactions()
1.2.1.1. Get transaction receipt
tx is each transaction object included in the block, we need to get the transaction receipt information through the Hash of the transaction object:
receipt, err := global.EthRpcClient.TransactionReceipt(context.Background(), tx.Hash())
The receipt structure is as follows:
type Receipt struct {
// Consensus fields: These fields are defined by the Yellow Paper
Type uint8 `json:"type,omitempty"`
PostState []byte `json:"root"`
Status uint64 `json:"status"`
CumulativeGasUsed uint64 `json:"cumulativeGasUsed" gencodec:"required"`
Bloom Bloom `json:"logsBloom" gencodec:"required"`
Logs []*Log `json:"logs" gencodec:"required"`
// Implementation fields: These fields are added by geth when processing a transaction.
TxHash common.Hash `json:"transactionHash" gencodec:"required"`
ContractAddress common.Address `json:"contractAddress"`
GasUsed uint64 `json:"gasUsed" gencodec:"required"`
EffectiveGasPrice *big.Int `json:"effectiveGasPrice"`
// Inclusion information: These fields provide information about the inclusion of the
// transaction corresponding to this receipt.
BlockHash common.Hash `json:"blockHash,omitempty"`
BlockNumber *big.Int `json:"blockNumber,omitempty"`
TransactionIndex uint `json:"transactionIndex"`
}
One of them defines Logs , which is the log event contained in each of our transactions. It is also an array type, and we need to parse it out.
Here, some students may have doubts, what is this log? I send a picture to illustrate:
This picture is taken from: Logs tab of blockchain browser data
From the figure, we can see that there will be multiple logs in a transaction. In fact, this log is the data we need to capture the most. Take the monitoring of an NFT contract Mint event as an example. We actually need to capture the NFT The Log event of the contract, and then analyze whether it is a Mint event one by one
1.2.1.1.1, Analyzing Log Events
Log structure definition:
type Log struct {
// Consensus fields:
// address of the contract that generated the event
Address common.Address `json:"address" gencodec:"required"`
// list of topics provided by the contract.
Topics []common.Hash `json:"topics" gencodec:"required"`
// supplied by the contract, usually ABI-encoded
Data []byte `json:"data" gencodec:"required"`
// Derived fields. These fields are filled in by the node
// but not secured by consensus.
// block in which the transaction was included
BlockNumber uint64 `json:"blockNumber"`
// hash of the transaction
TxHash common.Hash `json:"transactionHash" gencodec:"required"`
// index of the transaction in the block
TxIndex uint `json:"transactionIndex"`
// hash of the block in which the transaction was included
BlockHash common.Hash `json:"blockHash"`
// index of the log in the block
Index uint `json:"logIndex"`
// The Removed field is true if this log was reverted due to a chain reorganisation.
// You must pay attention to this field if you receive logs through a filter query.
Removed bool `json:"removed"`
}
What we need is Topics[] and Data[], where Topics[0] is the first 4 bytes encrypted by keccak256 of this method (that is, the function selector). Topics contains up to 4 data, which is declared as indexed Field (this involves solidity knowledge, you can have an impression, I will explain it in detail in the solidity tutorial, if you are interested, you can check the solidity course I released), Data contains the remaining data
1.2.1.2. Processing transaction data
1.2.1.2.1. Verify whether the contract is created
What if we want to know if a transaction is creating a contract? In fact, it is very simple. There is a to field in each transaction. If the to field is empty , it means that the transaction is a contract creation operation.
1.2.1.2.2. Verify whether the address is a contract address
We can verify whether an address is a contract address by the following methods:
// 判断一个地址是否是合约地址
func isContractAddress(address string) (bool, error) {
addr := common.HexToAddress(address)
code, err := global.EthRpcClient.CodeAt(context.Background(), addr, nil)
if err != nil {
return false, err
}
return len(code) > 0, nil
}
Judging by obtaining the code of the specified address, if the length of the code is not 0, the address is the contract address
1.3. Persistent on-chain data
In the above chapters, we have explained how to obtain block data on the chain and how to parse it. Next, we will persist the data on the chain to our database
1.3.1. Create entity class
1.3.1.1, create transaction.go
Create a transaction.go file in the internal/model directory to store transaction data:
type Transaction struct {
Id uint64 `json:"id" gorm:"primary_key;AUTO_INCREMENT"`
BlockNumber uint64 `json:"block_number"`
TxHash string `json:"tx_hash" gorm:"type:char(66)" `
From string `json:"from" gorm:"type:char(42)" `
To string `json:"to" gorm:"type:char(42)" `
Value string `json:"value" gorm:"type:varchar(256)" `
Contract string `json:"contract" gorm:"type:char(42)" `
Status uint64 `json:"status"`
InputData string `json:"input_data" gorm:"type:varchar(4096)"`
*gorm.Model
}
func (tx *Transaction) TableName() string {
return "transactions"
}
func (tx *Transaction) Insert() error {
if err := global.DBEngine.Create(&tx).Error; err != nil {
return err
}
return nil
}
1.3.1.2, create event.go
Create an event.go file in the internal/model directory to store event data:
type Events struct {
Id uint64 `json:"id" gorm:"primary_key;AUTO_INCREMENT" `
Address string `json:"address" gorm:"type:char(42)" `
Data string `json:"data" gorm:"type:longtext" `
BlockNumber uint64 `json:"block_number"`
TxHash string `json:"tx_hash" gorm:"type:char(66)" `
TxIndex uint `json:"tx_index" `
BlockHash string `json:"block_hash" gorm:"type:varchar(256)" `
LogIndex uint `json:"log_index"`
Removed bool `json:"removed"`
*gorm.Model
}
func (e *Events) TableName() string {
return "events"
}
func (e *Events) Insert() error {
if err := global.DBEngine.Create(&e).Error; err != nil {
return err
}
return nil
}
func (e *Events) GetEventByTxHash() (*Events, error) {
var event Events
if err := global.DBEngine.Where("tx_hash = ?", e.TxHash).First(&event).Error; err != nil {
return nil, err
}
return &event, nil
}
1.3.1.3, Create topic.go
Create a topic.go file in the internal/model directory to store the topic data of the event:
type Topic struct {
Id uint64 `json:"id" gorm:"primary_key;AUTO_INCREMENT" json:"id"`
EventId uint64 `json:"event_id"`
Topic string `json:"topic" gorm:"type:longtext" `
*gorm.Model
}
func (tc *Topic) TableName() string {
return "topics"
}
func (tc *Topic) Insert() error {
if err := global.DBEngine.Create(&tc).Error; err != nil {
return err
}
return nil
}
1.3.1.4, modify the MigrateDb method
Modify the MigrateDb() method in db.go as follows:
// MigrateDb 初始化数据库表
func MigrateDb() error {
if err := global.DBEngine.AutoMigrate(&models.Blocks{
}, &models.Transaction{
}, &models.Events{
}, &models.Topic{
}); err != nil {
return err
}
return nil
}
1.3.2. Preparations
1.3.2.1, Create block.go
In the pkg directory, create a new blockchain directory, and then create a new block.go file in the blockchain directory
1.3.2.2. Initialize block information
When we query block information, we need a block height parameter. When our project was first created, the database block table was empty, so we need to initialize the first block information first, in pkg/blockchain / The block.go file creates a new InitBlock() method:
// InitBlock 初始化第一个区块数据
func InitBlock() {
block := &models.Blocks{
}
count := block.Counts()
if count == 0 {
lastBlockNumber, err := global.EthRpcClient.BlockNumber(context.Background())
if err != nil {
log.Panic("InitBlock - BlockNumber err : ", err)
}
lastBlock, err := global.EthRpcClient.BlockByNumber(context.Background(), big.NewInt(int64(lastBlockNumber)))
if err != nil {
log.Panic("InitBlock - BlockByNumber err : ", err)
}
block.BlockHash = lastBlock.Hash().Hex()
block.BlockHeight = lastBlock.NumberU64()
block.LatestBlockHeight = lastBlock.NumberU64()
block.ParentHash = lastBlock.ParentHash().Hex()
err = block.Insert()
if err != nil {
log.Panic("InitBlock - Insert block err : ", err)
}
}
}
The above code mainly does several tasks:
- First query whether the block record already exists in the database
- If it does not exist, query the latest block height
- Query the latest block information through the block height
- Assemble the data and store it in the database
1.3.3, persistent data
1.3.3.1, Create a new task execution method
Create a new SyncTask() method in the pkg/blockchain/block.go file. We hope that the program can pull data from the chain at intervals. In the project, we can use a ticker to achieve this. Declare a ticker object, which is one second in the example Time interval, and then take the value through chan (channel) to perform timing operations:
func SyncTask() {
ticker := time.NewTicker(time.Second * 1)
defer ticker.Stop()
for {
select {
case <-ticker.C:
latestBlockNumber, err := global.EthRpcClient.BlockNumber(context.Background())
if err != nil {
log.Panic("EthRpcClient.BlockNumber error : ", err)
}
var blocks models.Blocks
latestBlock, err := blocks.GetLatest()
if err != nil {
log.Panic("blocks.GetLatest error : ", err)
}
if latestBlock.LatestBlockHeight > latestBlockNumber {
log.Printf("latestBlock.LatestBlockHeight : %v greater than latestBlockNumber : %v \n", latestBlock.LatestBlockHeight, latestBlockNumber)
continue
}
currentBlock, err := global.EthRpcClient.BlockByNumber(context.Background(), big.NewInt(int64(latestBlock.LatestBlockHeight)))
if err != nil {
log.Panic("EthRpcClient.BlockByNumber error : ", err)
}
log.Printf("get currentBlock blockNumber : %v , blockHash : %v \n", currentBlock.Number(), currentBlock.Hash().Hex())
err = HandleBlock(currentBlock)
if err != nil {
log.Panic("HandleBlock error : ", err)
}
}
}
}
The above code mainly completes the operation:
- Get the latest block height
- Query the latest stored block data from the database
- Determine whether the latest blockchain height stored in the database is greater than the latest block height queried
- If it is larger, jump out of the loop and do not perform subsequent operations, otherwise query the block information through the latest blockchain height stored in the database
- Process the latest block information (stored in the database) through the HandleBlock() method
1.3.3.2, processing block data
The HandleBlock() method is as follows:
// HandleBlock 处理区块信息
func HandleBlock(currentBlock *types.Block) error {
block := &models.Blocks{
BlockHeight: currentBlock.NumberU64(),
BlockHash: currentBlock.Hash().Hex(),
ParentHash: currentBlock.ParentHash().Hex(),
LatestBlockHeight: currentBlock.NumberU64() + 1,
}
err := block.Insert()
if err != nil {
return err
}
err = HandleTransaction(currentBlock)
if err != nil {
return err
}
return nil
}
The above code mainly completes the work:
- Process block data and store it in the database
- Call the HandleTransaction() method to process the transaction data contained in the block
1.3.3.3. Processing transaction data
Create a new transaction.go file in the pkg/blockchain directory:
// HandleTransaction 处理交易数据
func HandleTransaction(block *types.Block) error {
for _, tx := range block.Transactions() {
receipt, err := global.EthRpcClient.TransactionReceipt(context.Background(), tx.Hash())
if err != nil {
log.Error("get transaction fail", "err", err)
}
for _, rLog := range receipt.Logs {
err = HandleTransactionEvent(rLog, receipt.Status)
if err != nil {
log.Error("process transaction event fail", "err", err)
}
}
err = ProcessTransaction(tx, block.Number(), receipt.Status)
if err != nil {
log.Error("process transaction fail", "err", err)
}
}
return nil
}
func ProcessTransaction(tx *types.Transaction, blockNumber *big.Int, status uint64) error {
from, err := types.Sender(types.LatestSignerForChainID(tx.ChainId()), tx)
if err != nil {
log.Error("Failed to read the sender address", "TxHash", tx.Hash(), "err", err)
return err
}
log.Info("hand transaction", "txHash", tx.Hash().String())
transaction := &models.Transaction{
BlockNumber: blockNumber.Uint64(),
TxHash: tx.Hash().Hex(),
From: from.Hex(),
Value: tx.Value().String(),
Status: status,
InputData: hex.EncodeToString(tx.Data()),
}
if tx.To() == nil {
log.Info("Contract creation found", "Sender", transaction.From, "TxHash", transaction.TxHash)
toAddress := crypto.CreateAddress(from, tx.Nonce()).Hex()
transaction.Contract = toAddress
} else {
isContract, err := isContractAddress(tx.To().Hex())
if err != nil {
return err
}
if isContract {
transaction.Contract = tx.To().Hex()
} else {
transaction.To = tx.To().Hex()
}
}
err = transaction.Insert()
if err != nil {
log.Error("insert transaction fail", "err", err)
return err
}
return nil
}
1.3.3.4, processing event data
Create a new event.go file in the pkg/blockchain directory:
func HandleTransactionEvent(rLog *types.Log, status uint64) error {
log.Info("ProcessTransactionEvent", "address", rLog.Address, "data", rLog.Data)
event := &models.Events{
Address: rLog.Address.String(),
Data: "",
BlockNumber: rLog.BlockNumber,
TxHash: rLog.TxHash.String(),
TxIndex: rLog.TxIndex,
BlockHash: rLog.BlockHash.String(),
LogIndex: rLog.Index,
Removed: rLog.Removed,
}
err := event.Insert()
if err != nil {
log.Error("event.Insert() fail", "err", err)
return err
}
evt, err := event.GetEventByTxHash()
if err != nil {
log.Error("event.GetEventByTxHash() fail", "err", err)
return err
}
log.Info("Topics", "topic", rLog.Topics)
for _, tp := range rLog.Topics {
topic := &models.Topic{
EventId: evt.Id,
Topic: tp.String(),
}
err := topic.Insert()
if err != nil {
log.Error("topic.Insert() fail", "err", err)
return err
}
}
return nil
}
1.4. Verification
1.4.1, modified main.go
Modify the main() method:
func main() {
blockchain.InitBlock()
blockchain.SyncTask()
}
1.4.2. Execution
Execute the main() method, and the normal print results are as follows:
1.4.3. View database information
1.4.3.1, block table
1.4.3.2, transaction table
1.4.3.3, event table
1.4.3.4, topic table
The data has been correctly inserted into the database, indicating that our program is working normally
Through the study of this chapter, we have completed 1) data fetching on the chain, 2) data analysis on the chain, and 3) data persistence on the chain. In fact, the data analysis of the block can be further in-depth, such as judging whether it is ERC721/ERC20 The contract is created, and then it is processed differently according to the actual business. This chapter will not explain in detail. If you want to know more, you can private message me. At this point, the series of articles [Using go to obtain data on the chain] is all over. Any questions please leave me a message
Please pay attention to the official account: Web3_preacher (web3_preacher) , reply "go to get the data on the chain" to receive the complete code