Mike Tang
Compound (https://compund.finanice) officially launched its independent chain network Gateway (formerly known as Compound Chain) in March 2021. Gateway is an independent chain, and its goal is to become a unified Defi hub to achieve cross-chain Defi—you can pledge assets on chain A and borrow on chain B—thus connecting all chain assets, Improve asset utilization and liquidity across the network.
The emergence of Gateway is based on the following judgment: the future world must be a multi-chain world, that is, no single blockchain can solve all problems. There will be many chains and many possibilities in the future. So people's assets will be dispersed to various independent platforms. How to improve the utilization of these scattered assets is the problem that Gateway needs to study and solve.
Gateway's system architecture
Gateway architecture diagram
As shown in the architecture diagram, Gateway connects all chains by developing specific Starport (a set of contracts or components on Peer Chain) for various chains (Peer Chain).
This article will not describe too much about Gateway's Defi-related matters. This article mainly makes a brief analysis of why Gateway chose Substrate for development.
Why choose Substate
Compound’s official statement:
We chose Substrate so that we could focus on building application code, instead of inventing consensus algorithms; it’s a modern framework built on a modern language, Rust. (https://medium.com/compound-finance/gateway-623f6f48d2b6)
We chose Substrate so that we could focus on building application code instead of inventing consensus algorithms; and it's a modern framework built on top of the modern language Rust.
The Rust language is a modern security-oriented programming language. Regarding the good reputation of the Rust language, this article will not repeat it. Interested friends can read "2020 Developer Survey Report: Rust Becomes the Most Popular Language Again". Rust has almost become the language of choice for blockchain development today, such as: Polkadot, Near, Solana, Dfinity, the bottom layer of FileCoin, etc. are all implemented in Rust. The Rust language has the following notable features:
- Memory safety: Its unique ownership and lifetime design theoretically guarantees that memory errors will not occur. Solved 70% of the underlying system defects
- Concurrency safety: In concurrent programming (multi-thread, multi-coroutine), the safety of concurrency can be guaranteed
- High performance: performance at the same level as C/C++, currently the top of the mainstream languages
- Zero-overhead abstraction: This feature allows Rust to seamlessly connect to the existing ecology of the C language without performance loss
- Powerful abstract expression ability: Rust draws on the expressive power of many functional languages to make its expression more capable
- Modern engineering facilities: Modern auxiliary systems such as Cargo and crates.io make it easy for Rust to build complex projects
- Active and active community: Rust has been developed and iterated in the community mode from the very beginning, which is actually very interesting, and it is significantly different from the centralized development of go
After understanding these features of Rust, it is natural for us to understand the sentence "Rust language has almost become the language of choice for blockchain development today". The blockchain itself puts security first, which is completely consistent with Rust's design philosophy. A lot of calculations need to be done in the blockchain, and high performance is also required. Rust also does its part. Blockchain systems are generally more complex and have high requirements for engineering-related infrastructure. Rust's powerful engineering design makes it especially suitable for large-scale project collaboration in team and open source community development.
Paritytech, a company known for its emphasis on security, developed the parity client for Ethereum in the Rust language. Then I continued to use Rust to develop the new project Polkadot. During the development of Polkadot, a major decision was gradually formed: to disassemble all the functions of the blockchain into abstract designs, and realize an open source and general-purpose blockchain. In the framework, and use this framework as a tool to build Polkadot products. And the major achievement of this blockchain framework is the protagonist of this article - Substrate.
Substrate is a general-purpose blockchain development framework developed in the Rust language. Its design elements, such as cryptographic algorithms, storage structure MPT tree, account system, etc., are mostly borrowed from the most successful Ethereum infrastructure in history (this is understandable, Paritytech was originally started as an Ethereum client, Gavin Wood is also one of the co-founders of Ethereum). For a framework to be universal, it needs to be highly abstract. However, the cost of high abstraction often appears to be complex in structure and not easy to use. So Substrate also provides a lot of DSL (domain-specific language), which is convenient for novices to learn and use. To briefly summarize, Substrate has the following characteristics:
- General purpose. Its design is oriented to the general field, rather than an SDK specially developed for a certain chain. Each team can use Substrate to develop a completely independent chain that does not depend on any existing network (for example, the blockchain developed using Substrate can have nothing to do with Polkadot, which is also one of Paritytech's design goals)
- Full-featured. It can cover almost all scenarios of the blockchain, and can be said to be the most comprehensive blockchain framework currently on the market
- Runtime code is compiled into wasm for execution. Wasm is the mainstream choice of VM bytecode in the blockchain industry today
- Super customizable. Substrate itself is a bunch of scattered components, which can be freely replaced and combined under a set of specification constraints
Engineering design is about making trade-offs. When it emphasizes one aspect, on the other hand, there will be certain compromises. Substrate also has its own shortcomings:
- Some concept terms (such as: extrinsic) are newly introduced, which need to be relearned and understood, and there is a certain threshold for getting started
- Emphasis on generality leads to over abstraction in some aspects and complex code vision (for example, there are too many generic parameters). These abstractions fully demonstrate the advanced syntax features of Rust, and the code is noisy
- The entire project has a large amount of code and depends on a lot of packages (more than 1,000 at present), resulting in a relatively long compilation time (more than tens of minutes for ordinary notebooks), and high performance requirements for the development machine
As can be seen above, these deficiencies are mainly caused by the objective dependency complexity brought about by Substrate's design goal - general purpose.
The flaws are not hidden, as one of the strongest blockchain development frameworks so far, Substrate has been welcomed by more and more innovative teams, and it is logical for the Compound team to choose Substrate for Gateway development.
Functional components of Substrate
Substrate overall framework diagram
As can be seen from the architecture diagram, Substrate has the following major components:
- P2p Networking P2p network
- Runtime runtime
- Storage storage
- consensus
- RPC remote procedure call
- Telemetry client monitoring tool
The P2p network is an essential subsystem of the blockchain system. Multiple nodes are connected to form the same network through p2p. Messages are sent between nodes to transfer data, and message propagation in the network may arrive after propagation through multiple relay nodes. In Substrate, rust-libp2p (https://github.com/libp2p/rust-libp2p) is used, and this project is mainly maintained by Paritytech.
Runtime, runtime, is the implementation part of blockchain business logic. In other words, if you want to use the blockchain to do practical things, you need to write it in it. Substrate supports compiling Runtime code into wasm bytecode or native code to run (with the improvement of wasm vm running speed, the native code mode may be abandoned in the future, which can also reduce some internal complexity). Storage can be operated in Runtime to realize state changes, so Runtime as a whole is also called State Transition Function (STF).
Storage, the storage subsystem, is an indispensable part of the blockchain system. In Substrate, Storage is used to persist the state changes of Runtime's logic, and also supports the external RPC state reading interface. Before the Event in Runtime is sent, it will also be stored in Storage for a short time. The goal of the consensus system is also to reach a consensus on the state in Storage. The Storage subsystem uses the kv database rocksdb or paritydb at the bottom.
Consensus, the consensus subsystem, is used to reach an agreement on the state of the blockchain among the participants of the network, that is, "consensus". Because it is a distributed system, the state of the so-called blockchain does not have a macro view of God to see a unified macro state. This state is actually the state of each node. Each node has its own local state view and a "local" global view. Each node makes a decision by comparing the local view with the global view. The consensus system needs to reach a consistent state on each node to push the system forward.
RPC, Remote Procedure Call, an interface used to provide access to the outside of the node. A Substrate node itself can exist as a service, and the outside world can access the local state information of the node or submit a change request to the node through these rpc interfaces. Substrate provides both HTTP and Websocket rpc channels.
Telemetry, a client monitoring tool, is used to collect node running information and send it to the remote Prometheus server.
Substrate's development model
Substrate is a general-purpose development framework. It provides three development styles for different levels of developers.
One: Directly use the node that comes with Substrate to start the chain. For developers who want to start the chain quickly and experience the effect, they can directly use the implementation of the node preset by Substrate. You only need to modify a JSON configuration file, and you can run it. Specifically, you can customize the following content:
- State information of the genesis block
- Account Account
- BalanceBalance
- Staking ratio, etc.
Two: the development of Runtime FRAME pallets. This development mode implements business logic into the pallet by writing the pallet code in the Runtime. Then compile the custom pallets and the pallets that come with Substrate that they depend on into wasm bytecode to run, which is the choice of most Substrate developers.
Three: In-depth customization based on Substrate Core. Substrate has been implemented into discrete components, fully abstracted and decoupled. For some advanced developers, in some specific scenarios, these components can be completely reassembled from the bottom layer to achieve deep node customization. For example, you can do:
- Use different cryptographic curves and hashing algorithms
- Use a different serialization method
- Replace different consensus algorithms
- Completely remove the FRAME layer code and use another language to write the business, as long as it can be compiled to wasm and follow the Substrate specification
- etc.
The figure below shows the relationship between development difficulty and technical flexibility at three levels.
Using Substrate Node directly is the easiest, but the least flexible. Development based on Substrate Core is the most flexible, but the most difficult. Doing FRAME pallets development is in the middle. It is also the way most Substrate developers should adopt.
What makes Substrate great
There are many good things about Substrate's design, let's take a look.
Upgradable forkless runtime
Since Substrate's runtime code is compiled into wasm to run, then the wasm bytecode itself is directly submitted to the chain as transaction data, and then propagated through the p2p network of the chain itself to realize the update of business logic. After each node receives the updated version of wasm bytecode, it updates it to the code segment, and uses the new version of wasm to execute logic after a certain block.
With this hot update code mechanism, the upgrade of business code will no longer cause forks (soft forks or hard forks), that is to say, it will not cause network forks due to technical and objective reasons (man-made active Forking is still possible).
It should be noted that this upgrade is limited to the part that the wasm bytecode can cover—that is, the code in the Runtime—upgrade. If you change the node code itself (that is, the part outside the Runtime), you still need to notify all nodes to perform manual or devops replacement.
Alternative Cryptography Libraries
Substrate supports several cryptographic curves at the same time:
- ECDSA
- Ed25519
- SR25519
Several hashes are supported at the same time:
- Blake2
- xxHash
Developers can choose to use it according to their needs. If there is not what you want, it is not difficult to add new curves and Hash functions according to its architecture.
Account system with rich layers
Substrate's Account Key is divided into three levels:
- Stash Key
- Controller Key
- Session Keys
Stash Key is an account used to deposit funds, and its private key part should be stored in a cold wallet as securely as possible. The Controller Key is used to control the parameters of the Validator, and it is also an intermediate proxy account of the Stash Key, which is very useful when updating the validator set. Session Keys are used to sign consensus-related messages. There can be multiple Session Keys, each of which has its own special purpose, and can generally be used in combination, such as:
impl_opaque_keys! {
pubstruct SessionKeys {
pubgrandpa: Grandpa,
pubbabe: Babe,
pubim_online: ImOnline,
pubauthority_discovery: AuthorityDiscovery,
}
}
The above code combines 4 independent Session Keys: Grandpa session key, Babe session key, ImOnline session key, AuthorityDiscovery session key into one big SessionKeys.
Session Keys can be understood as the unique identification (Identification) during the running of Validator. It is best to replace Session Keys every Session, so as to ensure maximum security.
Through this layered Account Key design, Substrate not only ensures security, but also provides sufficient flexibility, which can basically cover the needs of all application scenarios.
Abstract switchable consensus engine
Substrate has designed a consensus framework, which is very impressive. It separates block production (proposal) from finalization (finalize), while accommodating Nakamoto-like (only probabilistic finalization, no deterministic finalization) consensus and BFT-like (deterministic finalization) consensus.
Substrate provides the following algorithms for block generation:
- Aura: slot mode, in a known authority set, use round robin mode to generate blocks in turn
- Babe: slot mode, in a known authority set, use the verifiable random function VRF to randomly select block producers (each slot may have more than one block producer)
- Pow: proof of work to produce blocks
Without the cooperation of Finalize Gadget, these kinds of block generation algorithms can only achieve probabilistic finalization (finalization), but cannot achieve deterministic finalization. Substrate provides Grandpa, the Finalize Gadget, for deterministic finalization.
The block generation algorithm and the finalization algorithm can be freely used together. So there are combinations of Aura/Grandpa, Babe/Grandpa, and even Pow/Grandpa. And in scenarios that do not require deterministic finalization, of course, Grandpa may not be used. Substrate gives developers full freedom.
The Substrate consensus framework also provides some other infrastructure:
- Fork Choince Rules: Longest Chain Rule or GHOST Rule, used to determine how to choose the best chain algorithm when the chain has a fork
- Import Queue: import queue
- Block Import Trait: block import interface
- Block Import Pipeline: block import pipeline
Substrate also provides the function of coordinating and controlling the consensus process in Runtime. For example, adjust the difficulty during the operation of POW, decide whether to rotate next in PoA, dynamically modify the weight of Stake in PoS, and so on.
Based on Substrate's complete set of infrastructure, if its own consensus engine cannot meet the needs of developers, developers can also develop their own consensus engines according to their specifications, introduce them into the framework, and use them in Runtime appropriate controls.
Off-Chain Features
The Off-Chain feature is a fairly powerful infrastructure provided in Substrate. After all, for the blockchain, the logical operation space on the chain is very limited, and some things must be done through the chain. Before Off-Chain Worker (OCW), this type of thing was usually done by Oracle. The oracle is an external service that submits transactions to the blockchain through the RPC interface of the blockchain node to transmit external information to the chain. Although this approach is feasible, it is still not good enough in terms of security, integration, scalability and infrastructure efficiency.
In order to make the integration of off-chain data more secure and effective, Substrate provides off-chain related features. Its architecture diagram is as follows:
Off-chain Worker architecture diagram
The Off-chain feature consists of three major components:
- Off-Chain Worker
- Off-Chain Storage
- Off-Chain Indexing
Off-Chain Worker is used to implement off-chain logic. Its code is written together with the Runtime code, compiled into the same wasm bytecode string, and propagated to the entire network in the same transaction. But during execution, the code of Off-Chain Worker is executed in an independent VM, that is, it is completely isolated from the execution of Runtime logic. Specifically, Off-Chain Worker can realize the following functions:
- Submit the calculated results to the chain in the form of transactions
- Contains a full-featured HTTP client capable of accessing data from external services
- Can access the keystore of the local node so that transactions can be verified and signed
- Can access the local KV database and share this database among all off-chain workers
- A local, secure source of entropy used to generate random numbers
- You can access the local time of this node
- Can sleep and resume work
Off-Chain Storage is a logically independent storage space under the chain, which is completely isolated from the Storage on the chain. It has the following characteristics:
- Can be read and written by Off-Chain Worker
- Stored locally in the node, it will not be passed to other nodes in the network, and will not participate in the network consensus
- Shared access by all concurrently running Off-Chain Workers (thus requiring lock operations). Therefore, it can be used to communicate between different Workers
- Can be written by runtime code, but not read. Therefore, certain on-chain and off-chain interactive functions can be realized based on it
- Can be read by code in node outside the wasm environment, so it can be read by RPC
Off-Chain Indexing provides the ability to write data to Off-Chain Storage in the Runtime environment. But the data in Off-Chain Storage cannot be read. This opens up the possibility for some new programming paradigms.
There are some others, for example, a complete OCW integration testing framework and so on.
Substrate's off-chain features are very powerful and impressive.
Complete and flexible Gas fee calculation mechanism
Friends who have experience in Ethereum development know that the gas fee mechanism is a very successful design, which is very important to the security and smooth operation of the chain. Almost all subsequent blockchains have borrowed directly from this design. In Substrate, a very complete and detailed mechanism and configuration parameters are provided to help developers design their own Gas fee algorithm. The following Gas calculation and configuration parameters are built into Substrate:
- Includsion Fee: 包含 length_fee 和 weight_fee
- Fee Multiplier
- Additional Fees, including
○Bonds
○Deposits
○Burns
○Limits
- Default Weight Annotations
- Dynamic Weights
- Post Dispatch Weight Correction
- Custom Fees
- Custom Inclusion Fee
We will not explain each item in detail here, please refer to: https://substrate.dev for the specific meaning.
It can be seen that Substrate's design for gas fee calculation is very comprehensive, even a little complicated. Its purpose is still to achieve the goal of a general-purpose blockchain framework-everything should be available, and it must be able to be used out of the box.
Substrate is very suitable for starting an independent chain or an Appchain for Web3.0 applications. Generally, these chains will put a lot of effort into user experience. For example, for ordinary users of the application, they may not be aware of the existence of the gas fee in the process of using the service. In this scenario, the above-mentioned rich gas fee mechanism provided by Substrate can set the gas fee to zero or set For payment. Such features have the opportunity to make Substrate one of the most suitable development frameworks for Web3.0 Apps.
Runtime API and RPC integration
Substrate provides a set of RPC extension framework, allowing developers to expand and develop their own RPC interfaces (in addition to the interfaces provided by Substrate by default). Since the RPC implementation code is in node, outside the Runtime, theoretically, a full-featured (do anything) RPC service can be developed in Substrate. This makes Substrate a powerful RPC development framework.
And often we need to interact with the state in the Runtime, then we need to use the Runtime API. The Runtime API is a bridge between inside and outside of Runtime, and it can also be said to be a bridge between on-chain and off-chain.
Therefore, the combination of RPC and Runtime API can open up all the processes from initiating external requests to obtaining the state on the chain. Cooperating with the custom Runtime API function and RPC extension interface function, it gives developers great flexibility and possibilities. It also lays the foundation for Substrate to become an integrated Web3.0 development framework.
How Gateway uses Substrate
In order to find out how Gateway uses Substrate, we directly pull its source code and analyze it briefly.
git clonehttps://github.com/compound-finance/gateway
cd gateway && ls
As you can see, there are the following main directories (we added a brief note).
Starport contract code on ethereum/Ethereum
ethereum-client/Ethereum-related type definitions and basic utility functions
gateway-crypto/Basic cryptographic-related utility functions used by Gateway
integration/integration test code
Custom node code for node/Substrate
The std code of our-std/Gateway is a simple package of sp_std with a little extra
The main business logic code of the pallets/Gateway independent chain
Runtime build code for runtime/Substrate
Type-dependent procedural macros used by types-derive/Gateway
It can be seen that currently Gateway only realizes the startport connection to ethereum, and the connection of other chains is still under development and will be added gradually.
The main business code is under the pallets/ directory, and there are three subdirectories under this directory.
cash/main business logic
oracle/ code to get price from price feeder
runtime-interfaces/Some runtime interfaces
Let's start at pallets/cash/src/lib.rs. This is a standard pallet file with the structure
- Definition of trait Config
- decl_storage! Define the on-chain storage part
- decl_event! Define event output section
- decl_module! defines the module implementation part
In the module implementation part, there is the function fn offchain_worker(block_number: T::BlockNumber) {}. Its implementation is mainly these two function calls:
internal::events::track_chain_events::()
internal::notices::process_notices::(block_number)
We know that Substrate's offchain_worker entry is called when each block is imported into the local state database. That is to say, the function fn offchain_worker(block_number: T::BlockNumber) will be executed by each block, and this function will be executed once for each block. Therefore, each time a block is added, the above code will process events and notices separately, and the codes of these two blocks are defined in the internal module.
After continuing to track the pallets/cash/src/internal/events.rs and pallets/cash/src/internal/notices.rs files, the general logical points can be sorted out:
- Offchain_worker called once per block
- In the offchain_worker code, it will call the Ethereum basic types and auxiliary tools provided by ethereum-client/, send http requests to the Ethereum event server, obtain Ethereum events in batches, and classify and process them
- The processed result will be submitted to the chain from offchain_worker in the form of a transaction
- Enter the Runtime business logic for processing
The logic flow in pallets/oracle/ is also similar: in offchain_worker, use http request to obtain data from the price feeder address, and then submit it to the chain for use.
After understanding the above process, it is relatively easy to understand how Compound's Compound Governance Proposal on Ethereum affects the update of the validator node set on the Gateway chain. It still throws information out through the events of Ethereum, and the event server caches the events after collecting them. The Offchain worker on the Gateway side fetches the latest events in batches from the event server every time a new block is added. After obtaining the latest event, iterative processing according to the corresponding logic will do. In order to update the validator set in Substrate, you need to use Controler account, and related functions provided by Substrate such as SetKeys.
Due to space limitations, our analysis here ends here. Here we only briefly analyze the logic flow of the Gateway Pallet's thick lines, and then focus on the Offchain Worker part.
It can be seen that Gateway makes full use of the infrastructure provided by Substrate to easily realize the business logic, the integration of on-chain code and off-chain code, and the interaction with other chains. The whole code is implemented very clearly, which is worth learning from.
Summarize
In this article, we introduce the overall architecture of Gateway, the functional modules of Substrate, the characteristics of the Substrate framework, and how Gateway makes full use of the infrastructure provided by Substrate for the development of independent chains.
Due to space limitations, many places can only be clicked. Some links are attached at the end of the article for your extended reading.
The author's level is limited, readers are welcome to contact me by using the following email: [email protected].
References
1.Introducing Gateway: https://medium.com/compound-finance/gateway-623f6f48d2b6
2.Gateway Document:https://docs.google.com/document/d/1bp4uLYHt0_lUNO3dveFGelXYAC-M7RojqY-5jPDyUlM/edit#heading=h.ymt4jrq294bj
3.Compound’s Gateway: a deep dive into setting up a validator node.https://medium.com/ethereum-on-steroids/compounds-gateway-a-deep-dive-into-setting-up-a-validator-node-399a2817702d
4.https://github.com/compound-finance/gateway
5.Substrate.dev: https://substrate.dev/docs/en/
6. "2020 Developer Survey Report: Rust Becomes the Most Popular Language Again" https://mp.weixin.qq.com/s/xGZqrXfzfthVSIUWIQWENA