笔者注：2018年12月12日Hyperledger基金会在瑞士巴塞尔举办了第一届Hyperledger Global Forum大会并吸引了全球顶级区块链厂商和客户企业（包括IBM、Intel、微软、埃森哲、SAP、R3等等）。阿里巴巴提交的topic “Deploying Hyperledger Fabric on Cloud – Storage and Network Deep-dive”有幸获选并成为了国内唯一正式演讲的企业。本文基于此英文演讲内容整理而成。
大会官方演讲视频回放@Youtube: https://www.youtube.com/watch?v=NJb518txwUw
Hello everyone! This is Shan Yu from Alibaba Cloud. Today the topic I am going to share is called "Deploying Hyperledger Fabric on Cloud - Storage and Network Deep-dive".
First of all, I'd like to share what blockchain is from our perspective. In a narrow sense, just as everybody knows, blockchain is a kind of distributed shared ledger technology, based on smart contracts, all participants can reach consensus on transactions, and the transaction history in ledger cannot be tempered.
Actually this is more like a definition from a functional and purely technical angle.
But what is really attracting us, is in the broad sense, we think blockchain represents a new schema and collaboration model for all organizations, individuals, machines to construct a distributed trust network, connecting all trusted data, and let the value start flowing among them. It's just like "A Trusted Internet of Value". This will be a promising future for most industries and the whole society.
Therefore, Alibaba Cloud started the journey of blockchain, and we have chosen Hyperledger as our major direction. Here are the key reasons.
The first one is the openness of Hyperledger project and the technologies. As we know the purpose of blockchain is to solve trust problem. But if the blockchain technology itself is not open and runs like a black box, how can people trust the technology itself? So this first reason ensure our trust in the technology. Alibaba Cloud is dedicated to providing trusted infrastructure for everyone, so we choose the technology that people can trust.
The second one is the influence of Hyperledger. This can be proved by everyone of you coming from all around the world to join this wonderful conference. This influence helps build up a large global technical and business ecosystem.
The third one is that, the technology of hyperledger, take Fabric for example, is born with enterprise-oriented design. That means, from architecture to functions, it offers high standard of security, stability, availability, scalability, etc. These are the capabilities that enterprise users care most. Alibaba Cloud's mission is to help enterprise with digital transformation, so we also highly value this.
In a word, these are not only the reasons why we have chosen Hyperledger, but also the key factors for the future success of blockchain.
So what have we done in our blockchain journey with Hyperledger in the past two years?
Early in Oct 2017, we released a Blockchain Solution of Alibaba Cloud Container Service on Docker Swarm, helping users to deploy distributed Fabric network with just one click, reducing the time from days or hours into just several minutes.
Then in Dec, we upgraded the Blockchain Solution to support Kubernetes, and also incorporated Hyperledger Explorer.
In July this year, we launched Blockchain as a Service for China market, and then in Sep, together with Brian Behlendorf in Hangzhou, we make BaaS available for global market. Alibaba Cloud BaaS is globally the first blockchain product combining Intel SGX technology with Fabric together.
And just as announced this morning, we are very honored to officially become one member of Hyperledger family. And we will closely work with Hyperledger for the development of blockchain community and ecosystem. Let's look forward to more achievements together in 2019.
We are fortunate that Alibaba Group has a very extensive coverage of customers and industries in China. So Alibaba's blockchain has already been applied in many scenarios.
For example, our customer uses blockchain to encourage safe driving and reduce accidents, creating value out of people's driving data. Another example is that we are working with AI team to build a trusted digital content platform for copyright protection and trading. The third example is that we are working with IoT team to provide trusted device access and data for Industrial Internet.
In our vision, we are not only using blockchain to create incremental value, like improving efficiency and reducing cost, but we are also keen on creating transformational value for our customers, that is, the new scenarios, new business models and new economic which will not happen if without blockchain.
When we apply blockchain in enterprise scenarios, we still encountered quite some technical challenges in these areas.
Today we will mainly discuss the problems around storage and network.
First, let's dive into storage.
But why is storage important? Because the core of blockchain is ledger, and storage is where the ledger data resides.
In our engagements with customers, there are two typical problems or questions.
The first one, is how to do storage capacity sizing or planning? This was hard to answer because we had not seen such guidance or knowledge available before. But for production deployment of Hyperledger Fabric, especially in private cloud and local data center, we have to help our customer to do the planning at an early stage.
The second one, is how to address storage challenges, which we will talk about later.
Here is how we solved the problem.
In our BaaS, though Fabric is deployed on Docker containers and Kubernetes, we will store important data and configurations in external storage for persistence.
So we summarized a list of data paths and contents of different type of Fabric nodes, like Peer, Orderer, Kafka, ZooKeeper and CouchDB.
And now we know what data is inside the storage and where it is.
Then, we have done a trend analysis of the data growth in Fabric.
Here is an architecture diagram of a Fabric network. The red spots indicate the nodes will have high storage consumption growth, and the yellow spots indicate medium growth. This is because orderer, peer will keep a series of append-only blockchain files, which are the transaction history data, and it will keep growing as transactions grow. While Kafka broker also stores transaction data in topic, but we can set the retention period of topic data. And StateDB also stores latest state which is related with transaction, but it's not always growing. We will explain it later.
By doing source code analysis of Fabric and doing quite some testing, we have worked out this second version of formula, for estimating Hyperledger Fabric storage consumption. Please note this is for a single channel
And you can also use the excel format of the formula, and customize the inputs according to your actual deployment and scenario.
Next, I will explain how we came up with these magic numbers and variables.
As highlighted in yellow color,
3.34 KB is the rough size for transaction plus footprint in index DB and history DB of ledger. This number is based on statistics of our test results.
2.9 KB is the net size of a transaction, which is based on source code analysis and test results.
1.9 KB is the net size of a block, which is also based on source code analysis and test results.
Why do we have transaction playload multiply by two? Because each transaction contains "chaincode proposal payload" and " proposal response payload", both of which contain transaction payload.
Number of blocks, this is not a number which can be directly determined. Because a block will be cut when any of below conditions is satisfied:
- When a block has been kept waiting for long enough
- When there are enough number of transactions in a block
- When there are enough bytes of transaction data in a block
Therefore depending on the pattern of workload and scenario, we cannot give an one-for-all rule.
But we can give an example, like in a high-throughput scenario, which means the transactions keep coming in a very high speed, assuming MaxMessageCount is 10, then for 10,000 transaction, there will be 1,000 blocks generated.
Number of Kafka Topic Replicas: Please note this is not the num of Kafka brokers, but actually the topic replicas. In Fabric by default it's set to three.
CouchDB Compression Rate: Because transaction payload is stored as value in StateDB. And one of the StateDB implementation - CouchDB supports using Google's snappy algorithm for compression by default, as shown in the bottom right screenshot. But compression rate varies depending on payload contents. While for the other implementation of StateDB - LevelDB, because it does not support compression, so the value is one.
Scenario Type: For create scenario, in which we write some new key-value pair onto the chain, new key-value pair will be created in StateDB, so we choose will use 1 for such scenario.
For update scenario, in which we just modify an existing key-value pair on chain, there will be no new key-value created in StateDB. So we use 0 for such scenario.
In the very beginning, we only intended to get a rough level of Fabric storage consumption, like knowing whether it will be dozens of GB or hundreds of GB.
But after we worked out this version 2 of formula, when we compare the estimate result, marked in orange, with actual size, marked in blue, to our surprise, it's already much more accurate than we expected. So you can see the two lines are very close and even overlapped.
So this proves the formula is quite accurate in estimation.
So now we can understand the challenges in storage more clearly.
The first is that storage consumption keeps growing fast, as we can see from the formula, it's quite many times of the transaction number and payload size.
I remember there is a feature and a design in Fabric JIRA, proposing the pruning or cleanup of transaction history, specially those invalid transactions. But to be honest, even there are only valid transactions, as shown in our test results, the consumption growth is still quite horrible, especially when some of our customers want to apply blockchain to a national scale business and run it for years.
The second challenge is that, how to ensure stability and reliability of storage, and minimize impact to business applications. We need to take this seriously because if not handling properly, it will lead to service unavailable or even business data lost.
Thanks to using cloud, we have found some promising solution. In our deployment of Fabric on cloud, we choose NAS and cloud disk for data persistence. Both of them can provide very high reliability. And we particular choose SSD type to ensure I/O performance.
Kubernetes provides a very good abstraction of storage resources, so we are using persistent volume and persistent volume claim to request and mount these storage into Fabric pods.
And here we also found Alibaba Cloud NAS is a good fit for blockchain ledger:
- The first is that it support seamless and dynamic storage expansion, and you don't have to restart or interrupt Fabric nodes or your applications.
- The second one is that, some customers concern that when file system gets large, will it become slow? But to the contrary, the larger the NAS volume is, the higher throughput it will be.
Today some of our customers have been running blockchain system in production environment for months, achieving millions of business transactions, and hundreds of GB of ledger data. If not on cloud, it will be much more difficult for both customers and our team to maintain the storage.
Next, let's dive into the network of Hyperledger Fabric. Actually this is another big topic, and today I will mainly share some typical network connection patterns, based on our practice.
Maybe some of you have already know the various connections within a Fabric network, but to most users it's still not so clear or is still quite complex. So I managed to summarize it and draw it in this model.
Generally there are two types of connections.
The first type is transaction flow related, which happens in sequence. For example:
- Client SDK application connects to Fabric CA for client user registration or enrollment.
- Then client connects to peers of different organizations for transaction endorsement.
- After getting endorsement, client broadcasts transactions to orderers.
- Then Orderer will deliver blocks to all peers of the channel for validation and commit.
The above connections usually happen through a transaction flow.
And the second type of connection happens all the time.
- Like each org's leader peer will pull data from orderer, and initialize the dissemination to a random number of peer inside the org.
- Anchor peer will be connected for discovering peers in different organizations.
-
And among all peers, there will be gossip communications, mainly for three purposes：
- Detecting peers
- Disseminate ledger data
- Ledger transfer for newly joined peer
After this connection model analysis, we now have a conclusion, for Fabric network to function normally, we need to ensure a full-mesh network topology, so that client can connect to any peer and orderer, orderer can connect to any peer, and peer can connect to any other peer.
Next, I've summarized some typical patterns for these Fabric service connection.
The first one is very simple, if you deploy the whole Fabric network within a single Kubernetes cluster, Kubernetes will resolve service names to pod's cluster IP automatically.
Behind the scene, first it's kube-dns to resolve service name to service cluster IP, then kube-proxy to resolve service cluster IP to pod's cluster IP.
What you need to do, is just to define Fabric orderers, peers, CA etc using Kubernetes service names, in your configtx.yaml, crypto-config.yaml or any similar configurations.
This way is suitable for development and testing environment.
The second pattern is having Fabric services in different regions, like peer in region A, and orderer in region B. Here we still use Kubernetes as the deployment infrastructure.
In this pattern, we can define an external service and endpoint in region A, pointing to the external address and port of orderer in region B. In this way, peer in region A can still use Kubernetes service name to visit orderer service.
However, this way requires a bit more configuration efforts and is not so flexible. So we mainly use it for prototype or PoC.
The third pattern, is to use global DNS and external address for every Fabric service.
In this way, we will define global host name for all Fabric services, and register in a global DNS. These global host names will resolve to public IP assigned to these services.
This way is very flexible and globally usable, and has been adopted as the default option of Alibaba Cloud's BaaS. It can satisfy most of the basic customer requirements.
The fourth one is a flagship edition of the solution for cross-region Fabric network.
The scenario is that different customers have fabric nodes deployed in their own VPC on cloud and local data centers, and they want to establish a consortium together but without exposing to the public network.
For such advanced requirement, we have used Alibaba Cloud's CEN, which is cloud enterprise network, to connect all these VPC in a ring. In this way, they can connect to each other by using internal IP. Furthermore, to provide flexibility for applications, we can still use host name for services, and no need to expose to public DNS or network, just use DNS PrivateZone, which can ensure the host name resolution just within the range of these VPC networks.
For those services in local data center, we can use either VPN or leased line to connect them into any VPC then into the CEN ring.
In this way, we have come up with an advanced solution for secure and flexible consortium network among multiple enterprises, which is already supported by Alibaba Cloud BaaS.
In the end, I'd like to do a quick summary. By solving the problems in storage and network, we realize that the relationship between cloud and blockchain is a kind of perfect match.
By collaborating with Hyperledger, we hope the strengths of both sides can be reinforced, and we also welcome more partners and friends to join us, so that we can accelerate the development of blockchain ecosystem, to a greater success.
That's all for my sharing today. Thank you for listening!