The realization of cloud computing allows me to process large-scale data and obtain the information we need. Through cloud computing, we can quickly and accurately get the information we want from thousands of data. However, one thing we cannot ignore is that the implementation of cloud computing relies on massive data input while relying on large-scale distributed computing clusters. It can be said that without "big data", "cloud computing" cannot be achieved. So, how do we provide the massive data required by cloud computing? Obviously, storing this data is a prerequisite for us to be able to provide the data. Here, Google provides us with a reliable and convenient file system to store the massive data we need. It is GFS (Google File System).
One: What is GFS?
According to the official statement: GFS is a distributed structured data storage system, which is used to process massive data (usually PB-level data distributed on thousands of ordinary servers).
Obviously, GFS, as a file system for storing data, ensures the stability and integrity of data, and realizes fast reading and writing of data is a must-have feature of this file system. Since it is aimed at petabyte-level data, it means that the stored files are generally relatively large, and GB-level files are very common. Under this condition, special considerations are required for the size of the basic unit of storage (the size of the BLOCK), I/O operations, and so on. Second, as a distributed file system for processing petabytes of data, GFS will be deployed on hundreds to thousands of ordinary, inexpensive storage devices, so component failure is considered a norm rather than an accident event.
Two: GFS design expectations
1. As mentioned above, GFS will be deployed on a cluster with cheap storage, then machine failure will be a norm, so GFS must have strong error detection, disaster redundancy and The performance of automatic recovery of stale data.
2. GFS must have the ability to store a certain amount of large asking prices. At the beginning of the design, GFS engineers expected that there would be several million files, and the size of the files would be more than 100MB, and even a few GB files would appear frequently. Therefore, GFS must not only have the ability to store these data files, but also must be able to quickly provide reading and writing of these data.
3. The consistency of the documents needs to be ensured. Obviously, petabytes of data will not be able to be manipulated by a single client. The actual operation of GFS will face thousands of different client groups from different network environments. So how to ensure data consistency when different clients read the same price at the same time is the main difficulty for GFS to process data. In addition, for the consideration of data security, files in GFS are usually copied into three parts to form three copies. So when three copies are accessed by three clients at the same time, how to ensure the consistency of copies is also a problem that GFS needs to consider and deal with.
4. In the case of limited network bandwidth, it must be possible to process data in large batches at a high rate. Compared with the abundant storage space, the network will be a very scarce resource, so GFS must ensure that the entire cluster can quickly exchange data and transmit information to ensure that the system can process data at high speed.
Three: Design details
In order to achieve the above design expectations, GFS has made considerable adjustments on the ordinary distributed file system. A lot of details have also been carefully designed.
1. The design of the master node. A single master node is used in GFS to manage the entire system. Such a decision greatly simplifies the design of GFS. A single master node can accurately locate the location of the Chunk (the actual unit used to store data) and make decisions about replication and modification operations through global information. In the master node, the metadata of each Chunk (including the location of data storage, the replication factor of the replica, the log information of the modification operation, etc.) will be stored, and the heartbeat information will be used to periodically communicate with the CHUNK server to obtain the Chunk. Status information for the server.
2. The way the client accesses the data. Considering the limitation of network bandwidth, if the client keeps communicating with the master node for a long time, the master node will become a hotspot when a large number of clients access at the same time, affecting the read and write rate of system data. So in practice, the client will only communicate briefly with the master node when reading the file. After obtaining the element data of the chunk where the file to be accessed is located, the client will disconnect the communication with the master node and communicate directly with the chunk server instead. Because of this design, in actual use, the communication pressure of the master is quite small, and the required network bandwidth is always within the acceptable range.
3, Chunk size design. Chunk is the smallest unit for storing data, so the design of chunk size becomes an important factor affecting the efficiency of the system. In GFS, 64MB was selected. This choice is mainly based on the premise that GFS is used to handle large files (generally GB-level and above). The larger Chunk size ensures the efficiency of data read, write and search. At the same time, since the file size is generally large, the space waste caused by the large chunk size will be negligible. For the processing of small files, it is generally converted into larger files through data merging, thereby ensuring the processing efficiency of the system for small files.
4. Create an operation log. If the replica is a way to ensure data stability after an exception occurs on the chunk server, the operation log is used to ensure that the master server can quickly restore system functions after an exception occurs. Every operation performed by the master node will be recorded by the operation log. Operation log information is replicated to multiple remote hosts to ensure that master information is never lost. In order for the master to recover as soon as possible, the operation log must be kept small enough. Therefore, when the operation log grows to a certain amount, the master server will perform a checkpiont on the system and write the operation log information into a Checkpoint file. During disaster recovery, the master server will read these files from disk and replay a limited number of operation log files to recover the system.
5. A lease is signed between the master and the chunk. This is actually to ensure consistency between different chunk replicas. When operating on chunks, if different clients access different replicas and perform write operations separately, it is difficult for different replicas to ensure consistency. For this kind of problem, GFS chooses the method of signing a lease between the master node and the chunk node. When operating on the replicas of the chunk, the master node selects one of the replicas and signs a lease with it. This copy is called the primary copy. All operations on chunk replicas are serialized by the primary replica, and all replicas follow this order for modification operations. This setting ensures the consistency between replicas and greatly reduces the management burden of the master node.
Four: Summary
GFS shows us the characteristics of a system that supports large-scale data processing using common hardware. Through GFS, we can utilize ordinary computer clusters to achieve stable storage and fast reading of huge data. However, this file system is not without flaws, and a high degree of customization is a factor that affects its suitability for all projects. For different products within Google, GFS will be customized according to its special needs, so that GFS can meet the different requirements of different systems. Therefore, in actual use, we generally do not use GFS directly, but use the same alternative (HDFS, Hadoop Distribute File System). This file system for Hadoop is functionally the same as GFS, but it is more general and more suitable for the construction of actual cloud computing platforms.