1, the original architecture: TB-level data on a machine
We will use a distributed storage system, for example, to talk about the design of fault-tolerant architecture. First, let's look, in the end what is a distributed storage system?
In fact, especially simple, we will use the database of a table, for example.
For example, you happen to have a database, the database has a particularly large table, there are billions, or even billions of data.
Furthermore, assuming that the amount of data that a table of up to tens of TB, even hundreds of TB, then you feel Zeyang?
Of course, it is the heart of panic and helplessness, because if you use the MySQL database and the like, a disk on a single database server may be enough to put this data in a table!
Let's take a look at this picture below, to feel.
2, in the end what is the distributed storage?
So, if you happen to have a large dataset, hundreds of TB! That you still do not consider traditional database technology to store up.
Because with a database server may not fit all, so we consider a distributed storage technology? correct! This is the way to solve this problem.
We can do multiple machines thing! Such as engaging in 20 machines, data 1/20 to put on each machine.
For example, such as a total of 20TB of data, as long as it 1TB, 1TB should be okay on each machine? Each machine can be breezy put down so much the data.
Therefore, to split a large data set into a plurality of pieces, to put up multiple machines, this is called distributed storage .
Let's look at the following chart.
3, then what is a distributed storage system?
It distributed storage system is Shane?
Distributed storage system, of course, is responsible for a large data set to split into multiple pieces and then put up a plurality of storage machines, followed by a unified management system data in these dispersions stored on multiple machines.
The classic example is the hadoop such systems, then fastdfs also similar.
If you can open the brain-hole, starting from the idea of the nature of the common level, you will find that, in fact, similar elasticsearch, redis cluster system and so on, he is essentially true.
These are based on a distributed system architecture, the large split into multiple pieces of data you stored on multiple machines.
Let this article is from the distributed system architecture level, not rigidly adhere to any kind of technology, it can be tentatively set: This distributed storage system, there are two processes.
一个进程是Master节点，就在一台机器上，负责统一管控分散在多台机器上的数据。
另外一批进程叫做Slave节点，每台机器上都有一个Slave节点，负责管理那台机器上的数据，跟Master节点进行通信。
咱们看看下面的图，通过图再来直观的看看上面的描述。
4、那某台机器宕机了咋办？
这个时候又有一个问题了，那么万一上面那20台机器上，其中1台机器宕机了咋整呢？
这就尴尬了，兄弟，这会导致本来完整的一份20TB的数据，最后有19TB还在了，有1TB的数据就搞丢了，因为那台机器宕机了啊。
所以说你当然不能允许这种情况的发生，这个时候就必须做一个数据副本的策略。
比如说，我们完全可以给每一台机器上的那1TB的数据做2个副本的冗余，放在别的机器上，然后呢，万一说某一台机器宕机，没事啊，因为其他机器上还有他的副本。
我们来看看这种多副本冗余的架构设计图。
上面那个图里的浅蓝色的“1TB数据01”，代表的是20TB数据集中的第一个1TB数据分片。
图中可以看到，他就有3个副本，分别在三台机器中都有浅蓝色的方块，代表了他的三个副本。
这样的话，一份数据就有了3个副本了。其他的数据也是类似。
这个时候我们假设有一台机器宕机了，比如下面这台机器宕机，必然会导致“1TB数据01”这个数据分片的其中一个数据副本丢失。如下图所示：
那这个时候要紧吗？不要紧，因为“1TB数据01”这个数据分片，他还有另外2个副本在存活的两台机器上呢！
所以如果有人要读取数据，完全可以从另外两台机器上随便挑一个副本来读取就可以了，数据不会丢的，不要紧张，大兄弟。
5、Master节点如何感知数据副本消失？
现在有一个问题，比如说有个兄弟要读取“1TB数据01”这个数据分片，那么他就会找Master节点，说：
“你能不能告诉我“1TB数据01”这个数据分片人在哪里啊？在哪台机器上啊？我需要读他啊！”
我们来看看下面的图。
那么这个时候，Master节点就需要从“1TB数据01”的3个副本里选择一个出来，告诉人家说：
“兄弟，在哪台哪台机器上，有1个副本，你可以去那台机器上读“1TB数据01”的一个副本就ok了。”
但是现在的问题是，Master节点此时还不知道“1TB数据01”的副本3已经丢失了，那万一Master节点还是通知人家去读取一个已经丢失的副本3，肯定是不可以的。
所以，我们怎么才能让Master节点知道副本3已经丢失了呢？
其实也很简单，每台机器上负责管理数据的Slave节点，都每隔几秒（比如说1秒）给Master节点发送一个心跳。
那么，一旦Master节点发现一段时间（比如说30秒内）没收到某个Slave节点发送过来的心跳，此时就会认为这个Slave节点所在机器宕机了，那台机器上的数据副本都丢失了，然后Master节点就不会告诉别人去读那个丢失的数据副本。
大家看看下面的图，一旦Slave节点宕机，Master节点收不到心跳，就会认为那台机器上的副本3就已经丢失了，此时绝对不会让别人去读那台宕机机器上的副本3。
那么此时，Master节点就可以通知人家去读“1TB数据01”的副本1或者副本2，哪个都行，因为那两个副本其实还是在的。
举个例子，比如可以通知客户端去读副本1，此时客户端就可以找那台机器上的Slave节点说要读取那个副本1。
整个过程如下图所示。
6、复制副本以保持足够副本数量
这个时候又有另外一个问题，那就是“1TB数据01”这个数据分片此时只有副本1和副本2这两个副本了，这就不足够3个副本啊。
因为我们预设的是每个数据分片都得有3个副本的。大家想想，此时如何给这个数据分片增加1个副本呢？
很简单，Master节点一旦感知到某台机器宕机，就能感知到某个数据分片的副本数量不足了。
此时，就会生成一个副本复制的任务，挑选另外一台机器来从有副本的机器去复制一个副本。
比如看下面的图，可以挑选第四台机器从第二台机器去复制一个副本。
但是，现在这个复制任务是有了，我们怎么让机器4知道呢？
In fact, very simple, 4 machine will not send a heartbeat every second it? 4 When the machine sends heartbeats past, Master node to send the machine to copy this task 4 responds by heart, let the machine 4 from a copy of the copy machine 2 well.
Again, we come a map, look at this process:
Figure fancy, now the machine is not the addition of a 4 "1TB data 01," a copy of the 3? So "1TB data 01" This is not another piece of data points become three copies of?
7, delete the extra copies
That, in turn, if this time machine 3 suddenly restored, he also has a top "1TB data 01," a copy of the 3, the equivalent of at this time "01 1TB of data," there are four copies of the copy not to excess yet?
It does not matter, once the Master node perceived machine 3 resurrection, you will find an excessive number of copies, then generates a deleted copy of the task.
He will send 3 when the machine heartbeat, issued a copy of the delete command, the machine 3 delete your local copy of the excess on it. In this way, we can keep the number of only three copies.
Like, we look at the following chart.
8, summary
Well, here, by super vernacular to explain, as well as progressive evolution explain dozen or figure, I believe we did not understand before even distributed systems, absolutely can understand a distributed system of complete data fault-tolerant architecture is how to design a.
In fact, this fragmentation of data storage, multiple copies of redundant, downtime perception, automatically copies the migration, delete the extra copy, this mechanism for hadoop, elasticsearch and many other systems, it is similar.
So here I strongly suggest that you must take to absorb about distributed systems, middleware ideological fault-tolerant system architecture of the underlying data.
In this way, time after learning some similar technology on their principles, thoughts will feel a sense of deja vu.