The original text is synchronized to https://waylau.com/talk-about-distributed-system/
When it comes to "distributed system", everyone's first impression is that it is very tall and unfathomable. Most of the speeches or books about distributed systems by various Daniels are dumbfounded. This article hopes to use easy-to-understand vernacular to discuss what is a distributed system, what are the advantages of a distributed system, what challenges will a distributed system face, and how to design a distributed system.
What is a distributed system
Regarding the definition of "distributed system", let's first see what foreigners say. The book "Principles and Paradigms of Distributed Systems" defines a distributed system as follows: "A distributed system is a collection of independent computers that appear to users as a single related system."
Regarding this definition, our intuitive feeling is:
- First of all, this kind of system is relatively powerful, at least composed of several mainframes. For service providers such as Google and Amazon, their data centers are supported by tens of thousands of hosts.
- Secondly, although it is very awesome, for outsiders, it is impossible to feel the existence of these hosts. That is, we only see one system at work. Take the recent "Amazon S3 downtime incident" as an example. Usually, we don't know how many hosts are behind the services provided by Amazon, but we don't know until the S3 downtime. This product already accounts for half of the Internet world. .
From the process point of view, two programs run on the processes of two hosts respectively, and they cooperate with each other to finally complete the same service (or function), then theoretically, the system composed of these two programs can also be called "" Distributed Systems".
Of course, the two programs can be different programs or the same program. If it is the same program, we can call it a "cluster". The so-called cluster is to expand the same program horizontally to improve the service capacity.
The definitions of "distributed system" and "cluster" are simple enough.
(The picture is selected from "Common Technologies and Case Analysis of Distributed Systems" )
What are the advantages of distributed systems
So, why do we use distributed systems?
Speaking of distributed systems, we have to talk about the ancestor of distributed systems - centralized systems. Centralized systems and distributed systems are two completely opposite concepts. A centralized system is a way of concentrating all programs and functions on one host, thereby providing services to the outside world.
Centralized systems are the easiest to understand. For example, in our host PC or mobile phone, we install all kinds of software on one machine, and when I need any function, I will get it from this machine. For another example, for the course design or development of small applications we did in our student days, we installed the Web server, database, etc. on a computer. The advantage is that it is easy to understand and easy to maintain. I put everything I want in one place, and it is easy to find things. Of course, the disadvantages are also obvious. If the machine crashes or the hard disk is broken, the entire system will collapse, and if the backup is also on this hard disk, it is equivalent to a disaster.
That’s why Buffett has a famous saying about investing, “Don’t put your eggs in one basket.” The same is true for the system. The manufacturer's machine cannot always guarantee that it will never be broken, and we cannot guarantee that hackers will not come to our system. The most important thing is that we cannot guarantee that our programs will not have bugs. So problems are unavoidable and mistakes are inevitable. We can only spread the eggs into different baskets to mitigate the risk of one pot. This is why a distributed system is needed.
Another reason to use distributed systems is scalability. After all, any host (even a minicomputer, supercomputer) will have a performance limit. The distributed system can achieve horizontal horizontal performance expansion by continuously expanding the number of hosts. Everyone also knows that Google's server hosts are mostly made up of outdated second-tier machines.
What challenges will distributed systems face?
There is no doubt that distributed systems are more complex to implement than centralized systems. Distributed systems will be harder to understand, design, build, and manage, which means that the root cause of an application is harder to find.
When designing distributed systems, it is often necessary to consider the following challenges:
- Heterogeneity: Because distributed systems are constructed based on different networks, operating systems, computer hardware and programming languages, a common network communication protocol must be considered to shield the differences between heterogeneous systems. These differences are typically handled by middleware.
- Lack of a global clock: When programs need to cooperate, they coordinate their actions by exchanging messages. Tight coordination often relies on consensus on when program actions take place, but in practice the accuracy of synchronizing clocks among computers on the network is greatly limited, ie there is no global notion of correct time. This is a direct consequence of the fact that messages are sent over the network as the only means of communication.
- Consistency: Data is scattered or replicated on different machines, and how to ensure the consistency of data between hosts will become a difficulty.
- Independence of failures: Any computer can fail, and failures vary. The timing of failure between them is also independent of each other. Generally, distributed systems are designed to allow partial failures without affecting the normal use of the entire system.
- Concurrency: The purpose of distributed systems is to better share resources. Then every resource in the system must be designed to be safe in a concurrent environment.
- Transparency: The failure of any component in the distributed system, or the upgrade or migration of the host is transparent and invisible to the user.
- Openness: In a distributed system, different programmers write different components. If the components are ultimately integrated into a system, the interfaces published by the components must comply with certain specifications and can be understood by each other.
- Security: Encryption is used to provide appropriate protection for shared resources, and all sensitive information transmitted on the network needs to be encrypted. Denial of service attacks are still an open problem.
- Scalability: The system should be designed so that as the business volume increases, the corresponding system must also be able to expand to provide corresponding services.
how to design distributed
The essence of designing a distributed system is "how to reasonably split a system into multiple subsystems and deploy them on different machines". So the primary consideration is how to split the system reasonably. Since the split subsystems cannot exist in isolation, they must be connected and interacted through the network, so how to communicate between them becomes particularly important. Of course, in the communication process, it is necessary to identify "enemy and friend" to prevent information from being intercepted and tampered with during the transmission process, which involves security issues. For distributed systems to adapt to growing business demands, scalability must be considered. Distributed systems must also ensure reliability and data consistency.
In general, when designing a distributed system, the following issues should be considered:
- How is the system split into subsystems?
- How to plan the communication between subsystems?
- How to consider the security in the communication process?
- How to make the subsystem expandable?
- How to ensure the reliability of the subsystem?
- How is data consistency achieved?
In fact, each of the above problems is not a simple problem. Fortunately, we would like to thank open source, so that the technology of this era can be shared, and the cost of implementing complex systems is getting lower and lower. For example, when we design communication, we can use message-oriented middleware, such as Apache ActiveMQ, RabbitMQ, Apache RocketMQ, Apache Kafka, etc. There are also RPC frameworks similar to Google Protocol Buffer and Thrift. When designing distributed computing, we can use MapReduce, Apache Hadoop, Apache Spark, etc. for distributed computing. For big data and distributed storage, we can choose Apache HBase, Apache Cassandra, Memcached, Redis, MongoDB, etc. In terms of distributed monitoring, commonly used technologies include Nagios, Zabbix, Consul, ZooKeeper, etc.
Of course, this article is only a guide, and it is impossible to cover everything. I hope readers have different opinions, welcome to discuss.
references
- 《分布式 Java》：https://github.com/waylau/distributed-java
- "Common Technologies and Cases Analysis of Distributed Systems": https://github.com/waylau/distributed-systems-technologies-and-cases-analysis or Code Cloud http://git.oschina.net/waylau/distributed-systems -technologies-and-cases-analysis