cluster

Cluster (cluster), we know that a disk is composed of sectors one by one, and several sectors are combined into a cluster, and file access is based on clusters, even if the file is only 1 byte. Each cluster has a corresponding entry in the file allocation table, the cluster number is the entry number, and each entry occupies 1.5 bytes (disk space is less than 10MB) or 2 bytes (disk space is more than 10MB) ).

Files occupy disk space, and the basic unit is not bytes but clusters. Under normal circumstances, each cluster of a floppy disk is 1 sector, and the number of sectors per cluster of a hard disk is related to the total capacity of the hard disk, which may be 4, 8, 16, 32, 64... The data of the same file is not necessarily complete It is stored in a continuous area of ​​the disk, and is often divided into several segments, which are stored like a chain. This storage method is called chained storage of files. Since the connection information between segments (that is, FAT) is stored on the hard disk, the operating system can always accurately find the position of each segment and read it correctly when reading a file. In order to realize the chained storage of files, the hard disk must accurately record which clusters have been occupied by the file, and the cluster number of the next cluster that stores the subsequent content must also be indicated for each occupied cluster. For the last cluster of a file, it is necessary to indicate that this cluster has no successor clusters. These are all stored by the FAT table. There are many entries in the table, and each entry records the information of one cluster. Due to the importance of FAT for file management, for the sake of safety, FAT has a backup, that is, a new FAT is built behind the original FAT. All items in the initially formed FAT are marked as "unoccupied", but if the disk is partially damaged, the formatter will detect the damaged cluster and mark it as "bad cluster" in the corresponding item, which will not be used when saving files later. Will use this cluster again. The number of FAT items is equivalent to the total number of clusters on the hard disk, and the number of bytes occupied by each item should also be adapted to the total number of clusters, because the cluster number needs to be stored.

A computer cluster refers to a group of connected computers, which share the same Work is like a computer to the outside world. Clusters are generally connected by a local area network, but there are exceptions. Clusters are generally used for high-performance computing that cannot be completed by a single computer, and have high cost performance.

The background server of baidu and google is a cluster

cluster technology has been developed for many years, but in fact there is no very accurate definition and classification, and different people have different understandings.

In fact, it doesn't matter what it is called, as long as it can be beneficial to users. :-)

As far as personal understanding is concerned, there are the following types of clusters. Of course, as mentioned earlier, different people have different understandings, and you can fully discuss them. These categories of mine are more engineering than technical.

  
HA cluster

achieves high availability, but does not improve the performance of a single application. Most products on the market belong to this category and are technically simpler.

  


IP load balancing cluster

utilizes IP technology to support general IP applications. This technology is not very new. It was first used in hardware. After the emergence of Linux, there are many pure software models. This is also the benefit of open source.



Parallel computing cluster

includes some information such as PVM and beowulf. Delivery mechanism and API library, as well as task scheduling products, of course, the most difficult technically is the more intelligent products such as parallel compilation/parallel system



Application load balancing cluster

Although the highest purpose of cluster is to achieve real application-independent dynamic load balancing , but due to technical limitations, it can only be implemented in special applications now, and the application needs to be modified, so there is no general product, and most manufacturers have their own parallel versions. For example, oracle paralleler server. The

above is basically divided according to the perspective of engineering or product, and there should be a certain difference from the technical division.


A cluster is a parallel or distributed system composed of some interconnected computers. From the outside, they are just a system that provides unified services to the outside world.

There are many classifications of cluster technology itself, and there are many products on the market, and there is no very standard definition. Generally, it can be divided into the following categories:

  


Cluster based on redundancy

Strictly speaking, this kind of redundant system cannot be called a real cluster, because it can only improve the availability of the system, but cannot improve the overall performance of the system.

There are the following types.

A. A fault-tolerant machine is

characterized by the redundancy of all its hardware components (including hard disks, control cards, buses, power supplies, etc.) within a machine.

It can basically be independent of the software system and can achieve seamless switching, but the price is extremely expensive.

Typical market products: Compaq NonStop(Tandem), Micron(NetFrame), Straus

B. Dual-computer system based on system mirroring is

characterized by using dual-computer to mirror the system data and operating status (including data in memory), thereby realizing The purpose of hot backup.

It can achieve seamless switching, but because of the software control, it takes up a lot of system resources, and because the two machines need the exact same configuration, the performance-price ratio is too low.

Typical market products: Novell SFT III, Marathon Endurance 4000 for NT

C. Dual-computer system based on system switching The

characteristic is to use the dual-computer to mirror the system data (only refers to the hard disk data), and in the case of the failure of the host computer, the slave computer will Perform system-level handovers.

The price-performance ratio is moderate, but seamless switching is not possible.

Typical market products: Legato (Vinca) StandbyServer for NetWare, Savoir (WesternMicro) SavWareHA (Sentinel), Compaq StandbyServer

  


cluster based on application switching The

feature is that when a node in the cluster fails, other nodes can perform application-level Switching, so all nodes can provide their own services to the outside world under normal conditions, which is also known as a static load balancing method.

The performance-price ratio is high, but it cannot achieve seamless switching, and it cannot achieve load balancing for a single application itself.

Typical market products: Legato(Vinca) Co-StandbyServer for NT, Novell HA Server, Microsoft Cluster Server, DEC Cluster for NT, Legato Octopus, Legato FullTime, NeoHigh Rose HA, SUN Clusters, Veritas Cluster Server (FirstWatch), CA SurvivIT, 1776

  


cluster based on parallel computing is

mainly used in scientific computing, computing with large workload and other environments. There are various implementation methods such as parallel compilation, process communication, and task distribution.

Typical market products: TurboLinux enFuzion, Beowulf, Supercomputer Architectures, Platform

  


Cluster based on dynamic load balancing

All nodes provide the same external services, which can achieve load balancing for a single application and provide high availability at the same time.

The price-performance ratio is extremely high, but it cannot currently support the database.

Typical market products: TurboCluster Server, Linux Virtual Server, F5 BigIP, Microsoft Windows NT Load Balance Service

  


Load balancing

Load balancing is a cutting-edge technology to improve system performance. Still using the previous example, the processing capacity of an IA server is tens of thousands per second, obviously it cannot process hundreds of thousands of requests in one second, but if we can have 10 such servers to form a system, if there are By distributing all requests equally to all servers, the system has the ability to handle hundreds of thousands of requests per second. This is the basic idea of ​​load balancing.

In fact, there are load balancing products from multiple vendors on the market today. Due to the different main technologies used, they have different characteristics and different performances.

  


Round- robin DNS

The round-robin DNS scheme can be said to be the simplest and most intuitive technically. Of course, this solution can only achieve the function of load balancing, but cannot guarantee high availability.

Its principle is to set the mapping of multiple IP addresses to the same Internet host name in the DNS server. In this way, when DNS receives a request for querying a host name, it will cyclically return all corresponding IP addresses one by one. In this way, different client connections can be located on different IP hosts, and a relatively simple load balancing function can be implemented. However, this scheme has two fatal shortcomings:

l It can only achieve load balancing based on Internet hostname requests, and can do nothing if it is a request based directly on IP addresses.

l In the case of a node failure in the cluster, the DNS server will still return the IP address of the node to the querying party, and there will continue to be client requests trying to establish a connection with the failed node. In this case, even if you manually modify the corresponding settings of the DNS server and delete the faulty IP address, since all DNS servers on the Internet have a caching mechanism, there will still be thousands of clients that cannot connect to the cluster, unless Wait until all DNS caches have timed out.

  


Hardware Solutions

Some manufacturers provide hardware solutions for load balancing, creating high-end routers or switches with NAT (Network Address Translation) functions to achieve load balancing. The principle of NAT itself is to realize the translation of multiple private IP addresses to a single public IP address. The representative products are some high-end hardware switch series from Cicso and Alteon. This solution has the following disadvantages:

l Due to the use of special hardware, there are non-industry standard components in the entire system, which greatly affects the expansion, maintenance and upgrade of the system.

l The price is extremely expensive, and the software solution is basically an order of magnitude difference.

l Generally, only the status check at the node system level can be implemented, and the check at the service level cannot be refined.

l Due to the adoption of the NAT mechanism, the cluster management node itself has a large workload, which can easily become the bottleneck of the entire system.

l This particular piece of hardware is itself a single point of failure.

l It is very difficult to implement a cluster of remote nodes.

  


The principle of negotiation processing (parallel filtering)

is that the client request will be received by all nodes at the same time, and then all nodes negotiate and decide which node to process the request according to certain rules. The more significant feature of this scheme is that there is no significant management node in the entire cluster, and all decisions are made through consultation by all working nodes. The representative product is Microsoft Load Balancing Service of Microsoft. The features of this solution are:

l Due to the large amount of communication between nodes, which increases the burden on the network, it is generally necessary to increase the dedicated network for node communication, which increases the Difficulty and expense of installation and maintenance.

l Since each node has to receive and analyze all client requests, it greatly increases the burden on the network driver layer and reduces the work efficiency of the node itself. At the same time, the network driver layer can easily become a node system. bottleneck.

l Since the program of the network driver layer needs to be changed, it is not a general solution, and can only support special platforms.

l In the case of a small number of nodes, the negotiation efficiency is acceptable. Once the number of nodes increases, the communication and negotiation will become extremely complex and inefficient, and the performance of the entire system will decrease significantly in a nonlinear manner. Therefore, such schemes generally only allow up to a dozen nodes in theory.

l A cluster of remote nodes cannot be implemented.

l Since there is no unified manager in the cluster, chaotic abnormal phenomena may occur.

  


Traffic distribution

The principle of traffic distribution is that all user requests first reach the management node of the cluster, and the management node can decide to distribute the request to a certain service node according to the processing capability and status of all service nodes. When a service node fails due to hardware or software reasons, the management node can automatically detect and stop distributing traffic to this service node. In this way, the performance and processing capability of the entire system are increased by sharing the traffic, and the availability of the system can be well improved.

By making the management node itself a sub-cluster, the single point of failure due to the singularity of the management node itself can be eliminated. Some traditional technicians believe that because all customer traffic will pass through the management node, the management node can easily become the bottleneck of the whole system. But TurboCluster Server cleverly solves the problem through advanced direct routing or IP tunnel forwarding mechanism. All traffic responding to the client is directly returned to the client by the service node without passing through the management node again. As we all know, for service providers, the incoming traffic is much smaller than the outgoing traffic, so the management node itself will no longer be the bottleneck.

The specific implementation methods of traffic distribution include direct routing, IP tunneling and network address translation. TurboCluster Server currently supports the first two with the highest efficiency. Due to this advanced structure and technology, there is no upper limit on the number of service nodes in the TurboCluster Server cluster, and the efficiency of cooperative work for a large number of nodes can also be very well guaranteed.

  


Market Prospect

Cluster technology has been developed for many years, and its branches are also very many. At present, cluster technology is gradually moving towards a hierarchical structure, and there will definitely be cluster products dedicated to the front-end and back-end of users.

With the gradual improvement of the status of computer application and the increasing security and importance of the system, the cluster technology will certainly have a very broad application prospect.