Outline
This article is a summary of learning technology architecture of large-scale distributed sites. A framework for high-performance, highly available, scalable and extensible distributed a summary description of the site and gives a reference architecture. This paper is part of the study notes, is part of a personal experience summary, there is a good reference for large-scale distributed site architecture.
About the Author
Rotten pig skin, more than ten years of work experience, worked at Google and other foreign companies over the past few years, proficient in Java, distributed architecture, micro-architecture and database services, recently working on big data and block chain, hoping to break through to a higher realm
First, large-scale distributed site architecture technology
1, the characteristics of large sites
Multi-user, widely distributed
high-volume, high concurrency
huge amounts of data, service availability
harsh security environment, the network vulnerable to attack
multi-function, quick change, release often
from small to large, progressive development of
user-centered
free service, paid experience
2, large sites architectural goals
High performance: provides fast access experience.
Availability: Web services can have normal access.
Scalable: hardware increase / decrease, increase / reduction processing capability.
Security: Web site provides secure access and data encryption, secure storage and other strategies.
Scalability: conveniently prepared by adding / removing embodiment, increase / decrease of new functions / modules.
Agility: demand, rapid response;
3, large sites architectural patterns
Layered: generally divided into an application layer, the service layer, the data layer, layer management and analysis;
segmentation: Usually divided according to the service / module / functional characteristics, such as the application layer is divided into home, the user centers.
Distributed: The application deployed separately (such as multiple physical machines), work together through long-distance calls.
Cluster: An application / module / function deploy multiple copies (such as: multiple physical machines), to provide external access through a common load balancing.
Cache: The data on the position closest to the application or user, quicker access.
Induction: The induction of synchronous operation. After the client request without waiting for the server response, and other server-side processing is completed, using a polling manner or notification informing the requester. Generally refers to: request - in response - notification mode.
Redundancy: Increase copy, improve availability, security and performance.
Security: There are effective solutions to known problems, establish discovery and defense mechanisms of the unknown / potential problems.
Automation: duplicate, things do not need human intervention, by means of the tool, using the machine to complete.
Agility: positive acceptance requirements change, rapid response to business needs.
4, high-performance architecture
User-centric, providing fast web access experience. The main parameter have a shorter response time, a large concurrent processing, higher throughput and stability of performance parameters.
Optimization can be divided into a front end, application-level optimization, code optimization layer and the memory layer optimization.
Optimization of the front end: the business logic part before the site;
browser optimization: reducing the number of HTTP requests using the browser cache, compression is enabled, CSS JS position, JS asynchronous, Cookie reducing transmission; the CDN acceleration, the reverse proxy;
Application Layer Optimization: Processing server website business. Using the cache, asynchronous, cluster
code optimization: reasonable structure, multithreading, resource multiplexing (object pooling, thread pool), good data structure, the JVM tuning, a single embodiment, Cache like;
storage optimization: cache, SSD , optical fiber transmission, read and write optimization, redundant disk, distributed storage (HDFS), NoSQL like.
5, high availability architecture
Large sites should be able to properly access at any time, the normal provision of external services. Because of the complexity of large sites, distributed, low-cost servers, open source database, operating system, etc., to ensure high availability is very difficult, to say the fault site is inevitable.
How to improve the availability, the problem is the need to be urgently addressed. First, we need to consider the architecture level, when planning, consider usability. General availability of the industry represented by several indicators 9, such as four 9 (99.99), is not available within a year to allow time was 53 minutes.
Use different strategies at different levels, the general redundancy and high availability failover to solve the problem.
Application Layer: general design for the stateless, for each request, using which server process is not affected. The general use of load balancing technology (Session synchronization issues need to be resolved) to achieve high availability.
Service Layer: load balancing, hierarchical management, rapid failure (timeout setting), asynchronous calls, service degradation, power and other design.
Data layer: redundancy (cold, hot-standby [synchronous, asynchronous], warm backup), failover (acknowledgment, transfer, resume). Data availability is based on the theoretical aspects of the famous CAP theory (persistence, availability, data consistency [strong consistent user agreement, final agreement])
6, scalable architecture
Scalability refers to the original without changing the architecture, by adding / reducing hardware (server) manner, increase / decrease system capacity.
Application Layer: application segmentation vertically or horizontally. Then load balancing (DNS, HTTP [reverse proxy], IP, link layer) for a single function.
Service layer: Similar to the application layer;
data layer: sub-libraries, sub-table, the NoSQL like; the Hash algorithm used, consistency Hash.
7, scalable architecture
New functional blocks may be conveniently / removal, providing code / module level good scalability.
Modular, component: high cohesion, low coupling, improved reusability, scalability.
Interface stability: stable interface defined, in the case of the same interface, the internal structure can be "freely" changes.
Design Patterns: application of object-oriented ideas, principles, use of design patterns, code-level design.
Message Queuing: modular system interact through message queues, so that the dependence between the decoupling module.
Distributed service: Public service module technology, providing other systems, improve reusability, scalability.
8, security architecture
There are effective solutions to known problems, establish discovery and defense mechanisms of the unknown / potential problems. For security issues, to improve safety awareness, to establish a safe and effective mechanism of safeguards from the policy level, the organizational level, such as the server can not divulge the password, the password update every month, and can not be repeated three times within; security scan every week and so on. Institutionalized way, strengthen the security system. At the same time, we need to pay attention to all aspects of security-related. Security issues can not be ignored, including the infrastructure, security, application security, data privacy and security.
Infrastructure Security: hardware procurement, operating systems, network security environment. Generally use the formal channels to buy high-quality products, choice of operating system security, timely repair vulnerabilities, install antivirus software firewall. Protection against viruses, backdoors. Set up a firewall policy, the establishment of DDOS defense system, the use of attack detection system, the subnet isolation means.
Application Security: When program development, commonly known problem, use the right way to get rid of the code level. Prevent cross-site scripting attacks (XSS), injection attacks, cross-site request forgery (CSRF), error messages, HTML comments, file upload, path traversal. You can also use Web application firewall (such as: ModSecurity), vulnerability scanning and other security measures to strengthen the application-level security.
Data confidentiality Security: Storage Security (stored in the secure device, real-time, scheduled backups), save the security (important information encrypted and stored, to select the appropriate complex personnel saving and detection, etc.), transmission security (to prevent data theft and tampering);
commonly used encryption algorithm (encryption single hash [MD5, SHA], symmetric encryption [DES, 3DES, RC]) , asymmetric encryption [the RSA] and the like.
9, agility
Site architecture design, operation and maintenance management to adapt to the changes, providing high scalability, high scalability. Easy to deal with rapid business development, the sudden increase in traffic requires access.
In addition to the architectural elements described above, but also we need to introduce agile management, agile development ideas. The business, product, technology, operation and maintenance unite, demand, quick response.
10, for example large-scale architecture
Logical architecture using more than seven, the customer first layer and second layer distal optimization layer, the third layer is the application layer, the fourth layer service layer, data storage layer a fifth layer, the sixth layer having a large data storage layer, the seventh layer large data processing layer.
Client layer: supports PC browsers and mobile phone APP. The difference is the phone APP can be accessed directly through the IP, the reverse proxy server.
Front-end layer: Using DNS load balancing, CDN local acceleration and reverse proxy services;
Application Layer: Web Application Clusters; vertical splits on business, such as product applications, and other members of the center;
the service layer: provide public services, such as customer service, order service, payment services;
data layer: support for relational database cluster (supports read and write separation), NOSQL clusters, distributed file systems in the cluster; and distributed Cache;
large data storage layer: support service layer and application layer log data collection, structured and semi-structured data in a relational database and the database collection NOSQL;
large data processing layer: off-line data analysis or real-time data analysis Mapreduce Storm, and stores the processed data relational database. (Actual use, offline data and real-time data will be carried out in accordance with the operational requirements of the classification process, and stored in different databases, for use by the service layer or layers).
Second, the major electricity supplier website system architecture evolution
A mature large sites (such as Taobao, Lynx, Tencent, etc.) of the system architecture not start out with a complete high-performance, high availability, high-elasticity and other characteristics of the design, it is with the increase in volume of users, business extended functions evolved sound, in the process, development model, technical architecture, design has undergone great changes, even the technical staff has developed from a few people to a department or even a product line.
So mature system architecture, with the expansion of business and gradually improve, not overnight; characteristics of different business systems, have their own focus, such as Taobao, to solve the massive search for product information, orders, payments; Tencent e.g., hundreds to solve real-time transmission of a user message; Baidu it to process the search request massive.
They have their own business features, system architecture is different. Nevertheless we can from these different sites in the background, to find out which common technologies and tools widely used in large-scale site system architecture, the following procedure describes the evolution of large sites through the system, to understand these technologies and means.
Architecture technology programmer is not open around the topic, by the way here to recommend a framework for technical exchange group: 650 385 180, which will share some video recordings senior architect: There are Spring, MyBatis, Netty source code analysis, high concurrency, high the principle of performance, distributed micro-service architecture, JVM performance optimization of these knowledge necessary to become an architect. Also receive free learning resources, I believe that what has been working and encountered technical bottleneck of the Friends of code, there will be in this group you need to.
1, the beginning of site architecture
The original architecture, applications, databases, files are deployed on a single server, as shown:
2, applications, data, document separation
With the expansion of the business, a server can not meet the performance requirements, so that the applications, databases, files are each deployed on separate servers, and different hardware configuration of the server according to the use, for optimal performance results.
3, the use of cache to improve site performance
At the same time optimizing the performance of the hardware, but also software performance optimization, most of the sites in the system, will use technology to improve performance caching system, mainly due to the presence of hot spots using the cache data, most of the site visit follow the 28 principles (i.e. 80% of the access request falls on the final 20% of the data), so we can cache the hot data, reducing the access paths to these data and improve the user experience.
Cache achieve common way is local cache, distributed cache. Of course, CDN, such as a reverse proxy, the back talk anymore. Local cache, as the name suggests is an application server to cache data locally, can be stored in memory, you can also file exists, OSCache local cache that is commonly used components. The local cache is characterized by fast, but because of the limited space of the local cache so the amount of data is limited. Characterized distributed cache, may be
in a massive cache data, and expands very easily, are often used in the portal site in the category, the local cache is not reasonably fast speed, it is commonly distributed caching Memcached, Redis.
4, application server clusters to improve performance
Application server as a portal, it will take a lot of requests, we tend to share the number of requests by the application server cluster. In front of the application server load balancing server deployment scheduling a user request, according to distribution policy to distribute requests to the plurality of application server nodes.
Commonly used hardware load balancing technology has F5, more expensive, the software has LVS, Nginx, HAProxy. LVS load balancing is four, according to the destination address and port selection internal server, Nginx and HAProxy are seven load balancing, you can select the internal server according to the message content, the distribution path LVS and Nginx better than HAProxy, performance is higher, and Nginx HAProxy and the more configurable, as may be used for static and dynamic separation (the feature request message, selecting static resource server or application server).
5, the database sub-library separate read and write sub-table and
With the increase of users, the database is the largest bottleneck in improving performance is a common means for reading and writing the database and the sub-library separate sub-table, database name suggests is separate read and write into the read and write database library, standby function is achieved by data synchronization. Sub-library sub-table is divided into horizontal and vertical segmentation segmentation, segmentation level is large to a database table split, for example, the user table. Vertical segmentation is to cut according to different service points, such as user services, goods traffic related tables in different databases.
6, CDN and reverse proxy to improve site performance
If our servers are deployed room in Chengdu, Sichuan, for users who access is faster, and for users to access Beijing is slow, which is due to Sichuan and Beijing belong to different developed areas Telecom and China Unicom, Beijing user access need to go through a longer path to access the router via the Internet server to Chengdu, the return path is the same, so the data transmission time is relatively long. In this case, often used to solve CDN, CDN content caching data to the operator's room, starting with the most recent acquisition of data when users access operators, thereby greatly reducing the network access path. More professional CDN operators blue flood, network places.
The reverse proxy, is deployed in the engine room of the site, first access the reverse proxy server when a user request arrives, the reverse proxy server will cache data back to the user, if there is no cache data will continue to get access to the application server, so It does reduce the cost of acquiring the data. There is a reverse proxy Squid, Nginx.
7, Distributed File System
Users increase day by day, more and more traffic, more and more documents produced, single file server can not meet the demand, then we need to support distributed file system. Commonly used distributed file system GFS, HDFS, TFS.
8, and using the NoSQL search engine
For inquiries and analysis of massive data, we use NoSQL database plus search engines can achieve better performance. Not all data should be placed relational data. There are commonly used NoSQL MongoDB, HBase, Redis, search engines Lucene, Solr, Elasticsearch.
9, the application server for business Split
With the further expansion of the business, the application has become very bloated, then we need to split the business application, such as Baidu divided into news, web pages, images, and other services. Each business application responsible for independent business operations. Achieved between the business communication or message via the shared database.
10, build distributed service
Then we found various business applications will use some basic business services, such as customer service, order services, payment services, security services, which are essential elements to support various business applications. We will use these services extracted Division-service framework to build distributed services. Ali Dubbo is a good choice.
Third, a diagram illustrates the architecture of the electricity supplier
Fourth, the major electricity supplier site architecture Case
1, electric business case reasons
Distributed large sites, the moment there are a few main categories:
Large portals such as Netease, Sina and so on;
the SNS sites, such as school, happy network;
electricity supplier sites, such as Alibaba, Jingdong Mall, Gome online car home and so on.
Large general news information portal, you can use CDN, static, etc. optimized, more interactivity happy net may introduce more NoSQL, distributed cache, using high-performance communications framework. Electric's Web site have more than two types of characteristics, such as product details, can be used CDN, static, highly interactive require the use of NoSQL technologies. Therefore, we use electricity supplier website as a case analysis.
2, demand for electricity supplier website
customer demand:
The establishment of a whole category of e-commerce sites (B2C), users can purchase merchandise online, you can pay online, you can also cash on delivery;
can communicate with customer service online when users purchase;
users receive the goods, you can merchandise scoring, evaluation;
there are mature Invoicing system; the need butt and website;
hope to be able to support three to five years, the development of business;
projected 3 to 5 years the number of users reached 10 million;
organize regular double 11, double 12, 38 men's Day, etc. activities;
other functions or reference Jingdong Gome online and other sites.
The customer is the customer, we will not tell you what you would be, would only tell you what he wants, many times we have to guide, tap the customer's needs. Fortunately, it provides a clear reference site. Therefore, the next step is to carry out extensive analysis, combined with industry, as well as reference sites, to provide solutions.
Demand function matrix
Demand Management Traditionally, will be described using the embodiment of FIG demand or by a block diagram (wanted list). Doing so often overlooked a very important demand (non-functional requirements), it is recommended that you use the demand function matrix, a requirement description.
This website needs electricity supplier matrix as follows:
3, the primary site architecture
General site, the beginning of practice, are three servers, one application deployment, a deployment database, an NFS file system to deploy.
This is a more traditional approach a few years ago, before seeing a website more than 100,000 members, the vertical portal costume design, N number of pictures. We used a deployed application server, database and picture storage. There have been many performance problems.
As shown below:
However, the current mainstream site architecture earth-shaking changes have taken place. Usually by way of a cluster, a high-availability design. At least it looks like this:
Using a cluster of redundant application server, high availability; (load balancing device may be an application deployment)
using database standby mode, and data backup availability;
4, system capacity forecast
Estimated steps:
Registered users - UV daily amount - the amount of daily PV - concurrency per day;
peak estimates: the usual amount of 2 to 3 times;
(concurrent, number of transactions), the storage capacity is calculated in accordance with system capacity concurrency.
According to customer demand: 3 to 5 years the number of subscribers reached 10 million registered users, number of concurrent can do per second estimate:
Daily UV 2 million (28 principles);
daily Click Browse 30 times a day;
the PV amount: 200 * 30 = 60 million;
centralized views: there will be 60 million * 0.8 * 0.2 = 48,000,024 = 4.8 hours ( 28 principles);
minute concurrency value: 4.8 * 288 = 60 min, access 4800/288 = 167,000 per minute (approximately equal);
concurrent per second: 167000/60 = 2780 (approximately equal);
assumptions: peak period is three times the normal value, the number of concurrent up to 8340 times per second.
1 = 1.3 ms access time;
not a good school mathematics regret it? ! (I do not know if there are more than count errors, Oh ~ ~)
Estimated :( tomcat server to server, for example)
By a web server that supports 300 concurrent computation per second. 10 servers usually required (approximately equal); [Tomcat default configuration 150], a 30 peak servers;
Estimated capacity: 70/90 principle
The system CPU is generally maintained at about 70% level, 90% of the peak level, is not a waste of resources, and relatively stable. Memory, IO similar.
Above estimates are for reference only, because the server configuration, complexity of business logic and so influential. The CPU, hard disk, network, etc. will not be assessed.
5, site architecture analysis
Based on the above estimates, there are several issues:
Need to deploy large numbers of servers, the peak of the calculation, you may want to deploy 30 Web server. And thirty server, only spike, activities will be used when there is a lot of waste.
All applications deployed on the same server, the coupling between applications seriously. The need for vertical and horizontal segmentation segmentation.
There is a lot of redundant code application
server Session synchronization vast memory and network bandwidth
data requires frequent access to the database, the database access enormous pressure.
Architecture technology programmer is not open around the topic, by the way here to recommend a framework for technical exchange group: 650 385 180, which will share some video recordings senior architect: There are Spring, MyBatis, Netty source code analysis, high concurrency, high the principle of performance, distributed micro-service architecture, JVM performance optimization of these knowledge necessary to become an architect. Also receive free learning resources, I believe that what has been working and encountered technical bottleneck of the Friends of code, there will be in this group you need to.
Large sites are generally required to do the following schema optimization (optimization is architecture design, it is necessary to consider, from the general solution architecture / code level, tuning mainly to adjust simple parameters, such as the JVM tuning; if tuning involves a lot of code transformation, it is not a tune, belongs to reconstruction):
Split Business
Application Clusters deployment (distributed deployment, a cluster deployment and load balancing)
multi-level cache
single sign-on (Distributed Session)
database cluster (separate read and write, sub-library sub-table)
Services of
message queue
other technologies
6, site architecture optimization
6.1 Business split
according to business property vertical segmentation, product is divided into subsystems, subsystems shopping, payment subsystem, comment subsystem, customer service subsystem interface subsystem (such as docking Invoicing, SMS and other external systems).
Be defined in terms of level of service subsystem, the system can be divided into core and non-core systems. Core Systems: product subsystem, subsystem shopping, payment subsystem; non-core: Comment subsystem, customer service subsystem interface subsystem.
Business Split Role: promoted by specialized subsystem is responsible for teams and departments, professional people to do professional things, to solve the problem of coupling and scalability between modules; each subsystem alone deployment, avoid centralized deployment results in an application hang, all application problems unavailable.
Level definition effect: when used for burst traffic, protect critical applications and achieve graceful degradation; protecting critical applications are not affected.
After the split architecture diagram:
Reference deployment scenario 2
FIG deployed individually as each application, the system core and non-core systems deployed in combination
6.2 Application Clusters deployment (distributed, clustering, load balancing)
Distributed deployment: application of business after the split alone deployment, remote communication applications directly through the RPC;
cluster deployment: high availability requirements of the electricity supplier site, each application to deploy at least two servers clustered deployment;
load balancing: high the system must be used, and the general application of high availability through load balancing, high availability service distributed through the built-in load balancing, high availability relational database by standby mode.
After a clustered deployment architecture diagram:
More than 6.3-level cache
cache storage position in accordance with generally divided into two types of local caching and distributed cache. This case uses two cache way, cache design. A local cache buffer, the secondary cache is distributed cache. (And page caching, fragment caching, it is more fine-grained partitioning)
a cache, cache data dictionary, data, and common hot substantially non-variable / change rules information, all cache secondary cache buffer required. When a cache expires or is unavailable, the secondary cache access data. If there is no secondary cache, then access the database.
The proportion of the cache, typically 1: 4, to consider using the cache. (Theoretically 1: 2 can).
According to business characteristics cache expiration policy can use the following:
Cache expire;
cache triggering expired;
6.4 single sign-on (Distributed Session)
system is divided into several subsystems, the independent deployment, will inevitably encounter problems session management. Session is generally employed, Cookies, Session distributed manner. General Electric's Web site uses a distributed Session achieve.
Further distributed according to Session, establish a sound or a single sign-on account management systems.
Flow Description
When users first log in, the session information (user Id and user information), such as user Id Key, written Distributed Session;
when the user logs on again, get distributed Session, whether there is session information, if not the tone to the login page;
the general Cache middleware is recommended to use Redis, so it has a persistent feature, the convenient downtime distributed session, session information can be loaded from persistent storage;
when stored in session, you can set the session held time, say 15 minutes out automatically after exceeding;
binding Cache middleware, realization of distributed session, session session can be well simulated.
6.5 database cluster (separate read and write, sub-library sub-table)
large sites need to store vast amounts of data, to achieve mass data storage, high availability, high-performance general system design of redundant manner. There are two general ways and separate read and write sub-library sub-table.
Separate read and write: Read the general solution ratio is far larger proportion of the writing scene, it can be a main one, a multi-master or multi-master multi-standby standby mode.
In this case the split service based on the combination of sub-library and separate read and write sub-table. As shown below:
After service splitter: each subsystem requires separate library;
if the individual library too, can according to the service characteristics, is divided again libraries, such as commodity classification database product library;
the sub-libraries, if there is a large amount of data in the table , the table is divided, generally can be divided in accordance with table Id, time and the like; (advanced usage is consistent Hash)
on the basis of the sub-library, the sub-table, separate read and write;
related middleware of Cobar refer to (Ali , has not maintained), TDDL (Ali), Atlas (Qihoo 360), myCat.
Problem after sub-library sub-table sequence, JOIN, the problem affairs, will share sub-library sub-table topic, the introduction.
6.6 Service of
the plurality of common sub-functions / modules, extraction, used as a public service. For example, in this case members of the subsystem can be extracted as a public service.
6.7 Message Queuing
message queue can be solved coupling between subsystems / modules, asynchronous, high availability, high-performance systems. Is standard distributed systems. In this case, the message queue is mainly used in shopping and distribution sectors.
After a single user, writes a message queue, the client returns directly;
Stock subsystem: read the message queue information, Save complete inventory;
Distribution Subsystem: read the message queue information for distribution;
There currently using more MQ Active MQ, Rabbit MQ, Zero MQ , MS MQ the like, needs to be selected according to specific business scenarios. Recommendations can be studied Rabbit MQ.
6.8 Other infrastructure (technology)
in addition to split the business described above, application clustering, multi-level cache, single sign-on, database cluster, service oriented, message queues outside. There CDN, reverse proxy, distributed file systems, large data processing systems.
Not described in detail here, you can ask of your mother / Google, there is a chance for everyone can share.
7, Architecture Summary
Large site architecture is based on the business needs of continuous improvement, according to different business characteristics and will do specific design considerations, this article only describes some of the techniques and tools will involve a conventional large site, I hope we can bring inspiration.