Often when programmers are interviewing, the company's interview qualifications always have experience in the development of a large-scale system website. Let's take a look at a few screenshots of interview recruitment information...
large website definition
First of all, we need to think about a question, what kind of website is a large website, considering this problem from the perspective of the technical indicators of the website, people easily make the mistake of thinking that the traffic volume of the website is an indicator of measurement. People who know a little bit may think that It is the size of the concurrency of the website per unit time as an indicator. If these standards are used, then a website like hao123 is a large website, as shown in the following figure:
In fact, this kind of website has a very large number of visits and a very high number of concurrency, but it can be implemented with the simplest Web technology: as long as we keep the website fully static and deploy a few more servers, even if everyone on the earth is Both use it and the website works fine.
A large website is a combination of technology and business. As long as a website that meets the needs of certain users has great difficulty in both technology and business, it will inevitably make the enterprise invest more and better labor costs to realize it. Then such a website It's called a large website. 0
How to build a large website system step by step
In the Internet age, how to build a large website is an indispensable skill. Of course, the websites I'm currently in contact with are far more reading than writing. This article will explain step by step, how to use lamp to build and improve a large website (reading is greater than writing).
Website architecture, I personally think the most important considerations are two aspects: computing and storage. Some are computationally intensive and some are IO intensive. So the following will revolve around computing and storage.
1. The simplest construction
Suppose we started our own business, then we may need to build a website ourselves.
At this time, we need to rent a host (such as Alibaba Cloud's virtual host, etc.). For a website, data is the most important, so there needs to be a backup. But the daily pv is definitely not high, so theoretically only one computing machine is needed. Therefore, we only need 3 machines to complete a complete architecture.
As can be seen from the above figure, there are two main components deployed on the computer, one is webserver (for lightweight, niginx, lighttpd, etc. can be considered), the other is UI logic processing, and the lamp architecture uses php language to solve this problem. Because the data is the most important, the database obviously needs 2 machines, one host and one for redundant backup. lamp uses mysql to store data.
2 Increase the data cache
As the popularity of our website becomes higher, the PV is getting bigger and bigger every day, which leads to the problem that the pressure on the database is getting bigger and bigger. Obviously, for most websites, read traffic is much higher than write traffic. Even if we open the query cache of mysql, it can only reduce the pressure on the DB server to a certain extent by reducing the operations of the DB machine I. More reliable is to reduce the request to the DB server. Then you need to use cache at this time.
The cache is a non-relational kv storage, which is generally a memory operation during use. The following architectural improvements are as follows.
It can be seen that the ui write data is still directly written to the database, but the read is first read from the cache, and the cache cannot read it and then read from the database. Because it is very likely that most of the data can be accessed directly from the cache, which can greatly reduce the pressure on the database.
3. Increase the computing machine cluster (computing direction)
As the entire system pv continues to rise. A single computing machine can no longer meet the requirements. At this time, it is necessary to use additional computing machines to solve the problem. For convenience, this machine can be put into a cluster for unified management.
At this time, we may need to consider two issues: load balancing and data synchronization. The load balancing system is relatively difficult, but it is essential. The easiest way is to manage the configuration files in a unified manner through zookkeeper. For several machines under a node, requests can be distributed simply by probability. Data synchronization is also a difficulty, such as session synchronization, file operations, etc.
It should be noted that the result of a good architecture is as follows: the PV that N machines can support is X, then T*N machines can basically support T*X PV. In other words, the architecture must be able to support horizontal scaling. If the machine is doubled, but the peak pv supported cannot be (nearly) doubled, that architecture is a failed architecture, not a scalable architecture.
It can be seen that many machines can be hung under the load balancing system. A good expansion is that the number of times the machine is added, the corresponding increase in computing power (not considering the storage bottleneck for the time being).
4. Build a simple database cluster (storage direction)
As traffic increases and computing power increases, database capabilities also need to be improved. At this time, we can use read-write separation. There is also a master-slave theory. The main library can write, of course, it can certainly provide writing, and the slave library can only provide reading. Our current master-slave delay is within 20ms. At present, there are many such tools, such as mysql proxy and so on. (The following figure should be ui logic accessing dbproxy, there are some errors in the figure, but it does not affect the understanding).
As shown above, dbproxy has three main functions:
1. Separation of reading and writing. Read the main read from the library, write can only be written to the main library. We need to consider the master-slave delay in the actual design. For example, transaction reading must read the main library, and it is best to read the main library within a few seconds after writing.
2. Load balancing. He can automatically perform load balancing according to the db hanging under dbproxy.
3. dbproxy maintains the sql connection pool, which has long connections with db. After the request comes, you can directly take the connection from the connection pool.
5. Cross-regional caching of static pages
Obviously, our site has a lot of static pages that change every few days. However, due to the problem of cross-region and cross-machine room, users from other places may have slow access, so we can cache static pages through technologies such as cdn. In this way, the request to the server can be reduced, and the access time of users in different places and different computer rooms can be accelerated at the same time.
As shown above, static page caching is added
6. Cross-regional and cross-machine room design
When our business further expands, we may need to deploy machines across regions. At present, we are mainly divided into North China (Beijing) and East China computer rooms (Hangzhou, Nanjing). Cross-regional deployment can speed up the problem of slow access caused by regions. For example, Guangzhou's access to Beijing's computer room data is not as fast as Beijing's access to Beijing's computer room. At this time, it is mainly divided into two aspects: calculation and storage.
6.1 Calculation direction
Except for the label of the equipment room, the machine deployment in each equipment room should be exactly the same.
6.2 Storing Orientation
In my opinion, for a system with far more reads than writes, it is best to have only one master library and several slave libraries. Therefore, you only need to build a slave library in other computer rooms, so that the slave library can synchronize data from the main library. Of course, the price of this is that the master-slave time ratio is relatively long. In the case of unstable data link, the master-slave synchronization may be more than 400ms, so the design needs to consider this.
Of course, cache, etc. also need to be deployed across regions and computer rooms.
The figure briefly outlines a deployment plan across regions and computer rooms.
7. Use of general services
As the business expands, we may have some modules or businesses that need to consider new energy.
For example, for search business, we cannot directly implement the database's select like, so we need to use C and other compiled languages to build other systems. Therefore, we need to adjust the architecture according to the business to use some general high-performance computing services through http and other services.
Similarly, due to factors such as business development, we need to use in-memory databases, such as redis, which are general-purpose services in the storage direction.
Some of these services can be deployed across computer rooms, each computer room is not coupled, and some are coupled with each other, such as the main library and slave library similar to the database.
8. Other considerations
In addition to this, we also need to consider other factors. If you need to know, you can add the JAVA architect academic exchange group: 587372254
8.1 Website data
This is mainly for example uv/pv. There are several ways to do this. The first is to use third-party statistical attacks, such as Baidu statistics and Google statistics. The second is to count the logs of our existing system, and at the same time, further data mining can be carried out.
8.2 Security
This needs to consider whether the website has sql injection, xss vulnerability, csrf vulnerability, etc. This aspect is very critical for the website. Once hackers break in, the consequences are unimaginable.
For the administrator background, it is best not to open the external network permission, and only access it through the internal network.
8.3 seo etc.
Search engine optimization is self-evident. Follow-up may be specifically for Baidu SEO to conduct some analysis.