Author: I painted you Allure
Source: https://juejin.im/post/5d84e21f6fb9a06ac8248149
“
Every holiday period, people in first- and second-tier cities who return home and go out to play almost face a problem: grab train tickets!
12306 Ticket grabbing, thinking about extreme concurrency
Although tickets can be booked in most cases now, the scene of no tickets at the moment of ticket issuance is believed to be well understood by everyone.
Especially during the Spring Festival, everyone not only uses 12306, but also considers "Zhixing" and other ticket grabbing software. Hundreds of millions of people across the country are grabbing tickets during this time.
"12306 Service" withstands a QPS that cannot be surpassed by any spike system in the world, and millions of concurrency can't be more normal!
The author specifically studied the server architecture of "12306" and learned many highlights of its system design. Here, I will share with you and simulate an example: how to grab 1 million train tickets at the same time, the system provides normal And stable service.
Github code address:
https://github.com/GuoZhaoran/spikeSystem
Large and highly concurrent system architecture
Highly concurrent system architectures will be deployed in distributed clusters. The upper layer of the service has layer-by-layer load balancing, and provides various disaster recovery methods (dual fire room, node fault tolerance, server disaster recovery, etc.) to ensure high availability of the system, and the traffic will also be based on Different load capacities and configuration strategies are balanced on different servers.
Below is a simple diagram:
Introduction to load balancing
The above figure describes the three-layer load balancing that the user requests to the server. The three load balancing are briefly introduced below.
① OSPF (Open Shortest Link First) is an Interior Gateway Protocol (Interior Gateway Protocol, IGP for short)
OSPF establishes a link-state database by advertising the state of network interfaces between routers to generate the shortest path tree. OSPF automatically calculates the Cost value on the routing interface, but you can also manually specify the Cost value of the interface. Automatically calculated value.
Cost calculated by OSPF is also inversely proportional to the interface bandwidth. The higher the bandwidth, the smaller the Cost value. The path to the same Cost value of the target can perform load balancing, and up to 6 links can perform load balancing at the same time.
②LVS （Linux Virtual Server）
It is a cluster technology that uses IP load balancing technology and content-based request distribution technology.
The scheduler has a good throughput rate, and transfers requests to different servers for execution in a balanced manner, and the scheduler automatically shields the server from failures, thereby forming a group of servers into a high-performance, highly available virtual server.
③Nginx
Everyone must be familiar with it. It is a very high-performance HTTP proxy / reverse proxy server. It is also often used in service development for load balancing.
There are three main ways for Nginx to achieve load balancing:
-
polling
-
Weighted polling
-
IP Hash polling
Next, we will do special configuration and test for Nginx's weighted polling.
Demo of Nginx weighted polling
Nginx achieves load balancing through the Upstream module. The configuration of weighted polling can add a weight value to the related services. During configuration, the corresponding load may be set according to the performance and load capacity of the server.
The following is a configuration of weighted polling load. I will listen to ports 3001-3004 locally and configure the weights of 1, 2, 3, and 4 respectively:
#配置负载均衡
upstream load_rule {
server 127.0.0.1:3001 weight=1;
server 127.0.0.1:3002 weight=2;
server 127.0.0.1:3003 weight=3;
server 127.0.0.1:3004 weight=4;
}
...
server {
listen 80;
server_name load_balance.com www.load_balance.com;
location / {
proxy_pass http://load_rule;
}
}
I configured the virtual domain name address of www.load_balance.com in the local / etc / hosts directory.
Next, use the Go language to open four HTTP port monitoring services. The following is the Go program that listens on port 3001. Others only need to modify the port:
package main
import (
"net/http"
"os"
"strings"
)
func main() {
http.HandleFunc("/buy/ticket", handleReq)
http.ListenAndServe(":3001", nil)
}
//处理请求函数,根据请求将响应结果信息写入日志
func handleReq(w http.ResponseWriter, r *http.Request) {
failedMsg := "handle in port:"
writeLog(failedMsg, "./stat.log")
}
//写入日志
func writeLog(msg string, logPath string) {
fd, _ := os.OpenFile(logPath, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0644)
defer fd.Close()
content := strings.Join([]string{msg, "\r\n"}, "3001")
buf := []byte(content)
fd.Write(buf)
}
I wrote the requested port log information to the ./stat.log file, and then used the AB pressure test tool to do the pressure test:
ab -n 1000 -c 100 http://www.load_balance.com/buy/ticket
According to the results in the statistics log, ports 3001-3004 received 100, 200, 300, and 400 requests, respectively.
This is in good agreement with the weight ratio I configured in Nginx, and the traffic after load is very uniform and random.
For specific implementation, you can refer to the Nginx Upsteam module implementation source code. Here I recommend an article "Load balancing of the Upstream mechanism in Nginx":
https://www.kancloud.cn/digest/understandingnginx/202607
Type selection of panic buying system
Back to the question we originally mentioned: How does the train ticket spike system provide normal and stable services under high concurrency?
From the above introduction, we know that user spike traffic is evenly distributed to different servers through layers of load balancing. Even so, the QPS suffered by the single machine in the cluster is also very high. How to optimize the stand-alone performance to the extreme?
To solve this problem, we must think about one thing: usually the ticket booking system has to deal with the three basic stages of generating orders, deducting inventory, and paying by users.
What our system has to do is to ensure that train ticket orders are not oversold and many sold. Each ticket sold must be paid to be valid, and the system must also withstand extremely high concurrency.
How should the sequence of these three stages be allocated to make it more reasonable? Let's analyze:
Order minus inventory
When a user's concurrent request arrives at the server, an order is first created, and then the inventory is deducted, waiting for the user to pay.
This order is the solution that most of us will think of in the first place. In this case, we can also ensure that the order will not be oversold, because the inventory will be reduced after the order is created. This is an atomic operation.
But this will also cause some problems:
-
In the case of extreme concurrency, the details of any memory operation are critical to performance. Especially logic such as order creation generally needs to be stored in a disk database. The pressure on the database is conceivable.
-
If the user has maliciously placed an order, only placing an order and not paying it will reduce the inventory and sell a lot of orders. Although the server can limit the number of IP and user purchase orders, this is not a good method.
Payment minus inventory
If you wait for users to pay for orders and reduce inventory, the first feeling is that they will not sell less. But this is a taboo for concurrent architecture, because in extreme concurrency, users may create many orders.
When the inventory is reduced to zero, many users find that the orders they grab cannot be paid, which is also called "oversold". Can not avoid concurrent operation of database disk IO.
Withholding inventory
From the consideration of the two options above, we can conclude that as long as the order is created, the database IO must be operated frequently.
So is there a solution that does not require direct manipulation of database IO, this is the withholding inventory. First deduct the inventory to ensure that it is not oversold, and then generate user orders asynchronously, so that the response to the user will be much faster; then how to guarantee a lot of sales? What if the user gets the order and doesn't pay?
We all know that orders now have an expiration date. For example, if the user does not pay within five minutes, the order will expire. Once the order expires, new inventory will be added. This is also the solution that many online retail companies use to ensure that many products are sold.
The generation of orders is asynchronous, and they are generally processed in the instant consumption queues such as MQ and Kafka. When the order quantity is relatively small, the generation of orders is very fast, and users almost do not have to queue.
The art of deduction
From the above analysis, it is clear that the plan to withhold inventory is the most reasonable. We further analyze the details of deduction inventory, there is still a lot of room for optimization, where does the inventory exist? How to ensure the correct deduction of inventory under high concurrency, and can quickly respond to user requests?
In the case of low concurrency on a single machine, we usually deduct inventory like this:
In order to ensure the atomicity of deduction of inventory and the generation of orders, you need to use transaction processing, then take inventory judgment, reduce inventory, and finally submit transactions, the entire process has a lot of IO, and the operation of the database is blocked.
This method is simply not suitable for highly concurrent spike systems. Next, we optimize the stand-alone deduction plan: local deduction inventory.
We allocate a certain amount of inventory to the local machine, reduce the inventory directly in memory, and then create orders asynchronously according to the previous logic.
The improved stand-alone system looks like this:
This avoids frequent IO operations on the database and only performs operations in memory, which greatly improves the ability of the single machine to resist concurrency.
However, a single machine with millions of user requests cannot be resisted anyway. Although Nginx uses the Epoll model to process network requests, the c10k problem has already been solved in the industry.
However, under the Linux system, all resources are files, and so are network requests. A large number of file descriptors can make the operating system instantly lose response.
We mentioned the weighted balancing strategy of Nginx above. We might as well assume that the 100W user requests are evenly balanced to 100 servers, so that the concurrency that the single machine bears is much smaller.
Then we locally stock 100 train tickets per machine, and the total inventory on 100 servers is still 10,000, which ensures that the inventory orders are not oversold. The following is the cluster architecture we describe:
The problems followed one after another. In the case of high concurrency, we can't guarantee the high availability of the system now. If two or three of the 100 servers are down because they cannot handle concurrent traffic or other reasons. Then the orders on these servers will not be sold, which results in fewer orders.
To solve this problem, we need to do unified management of the total order volume, which is the next fault-tolerant solution. The server must not only reduce inventory locally, but also reduce inventory uniformly remotely.
With the remote unified inventory reduction operation, we can allocate some extra "Buffer inventory" to each machine according to the load of the machine to prevent machine downtime in the machine.
Let's analyze it in detail with the following architecture diagram:
We use Redis to store unified inventory, because Redis's performance is very high, claiming that the stand-alone QPS can resist 10W concurrency.
After the local inventory reduction, if there is an order locally, we will request Redis remote inventory reduction. Both the local inventory reduction and the remote inventory reduction are successful. Only then will the user be reminded of the success of the ticket grab, which can effectively guarantee that the order will not Oversold.
When a machine is down in the machine, because there is a reserved Buffer ticket on each machine, the remaining ticket on the machine that is down can still be compensated for on other machines, ensuring a lot of sales.
How much is the appropriate Buffer balance? Theoretically, the more Buffer is set, the more machines the system can tolerate downtime. However, if the Buffer is set too large, it will also have a certain impact on Redis.
Although the Redis in-memory database has a very high anti-concurrency capability, the request will still go to the network IO once. In fact, the number of requests for Redis during the ticket grabbing process is the total amount of local inventory and Buffer inventory.
Because when the local inventory is insufficient, the system directly returns the user's "sold out" information prompt, and will no longer follow the logic of deduction of inventory.
To a certain extent, it also avoids the huge network request volume to overwhelm Redis, so how much the Buffer value is set requires the architect to carefully consider the load capacity of the system.
Code demo
The Go language is originally designed for concurrency. I use the Go language to show you the specific process of single ticket grabbing.
Initialization work
The Init function in the Go package is executed before the Main function. At this stage, some preparatory work is mainly done.
The preparation work that our system needs to do is: Initialize the local inventory, initialize the Hash key value of the remote Redis storage unified inventory, and initialize the Redis connection pool.
In addition, an Int Chan with a size of 1 needs to be initialized in order to realize the function of distributed lock.
You can also use read-write locks or Redis and other methods to avoid resource competition, but using Channel is more efficient. This is the philosophy of Go language: Do not communicate through shared memory, but share memory through communication.
The Redis library uses Redigo, and the following is the code implementation:
...
//localSpike包结构体定义
package localSpike
type LocalSpike struct {
LocalInStock int64
LocalSalesVolume int64
}
...
//remoteSpike对hash结构的定义和redis连接池
package remoteSpike
//远程订单存储健值
type RemoteSpikeKeys struct {
SpikeOrderHashKey string //redis中秒杀订单hash结构key
TotalInventoryKey string //hash结构中总订单库存key
QuantityOfOrderKey string //hash结构中已有订单数量key
}
//初始化redis连接池
func NewPool() *redis.Pool {
return &redis.Pool{
MaxIdle: 10000,
MaxActive: 12000, // max number of connections
Dial: func() (redis.Conn, error) {
c, err := redis.Dial("tcp", ":6379")
if err != nil {
panic(err.Error())
}
return c, err
},
}
}
...
func init() {
localSpike = localSpike2.LocalSpike{
LocalInStock: 150,
LocalSalesVolume: 0,
}
remoteSpike = remoteSpike2.RemoteSpikeKeys{
SpikeOrderHashKey: "ticket_hash_key",
TotalInventoryKey: "ticket_total_nums",
QuantityOfOrderKey: "ticket_sold_nums",
}
redisPool = remoteSpike2.NewPool()
done = make(chan int, 1)
done <- 1
}
Local deduction inventory and unified deduction inventory
The logic of local inventory deduction is very simple. The user requests to come over, add sales volume, and then compare whether the sales volume is greater than the local inventory, and return the Bool value:
package localSpike
//本地扣库存,返回bool值
func (spike *LocalSpike) LocalDeductionStock() bool{
spike.LocalSalesVolume = spike.LocalSalesVolume + 1
return spike.LocalSalesVolume < spike.LocalInStock
}
Note that the operation of the shared data LocalSalesVolume is to use locks to achieve, but because the local deduction inventory and unified deduction inventory is an atomic operation, so use the Channel to achieve the top layer, which will be discussed later.
Unified deduction of inventory operation Redis, because Redis is single-threaded, and we want to achieve data from it, write data and calculate a series of steps, we must cooperate with Lua script packaging commands to ensure the atomicity of the operation:
package remoteSpike
......
const LuaScript = `
local ticket_key = KEYS[1]
local ticket_total_key = ARGV[1]
local ticket_sold_key = ARGV[2]
local ticket_total_nums = tonumber(redis.call('HGET', ticket_key, ticket_total_key))
local ticket_sold_nums = tonumber(redis.call('HGET', ticket_key, ticket_sold_key))
-- 查看是否还有余票,增加订单数量,返回结果值
if(ticket_total_nums >= ticket_sold_nums) then
return redis.call('HINCRBY', ticket_key, ticket_sold_key, 1)
end
return 0
`
//远端统一扣库存
func (RemoteSpikeKeys *RemoteSpikeKeys) RemoteDeductionStock(conn redis.Conn) bool {
lua := redis.NewScript(1, LuaScript)
result, err := redis.Int(lua.Do(conn, RemoteSpikeKeys.SpikeOrderHashKey, RemoteSpikeKeys.TotalInventoryKey, RemoteSpikeKeys.QuantityOfOrderKey))
if err != nil {
return false
}
return result != 0
}
We use the Hash structure to store the total inventory and total sales information. When the user requests to come over, we determine whether the total sales is greater than the inventory, and then return the relevant Bool value.
Before starting the service, we need to initialize Redis' initial inventory information:
hmset ticket_hash_key "ticket_total_nums" 10000 "ticket_sold_nums" 0
Respond to user information
We start an HTTP service and listen on a port:
package main
...
func main() {
http.HandleFunc("/buy/ticket", handleReq)
http.ListenAndServe(":3005", nil)
}
Above we have done all the initialization work, then the logic of handleReq is very clear, to determine whether the ticket grab is successful, and return the user information.
package main
//处理请求函数,根据请求将响应结果信息写入日志
func handleReq(w http.ResponseWriter, r *http.Request) {
redisConn := redisPool.Get()
LogMsg := ""
<-done
//全局读写锁
if localSpike.LocalDeductionStock() && remoteSpike.RemoteDeductionStock(redisConn) {
util.RespJson(w, 1, "抢票成功", nil)
LogMsg = LogMsg + "result:1,localSales:" + strconv.FormatInt(localSpike.LocalSalesVolume, 10)
} else {
util.RespJson(w, -1, "已售罄", nil)
LogMsg = LogMsg + "result:0,localSales:" + strconv.FormatInt(localSpike.LocalSalesVolume, 10)
}
done <- 1
//将抢票状态写入到log中
writeLog(LogMsg, "./stat.log")
}
func writeLog(msg string, logPath string) {
fd, _ := os.OpenFile(logPath, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0644)
defer fd.Close()
content := strings.Join([]string{msg, "\r\n"}, "")
buf := []byte(content)
fd.Write(buf)
}
As mentioned earlier, we need to consider the race condition when deducting inventory. Here we use Channel to avoid concurrent read and write, ensuring efficient sequential execution of requests. We have written the return information of the interface to the ./stat.log file to facilitate the pressure measurement statistics.
Stand-alone service pressure test
To start the service, we use the AB pressure test tool to test:
ab -n 10000 -c 100 http://127.0.0.1:3005/buy/ticket
The following is the pressure test information of my local low-end Mac:
This is ApacheBench, Version 2.3 <$revision: 1826891="">
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 127.0.0.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
Server Software:
Server Hostname: 127.0.0.1
Server Port: 3005
Document Path: /buy/ticket
Document Length: 29 bytes
Concurrency Level: 100
Time taken for tests: 2.339 seconds
Complete requests: 10000
Failed requests: 0
Total transferred: 1370000 bytes
HTML transferred: 290000 bytes
Requests per second: 4275.96 [#/sec] (mean)
Time per request: 23.387 [ms] (mean)
Time per request: 0.234 [ms] (mean, across all concurrent requests)
Transfer rate: 572.08 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 8 14.7 6 223
Processing: 2 15 17.6 11 232
Waiting: 1 11 13.5 8 225
Total: 7 23 22.8 18 239
Percentage of the requests served within a certain time (ms)
50% 18
66% 24
75% 26
80% 28
90% 33
95% 39
98% 45
99% 54
100% 239 (longest request)
According to the indicators, I can process 4000+ requests per second on a single machine. Normal servers are multi-core configurations, and there is no problem in processing 1W + requests.
And looking at the logs found that throughout the service process, the requests are normal, the traffic is even, and Redis is also normal:
//stat.log
...
result:1,localSales:145
result:1,localSales:146
result:1,localSales:147
result:1,localSales:148
result:1,localSales:149
result:1,localSales:150
result:0,localSales:151
result:0,localSales:152
result:0,localSales:153
result:0,localSales:154
result:0,localSales:156
...
Summary review
Overall, the spike system is very complicated. Here we just briefly introduce some strategies that simulate how single machines are optimized to high performance, how clusters avoid single points of failure, and ensure that orders are not oversold and many are sold
The complete order system also has an order progress check. Each server has a task that regularly synchronizes the remaining tickets and inventory information from the total inventory to the user, and the user does not pay within the order validity period, releases the order, and adds Stock and so on.
We have realized the core logic of high concurrency ticket grabbing. It can be said that the system design is very ingenious, which avoids the operation of DB database IO.
For high concurrent requests to Redis network IO, almost all calculations are done in memory, and it effectively guarantees that it is not oversold, many sold, and can tolerate the downtime of some machines.
I think two of them are particularly worth learning and summarizing:
① Load balancing, divide and conquer
Through load balancing, different traffic is divided into different machines, and each machine handles its own requests and maximizes its performance.
In this way, the system as a whole can withstand extremely high concurrency, just like a team working, everyone will bring their own value to the extreme, and the team growth is naturally great.
② reasonable use of concurrency and asynchronous
Since the Epoll network architecture model solves the c10k problem, asynchrony is more and more accepted by server-side developers. Work that can be done with asynchrony is done with asynchrony, which can achieve unexpected results in function disassembly.
This is reflected in Nginx, Node.JS, and Redis. The Epoll model they use to process network requests tells us in practice that single-threading can still exert powerful power.
The server has entered the multi-core era. The Go language, a language born for concurrency, perfectly uses the advantages of multi-core servers. Many tasks that can be processed concurrently can be solved using concurrency. For example, when Go processes HTTP requests, each request will be Executed in a Goroutine.
In short, how to reasonably squeeze the CPU so that it can play its due value is always the direction we need to explore and learn.
