1, due
A 18:40 pm start, Teng News cloud received SMS alarm, display database CPU utilization of more than 100%, while the number of slow query log has about 1500.
Under normal circumstances: CPU utilization rate between 30% -40%, the number of slow query log is 0.
2, identify the cause
Because when receive text messages, just on the way home, can not handle, so the only home reprocessing.
While on the road, she received several text messages and resume again abnormal messages. There is a telling moment, a moment to recover.
After arriving home, the console log Tencent cloud database, query monitoring, we found that indeed 145% CPU usage, and the duration is 20 minutes.
18:40-19:00
19:20-19:40
Then I log Tencent cloud phpMyAdmin console, want to try to look at what is happening query. Then execute the command show processlist, unfortunately, when I execute a command, the peak has passed, I do not see any queries.
So, I downloaded a 18: Slow query log file between 00 and want to find the problem: 00-19. After downloading, find the first time began to slow query exactly 18:41 minutes. And view the entire file, are found with a SQL statement, the query is the same table, and uid is always the same.
Since I am not very familiar with the business, so I shot these SQL statements sent to developers. Developers after seeing the SQL statement, find the corresponding code location, and told me that the Cause of the SQL query is a request for an address, / account / ....
Then I downloaded the load balancing access log, only downloaded 18: 00-19: 00 time period. I According to URI, query logs provided by the developer, found true record.
Then I try to filter the log to determine the access address. My first filter the entire log, the number of statistical / account / ... appears. The number found to be 7900. In order to know whether this 7900 18: Number: "519" string: "184" 00 appeared, so I log in with a single filter: the number of strings and 00-19. Found that the number two together for 7600. Description This 7900 request is indeed from 18: 40-19: 00 this time started it.
And it is clear that all requests from the same IP address. For obvious reasons, that is an idiot in initiating an exception request.
3, approach
First, limit the access of IP load balancing on the firewall. Then developers also added CSRF.