Author: JD Retail Dong Fangyou
introduction
Application health is an indicator for feedback of application health. It classifies system indicators into basic resources, containers, applications, alarm configurations, and links. It collects a series of system application indicators and scores them.
Each indicator of application health shows potential hidden dangers and security problems that may exist in a certain aspect of the system; therefore, improving application health is of great significance for system monitoring. Knowing what is happening requires knowing why, and understanding the hidden dangers behind the indicators in the application health is very helpful for us to understand and improve system security.
As a back-end R&D engineer, while promoting the improvement of the application health in the group, based on the problems encountered, combined with the analysis of the application health, I will summarize a series of hidden dangers of the application health one by one;
In the first article, let’s analyze the hidden dangers that may be caused by the database time zone setting items that are easily overlooked.
1. Application health check items
In the database connection pool configuration, obtained by parsing the source code, it supports four connection pools: DBCP1.X, DBCP2.X, Ali Durid, and HikariCP; configuration monitoring includes the following items
An example of risk is shown in the diagram below:
The three indicators of connectTimeout, SocketTimeout and time zone are analyzed from the url of the connection pool data source, such as: mysql://xxx.jd.com:3358/jdddddb_0?autoReconnect=true&useUnicode=true&characterEncoding=UTF-8&connectTimeout=1000&socketTimeout=3000&serverTimezone=Asia /Shanghai
Among them, the time zone setting is easy to be ignored; what kind of hidden dangers will be caused by ignoring the setting?
2. Problems encountered
1. Phenomenon
On March 12, 2023 (the second Sunday in March), the system UMP monitors and alarms, and the prompt is as follows
2. The cause of the problem
Mysql driver: after mysql-connector-java is upgraded to version 8. To parse the database time to java time, you need to get the time zone of the database. If the time zone is specified in the database connection, it will be used, otherwise the system time zone may be used
It can be queried through the select @@time_zone statement. If it returns SYSTEM, it means that the database has not set a time zone. Use the select @@system_time_zone statement to query and get the system default time zone, which is CST.
The CST time zone is Central Time in the United States, because the United States has daylight saving time and non-daylight saving time
CST non-daylight saving time corresponds to UTC-06:00, and summer time corresponds to UTC-05:00.
Daylight saving time in the United States starts at midnight on the second Sunday in March and ends at the early morning on the first Sunday in November.
Take 2023 as an example:
Before the start of daylight saving time adjustment: Sunday, March 12, 2023 at 02:00:00, the time is moved forward by one hour.
After adjustment: Sunday, March 12, 2023 03:00:00
Before the end of daylight saving time adjustment: 02:00:00 on Sunday, November 05, 2023, the time will be set back one hour.
After adjustment: Sunday, November 05, 2023 01:00:00
This means that: CST does not have a time interval of 2023-03-12 02:00:00~2023-03-12 03:00:00. There will be two intervals from 2023-11-05 01:00:00 to 2023-11-05 02:00:00.
Therefore, an "SQLException: HOUR_OF_DAY: 2 -> 3" exception will be thrown when getting the information.
3. Modify the plan
Set the data time zone in the database connection address: serverTimezone=Asia/Shanghai
3. Other hidden dangers related to time
1. According to the feedback from the research experiment, there may be performance problems when setting the time zone as the default, and it is often necessary to specify the time zone.
2. Pay attention to the time deviation when using the timestamp type:
The time range of the timestamp type is between '1970-01-01 00:00:01' and '2038-01-19 03:14:07'. If it exceeds this range, the value will be recorded as '0000-00-00 00:00:00' ', an important feature of this type is that the saved time is closely related to the time zone. The UTC (Universal Time Coordinated) standard refers to the standard time at 0 degrees longitude. The time zone in our daily life is the eighth in the Eastern Hemisphere where the capital Beijing is located. The zone is used as the benchmark, and the East 8 zone time (commonly known as Beijing time) is uniformly used, which is 8 hours earlier than UTC. The time zone setting also follows this standard, so the time range corresponding to the timestamp should be calibrated to '1970-01-01 08: 00:01' and '2038-01-19 11:14:07', that is to say, 1970-1-1 08:00:01 in East Eighth District is equivalent to UTC1970-1-1 00:00:01.
3. Try to use dateTime format instead of timestamp:
There are some situations where you need to pay attention not to use timestamp to store time:
• Birthday: the birthday must be earlier than 1970, which will exceed the range of timestamp
• Expiration date of validity: the maximum time of timestamp is 2038, if it is used to save the expiration date of similar ID card, the expiration date of business license, etc., it is not suitable.
• Business survival time: The Internet age is developing rapidly, and business time is likely to continue to operate in 2038.
4. Other hidden dangers of database connection settings
1. Number of connections setting
(1 Introduction
When the database connection pool is initialized, a certain number of database connections will be created and placed in the connection pool. The number of these database connections is restricted by the minimum number of database connections. Regardless of whether these database connections are used or not, the connection pool will always be guaranteed to have at least this many connections. The maximum number of database connections in the connection pool limits the maximum number of connections that the connection pool can occupy. When the number of connections requested by the application from the connection pool exceeds the maximum number of connections, these requests will be added to the waiting queue.
From this point of view, when the maximum number of connections to the database is not large enough, some reports or requests to query the database will fail, and an error will be reported because the number of connections is not enough and cannot be processed. When there are a large number of concurrent report requests and the maximum number of connections in the connection pool is not enough, some user requests cannot be processed, which affects the throughput of the entire project from another level and limits the performance and efficiency of the project. .
(2) Setting principles
It can not only guarantee the requirements for the number of database connections during normal use of the project, but also protect the security and stability of the DBS.
(3) Inquiry method:
Query the maximum number of connections command: show variables like'%max_connections%';
Query the number of established connections to the current database: show status like 'Threads_connected';
(4) Suggestion:
The MYSQL official website gives a suggested ratio for setting the maximum number of connections:
Max_used_connections / max_connections * 100% ≈ 85%
That is, the number of used connections accounts for about 85% of the total upper limit.
2. Timeout setting
(1 Introduction
A complete request includes three stages: 1. Establish connection 2. Data transmission 3. Disconnect
connect timeout: If the time required to establish a connection with the server (here refers to the database) exceeds ConnectionTimeOut, a ConnectionTimeOutException will be thrown, that is, the server connection timed out and the connection was not established within the specified time. In the database connection settings, connectTimeout indicates the timeout period for waiting to establish a socket connection with the MySQL database. The default value is 0, which means no timeout is set, and the unit is milliseconds.
socket timeout: If the connection with the server is successful, the data transmission will start. If the server takes too long to process data and exceeds SocketTimeOut, SocketTimeOutException will be thrown, that is, the server response timed out, and the server did not return data to the client within the specified time. In the database connection settings, socketTimeout indicates the timeout time for waiting when reading and writing sockets after the client and the MySQL database establish a socket. The default socketTimeout of the Linux system is 30 minutes.
(2) Hidden dangers
The access time to the database is too long, the amount of accessed data is large or the amount of scanned data is too large, causing the database to be unresponsive for a long time. The connection is occupied and cannot be released, which will cause the thread pool to be full. Therefore, in order to be able to release the occupied link in time, other businesses will not be affected by the database access, so the database access timeout period should be set reasonably.
The socket timeout of JDBC is very important when the database is suddenly stopped or a network error occurs (due to equipment failure, etc.). Due to the structure of TCP/IP, there is no way for the socket to detect network errors, so the application cannot actively discover that the database connection is disconnected. If the socket timeout is not set, the application will wait indefinitely before the database returns a result. This connection is called a dead connection.
In order to avoid dead connections, the socket must have a timeout configuration. The socket timeout can be set through JDBC. The socket timeout can avoid the endless waiting of the application when a network error occurs, and shorten the service failure time.
(3) Recommendations
In general, it is recommended to configure connectTimeout=60000, in milliseconds. It is recommended to configure socketTimeout=60000, in milliseconds. Specific configurations vary by system.
Summarize
The database connection application health check items that are easy to be ignored have hidden dangers that may be caused by improper setting of time zone, timeout period, and number of connections. Before it happens.