1. Problem Raising and Solving
At a certain pilot site, testers reported that the connection to the database failed and logs appeared too many connections. Since I deployed the database using docker, and the time was urgent, I needed to respond immediately. After asking, the version and configuration were no different from other pilots, except that the new pilot had some more connected microservices.
After investigation, it can be solved by adding connection parameters when the container starts, as follows:
command: ['mysqld', '--max-connections=1024', ‘其它参数’]
In order not to affect the launch, the modified configuration was provided to the testers at that time, and the test was normal after the modification. Since this area has not been studied yet, this article is written.
2. Server settings
2.1 Database service
docker-compose.yml file:
version: '2'
services:
ttmysql:
image: mysql:8.0
container_name: llmysql
restart: always
command: ['mysqld', '--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci', '--explicit_defaults_for_timestamp=false', '--lower_case_table_names=1', '--default_authentication_plugin=mysql_native_password'] ## , '--skip-ssl'
volumes:
- ./mysqldata:/var/lib/mysql
environment:
- TZ=Asia/Shanghai
- MYSQL_ROOT_PASSWORD=123456
- MYSQL_USER=latelee
- MYSQL_PASSWORD=123456
ports:
- "43306:3306"
networks:
- mysql-net
networks:
mysql-net:
driver: bridge
~
For simplicity, rootthe user password for the database is 123456.
start up:
docker-compose up -d
2.2 Configuration file analysis
Enter the container:
docker exec -it llmysql bash
View the default configuration file:
# cat /etc/mysql/my.cnf
[mysqld]
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
datadir = /var/lib/mysql
secure-file-priv= NULL
# Custom config should go here
!includedir /etc/mysql/conf.d/
As you can see, the customized configuration file should be placed in /etc/mysql/conf.d/the directory. Check it out:
# ls /etc/mysql/conf.d/
docker.cnf mysql.cnf
Among them, /etc/mysql/conf.d/mysql.cnf the document has no substantive content.
For container deployment, the custom configuration file can be mounted to the above directory. The example is as follows:
volumes:
- ./mysqldata:/var/lib/mysql
- ./my.cnf:/etc/mysql/conf.d/my.cnf
2.3 Parameter configuration description
According to the official documentation, system variables ( server-system-variables) can be passed either in the form of command line parameters or in the form of configuration files. From the author's experience, if most use the default configuration and only change individual parameters, the former is better, but the latter has strong customization capabilities and is suitable for high-end applications.
If you set the maximum number of connections, you can my.cnfmake the following modifications to the file:
[mysqld]
max_connections=1000
The command line method can also be used, the parameters are as follows:
command: ['mysqld', '--max-connections=1024', ‘其它参数’]
Note that when both exist at the same time, after testing, the command line method has the highest priority.
3. Client test
3.1 Connect to database
Use the following command to connect to the database:
mysql -h 127.0.0.1 -P 43306 -uroot -p123456 --ssl-mode=DISABLED
临时 创建test数据库：
create database test;
3.2 Query status
Maximum number of connections
Use the following command to query the maximum number of connections of the mysql server:
show variables like '%max_connections%';
Connection and query output log:
$ mysql -h 127.0.0.1 -P 43306 -uroot -p123456 --ssl-mode=DISABLED
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 216676
Server version: 8.0.21 MySQL Community Server - GPL
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show variables like '%max_connections%';
+------------------------+-------+
| Variable_name | Value |
+------------------------+-------+
| max_connections | 151 |
| mysqlx_max_connections | 100 |
+------------------------+-------+
2 rows in set (3.08 sec)
As you can see, the default maximum number of connections is 151. According to official documentation, max_connectionsthe value range is 1~100000.
The number of connections the server responded to
max_used_connections：
show global status like 'max_used_connections';
Output example:
mysql> show global status like 'max_used_connections';
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| Max_used_connections | 1 |
+----------------------+-------+
1 row in set (0.00 sec)
Current number of connections
SHOW STATUS LIKE 'Threads%';
Output example:
mysql> SHOW STATUS LIKE 'Threads%';
+-------------------+-------+
| Variable_name | Value |
+-------------------+-------+
| Threads_cached | 0 |
| Threads_connected | 1 |
| Threads_created | 1 |
| Threads_running | 2 |
+-------------------+-------+
4 rows in set (0.00 sec)
Current connection information
SHOW PROCESSLIST;
Output example:
SHOW PROCESSLIST;
+----+-----------------+---------------------+------+---------+------+------------------------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+-----------------+---------------------+------+---------+------+------------------------+------------------+
| 5 | event_scheduler | localhost | NULL | Daemon | 80 | Waiting on empty queue | NULL |
| 8 | root | 192.168.144.1:44052 | NULL | Query | 0 | init | SHOW PROCESSLIST |
+----+-----------------+---------------------+------+---------+------+------------------------+------------------+
2 rows in set (0.00 sec)
Note: Since there are no other connections, the actual number of connections is 1, and the executed command is SHOW PROCESSLIST.
max_used_connections / max_connections * 100%（理想值≈ 85%）
如果max_used_connections跟max_connections相同，那么就是max_connections设置过低或者超过服务器负载上限了，低于10%则设置过大。
3.3 Self-test program connection
This section uses golang to write a program for connection testing, and then queries the connection information of the server.
The database related closed code is as follows:
package main
import (
"context"
"database/sql"
"fmt"
"strings"
"time"
"errors"
"os"
// 导入mysql驱动
_ "github.com/denisenkom/go-mssqldb"
_ "github.com/go-sql-driver/mysql"
// _ "github.com/mattn/go-oci8"
_ "github.com/mattn/go-sqlite3"
)
const (
SQL_TIMEOUT_S = 10 // 连接超时时间 10 秒
)
// 根据传入参数解析
func CreateSQLDb(dbstr string) (SQLDB *sql.DB, DBType string) {
fmt.Println("connecting db...")
var err error
if len(dbstr) == 0 {
fmt.Println("sql string is empty")
os.Exit(0)
}
if strings.Contains(dbstr, "encrypt=disable") == true {
SQLDB, err = CreateSqlServer(dbstr)
DBType = "sqlserver"
} else if strings.Contains(dbstr, "@tcp") == true {
SQLDB, err = CreateMysql(dbstr)
DBType = "mysql"
} else if strings.Contains(strings.ToUpper(dbstr), "DB3") == true {
SQLDB, err = CreateSqlite3(dbstr)
DBType = "sqlite"
} else {
fmt.Printf("dbstr %v not support\n", dbstr)
os.Exit(0)
}
/*
else if strings.Contains(dbstr, "latelee_pdb") == true ||
strings.Contains(dbstr, "ORCLCDB") == true {
SQLDB, err = db.CreateOracle(dbstr)
DBType = "oracle"
}
*/
if err != nil {
fmt.Println("connect db error: ", err)
}
return
}
/
func IsExist(path string) bool {
_, err := os.Stat(path)
return err == nil || os.IsExist(err)
}
func CreateSqlServer(dbstr string) (sqldb *sql.DB, err error) {
connCh := make(chan int)
timeoutCtx, cancel := context.WithTimeout(context.Background(), time.Second*SQL_TIMEOUT_S)
defer cancel()
go func() {
sqldb, err = sql.Open("mssql", dbstr)
if err != nil {
err = errors.New("open database failed: " + err.Error())
connCh <- -1
return
}
// Open不一定会连接数据库，Ping可能会连接
err = sqldb.Ping()
if err != nil {
err = errors.New("connect database failed: " + err.Error())
connCh <- -1
return
}
connCh <- 0
}()
select {
case <-timeoutCtx.Done():
sqldb = nil
err = errors.New("connect to sqlserver timeout")
case ret := <-connCh:
if ret == 0 {
fmt.Println("connect to sqlserver ok")
err = nil
}
}
return
}
func CreateMysql(dbstr string) (sqldb *sql.DB, err error) {
connCh := make(chan int)
timeoutCtx, cancel := context.WithTimeout(context.Background(), time.Second*SQL_TIMEOUT_S)
defer cancel()
go func() {
sqldb, err = sql.Open("mysql", dbstr)
if err != nil {
err = errors.New("open database failed: " + err.Error())
connCh <- -1
return
}
err = sqldb.Ping()
if err != nil {
err = errors.New("connect database failed: " + err.Error())
connCh <- -1
return
}
connCh <- 0
}()
select {
case <-timeoutCtx.Done():
sqldb = nil
err = errors.New("connect to mysql timeout")
case ret := <-connCh:
if ret == 0 {
fmt.Println("connect to mysql ok")
err = nil
}
}
return
}
func CreateSqlite3(dbname string) (sqldb *sql.DB, err error) {
if !IsExist(dbname) {
return nil, errors.New("open database failed: " + dbname + " not found")
}
sqldb, err = sql.Open("sqlite3", dbname)
if err != nil {
return nil, errors.New("open database failed: " + err.Error())
}
err = sqldb.Ping()
if err != nil {
return nil, errors.New("connect database failed: " + err.Error())
}
fmt.Println("connect to ", dbname, "ok")
return
}
// note：暂时不使用oracle
func CreateOracle(dbstr string) (sqldb *sql.DB, err error) {
// connCh := make(chan int)
// timeoutCtx, cancel := context.WithTimeout(context.Background(), time.Second*SQL_TIMEOUT_S)
// defer cancel()
// go func() {
// sqldb, err = sql.Open("oci8", dbstr)
// if err != nil {
// err = errors.New("open database failed: " + err.Error())
// connCh <- -1
// return
// }
// err = sqldb.Ping()
// if err != nil {
// err = errors.New("connect database failed: " + err.Error())
// connCh <- -1
// return
// }
// connCh <- 0
// }()
// // log.Println("connect to ", dbParam.server, dbParam.database, "ok")
// select {
// case <-timeoutCtx.Done():
// sqldb = nil
// err = errors.New("connect to oracle timeout")
// case ret := <-connCh:
// if ret == 0 {
// fmt.Println("connect to oracle ok")
// err = nil
// }
// }
return
}
The main program is as follows:
package main
import (
"database/sql"
"fmt"
"os"
)
var SQLDB *sql.DB // 默认以此为准
var DBType string
var ConfDBServer string = "root:123456@tcp(10.0.153.12:43306)/test?charset=utf8&interpolateParams=true&parseTime=true&loc=Local"
func initDB(dbfile string) {
SQLDB, DBType = CreateSQLDb(ConfDBServer)
if SQLDB == nil {
os.Exit(0)
}
}
var user_table_sql string = `CREATE TABLE user (
id bigint(20) NOT NULL,
email varchar(128) DEFAULT NULL,
first_name varchar(128) DEFAULT NULL,
last_name varchar(128) DEFAULT NULL,
username varchar(128) DEFAULT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
`
var user_table_sql_1 string = `CREATE TABLE user (
id bigint(20) NOT NULL,
email varchar(128) DEFAULT NULL,
first_name varchar(128) DEFAULT NULL,
last_name varchar(128) DEFAULT NULL,
username varchar(128) DEFAULT NULL,
`
var user_table_sql_3 string = `
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
`
/
func createDBTable(sqldb *sql.DB) {
_, err := sqldb.Exec(user_table_sql)
if err != nil {
fmt.Printf("Exec sql failed: [%v] [%v] \n", err, user_table_sql)
}
}
func createDBTableLong(sqldb *sql.DB) {
count := 200
mysql := ""
for i := 0; i < count; i++ {
mysql += fmt.Sprintf("username%v int DEFAULT NULL,", i)
}
realsql := user_table_sql_1 + mysql + user_table_sql_3
// fmt.Println("dddd ", realsql)
// return
_, err := sqldb.Exec(realsql)
if err != nil {
fmt.Printf("Exec sql failed: [%v] [%v] \n", err, realsql)
}
for {
// 死循环查询连接数
}
}
func main() {
fmt.Println("db test...")
initDB(ConfDBServer)
createDBTableLong(SQLDB)
}
Query the current number of connections in the service library:
mysql> SHOW PROCESSLIST;
+----+-----------------+---------------------+------+---------+------+------------------------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+-----------------+---------------------+------+---------+------+------------------------+------------------+
| 5 | event_scheduler | localhost | NULL | Daemon | 1529 | Waiting on empty queue | NULL |
| 11 | root | 192.168.208.1:49212 | test | Query | 0 | init | SHOW PROCESSLIST |
| 21 | root | 10.0.11.23:60858 | test | Sleep | 23 | | NULL |
+----+-----------------+---------------------+------+---------+------+------------------------+------------------+
3 rows in set (0.00 sec)
summary
Due to time constraints, this article is only the beginning, and various tests will be carried out as needed based on the test program. It is planned to be used as a database testing tool, which can be used to adapt localized database options, such as: client library friendliness (such as the above code, can be directly used for MySQL and TiDB connections); the data table supports the maximum number of columns; one column Storage size; etc.
reference
mysql official configuration instructions: https://dev.mysql.com/doc/refman/8.1/en/server-system-variables.html