"Tencent Cloud TDSQL-C Product Evaluation Activity" is an activity aimed at database product evaluation and product experience launched by Tencent Cloud and CSDN. This event is mainly for TDSQL-C Serverless version.
This participation event can cover users of different technical levels, and can experience and evaluate the product in terms of automatic elasticity, automatic start-stop capability, compatibility, security, concurrency, reliability, etc. of TDSQL-C products. At the same time, you can not only gain practical experience in related technical fields, but also get rich activity incentives.
I. Introduction:
I have been in contact with and engaged in Internet development for nearly 7 years. From PHP and Swoole at the beginning, to languages that I gradually came into contact with later, such as java, go, python, Node, etc., I have been working with databases (such as: Mysql, MariaDB, PostgreSQL) to deal with, the most engaged in the industry is the development of e-commerce project systems.
The above is the iterative evolution process of the company from 0 to 1 e-commerce system, considering its own business characteristics, as well as the resources it owns or can deploy. Only after these are clarified can an appropriate architecture be appropriately designed to ensure that stable services can be provided for the application. From stand-alone architecture, dynamic and static separation architecture, application and data separation architecture, database master-slave architecture, load balancing architecture, MySQL will be widely used. No matter how complex the architecture is, MySQL will basically be used.
In a certain procedure, SQL-related knowledge reserve, application, and optimization are important indicators to measure a back-end developer. If you are still only staying in CURD, you just feel that you want to store data in the database, modify, update, and delete some data. It may be necessary to learn more about SQL and get in touch with larger and wider scenarios. Of course, this is relative to a certain stage. Of course, I am also constantly learning, so as to constantly summarize and improve.
Recently, I have come into contact with many useful products of Tencent Cloud through CSDN activities. In my previous work, I have been using a self-built database system or a cloud database (such as Alibaba Cloud) for system development. Now let us understand and practice Tencent Cloud. TDSQL-C MySQL Serverless version database.
2. Introduction to TDSQL-C:
Due to limited space and energy, this article only evaluates the TDSQL-C MySQL Serverless version database.
2.1 Introduction to TDSQL-C:
TDSQL is Tencent Cloud's enterprise-level distributed database. It covers a complete database product system with financial-level distributed, cloud-native, analytical and other multi-engine integrations. It provides industry-leading financial-level high availability, separation of computing and storage, data warehouse, and enterprise-level Security and other capabilities, as well as a complete product service system such as an intelligent operation and maintenance platform and a serverless version.
Cloud Native Database TDSQL-C (Cloud Native Database TDSQL-C) is called TDSQL-C for short. TDSQL-C is a new generation of high-performance and high-availability enterprise-level distributed cloud database self-developed by Tencent Cloud. Integrating the advantages of traditional database, cloud computing and new hardware technology, it is 100% compatible with MySQL and PostgreSQL, achieving a high throughput of over one million QPS, and 128TB massive distributed intelligent storage to ensure data security and reliability.
TDSQL-C MySQL Serverless Edition (TDSQL-C for MySQL) is a distributed database product created by Tencent. It has strong consistent high availability, global deployment architecture, distributed horizontal expansion, high performance, and enterprise-level security. Intelligent DBA, automatic operation, monitoring and alarming and other supporting facilities provide customers with a complete distributed database solution.
Above, some introductions have been made to TDSQL-C MySQL from big to small. Let’s make a brief summary below. First, we have a general understanding of TDSQL-C MySQL.
3. Participate in evaluation activities:
Fortunately, I had the opportunity to conduct a database evaluation activity. The requirement of the activity is to consume the performance and storage of the database by various means.
3.1 Purchase TDSQL-C MySQL database:
Open the (TDSQL-C) [https://cloud.tencent.com/product/cynosdb] product link, in the "instance form", you must select the Serverless version, which is very critical, and the selection range for "computing power configuration" is 32- 64.
After purchasing a database instance, you need to purchase a "storage resource package" and select a storage capacity of 10TB.
In the tab "Resource Pack" in the TDSQL-C MySQL Serverless database instance, just carry out the "storage resource pack" you just purchased.
3.2 Use mysql2 extension + PM2 to write data for multiple processes of Node service:
const mysql = require('mysql2/promise');
// 初始化数据库连接
let connection = null
async function initDb () {
connection = await mysql.createConnection({
host: 'gz-xxxxx.sql.tencentcdb.com', // TDSQL外网网址
user: 'root', // 数据库用户名
password: 'Mysql123..', // 数据库密码
database: 'test_db', // 数据库名称
port: 22395, // 数据库端口
});
}
async function Run () {
await initDb()
for (let i = 0; i < 99999; i++) {
let sql = "INSERT INTO `use` (name, age) VALUES (?, ?)";
connection.execute(sql, ['11111', 22]).then(result => {
console.log("插入数据成功，插入ID值为", result[0].insertId)
})
}
}
Run()
Run the script on the server to detect, the following is the prompt that the script inserts data successfully.
Use PM2 to start 5 processes and use Node service to asynchronously insert 10w pieces of data, try the effect first.
Judging from the cost details, the effect is not obvious, and only about 50M of data is stored, which is obviously far from enough to achieve the purpose of consumption.
Modify the script to insert 100w data, the script will be disconnected directly, and PM2 will often go offline after opening multiple processes.
3.3. Use Gin + Gorm V2 coroutine to write data:
CREATE TABLE `use` (
`id` bigint(1) NOT NULL AUTO_INCREMENT,
`name` longtext,
`age` int,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4;
dns := fmt.Sprintf("%s:%s@tcp(%s)/%s?charset=%s&parseTime=True&loc=Local", "root", "Ddb123..", "gz-xxxxxxxxx.com:22395", "test_db", "utf8")
db, _ := gorm.Open(mysql.Open(dns), &gorm.Config{
Logger: logger.Default.LogMode(logger.Silent), # 关闭Sql在控制台显示日志
})
func insertDB(db *gorm.DB, text string) {
var users = []DataScores {
{
Name: "xxxx", Score: "11", },
{
Name: "xxxx", Score: "11", },
}
db.Create(&users)
}
From the above figure, it can be analyzed that the storage cost of nearly 40G storage space is still relatively low, and the inserted data is still slow, but there is an extra CCU settlement list, which is a consumption parameter related to computing power.
First of all, let’s analyze that in the table insertion statement used, I used a large text for the name. In this case, the amount of data I insert will be relatively large, otherwise some performance will be wasted during MySQL connection and network transmission. Among them, it is also found that the bandwidth traffic of the server is also relatively high, because it is also opened with elastic bandwidth, which also requires fees.
Try to improve the efficiency of insertion. After finding that the addition of the Gorm Create batch function still has efficiency problems, replace it with the Exec method to directly execute native SQL, and find that the efficiency has increased a lot.
CREATE TABLE `tests` (
`id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'id',
`name1` longtext,
`name2` longtext,
`name3` longtext,
`name4` longtext,
`name5` longtext,
`name6` longtext,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4
func selectDb(db *gorm.DB) {
for {
result := db.Exec("INSERT INTO `tests` (`name1`, `name2`, `name3`, `name4`, `name5`, `name6`) VALUES ('大段文本', '大段文本', '大段文本', '大段文本', '大段文本', '大段文本')")
fmt.Println("插入name表成功", result.RowsAffected)
}
}
In the DMC database management console that comes with TDSQL-C, execute the following statement to simulate CPU consumption. When you see the waiting timeout, guess the possible reason is that the number of connection processes is waiting, resulting in timeout.
Immediately check the TDSQL-C instance, in the "Parameter Settings", find "max_connections" and modify it to 100000. It is also more convenient here, no need to modify in the mysql.conf or mysql.ini configuration file, and then restart, all support hot reload, no need to stop and restart, to prevent impact on production, at the same time, the configuration also greatly simplifies the professional Operation and maintenance capabilities, lowering the threshold.
Moreover, we improve the effect of inserting scripts by setting the maximum number of connections to the Gorm DB database.
dns := fmt.Sprintf("%s:%s@tcp(%s)/%s?charset=%s&parseTime=True&loc=Local", "root", "Ddb123..", "gz-cynosdbmysql-grp-3awgm0zf.sql.tencentcdb.com:22395", "test_db", "utf8")
db, _ := gorm.Open(mysql.Open(dns), &gorm.Config{
Logger: logger.Default.LogMode(logger.Silent),
})
# 设置数据库的最大连接数
sqlDB, _ := db.DB()
// SetMaxIdleConns sets the maximum number of connections in the idle connection pool.
sqlDB.SetMaxIdleConns(1000)
// SetMaxOpenConns sets the maximum number of open connections to the database.
sqlDB.SetMaxOpenConns(10000)
At the same time, execute the above query in the script, and you can see that the CPU is gradually increased from 0 to about 9. SQL statements, mainly by checking themselves, generate some temporary table aggregations, because the fields are not indexed, which is basically a full table scan. At the same time, like also destroys the left-principle query strategy, resulting in the need for a lot of data Querying in , the CPU will gradually increase, and at this time, the corresponding CCU will also be fully increased.
Looking back, the cost details have also risen.
There is a point to note here. You must associate the storage resource package. After the association, the storage will not be billed according to the volume, but will be billed directly by the traffic package.
Among them, the bandwidth traffic was really unbearable. Switching to a friend's company test environment, the server consumed nearly 108G of bandwidth in about 3 days.
3.4 Test server configuration related parameter information:
The main server is 2-core 4G, the elastic bandwidth is 5Mbps peak, and the CPU consumption is very little, mainly because the bandwidth cannot keep up.
3.5 Summary:
At this point, the initial experience of TDSQL-C MySQL Serverless from 0 to 1 is completed. Without any learning, you can quickly use TDSQL-C MySQL Serverless for development, and it is 100% compatible with the basic usage of MySQL.
In the past, I only carried out pressure testing on the interface, and I had no actual pressure testing experience on the database, which also allowed me to further consolidate the knowledge system related to the database. The Go program written by using the top command on the server to view the CPU and memory is relatively small. TDSQL-C MySQL Serverless also supports a large number of business scenarios. MySQL used to be the bottleneck of the project, but now it has become the bottleneck of bandwidth.
3.6 Notes for testing:
Because the billing has a certain delay and is not real-time, so when the cost consumption is relatively low, it is not easy to ask him to get up and stop the service at night because of the use of a friend's server. Therefore, the following automatic Even if you stop the database in the same way, you will still incur costs. Finally, the database instance will be destroyed so that you will not continue to incur costs.
3.7 Extension - How to conduct intranet communication in the follow-up study?
Because the bandwidth generated by the test is relatively large, I bought a Tencent Cloud server to see if I can connect to the intranet, but the test failed. The intranet IP segment of the server is 10, while the intranet IP segment of the database instance is 172.
4. Advantages of TDSQL -C MySQL:
(DB-Engines Ranking)[https://db-engines.com/en/ranking] ranks database management systems by popularity.
The analysis of this list aims to provide a technical direction reference for database practitioners, and the ranking involved is not based on factors such as the technological advancement of products or market share. It can be seen that MySQL is still very popular in use. However, regardless of the ranking, the most important thing is to choose a technology that is suitable for the business needs of the enterprise.
Facing such a version of MySQL, what advantages does TDSQL -C MySQL have?
ApsaraDB for TDSQL-C MySQL Serverless is the core product of Tencent Cloud's PaaS product line. It has made efforts in enterprise-level kernel, enterprise-level scalability, enterprise-level high availability, high reliability, data security, and intelligent operation and maintenance, making it a A database with enterprise-level capabilities, not just open source MySQL.
4.1 Advantages:
-(1) Supports SQL syntax, 100% compatible, easy to use;
-(2) Supports batch data operations, which improves data processing efficiency;
-(3) Supports distributed architecture, which can be easily expanded (in Section 5 Theory explain).
4.2 SQL syntax support:
The SQL statement using a certain project, including the table creation statement and the insertion statement, can be fully executed. It is also possible to output the log on the program, and no problems have been found for the time being.
Through the operation and actual operation of the project, there is no problem with SQL running error reporting. The following is the printed log:
4.3 Check the batch data operation effect:
Using the above table creation statement and using the Gin coroutine to insert a large text for 1 minute, TDSQL-C MySQL inserted 1095 records, while the self-built MySQL 5.7 had 701 records. The efficiency of TDSQL-C MySQL is that of self-built MySQL 5.7 1.57 times, the efficiency is relatively high.
Of course, this is just a simple test, and does not consider the strict comparison of all parameters such as bandwidth, server configuration, and database instance configuration.
4.4 Summary:
Five, database management tools:
5.1 Third-party database connection tools:
In normal work development, connecting and managing databases is a crucial task. MySQL is a very popular open source relational database. Therefore, there are many MySQL databases used to connect and manage. Navicat for MySQL is the most commonly used in daily work.
- Host: It is the external network address opened on TDSQL-C MySQL Serverless
- Port: It should be noted that it is not port 3306. For details, please refer to the port given when opening
- Test Connection: Click to test whether the connection is successful
But in our team management, there will be a relatively big problem, that is, the control of database permissions.
5.2 Database Management (DMC):
Database Management (DMC) is an efficient and reliable one-stop database management platform, which provides users with database management services integrating database table-level operations, real-time monitoring, instance session management, SQL window, and data management, helping you to be more convenient, Manage multiple database instances in a standardized manner.
- Create new database tables, views, stored procedures, etc.
- Data import and export
- SQL execution and security audit
- Authority control, data change approval
In addition to satisfying the above regular functions of Navicat for MySQL, there are other more practical functions, such as viewing some illegal connections, sessions and other information, and experienced it, which basically meets the needs of actual development.
A parameter template is provided, and you can customize the parameter template that meets your business needs. If you need to purchase a database instance next time, you can directly use the newly created template.
5.3 Intelligent database management DBbrain:
TencentDB for DBbrain (DBbrain) is an autonomous database cloud service launched by Tencent Cloud that provides users with functions such as database performance, security, and management. DBbrain uses machine learning, big data methods, and expert experience engines to quickly replicate the mature experience of senior database administrators, intelligentize a large number of traditional manual database operation and maintenance work, serve cloud and cloud enterprises, and effectively guarantee the security of database services. Stable and efficient operation.
The currently better experience is "Diagnostic Optimization", which allows you to see the status of CPU, memory, storage, and traffic. In particular, there is a "Health Score" to see if there is any abnormality.
The performance trend can be viewed based on some indicators. You can always pay attention to the TPS and QPS of the project, whether the database design meets the requirements, and which indicators will have the risk of bottlenecks.
Slow SQL analysis is also one of the indicators that need to be paid attention to in normal development. When the execution time exceeds the set long_query_time, it will be recorded in the slow SQL log, so that you can pay attention to which statements in the project do not meet the requirements in time. index?
SQL optimization is a very powerful function that can help developers optimize SQL, especially for complex SQL queries. From the perspective of statements, it is to perform visual analysis on the results of EXPLAIN, and you can see the problem very intuitively. The original return value in English can be displayed in Chinese, which is very helpful for non-DBA professionals to quickly analyze and improve.
Click to display a more detailed parameter description:
5.4 Encountered errors:
5.5 Summary:
In daily use and operation and maintenance, users may care about database status, gateway status, each node status, Zookeeper, OSS status, etc. However, in previous database products, much of this information could not be directly obtained, and it was necessary to find an account with operation and maintenance or higher authority to obtain relevant information. But in the TDSQL platform, it can be directly obtained through some DMC and database intelligent management DBbrain.
DMC and database intelligent management DBbrain can help users quickly use, manage, and operate TDSQL clusters. Through different role settings, such as platform administrator, instance administrator, tenant administrator, etc., different permissions are given to help users perform instance management, cluster management, resource management, monitoring management, alarm management, log management, performance analysis, etc. Azimuth operation and maintenance management function.
6. Principle analysis of TDSQL -C MySQL:
6.1 TDSQL -C MySQL classification:
If the database types are classified according to relational type and NoSQL type, the type of TDSQL-C for MySQL Serverless version will be classified into the type of NewSQL.
6.2 TDSQL -C MySQL Serverless architecture:
The evolution of NewSQL from a traditional database to a cloud-native database is also a process in which the database architecture continues to evolve with the iterative innovation of business and technology. The overall architecture of TDSQL-C MySQL Serverless actually completely separates the computing layer from the storage layer, realizes the stateless way of computing nodes, and is 100% compatible with the MySQL protocol.
6.3 Comparison with MySQL:
Different users have different specifications and business conditions, whether their parameters can adapt to the maximum state of their own database operation. TDSQL-C MySQL Serverless uses deep learning algorithms to prioritize business analysis, recommends to users which parameters should be adjusted to what value in the current database, how much performance can be improved, and proposes intelligent tuning solutions to users.
6.4 Summary:
TDSQL -C MySQL Serverless provides more choices in terms of flexibility and cost, more flexible configuration, and lower cost architecture, such as supporting single node, writing log history library or merging library when the performance does not need to be so strong , In other scenarios, such as requiring a large capacity but not high performance, this method can be used to solve the problem.
7. Applicable scenarios:
TDSQL-C MySQL Serverless is a distributed database compatible with the MySQL protocol developed by Tencent. It is suitable for OLTP scenarios with large concurrency, high performance, and large capacity. TDSQL is divided into centralized and distributed versions. The distributed version can support distributed transactions, but the performance is not as good as stand-alone transactions, and the performance will suffer a certain loss. If you want to use it, you need to conduct actual test results to decide whether to use it.
-(1) Industrial Internet, IoT, security monitoring and other fields;
-(2) Scenarios that need to process a large amount of time series data;
-(3) Application scenarios that need to efficiently process real-time data collection.
Eight, the company's business scenario considerations:
8.1 Sub-database and sub-table:
Since TDSQL-C MySQL Serverless is used in storage, the advantages of using the cloud-native storage-distributed separation architecture are relatively large, and the maximum specification storage supports TB and PB levels. For mass storage services, there is no need to split databases and tables frequently For data tables, the capacity of a single database and a single table can fully meet the business needs of users.
8.2 Storage capacity:
As above, using the cloud-native storage-sparse separation architecture can be expanded infinitely, and there is no limit to the capacity. In the past, before using MySQL, the estimated capacity of the data volume will be estimated, and the capacity will be expanded after reaching the capacity warning point. However, in TDSQL-C MySQL Serverless, storage is billed by volume, providing users with a way to calculate how much they use.
8.3 Performance:
In the above experience, I also saw many advantages of TDSQL-C MySQL Serverless, such as the database instance is disabled, and when there is data communication, it will be quickly enabled. Inserting data is also much faster than MySQL.
8.4 Add column efficiency:
If you added a column to a large MySQL table before, you need to remove the index, then add the column, and then add the index. Otherwise, if you operate tens of T, hundreds of T of data, once you encounter this kind of table, you need to add a column, DDL The operation time is uncontrollable, and the storage space may explode. TDSQL-C MySQL provides an Instant DDL function, which can solve the problem of users adding columns quickly, and the adding time is basically completed in seconds.
8.5 Case Usability Analysis:
TDSQL-C products have undergone continuous iterations in recent years. Tencent Cloud TDSQL-C products have served many joint-stock banks such as WeBank, Zhangjiagang Rural Commercial Bank, Futu Securities, Huaxia Bank, and many large Internet innovation insurance companies. More than 600 financial and government and enterprise customers. With practical experience in many fields, it is enough to deal with the company's business development.
9. Summary:
The TDSQL-C for MySQL Serverless architecture cloud database can reflect the core value brought by the cloud native architecture, such as some massive storage scenarios, storing data and files, and the performance has also been greatly improved, such as the ability to support parallel computing , improve SQL performance, including parallel scanning, parallel multi-table join, sorting, grouping, aggregation and other parallel capabilities.
At the same time, in terms of ease of use, it also provides many parameter templates, visual operations such as monitoring indicator optimization, backup and log aspects, which greatly reduces operation and maintenance costs.
Through understanding and learning from 0 to 1, TDSQL-C MySQL Serverless builds a product panorama capability map, which is not just a database that is usually recognized, but can monitor everything through consoles, clients, credential management, and auditing. To provide users with a very safe access environment. Then through parameter tuning to assist and assist users to use the database better, use the database better, and make the business run faster, instead of upgrading the database when it is not working. Then to the overall architecture, it supports physical machine and cloud server architecture, supports remote disaster recovery, and builds to meet the company's enterprise-level business needs, bringing cost reduction and efficiency improvement to the company.