MySQL is the world's most popular open source database. It has long been ranked second in the DB-Engines Ranking list and has a large number of enterprise users and developers around the world. However, as time passes, MySQL users are facing new challenges. Oracle officially announced that it will terminate official technical support for MySQL 5.7 version in October 2023. According to third-party statistics, more than half of MySQL servers are still running version 5.7. In the next few months, a large number of MySQL instances must be upgraded to version 8.0 and higher, otherwise they will not be able to enjoy the technical support and important patch updates provided by Oracle, and enterprise users will face a major challenge.
As a new generation of distributed relational database, TiDB has embraced the MySQL ecosystem from the first day of its birth and has been continuously compatible with MySQL 5.7 and MySQL 8.0, bringing users a smoother migration and usage experience. This article will introduce the new progress of TiDB 7.4 DMR in terms of MySQL 8.0 compatibility, and explore how TiDB can fundamentally solve the various challenges faced by MySQL users.
○ The upgrade affects business continuity . Upgrading MySQL running in single instance or "master-slave mode" will cause the database service to stop, which may have an impact on business operations. Enterprise-level users running a large number of MySQL instances need to invest a lot of manpower and material resources in testing and drills to deal with the potential risks of upgrades.
○ Difficulty in expanding business scale . As business scale expands and data usage scenarios increase, users usually need to make a trade-off between single-machine capacity limitations and shard management complexity. The difficulty of database expansion restricts business scale and development speed.
○ Lack of extremely high availability solutions . For the MySQL database that supports core business scenarios, if an unpredictable downtime event occurs, business recovery becomes complicated, and achieving extremely low recovery time objectives (RTO) becomes a challenge for database administrators.
○ Insufficient real-time analysis capabilities . MySQL's performance when processing large-scale data real-time analysis is not as good as in OLTP (online transaction processing) scenarios. This is a challenge for businesses that require complex queries and data analysis.
○ Original factory hosting services are limited . Although cloud service providers will provide MySQL hosting services, most of them lack official support from Oracle. This means that users cannot get quick feedback and support from the original database manufacturer when dealing with deep product issues and discovering common functional requirements.
Therefore, it is undoubtedly a wise move to migrate to a mature product and solve the above problems in one fell swoop. TiDB is an ideal choice for comprehensive upgrade of MySQL. By choosing TiDB, you can not only get rid of the dilemma of MySQL upgrade and scalability, but also enjoy many additional benefits such as HTAP and database integration.
TiDB is an enterprise-level distributed relational database independently developed by PingCAP. It has important features such as horizontal expansion and contraction, financial-level high availability, real-time HTAP, cloud native, and compatibility with the MySQL 5.7 protocol and ecosystem. TiDB adopts a native distributed architecture design and has flexible elastic scalability. The entire process is transparent to the business and requires no manual intervention. TiDB's multi-copy storage and Multi-Raft protocol ensure strong consistency and high availability of data, and data availability will not be affected when some copies fail. TiDB minimizes the impact of version updates through rolling upgrades. In addition, temporary nodes can be added to ensure that TiDB's performance fluctuations and connection interruptions during the upgrade process are controlled within 5%, significantly reducing the impact of the upgrade on the business. .
In addition, as the creator of TiDB, PingCAP has launched TiDB Cloud, a database hosting service based on the world's leading cloud service providers. Service support covers complex problem diagnosis, upgrade support, emergency rescue, etc., fully reflecting the advantages of original services.
From the early days of the project, TiDB's product strategy of embracing the MySQL ecosystem has continued to this day. TiDB is compatible with MySQL's wire protocol and syntax commands, which means that the MySQL client, MySQL driver, and some MySQL tools can run directly on TiDB. For the vast majority of applications running on MySQL, almost no code changes are required.
With the release of MySQL 8.0, TiDB has actively expanded its compatibility with MySQL 8.0 on the basis of being compatible with MySQL 5.7. TiDB v7.4.0 releases support for common functions of MySQL 8.0, making it easy to smoothly migrate MySQL 8.0 applications. This article lists some functions:
3.1 Common table expression (CTE)
As an important capability introduced in MySQL 8.0, TiDB supports the ANSI SQL 99 standard CTE and its recursive writing method starting from version 5.1. When writing complex queries, you can use common table expressions (CTE) to build a temporary intermediate result set, which can be referenced multiple times in SQL statements to improve SQL statement writing efficiency, readability, and execution efficiency. In the current version, TiFlash also supports CTE.
For example, the table authors stores author information, and book_authors records the correspondence between author IDs and the IDs of the books they write.
mysql> desc authors;
+------------+--------------+------+------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+--------------+------+------+---------+-------+
| id | bigint(20) | NO | PRI | NULL | |
| name | varchar(100) | NO | | NULL | |
| gender | tinyint(1) | YES | | NULL | |
| birth_year | smallint(6) | YES | | NULL | |
| death_year | smallint(6) | YES | | NULL | |
+------------+--------------+------+------+---------+-------+
mysql> desc book_authors;
+-----------+------------+------+------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+------+---------+-------+
| book_id | bigint(20) | NO | PRI | NULL | |
| author_id | bigint(20) | NO | PRI | NULL | |
+-----------+------------+------+------+---------+-------+Using CTE, you can easily write SQL to list how many books the 50 oldest authors have written.
mysql> WITH top_50_eldest_authors_cte AS (
-> SELECT a.id, a.name, (IFNULL(a.death_year, YEAR(NOW())) - a.birth_year) AS age
-> FROM authors a
-> ORDER BY age DESC
-> LIMIT 50
-> )
-> SELECT
-> ANY_VALUE(ta.id) AS author_id,
-> ANY_VALUE(ta.age) AS author_age,
-> ANY_VALUE(ta.name) AS author_name,
-> COUNT(*) AS books
-> FROM top_50_eldest_authors_cte ta
-> LEFT JOIN book_authors ba ON ta.id = ba.author_id
-> GROUP BY ta.id;
+-----------+------------+----------------------+-------+
| author_id | author_age | author_name | books |
+-----------+------------+----------------------+-------+
| 524470241 | 80 | Alexie Kirlin | 7 |
| 67511645 | 80 | Bridgette Tromp | 9 |
...
| 48355494 | 80 | Audrey Mayert | 7 |
+-----------+------------+----------------------+-------+
50 rows in set (0.23 sec)
Related documentation: https://docs.pingcap.com/zh/tidb/stable/dev-guide-use-common-table-expression
3.2 Window function
Window Function, also called analytic function, is used when analyzing, summarizing, and sorting data. Window functions can be written in SQL form to complete some complex data sorting work and help users explore the value of data. For example, data grouping and sorting, change trend analysis, etc.
TiDB currently fully supports the window functions provided by MySQL 8.0, and most of them can be pushed down to TiFlash to run.
Related documents: https://docs.pingcap.com/zh/tidb/stable/window-functions
3.3 Resource management and control
TiDB introduced resource management and control in version 7.1, with the purpose of rationally allocating cluster resources, improving database stability, and reducing database usage costs. In scenarios where multiple applications share a TiDB cluster, resource isolation can effectively reduce the impact of application load changes on other applications. Resource management can also solve the impact of batch jobs and background tasks on core business, as well as sudden SQL performance issues. Slowing down the entire cluster is an important ability to improve the stability of large clusters.
Although there are differences in implementation methods from MySQL, TiDB is compatible with MySQL's syntax and hints for specifying resource groups, reducing user learning costs and migration costs. In addition, TiDB's resource isolation can more effectively control the most important I/O resources, achieving the same or even better results than MySQL.
The following shows the use of resource management to control all resources used by usr1 within 500 RU per second.
● Estimated cluster capacity
mysql> CALIBRATE RESOURCE● Create app1 resource group with a limit of 500 RU per second.
mysql> CREATE RESOURCE GROUP IF NOT EXISTS app1 RU_PER_SEC = 500;● Associate the user with the resource group, and the session of usr1 is automatically associated with the resource group app1
mysql> ALTER USER usr1 RESOURCE GROUP app1;You can also modify the resource group to which the session belongs
mysql> SET RESOURCE GROUP `app1`;Or use hint RESOURCE_GROUP() to specify the resource group to which the statement belongs.
mysql> SELECT /*+ RESOURCE_GROUP(rg1) */ * FROM t limit 10;
Related documents: https://docs.pingcap.com/zh/tidb/stable/tidb-resource-control
3.4 Role-based permission management
TiDB supports MySQL-compatible role management. Role-based authorization can simplify permission management and reduce the risk of errors. Access to the database can be better controlled by associating permissions with roles. Customers can classify work in different scenarios, create corresponding roles, and grant roles to authorized database users. During actual operations, database users can switch roles according to different scenarios to reduce the possibility of misoperation.
Here is an example of using roles to split permissions. As an application administrator, user dev_adm_usr needs to operate the data in the database app_db. In most cases, it is just a query. Occasionally, it will make modifications when data correction is needed. In order to prevent misoperation of dev_adm_usr, we separate the two parts of permissions and use roles, and only give ourselves the read and write roles when necessary.
● Create roles app_read_role and app_write_role
mysql> CREATE ROLE 'app_read_role', 'app_write_role';● Grant the corresponding permissions to the roles. Here, the two roles are granted read and write permissions for app_db respectively.
mysql> GRANT SELECT ON app_db.* TO 'app_read_role'@'%';
mysql> GRANT INSERT, UPDATE, DELETE ON app_db.* TO 'app_write_role'@'%';● Grant both roles to user dev_adm_usr
mysql> GRANT 'app_read_role','app_write_role' TO 'dev_adm_usr'@'localhost';● Set app_read_role as the default role of dev_adm_usr, so that user dev_adm_usr has read-only permissions by default when logging in.
mysql> SET DEFAULT ROLE 'app_read_role' TO 'dev_adm_usr'@'localhost';● When dev_adm_usr needs to modify data, enable the role app_write_role
mysql> SET ROLE app_read_role,app_write_role;Or enable all roles
mysql> SET ROLE ALL;
Related documents: https://docs.pingcap.com/zh/tidb/stable/role-based-access-control
3.5 Enhanced uft8mb4 character set
An important change in MySQL 8.0 is that the default character set changes to the more general uft8mb4, and the default sorting method changes to utf8mb4_0900_ai_ci. TiDB has also added the utf8mb4_0900_ai_ci sorting method in the new version to make system migration easier.
In order to be compatible with both MySQL 5.7 and MySQL 8.0, TiDB supports the MySQL-compatible variable default_collation_for_utf8mb4. Allows the user to adjust the default sorting of the utf8mb4 character set. This approach ensures TiDB's smooth transition between different MySQL versions and can adapt to the needs of different applications.
If migrating from MySQL 8.0, set to 8.0 default sorting utf8mb4_0900_ai_ci
set global default_collation_for_utf8mb4='utf8mb4_0900_ai_ci';If migrating from MySQL 5.7, set utf8mb4_general_ci to 5.7 as the default sort for utf8mb4
set global default_collation_for_utf8mb4='utf8mb4_general_ci';3.6 JSON Multi-valued Index (Multi-valued Index)
After supporting the complete functions of MySQL 5.7, TiDB continues to add support for newly released functions of MySQL 8.0. Recent versions support "multi-value indexing", allowing indexing of an "array" in the JSON type, thereby improving the efficiency of retrieval of JSON data. Usage is exactly the same as MySQL, which means that during the migration process, there is no need to modify the data modeling or application, and users can continue to operate JSON data in a familiar way.
Multivalued indexes are an extension of ordinary index structures. Different from the 1:1 correspondence between ordinary indexes and tables, the correspondence between multi-valued indexes and tables is N:1. The same as MySQL, when using MEMBER OF(), JSON_CONTAINS(), JSON_OVERLAPS() in the condition to retrieve, a multi-value index may be selected.
For example, we assume that there is a customer information table, and all detailed information is compiled into a JSON type column in JSON format, and there is an array structure that stores the cities where the customer is located.
When we need to retrieve which customers are in Beijing, if there is no multi-valued index, this query needs to scan the entire table.
SELECT name FROM customer
WHERE 'beijing' MEMBER OF $.city;At this time, we can create a multi-valued index for the city array, and the above query can use the index to retrieve matching records, greatly improving query performance.
ALTER TABLE customers add index idx_city (name, (CAST(custinfo->'$.city' AS char(20) ARRAY)));Like ordinary indexes, when the optimizer does not select a multi-valued index, you can use the optimizer prompt USE_INDEX() or USE_INDEX_MERGE() to force the optimizer to make a selection.
Related documents: https://docs.pingcap.com/zh/tidb/stable/choose-index#Use multi-value index
3.7 Modify the hint of session variables (SET_VAR())
MySQL 8.0 introduced a special hint SET_VAR(). Using this hint, you can modify a session-level system variable while the statement is running. TiDB also supports this hint in v7.4.0, which improves the flexibility of system variable settings and can be "customized" for SQL statements. Multiple variables related to the optimizer and execution time can be modified using hints.
For example, for the analysis and processing of large tables, appropriately increase the execution parallelism of SQL.
SELECT /*+ set_var(tidb_executor_concurrency=20) */
l_orderkey,
SUM(
l_extendedprice * (1 - l_discount)
) AS revenue,
o_orderdate,
o_shippriority
FROM
customer,
orders,
lineitem
WHERE
c_mktsegment = 'BUILDING'
AND c_custkey = o_custkey
AND l_orderkey = o_orderkey
AND o_orderdate < DATE '1996-01-01'
AND l_shipdate > DATE '1996-02-01'
GROUP BY
l_orderkey,
o_orderdate,
o_shippriority
ORDER BY
revenue DESC,
o_orderdate
limit 10;You can also use this hint to force the previous query to select TiFlash, while other queries remain unchanged.
SELECT /*+ set_var(tidb_isolation_read_engines='tidb,tiflash') */
l_orderkey,
SUM(
l_extendedprice * (1 - l_discount)
) AS revenue,
o_orderdate,
o_shippriority
FROM
customer,
orders,
lineitem
WHERE
c_mktsegment = 'BUILDING'
AND c_custkey = o_custkey
AND l_orderkey = o_orderkey
AND o_orderdate < DATE '1996-01-01'
AND l_shipdate > DATE '1996-02-01'
GROUP BY
l_orderkey,
o_orderdate,
o_shippriority
ORDER BY
revenue DESC,
o_orderdate
limit 10;
Related documentation: https://docs.pingcap.com/zh/tidb/v7.4/optimizer-hints#set_varvar_namevar_value
3.8 CHECK constraints
CHECK constraint is a type of consistency constraint check and is used to maintain the accuracy of data. CHECK constraints can be used to restrict the value of a field in a table to meet specified conditions. After adding a CHECK constraint to a table, TiDB will check whether the constraint conditions are met when inserting or updating data. If not, an error will be reported.
Before MySQL 8.0, it only supported the syntax of CHECK constraints and did not actually check it in actual operation. It was fully supported after 8.0. TiDB has also added this feature in the new version. In order to prevent customers from having residual CHECK conditions in their DDL, which may cause problems due to this feature, TiDB does not enable CHECK constraint checking by default, but manually enables it through the variable tidb_enable_check_constraint. This fully reflects TiDB's product strategy of being compatible with both MySQL 5.7 and 8.0.
mysql> set global tidb_enable_check_constraint=on;
mysql> CREATE TABLE t
-> ( a INT CHECK(a > 10) NOT ENFORCED, -- 不生效 check
-> b INT,
-> c INT,
-> CONSTRAINT c1 CHECK (b > c)
-> );
mysql> insert into t values (20,20,20);
ERROR 3819 (HY000): Check constraint 'c1' is violated.
Related documents: https://docs.pingcap.com/zh/tidb/dev/constraints#check-%E7%BA%A6%E6%9D%9F
In order to reduce the complexity of user data migration, TiDB launched a tool TiDB Data Migration (DM). It can assist users in full data migration and incremental data synchronization from databases compatible with the MySQL protocol (MySQL, MariaDB, Aurora MySQL) to TiDB. DM supports DDL synchronization, merging of sub-databases and sub-tables, and has a variety of built-in filters to flexibly adapt to different scenarios, effectively improving the efficiency of data migration.
TiDB 7.4 will be the last DMR version of the TiDB 7 series and has made many optimizations for MySQL 8.0. As a comprehensive upgrade of MySQL, TiDB's technological leadership helps users cope with changing business data challenges and achieve continued business growth and innovation. While TiDB is highly compatible with the features of MySQL 5.7 and MySQL 8.0, it will also continue to provide technical support to ensure that users can smoothly migrate various business applications, thereby reducing the workload and risks during the migration process.