Take a quick look at the new features of mysql8.0 version, should your database be upgraded

Foreword

Presumably, many people are still using mysql5.7 or earlier versions, including the company I am in, which is also using version 5.7, after all, it is stable.
In fact, mysql8.0.1 was released in 2016, but until 8.0.11, which was released in 2018, it was considered the official version in the true sense.
From mysql5 to mysql8, there are actually mysql6 transition version and mysql7 cluster version, but for us, it can be directly understood as mysql upgrade from 5.7 to mysql8.0.11, and now the latest version of the official website has reached mysql8.0.18.
mysql download official website

New features of mysql8.0

1. Account Security

1.1. User creation and user authorization are implemented separately in
version 5.7. You can use the grant statement to create users and authorize
version 8.0. You cannot use grant statements to create users and authorize them. You need to create a statement to create users first, and then grant authorization
to do so. In order to make the statement more clear, each has his own duties.
1.2 Update of the identity authentication plug-in
View the identity authentication plug-in through the statement: show variables like 'default_authentication%'
version 5.7, the default identity authentication plug-in is: mysql_native_password
version 8.0, the default identity authentication plug-in is changed to: caching_sha2_password
new version of the plugin, say It is a security and performance upgrade of the plug-in.
When we use the client to connect to the new version of mysql, if the client is not upgraded, an error may be reported and the connection authentication fails.

Of course, the new version of the authentication plug-in can also be changed back, you can change it in the mysql configuration file, and then restart.
(In /etc/my.cnf, search for mysql_native_password and adjust the comment of the line)
You can also dynamically modify the identity authentication plug-in of a user:
alter user 'user name' @ '%' identified with mysql_native_password by 'new password'
After modification, you can check by the following statement, the identity authentication plug-in of a single user has been modified:
select user, host, plugin from mysql.user;

1.3. The password management strategy is strengthened. A
new restriction function is added: the previous password is not allowed to be reused. The specific variables have the following meanings: it
cannot be repeated with the previous 3 passwords: password_history = 3; (default 0)
cannot be repeated with the password within 30 days: password_reuse_interval = 30; (default 0) The
current user password needs to be provided when changing the password (the root user is not restricted by this parameter): password_require_current = ON; (default OFF)
view variables: show variables like 'password%';

1.3.1. Modify these variables, you can directly modify the mysql configuration file, just add password_history = 3 at the end of the file, so that all users around the world are modified, and the service needs to be restarted

1.3.2, modify these variables, you can also modify the variables dynamically: set persist password_history = 3;
set persist is also a new feature of mysql8.0, the set variables are persistent, restart will also take effect.
This is more than the previous version of set global ... this kind of restart is more reliable.
In fact, the principle of set persist is that a new configuration file mysqld-auto.cnf is generated in the directory (/ var / lib / mysql) to achieve the purpose of persistence. In this file, the set variables are saved in json format Next time, restart mysql, in addition to reading its own configuration file, it will also read this.

1.3.3, modify these variables, you can also modify the user level: alter uesr 'user name' @ '%' password history 5;

The principle of this strategy is actually to add a historical password record table mysql.password_history, and every password change will be recorded in this table.

1.4. New function: role management
The concept of this role is the same as the role concept in RBAC thought, which is to solve the problem of management confusion when users correspond to many permissions.

The steps are as follows:
define the role: create role 'role name';
role binding permissions: grant insert, update, delete on database. * To 'role name';
user bind role (when the binding is completed, it needs to be opened to take effect): grant 'Role name' to 'user name';
open a default role for a user: set default role 'role name' to 'user name';
open all roles for a user: set default role all to 'user name';
also You can log in to the user and enable the role: set role 'role name';
last query to verify the user's permissions: show grants for 'role name';
you can also query the specific permissions that the user's role has: show grants for 'user name' using ' Role name ';
revoke role's permissions: revoke insert, update on database. * From' role name ';
after revoking role's permissions, users corresponding to the corresponding role will not have the permissions.

In fact, the role created exists in the mysql.user table, which means that the role of mysql or task is also a user, but there is no password.
Which role the user belongs to is in the mysql.default_role table.
Which users are granted the role is in the mysql.role_edges table.

2. Optimizer index

2.1, hidden index (invisible)

Also known as Invisible Index
8.0, the hidden index will not be used by the optimizer, but background maintenance is still required when data is modified.
You can see the effect by looking at the sql query execution plan (explain sql).
Set the index to visible: alter table table name alter index index name visible;
set the index to invisible: alter table table name alter index index name invisible; the
primary key can not be set to invisible.
Application scenario: soft delete index, grayscale release
If you need to delete the index in the production environment, but you need to create a new index if you are afraid of deleting it, you can first set this index as a hidden index, and confirm the deletion of the index when you are online When there is no impact, you can actually delete this index again.
In the production environment, if we want to add an index, we can first set it as a hidden index, and then release it in grayscale. At this time, the hidden index is not available, but we need to maintain it. At this time, you can modify a variable so that the optimizer of the current session can find the index to see the effect, but it will not affect other sessions. The relevant steps are as follows:
execute select @@ optimezer_switch \ G to
see there is a switch variable ： Use_invisible_indexes = off
Turn on this switch at the current session level: set session optimezer_switch = “use_invisible_indexes = on”;

2.2, descending index

In versions prior to 8.0, you can specify whether the index is descending or ascending when creating the index, but in the end, the ascending index is created.
Starting from 8.0, it supports true descending index, but only BTREE that supports InnoDB engine supports descending index.
The benefits of implementing a real descending index are: In
previous versions, if the index field a was sorted in ascending order and the index field b was sorted in descending order when querying, then in fact, when looking at the execution plan, you can find that it is not only used The index will also be used to sort other files.
But after 8.0, with a real descending index, this situation can be solved with only the index.
Prior to 8.0, there was an implicit sorting for the group by statement.
Starting from 8.0, because of the descending index, implicit sorting is not performed on the group by statement, because the previous version is in ascending order by default, the new version has both ascending and descending order, you can not specify the group by default, if there is sorting , You must use order by.

2.3, function index

Version 8.0.13 supports functional indexing.
This index also supports descending index and also supports indexing of JSON data nodes.
This function index is based on the virtual column function.
It is equivalent to this field for creating a function index. A new column is added. The value is the result of the corresponding field after the function operation. This is easy to understand. In fact, our previous version also had a similar solution. A column, each time a new column is added, this column always stores the result of a certain column through function operation, and finally you can query it according to this column:
alert table table name add column new column name type generated always as (function (original Column name));
This sql is to make the new column name always equal to the result of the original column name numerical operation function.
8.0.13 is just a package of the above scheme. Instead of adding a new column and specifying it, you can directly create a functional index of the original column to achieve the same purpose, making people look more concise.

Create ordinary index: create index index name on table name (field name)
Create function index: create index index name on table name (function name (field name))
function is some methods, such as upper is to convert the field to uppercase.

Usage scenario: The
field in the table is the product name product, and its value has uppercase and lowercase.
When we match the query, if no functional index is created, then query where upper (product) = 'ABCD', it must be a full table scan.
If we created an upper index for this field in advance, then we can use this index when querying.

JSON data, which field in JSON can be specified when creating an index. Personally feel that this feature is a bit tasteless. Because when we design the table structure, we generally avoid storing json data. Even if it is stored, it must be similar to text data, and it will not be queried based on a value in it. So this function can be ignored, it can only be used as an optimization solution for slow query recovery.

3. SQL statement enhancement

Since 8.0, the WITH clause is supported: with XX as ABC select * from XX. XX is equivalent to an alias.
In fact, it is equivalent to using the WITH keyword to define one or more variables. Different variables can specify a table, the previously defined variables, It can be used by the following variables, and then the tables specified by these variables can be used when querying later.

There is also a recursive WITH writing method, which is actually equivalent to the for loop statement in java (int i = 1; i <100; i ++),
so in fact , it is possible to write recursion in SQL. A simple example:

with recursive 自定义别名(i) AS
(
	select 1
	union all
	select i+1 from 自定义别名 where i<100
)
select * from 自定义别名

The above sql, finally returned a list [1,2,3 ... 100].
The i in the custom alias (i) is the loop variable, where i <100 is the final end condition.
Each time i + 1, it will judge whether it is <100. If the condition is met, it is equivalent to finding out the value of i until Meet the conditions, and finally output all the value sets.

For this sql enhancement, I don't see a scene that is particularly suitable for use at present, because the current development model only needs to provide simple additions, deletions, changes and transactions for the database. Other functions are relatively less used, so I concluded that this feature is not interesting at the moment.

4. New data analysis function

Add keyword OVER, usage:

OVER(
	partition by 字段名 
	order by 字段名 
	自定义
)
partition by用于分组，和我们常用的的group by一个意思
order by，这个直接连名字都不改了，就是对分完组的每个组内的数据排序
最后这个自定义，也是作用于分完组的每个组内，可以更详细的对组内的数据进行筛选，比如可以指定组内从第一个到当前，每次都可以取出从第一个到当前行的总和，等等功能。

And added some function methods (such as ranking function, etc.), and the previous like SUM can also be used as this analysis function.
In fact, it is a variant of some of the group by aggregation functions. Group by is a group of several results. This new analysis function is a few original data, just display a few, and there is no aggregation.

Overall, this is not a fundamental improvement, it is still enhanced on the original basis, and it seems to increase the complexity, the average learning cost is slightly higher, maybe for some report business, this will be more appropriate, but for most People, it is estimated that not many people will use this.

5. InnoDB enhancements

5.1. Optimized some metadata files

Under / var / lib / mysql / a database, a lot of file-based data information is deleted, such as .frm files, .MYD files, and .MYI files. In fact, there is a library name called these file information. ibd file is down

5.2. Change the system table mysql and data dictionary table to InnoDB engine

5.3, improve the access performance of the data dictionary table

5.4 DDL operations can be atomic

For example, DDL modifies two fields, the first one succeeds and the second fails, then the final result is both failure. Before version 8.0, the first one in this case was still successful, and the second one failed. This is the difference.

5.5. Persistence of self-increasing columns

Before version 8.0, the self-increasing column counter was stored in memory. After the service restarts, you need to re-read the maximum value of the self-increasing column and then generate the latest maximum value of the self-increasing column. This may be repeated under special circumstances. This is still A long-standing bug before version 8.0.

简答说下老版本这个bug：
比如自增列是id，已有id自增到10，某个时刻删除这个10的数据，然后重启，然后再新增一条数据
如果不重启，新增的数据的自增id，应该是11
但如果重启后，新增的数据的自增id，就还是10，因为重启后，系统扫描这个表，会发现最大id是9，然后新增的话自然就是10了
这其实是我们不愿意看到的情况，因为id为10的这条记录，虽然被删了，但如果新增数据，我们还是希望id是11，而不是10

In version 8.0, the self-increasing column counter is written to the redo log every time it changes. This persistence operation is actually to solve the bug that the self-increasing column may be repeated.

5.6, auto-sensing the maximum value of self-increasing columns

The self-increment column of the old version, for example, since it has been increased to 10, if you manually change the id of the data with the id of 1 to 11, then you will add an error again because the newly added data will increase the counter. The assigned id is 11, but in fact there are already 11 in the database.
The new version, optimized for this, will not report an error, and will perceive the operation that the id becomes 11, and then the new will start from 12.

5.7. Deadlock check

A new dynamic variable (innodb_deadlock_detect) was added in 8.0, which is used to control whether the system performs InnoDB's deadlock check. It is turned on by default.
After opening this deadlock check, if two things deadlock, the system will immediately fail one thing.
Deadlock check will have a more obvious impact on performance, in order to improve performance, you can turn off this feature.
In the old version, for this kind of deadlock situation, there is actually a timeout period, and beyond this time, it will also make a thing invalid. This timeout is controlled by a variable (innodb_lock_wait_timeout), it seems that the default is 50 seconds, I guess the new version of this deadlock check, is it equivalent to adjusting this value to a certain value, and then use another switch to control it .

5.8. New options for locking statements (only for row locks)

In the old version, there is a kind of sql:
select… for update.
If there is a change in the query, wait for the change and return.
The new version has been strengthened to deal with this situation. Two options, nowait and skip locked, can be added after the statement.
select… for update nowait;
nowait just does not wait, and returns immediately when it encounters a row lock.
select… for update skip locked;
skip locked is to skip this one that is being modified and only return the row without lock.
This function is also useful. For example, it is similar to the situation of querying the remaining tickets of train tickets. If you find that some tickets are being modified at the same time, you can return or skip these tickets directly.

5.9 Support some fast DDL

alert table… algorithm = instant In
this way, you can avoid some data copying, which is much faster than the ordinary DDL operation before. The online environment is very suitable for this.

5.10, InnoDB temporary table space optimization

The shared temporary table space ibtemp1 is used.

5.11 New static variable: innodb_dedicated_server

If there is a server dedicated to mysql only, then open this parameter, the system will automatically configure InnoDB memory parameters: innodb_buffer_pool_size, innodb_log_file_size, etc., will try to occupy system resources, thereby improving system performance

5.12, others

6. JSON related

6.1. New inline path operator

New JSON operator: column->> path is
equivalent to the old version: JSON_UNQUOTE (column-> path)
is also equivalent to the old version: JSON_UNQUOTE (JSON_EXTRACT (column, path))
column is the field name accessed by json , If there is a name attribute
path in json is $ .name

6.2. New json function

JSON_ARRAYAGG (): used to generate a json array, a column name in
parentheses JSON_OBJECTAGG (): used to generate a json object, multiple column names can be in brackets, separated by commas
JSON_PRETTY (): beautify the json output format
JSON_STORAGE_SIZE (): You can view the size of the json column, in bytes.
In fact, these four functions 5.7.22 added
JSON_STORAGE_FREE (): you can view the size of the json column released, in bytes, the actual occupied may be larger than this, this is the actual size.
JSON_MERGE_PATCH (): merge two json objects into one, the same node takes the value of the last node
JSON_MERGE_PRESERV (): merge two json objects into one, the same node retains
the function of the above function, which is the old version of JSON_MERGE () Function, so the new version abandons the JSON_MERGE () function
JSON_TABLE (): convert json data to a relational table, you can use the return result of this function as a normal table, and you can use sql to query
this in fact and the previous A second function is the opposite operation