一、问题描述
某次数据库关停过程中，遗留部分正在连接的进程（有相关网络连接）未正常关闭，因是生产非上线环境，进行了强制终止连接，导致MySQL再次启动报错：
ERROR! The server quit without updating PID file (/opt/deepinsight/mysql/data/Namenode.pid).
查看日志：
2022-05-18 22:59:50 21655 [ERROR] Fatal error: mysql.user table is damaged or in unsupported 3.20 format.
2022-05-18 23:01:42 22121 [Note] Plugin 'FEDERATED' is disabled.
/opt/deepinsight/mysql/bin/mysqld: Table 'mysql.plugin' doesn't exist
2022-05-18 23:01:42 22121 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
……
2022-05-18 23:01:42 22121 [Note] InnoDB: Using atomics to ref count buffer pool pages
2022-05-18 23:01:42 22121 [Note] InnoDB: The InnoDB memory heap is disabled
2022-05-18 23:01:42 22121 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-05-18 23:01:42 22121 [Note] InnoDB: Memory barrier is not used
2022-05-18 23:01:42 22121 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-05-18 23:01:42 22121 [Note] InnoDB: Using Linux native AIO
2022-05-18 23:01:42 22121 [Note] InnoDB: Using CPU crc32 instructions
2022-05-18 23:01:42 22121 [Note] InnoDB: Initializing buffer pool, size = 2.0G
2022-05-18 23:01:42 22121 [Note] InnoDB: Completed initialization of buffer pool
2022-05-18 23:01:42 22121 [Note] InnoDB: Highest supported file format is Barracuda.
2022-05-18 23:01:42 22121 [Note] InnoDB: The log sequence numbers 1625987 and 1625987 in ibdata files do not match the log sequence number 11933717 in the ib_logfiles!
2022-05-18 23:01:42 22121 [Note] InnoDB: Database was not shutdown normally!
2022-05-18 23:01:42 22121 [Note] InnoDB: Starting crash recovery.
2022-05-18 23:01:42 22121 [Note] InnoDB: Reading tablespace information from the .ibd files...
2022-05-18 23:01:42 22121 [Note] InnoDB: Restoring possible half-written data pages
2022-05-18 23:01:42 22121 [Note] InnoDB: from the doublewrite buffer...
2022-05-18 23:01:43 22121 [Note] InnoDB: 128 rollback segment(s) are active.
2022-05-18 23:01:43 22121 [Note] InnoDB: Waiting for purge to start
2022-05-18 23:01:43 22121 [Note] InnoDB: 5.6.51 started; log sequence number 11933717
2022-05-18 23:01:43 22121 [Note] Recovering after a crash using mysql-bin-master
2022-05-18 23:01:43 22121 [Note] Starting crash recovery...
2022-05-18 23:01:43 22121 [Note] Crash recovery finished.
……
2022-05-18 23:01:43 22121 [ERROR] Fatal error: mysql.user table is damaged or in unsupported 3.20 format.
……
022-05-18 22:48:49 19136 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2022-05-18 22:48:49 19136 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2022-05-18 22:48:49 19136 [Note] InnoDB: Unable to open the first data file
2022-05-18 22:48:49 7f5e2d97f780 InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2022-05-18 22:48:49 19136 [ERROR] InnoDB: Can't open './ibdata1'
2022-05-18 22:48:49 19136 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2022-05-18 22:48:49 19136 [ERROR] Plugin 'InnoDB' init function returned error.
2022-05-18 22:48:49 19136 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2022-05-18 22:48:49 19136 [ERROR] Unknown/unsupported storage engine: InnoDB
2022-05-18 22:48:49 19136 [ERROR] Aborting
环境：MySQL 14.14 Distrib 5.6.51
MySQL教程：单击查看；
二、处理
1）按照提示运行mysql_upgrade报错：
Looking for 'mysql' as: ./bin/mysql
Looking for 'mysqlcheck' as: ./bin/mysqlcheck
Error: Failed while fetching Server version! Could be due to unauthorized access.
FATAL ERROR: Upgrade failed
2）编辑my.cnf,添加skip-grant-tables，然后重启数据库就可以登录；
3）关于RSA private key file not found: /db/mysql5.6/data//private_key.pem. Some authentication plugins will not work；可按如下处理：
openssl genrsa -out mysql.pem 2048
openssl rsa -in mysql.pem -pubout -out mysql.pub
重启数据库后，执行：mysqlcheck --all-databases --repair，尝试，现场执行未果。
或执行：mysql_upgrade -u root -p 尝试修复，现场验证未果
更多使用参考 mycheck使用说明；
4）备份全库结构：
mysqldump -uroot -p -A -d > /home/mysql//mydb_schemal_bak.sql
最后删除了data目录下所有文件，重新初始化，执行：/usr/bin/mysql_install_db --user=mysql
，导入数据库重建。
参考如下：
1.备份全部数据库的数据和结构
mysqldump -uroot -p123456 -A > /data/mysqlDump/mydb.sql
2.备份全部数据库的结构（加 -d 参数）
mysqldump -uroot -p123456 -A -d > /data/mysqlDump/mydb.sql
3.备份全部数据库的数据(加 -t 参数)
mysqldump -uroot -p123456 -A -t > /data/mysqlDump/mydb.sql
4.备份单个数据库的数据和结构(,数据库名mydb)
mysqldump -uroot-p123456 mydb > /data/mysqlDump/mydb.sql
5.备份单个数据库的结构
mysqldump -uroot -p123456 mydb -d > /data/mysqlDump/mydb.sql
6.备份单个数据库的数据
mysqldump -uroot -p123456 mydb -t > /data/mysqlDump/mydb.sql
7.备份多个表的数据和结构（数据，结构的单独备份方法与上同）
mysqldump -uroot -p123456 mydb t1 t2 > /data/mysqlDump/mydb.sql
8.一次备份多个数据库
mysqldump -uroot -p123456 --databases db1 db2 > /data/mysqlDump/mydb.sql
5）登录mysql，执行grant授权时报错：ERROR 1805 (HY000): Column count of mysql.user is wrong. Expected 43, found 45. The table is probably corrupted
对比user表异常的字段，用以下命令删除多余的；也可用Navicat通过表设计来修改：
mysql -e "ALTER TABLE mysql.user DROP COLUMN is_role, DROP default_role, DROP max_statement_time" -uroot -p
#现场实际：
mysql -e "ALTER TABLE mysql.user DROP COLUMN account_locked, DROP password_lifetime, DROP password_last_changed" -uroot -p
回顾来看，这应该是导入的sql数据结构与当前数据库不一致所致，因sql文件过于久远，已无从考证。
6）如果MySQL日志报错类似： Table ‘mysql.slave_relay_log_info’ cannot be opened.
检查删除以下5张表并重建：
drop table if exists innodb_index_stats;
drop table if exists innodb_table_stats;
drop table if exists slave_master_info;
drop table if exists slave_relay_log_info;
drop table if exists slave_worker_info;
#然后
source ./mysql/share/mysql_system_tables.sql
更多参看：mysql.slave_master_info表不存在的解决方法；
三、附录：
1）一个MySQL备份脚本：
#!/bin/bash
#保存备份个数，备份31天数据
number=31
#备份保存路径
backup_dir=/root/mysqlbackup
#日期
dd=`date +%Y-%m-%d-%H-%M-%S`
#备份工具
tool=mysqldump
#用户名
username=root
#密码
password=123456
#将要备份的数据库
database_name=blue
#如果文件夹不存在则创建
if [ ! -d $backup_dir ];
then
mkdir -p $backup_dir;
fi
#简单写法 mysqldump -u root -p123456 users > /root/mysqlbackup/users-$filename.sql
$tool -u $username -p$password $database_name > $backup_dir/$database_name-$dd.sql
#写创建备份日志
echo "create $backup_dir/$database_name-$dd.dupm" >> $backup_dir/log.txt
#找出需要删除的备份
delfile=`ls -l -crt $backup_dir/*.sql | awk '{print $9 }' | head -1`
#判断现在的备份数量是否大于$number
count=`ls -l -crt $backup_dir/*.sql | awk '{print $9 }' | wc -l`
if [ $count -gt $number ]
then
#删除最早生成的备份，只保留number数量的备份
rm $delfile
#写删除文件日志
echo "delete $delfile" >> $backup_dir/log.txt
fi
2）cron任务配置示例：
1、每天早上6点
0 6 * * * echo “Good morning.” >> /tmp/test.txt //注意单纯echo，从屏幕上看不到任何输出，因为cron把任何输出都email到root的信箱了。
2.每两个小时
0 */2 * * * echo “Have a break now.” >> /tmp/test.txt
3.晚上11点到早上8点之间每两个小时和早上八点
0 23-7/2，8 * * * echo “Have a good dream” >> /tmp/test.txt
4.每个月的4号和每个礼拜的礼拜一到礼拜三的早上11点
0 11 4 * 1-3 command line
5.1月1日早上4点
0 4 1 1 * command line SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root //如果出现错误，或者有数据输出，数据作为邮件发给这个帐号 HOME=/
6.每小时执行/etc/cron.hourly内的脚本
01 * * * * root run-parts /etc/cron.hourly
7.每天执行/etc/cron.daily内的脚本
02 4 * * * root run-parts /etc/cron.daily
8.每星期执行/etc/cron.weekly内的脚本
22 4 * * 0 root run-parts /etc/cron.weekly
9.每月去执行/etc/cron.monthly内的脚本
42 4 1 * * root run-parts /etc/cron.monthly
注意: "run-parts"这个参数了，如果去掉这个参数的话，后面就可以写要运行的某个脚本名，而不是文件夹名。
10.每天的下午4点、5点、6点的5 min、15 min、25 min、35 min、45 min、55 min时执行命令。
5，15，25，35，45，55 16，17，18 * * * command
11.每周一，三，五的下午3：00系统进入维护状态，重新启动系统。
00 15 * * 1，3，5 shutdown -r +5
12.每小时的10分，40分执行用户目录下的innd/bbslin这个指令：
10，40 * * * * innd/bbslink
13.每小时的1分执行用户目录下的bin/account这个指令：
1 * * * * bin/account
3）删库跑路后紧急救援