centos7.5 offline installation and deployment of TiDB-6.5.0 distributed system
The official website is very clear. Please be sure to read the official website to learn. The official website address is at the bottom of the article. I am just recording the pitfalls. I hope it can help you, stranger.
1. Requirements, why should we deploy TiDB-6.5.0 distributed system?
At present, the business data of most enterprises are scattered in different systems without a unified summary. As the business develops, the decision-making level of the enterprise needs to understand the business status of the entire company in order to make timely decisions. Therefore, it is necessary to integrate the business data scattered in The data from various systems are gathered in the same system and undergo secondary processing to generate T+0 or T+1 reports. The traditional common solution is to use ETL + Hadoop, but the Hadoop system is too complex, and the operation and maintenance and storage costs are too high to meet the needs of users. Compared with Hadoop, TiDB is much simpler. Businesses synchronize data to TiDB through ETL tools or TiDB synchronization tools. Reports can be directly generated in TiDB through SQL.
2. Basic concepts of TIDB
2.1 What is TiDB
TiDB is an open source distributed relational database independently designed and developed by PingCAP. It is a converged distributed database product that supports both online transaction processing and online analytical processing (Hybrid Transactional and Analytical Processing, HTAP). It has horizontal expansion or contraction capabilities. Capacity, financial-grade high availability, real-time HTAP, cloud-native distributed database, compatibility with MySQL 5.7 protocol and MySQL ecosystem and other important features. The goal is to provide users with one-stop OLTP (Online Transactional Processing), OLAP (Online Analytical Processing), and HTAP solutions. TiDB is suitable for various application scenarios such as high availability, strong consistency requirements, and large data scale.
2.2 TiDB overall architecture
To deeply understand the horizontal expansion and high availability features of TiDB, you first need to understand the overall architecture of TiDB. TiDB cluster mainly includes three core components: TiDB Server, PD Server and TiKV Server. In addition, there are TiSpark components to solve users' complex OLAP needs and TiDB Operator components to simplify cloud deployment management.
2.3 TiDB Server
TiDB Server is responsible for receiving SQL requests, processing SQL-related logic, finding the TiKV address that stores the data required for calculation through PD, interacting with TiKV to obtain data, and finally returning results. TiDB Server is stateless. It does not store data itself, but is only responsible for calculations. It can be infinitely expanded horizontally and can provide a unified access address to the outside world through load balancing components (such as LVS, HAProxy or F5).
2.4 PD Server
Placement Driver (PD for short) is the management module of the entire cluster. It has three main tasks: one is to store the meta-information of the cluster (which TiKV node a certain key is stored on); the other is to schedule and load balance the TiKV cluster (such as data migration, Raft group leader migration, etc.); the third is to allocate a globally unique and increasing transaction ID.
PD ensures data security through the Raft protocol. Raft's leader server is responsible for processing all operations, and the remaining PD servers are only used to ensure high availability. It is recommended to deploy an odd number of PD nodes.
2.5 TiKV Server
TiKV Server is responsible for storing data. From the outside, TiKV is a distributed Key-Value storage engine that provides transactions. The basic unit of storing data is Region. Each Region is responsible for storing data of a Key Range (left-closed and right-open range from StartKey to EndKey). Each TiKV node is responsible for multiple Regions. TiKV uses the Raft protocol for replication to maintain data consistency and disaster recovery. Replicas are managed in units of Region. Multiple Regions on different nodes form a Raft Group and are replicas of each other. The load balancing of data among multiple TiKVs is scheduled by PD, which is also scheduled in Region units.
2.6 TiSpark
As the main component in TiDB to solve users' complex OLAP needs, TiSpark runs Spark SQL directly on the TiDB storage layer, while integrating the advantages of TiKV distributed clusters and integrating into the big data community ecosystem. At this point, TiDB can support both OLTP and OLAP through a system, eliminating the worry of user data synchronization.
2.7 TiDB Operator
TiDB Operator provides the ability to deploy and manage TiDB clusters on mainstream cloud infrastructure (Kubernetes). It combines the best practices of container orchestration in the cloud native community with TiDB's professional operation and maintenance knowledge, integrating one-click deployment, multi-cluster co-deployment, automatic operation and maintenance, fault self-healing and other capabilities, which greatly lowers the threshold for users to use and manage TiDB. and cost.
2.8 Compatibility comparison with MySQL
TiDB supports the MySQL transport protocol and most of its syntax. This means that your existing MySQL connectors and clients can continue to be used. In most cases, your existing applications can be migrated to TiDB without any code modifications.
The currently officially supported version of TiDB server is MySQL 5.7. Most MySQL operation and maintenance tools (such as PHPMyAdmin, Navicat, MySQL Workbench, etc.), as well as backup and recovery tools (such as mysqldump, Mydumper/myloader), etc. can be used directly.
However, because some features cannot be implemented well in a distributed environment, they are currently not supported or their performance is different from MySQL. Some MySQL syntax can be parsed and passed in TiDB, but no subsequent processing will be performed. For example, the Engine in the Create Table statement is parsed and ignored.
https://pingcap.com/docs-cn/v6.5.0/reference/mysql-compatibility/
Don’t wait, the installation and deployment will begin. You can read the above after the deployment is completed.
3. TiDB-6.5.0 distributed system installation and deployment
3.1 Download binary installation package
Enter the official download address, select the community version to download, and select the corresponding version
https://cn.pingcap.com/product-community/
3.2 Configure the basic environment
#创建tidb用户
adduser tidb
passwd tidb
vim /etc/sudoers
tidb ALL=(ALL) NOPASSWD:ALL
#进行相互之间免密操作
su - tidb
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
以下我都是在tidb用户下操作的，权限不足就加sudo
3.3 Deploy TiUP components in offline environment
After sending the offline package to the server, execute the following command to install the TiUP component:
#对压缩包的一些处理，其实也无所吊谓，处不处理都可以，想来诸位都是有linux一定基础的
sudo chmod +x tidb-community-server-v6.5.0-linux-amd64.tar.gz
sudo chown tidb:tidb tidb-community-server-v6.5.0-linux-amd64.tar.gz
#创建 tidb-6.5.0文件夹，以后所有关于tidb的数据都放到了这个文件夹里面
mkdir /opt/module/tidb-6.5.0
sudo tar -zxvf tidb-community-server-v6.5.0-linux-amd64.tar.gz -C /opt/module/tidb-6.5.0
sudo chown -R tidb:tidb ./tidb-6.5.0/
sh ./local_install.sh
source /home/tidb/.bash_profile
tiup playground
The local_install.sh script will automatically execute the tiup mirror set tidb-community-server- version − linux − amd 64 command to set the current mirror address to tidb − community − server − {version}-linux-amd64 command to set the current mirror address to tidb -community-server-version−linux−The am d 64 command sets the current mirror address to ti d b−community−server−{version}-linux-amd64。
3.4 Merge offline packages
If the offline software package is downloaded through the official download page, the TiDB-community-server software package and TiDB-community-toolkit software package need to be merged into the offline image. If the offline component package is manually packaged through the tiup mirror clone command, this step does not need to be performed.
Execute the following command to merge the offline components into the server directory.
tar xf tidb-community-toolkit-${version}-linux-amd64.tar.gz
ls -ld tidb-community-server-${version}-linux-amd64 tidb-community-toolkit-${version}-linux-amd64
cd tidb-community-server-${version}-linux-amd64/
cp -rp keys ~/.tiup/
tiup mirror merge ../tidb-community-toolkit-${version}-linux-amd64
If you need to switch the mirror to another directory, you can manually execute tiup mirror set to switch. If you need to switch to the online environment, you can execute tiup mirror set https://tiup-mirrors.pingcap.com.
3.5 Initialize cluster topology file
For two common deployment scenarios, you can also generate recommended topology templates through the following commands:
Hybrid deployment scenario: deploy multiple instances on a single machine
tiup cluster template --full > topology.yaml
Cross-computer room deployment scenario: Deploy TiDB cluster across computer rooms
tiup cluster template --multi-dc > topology.yaml
Execute vim topology.yaml and view the contents of the configuration file (I have a single node for testing. The single node will be deployed, and the cluster should also be deployed. I will not comment on the tutorials I will copy others later.):
global:
user: "tidb"
ssh_port: 22
deploy_dir: "/opt/module/tidb-6.5.0/tidb-deploy"
data_dir: "/opt/module/tidb-6.5.0/tidb-data"
server_configs: {
}
pd_servers:
- host: 192.168.2.223
tidb_servers:
- host: 192.168.2.223
tikv_servers:
- host: 192.168.2.223
monitoring_servers:
- host: 192.168.2.223
grafana_servers:
- host: 192.168.2.223
alertmanager_servers:
- host: 192.168.2.223
3.6 Execute deployment command
注意
通过 TiUP 进行集群部署可以使用密钥或者交互密码方式来进行安全认证：
如果是密钥方式，可以通过 -i 或者 --identity_file 来指定密钥的路径。
如果是密码方式，可以通过 -p 进入密码交互窗口。
如果已经配置免密登录目标机，则不需填写认证。
一般情况下 TiUP 会在目标机器上创建 topology.yaml 中约定的用户和组，以下情况例外：
topology.yaml 中设置的用户名在目标机器上已存在。
在命令行上使用了参数 --skip-create-user 明确指定跳过创建用户的步骤。
Before executing the deployment command, use the check and check --apply commands to check and automatically repair potential risks in the cluster:
1. Check the potential risks in the cluster:
tiup cluster check ./topology.yaml --user root
2. Potential risks in automatically repairing clusters:
tiup cluster check ./topology.yaml --apply --user root
3.6.1.1 The following are solutions to problems when checking potential risks in the cluster
Check cluster commands
tiup cluster check ./topology.yaml --user tidb
问题一：CPU frequency governor is conservative, should use performance, auto fixing not supported
Select performance mode for the cpufreq module that adjusts the CPU frequency. Fix the CPU frequency at the highest supported operating frequency without dynamic adjustment to obtain the best performance.
Solution: Read and use
Linux internally has five frequency management strategies: userspace, conservative, ondemand, powersave (power saving mode) and performance (performance mode).
View the current CPU mode:
view cpu0
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
view all cpu
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
View all CPU frequencies:
cpupower -c all frequency-info
Set CPU mode:
cpupower frequency-set -g performance #设置成性能模式
cpupower frequency-set -g powersave #设置成省电模式
Linux error: -bash: cpupower: command not found
这个错误提示表明系统中没有安装cpupower命令。可以通过以下命令安装cpupower：
在Debian/Ubuntu上：
sudo apt-get update
sudo apt-get install linux-tools-common linux-tools-$(uname -r)
在CentOS/RHEL上：
sudo yum install kernel-tools
安装完成后，再次尝试运行cpupower命令即可。
问题二： service irqbalance not found, should be installed and started
This error message indicates that the irqbalance service is not installed in the system. irqbalance can be installed with the following command:
#在Debian/Ubuntu上：
sudo apt-get update
sudo apt-get install irqbalance
#在CentOS/RHEL上：
sudo yum install irqbalance
#安装完成后，启动irqbalance服务：
sudo service irqbalance start
#或者：
sudo systemctl start irqbalance
如果需要开机自启动irqbalance服务，可以执行以下命令：
#在Debian/Ubuntu上：
sudo update-rc.d irqbalance defaults
#在CentOS/RHEL上：
sudo systemctl enable irqbalance
After the installation and startup are complete, try running the irqbalance service again.
问题三: Fail numactl not usable, /etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
/etc/profile.d/lang.sh: line 20: warning: locale: LC_COLLATE: cannot change locale(en_US.UTF-8): No such file or directory
/etc/profile.d/lang.sh: line 23: /etc/profile.d/lang.sh: line 26: warning: locale: LC_NUMERIC: cannot change locale (en_US.UTF-8): no such file or directory
/etc/profile.d/lang.sh: line 26: warning: locale: LC_NUMERIC: cannot change locale (en_US.UTF-8): 8):
/etc/profile.d/lang.sh: line 29: warning: locale: LC_TIME: cannot change locale (en_US.UTF-8)
:
sudo yum -y install numactl
If numactl can be used, then the first problem does not exist. Maybe it's just a locale setting issue. Use the following command from the command line to install missing language packs:
#安装glibc-common包：
sudo yum install -y glibc-common
#生成en_US.UTF-8 locale：
sudo localedef -c -i en_US -f UTF-8 en_US.UTF-8
#编辑/etc/locale.conf文件，并添加以下内容：
LANG=en_US.utf8
LC_ALL=en_US.utf8
#编辑/etc/profile文件，并添加以下内容：
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
#重新启动以使更改生效。
source /etc/profile
Question 4: 192.168.2.221 listening-port Fail port 2380 is already in use
Solution: Verify if any other program is using port 2379:
sudo lsof -i :2379
If a program uses this port, you will see information about the related process.
Stop the program process using this port:
sudo kill -9 <PID>
where PID is the process ID using the port. If multiple processes are using the port, you will need to repeat this step to stop them.
Question 5: Warn The NTPd daemon may be not start
TiDB is a distributed database system that requires time synchronization between nodes to ensure transaction linear consistency of the ACID model. At present, the common solution to solve the problem of timing is to use NTP service. You can use the pool.ntp.org timing service on the Internet to ensure the time synchronization of nodes. You can also use the NTP service built by yourself in the offline environment to solve the problem of timing.
1. Use the following steps to check whether the NTP service is installed and synchronized with the NTP server normally:
Execute the following command. If running is output, it means the NTP service is running:
sudo systemctl status ntpd.service
If the error message Unit ntpd.service could not be found. is returned, please try executing the following command to check whether the system configuration used for clock synchronization with NTP is chronyd or ntpd:
sudo systemctl status chronyd.service
If it is found that neither chronyd nor ntpd is configured on the system, it means that neither service has been installed on the system. At this time, you should install one of the services first and ensure that it can start automatically. By default, ntpd is used.
2. Execute the ntpstat command to check whether it is synchronized with the NTP server:
ntpstat
If synchronized to NTP server is output, it means that the NTP server is being synchronized normally:
The following situations indicate that the NTP service is not synchronized normally:
unsynchronised
The following conditions indicate that the NTP service is not functioning properly:
Unable to talk to NTP daemon. Is it running?
3.7 Re-execute the check command
tiup cluster check ./topology.yaml --user tidb
Found all no problems, ok
3.8 Deploy TiDB cluster:
tiup cluster deploy harbor-tidb1 v6.5.0 ./topology.yaml --user tidb -p root
In the above deployment example:
harbor-tidb1 is the name of the deployed cluster.
v6.5.0 is the deployed cluster version. You can view the latest available version supported by TiUP by executing tiup list tidb.
The initialization configuration file is topology.yaml.
–user root means logging in to the target host as the root user to complete cluster deployment. This user needs to have ssh permissions to the target machine and sudo permissions on the target machine. Deployment can also be completed using other users with ssh and sudo permissions.
[-i] and [-p] are optional and do not need to be filled in if password-free login to the target machine has been configured. Otherwise, just choose one of them. [-i] is the private key of the root user (or other user specified by --user) who can log in to the target machine. You can also use [-p] to interactively enter the user's password.
It is expected that the keyword Deployed cluster harbor-tidb1successfully will be output at the end of the log, indicating that the deployment is successful.
3.9 Check the cluster status managed by TiUP
tiup cluster list
TiUP supports managing multiple TiDB clusters. This command will output all cluster information currently managed through TiUP cluster, including cluster name, deployment user, version, key information, etc.
3.10 Check the deployed TiDB cluster
For example, execute the following command to check the harbor-tidb1 cluster:
tiup cluster display harbor-tidb1
The expected output includes the instance ID, role, host, listening port and status (because it has not been started yet, so the status is Down/inactive) and directory information in the tidb-test cluster.
3.11 Start the cluster
Secure startup is a new startup method introduced by TiUP cluster starting from v1.9.0. Using this method to start the database can improve database security. Safe boot is recommended.
After secure startup, TiUP will automatically generate the password of the TiDB root user and return the password on the command line interface.
注意
使用安全启动方式后，不能通过无密码的 root 用户登录数据库，你需要记录命令行返回的密码进行后续操作。
该自动生成的密码只会返回一次，如果没有记录或者忘记该密码，请参照忘记 root 密码修改密码。
Method 1: Safe boot
tiup cluster start harbor-tidb1 --init
The expected results are as follows, indicating successful startup.
Method 2: Normal startup
tiup cluster start harbor-tidb1
The expected result is Started cluster harbor-tidb1successfully, indicating that the startup is successful. After using the normal startup mode, you can log in to the database through the root user without a password.
3.12 Verify cluster running status
tiup cluster display harbor-tidb1
Expected result output: The Status information of each node is Up, indicating that the cluster status is normal.
3.13 Use navicat to connect
You can see that the port number of tidb is 4000 and the pd operation and maintenance port is 2379. We use port 4000 to connect to the database through tools like Navicat.
After logging in, we can change our database password
to TiDB and store the password in the mysql.user system database. Only users with CREATE USER permissions, or with mysql database permissions (INSERT permissions for creation, UPDATE permissions for updates) can set or modify passwords.
Changing the password for an existing account can be done through the SET PASSWORD FOR or ALTER USER statement:
SET PASSWORD FOR 'root'@'%' = '你的新密码';
#或者
ALTER USER 'test'@'localhost' IDENTIFIED BY '你得新密码';
For details, please see the official website:
https://docs.pingcap.com/zh/tidb/stable/user-account-management#%E5%BF%98%E8%AE%B0-root-%E5%AF%86%E7%A0%81
3.14 Access general
http://IP:3000/login The account and password are both admin
3.15 Access Dashboard
Dashboard URL: http://IP:2379/dashboard
3.16 Key components
Several key component information:
Pd: metadata and control scheduling component
Tikv: storage component
Tidb: database instance component
Tiflash: flash memory component
Tidb Although similar to mysql, it is powerful in distribution. If you want to use mysql, after the database becomes larger, To consider sharding databases and tables, and using data routing tools such as mycat, Tidb is designed to be distributed from the bottom up, with a storage architecture similar to HDFS, making distribution a native architecture.
4. Configure Grafana
4.1 Add Prometheus data source
Log in to the Grafana interface.
Default address: http://localhost:3000
Default account: admin
Default password: admin
I won’t write too much here. Please refer to the official website address for details. The address is as follows.
https://docs.pingcap.com/zh/tidb/stable/deploy-monitoring-services
4.2 You can view the following cluster component information:
TiDB Server:
query 处理时间，可以看到延迟和吞吐
ddl 过程监控
TiKV client 相关的监控
PD client 相关的监控
PD Server:
命令执行的总次数
某个命令执行失败的总次数
某个命令执行成功的耗时统计
某个命令执行失败的耗时统计
某个命令执行完成并返回结果的耗时统计
TiKV Server:
GC 监控
执行 KV 命令的总次数
Scheduler 执行命令的耗时统计
Raft propose 命令的总次数
Raft 执行命令的耗时统计
Raft 执行命令失败的总次数
Raft 处理 ready 状态的总次数
5. Use of TiDB (will be updated later)
5.1
5.2
5.3
5.4
5.5
6. Use of Dashboard
6.1
7. Official website deployment address
https://docs.pingcap.com/zh/tidb/stable/production-deployment-using-tiup