One of the TiDB database from entry to proficiency series: TiDB database software and hardware environment requirements and system configuration check

1. Software and hardware configuration requirements

As an open source one-stack real-time HTAP database, TiDB can be well deployed and run in Intel architecture server environment, ARM architecture server environment and mainstream virtualization environment, and supports most mainstream hardware networks. As a high-performance database system, TiDB supports mainstream Linux operating system environments.

1. Operating system and platform requirements

For combinations of different operating systems and CPU architectures, TiDB provides support for different levels of quality standards.

On the following operating systems and corresponding CPU architecture combinations, TiDB can meet the requirements of enterprise-level production quality, and the product features have been comprehensively and systematically verified:

2. Suggested server configuration

TiDB supports the deployment and operation of 64-bit general-purpose hardware server platforms based on Intel x86-64 architecture or hardware server platforms based on ARM architecture. There are following requirements and suggestions for the server hardware configuration (excluding the occupation of the operating system OS itself) in the development, testing and production environments:

3. Network requirements

TiDB is an open source one-stack real-time HTAP database. Its normal operation requires the network environment to provide the following network port configuration requirements. Administrators can open relevant ports on the network side and host side according to the deployment scheme of TiDB components in the actual environment:

4. Disk Space Requirements

Client web browsers require
TiDB to provide a Grafana-based technology platform for visual display of various indicators of the database cluster. The monitoring portal can be accessed using newer versions of Microsoft IE, Google Chrome, Mozilla Firefox that support Javascript.

2. TiDB environment and system configuration check

1. Add data disk EXT4 file system mount parameters on the TiKV deployment target machine

For production environment deployment, it is recommended to use NVME type SSD disk with EXT4 type file system to store TiKV data files. This configuration scheme is the best implementation scheme, and its reliability, security, and stability have been proven in a large number of online scenarios.

Use the root user to log in to the target machine, format the data disk of the deployment target machine into an ext4 file system, and add nodelalloc and noatime mount parameters when mounting. nodelalloc is a mandatory parameter, otherwise TiUP will fail the detection during installation; noatime is an optional and recommended parameter.

Note
If your data disk has been formatted as ext4 and the disk is mounted, you can first execute the umount /dev/nvme0n1p1 command to unmount, start from the steps of editing the /etc/fstab file, add mount parameters and mount again.

Taking the /dev/nvme0n1 data disk as an example, the specific operation steps are as follows:

1. Check the data disk

fdisk -l

Disk /dev/nvme0n1: 1000 GB

2. Create partitions

parted -s -a optimal /dev/nvme0n1 mklabel gpt -- mkpart primary ext4 1 -1

Note
Use the lsblk command to view the device number of the partition: for nvme disks, the generated partition device number is generally nvme0n1p1; for ordinary disks (such as /dev/sdb), the generated partition device number is generally sdb1.

3. Format the file system

mkfs.ext4 /dev/nvme0n1p1

4. View data disk partition UUID

The UUID of nvme0n1p1 in this example is c51eb23b-195c-4061-92a9-3fad812cc12f

lsblk -f

NAME    FSTYPE LABEL UUID                                 MOUNTPOINT
sda
├─sda1  ext4         237b634b-a565-477b-8371-6dff0c41f5ab /boot
├─sda2  swap         f414c5c0-f823-4bb1-8fdf-e531173a72ed
└─sda3  ext4         547909c1-398d-4696-94c6-03e43e317b60 /
sr0
nvme0n1
└─nvme0n1p1 ext4         c51eb23b-195c-4061-92a9-3fad812cc12f

5. Edit the /etc/fstab file and add nodelalloc mount parameters

vi /etc/fstab

UUID=c51eb23b-195c-4061-92a9-3fad812cc12f /data1 ext4 defaults,nodelalloc,noatime 0 2

6. Mount the data disk

mkdir /data1 && \
mount -a

7. Execute the following command, if the file system is ext4, and the mount parameter contains nodelalloc, it means that it has taken effect

mount -t ext4

/dev/nvme0n1p1 on /data1 type ext4 (rw,noatime,nodelalloc,data=ordered)

2. Set the temporary space of the TiDB node (recommended)

Some operations of TiDB need to write temporary files to the server, so it is necessary to ensure that the operating system user running TiDB has sufficient permissions to read and write to the target directory. If the TiDB instance is not started with root authority, you need to check the directory authority and set it correctly.

• TiDB temporary workspace

  • Operations that consume a lot of memory, such as hash table construction and sorting, may write temporary data to disk to reduce memory consumption and improve stability. The disk location to write to is defined by the configuration item tmp-storage-path. By default, make sure that the user running TiDB has read and write permissions to the operating system temporary folder (usually /tmp).

• Fast Online DDL Workspace

  • When the variable tidb_ddl_enable_fast_reorg is set to ON (the default value is ON in v6.5.0 and above), Fast Online DDL will be activated, and some DDLs will read and write temporary files. The temporary file location is defined by the configuration temp-dir, and it is necessary to ensure that the user running TiDB has read and write permissions to this directory in the operating system. Take the default directory /tmp/tidb as an example:

Note
If there may be DDL operations for large objects in the business, it is recommended to configure an independent file system and a larger temporary space for temp-dir.

sudo mkdir /tmp/tidb

If the directory /tmp/tidb already exists, make sure you have write permission.

sudo chmod -R 777 /tmp/tidb

3. Detect and close the firewall of the target deployment machine

This section describes how to close the firewall configuration of the target host, because in the TiDB cluster, the access ports between nodes need to be opened to ensure the normal transmission of read and write requests, data heartbeats and other information. In a common online scenario, the network connection from the database to the business service and the database node is to complete the data interaction in the security domain. If there is no special security requirement, it is recommended to close the firewall of the target node. Otherwise, it is recommended to configure the port information into the white list of the firewall service according to the port usage rules.

1. Check the firewall status (take CentOS Linux release 7.7.1908 (Core) as an example)

sudo firewall-cmd --state
sudo systemctl status firewalld.service

2. Turn off the firewall service

sudo systemctl stop firewalld.service

3. Turn off the firewall auto-start service

sudo systemctl disable firewalld.service

4. Check the firewall status

sudo systemctl status firewalld.service

4. Detect and install NTP service

TiDB is a distributed database system that requires time synchronization between nodes to ensure transactional linear consistency of the ACID model. At present, the general solution to solve the time service is to use the NTP service. The time service of the pool.ntp.org in the Internet can be used to ensure the time synchronization of the nodes, or the NTP service built by the offline environment can be used to solve the time service.

Use the following steps to check whether the NTP service is installed and synchronized with the NTP server:

1. Execute the following command, if the output running indicates that the NTP service is running:

sudo systemctl status ntpd.service

ntpd.service - Network Time Service
Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
Active: active (running) since 一 2017-12-18 13:13:19 CST; 3s ago

If the error message Unit ntpd.service could not be found. is returned, please try to execute the following command to check whether the system configuration used for clock synchronization with NTP is chronyd or ntpd:

sudo systemctl status chronyd.service

chronyd.service - NTP client/server
Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-04-05 09:55:29 EDT; 3 days ago

If it is found that neither chronyd nor ntpd is configured in the system, it means that neither service has been installed in the system. At this point, one of the services should be installed first and ensure that it can be started automatically. By default, ntpd is used.

If the system configuration you are using is chronyd, please go to step 3 directly.

2. Execute the ntpstat command to check whether it is synchronized with the NTP server:

Note that the Ubuntu system needs to install the ntpstat package.

ntpstat

If synchronized to NTP server is output, it means that it is synchronizing with the NTP server normally:

synchronised to NTP server (85.199.214.101) at stratum 2
time correct to within 91 ms
polling server every 1024 s

The following conditions indicate that the NTP service is not synchronizing properly:

unsynchronised

The following conditions indicate that the NTP service is not functioning properly:

Unable to talk to NTP daemon. Is it running?

3. Execute the chronyc tracking command to check whether the Chrony service is synchronized with the NTP server.

chronyc tracking

If the command returns Leap status: Normal, it means the synchronization process is normal.

Reference ID    : 5EC69F0A (ntp1.time.nl)
Stratum         : 2
Ref time (UTC)  : Thu May 20 15:19:08 2021
System time     : 0.000022151 seconds slow of NTP time
Last offset     : -0.000041040 seconds
RMS offset      : 0.000053422 seconds
Frequency       : 2.286 ppm slow
Residual freq   : -0.000 ppm
Skew            : 0.012 ppm
Root delay      : 0.012706812 seconds
Root dispersion : 0.000430042 seconds
Update interval : 1029.8 seconds
Leap status     : Normal

If the command returns the following results, it means that there is an error in the synchronization process:

Leap status    : Not synchronised

If the command returns the following results, it means that the Chrony service is not running properly:

506 Cannot talk to daemon

If you want the NTP service to start synchronizing as soon as possible, execute the following command. You can replace pool.ntp.org with your NTP server:

sudo systemctl stop ntpd.service && \
sudo ntpdate pool.ntp.org && \
sudo systemctl start ntpd.service

If you want to manually install the NTP service on the CentOS 7 system, you can execute the following command:

sudo yum install ntp ntpdate && \
sudo systemctl start ntpd.service && \
sudo systemctl enable ntpd.service

5. Check and configure operating system optimization parameters

In the TiDB of the production system, it is recommended to optimize the configuration of the operating system as follows:

• Turn off transparent huge pages (that is, Transparent Huge Pages, abbreviated as THP). Database memory access patterns tend to be sparse rather than sequential. When high-order memory fragmentation is severe, there is a high latency in allocating THP pages.
• Set the storage medium's I/O scheduler to noop. For high-speed SSD storage media, the I/O scheduling operation of the kernel will cause performance loss. After the scheduler is set to noop, the kernel does not do anything, and directly sends I/O requests to the hardware to obtain better performance. At the same time, the noop scheduler also has good universality.
• Select performance mode for the cpufreq module that adjusts the CPU frequency. The best performance can be obtained by fixing the CPU frequency at the highest supported operating frequency without dynamic adjustment.

Use the following steps to check the current configuration of the operating system and configure system optimization parameters:

1. Execute the following command to view the enabled status of transparent huge pages.

cat /sys/kernel/mm/transparent_hugepage/enabled

[always] madvise never

Note that
[always] madvise never means that transparent huge pages are enabled and need to be turned off.

2. Execute the following command to view the I/O scheduler of the disk where the data directory is located. Assume that data directories are created on two disks, sdb and sdc.

cat /sys/block/sd[bc]/queue/scheduler

noop [deadline] cfq
noop [deadline] cfq

Note that
noop [deadline] cfq means that the disk I/O scheduler uses deadline and needs to be modified.

Run the following command to view the unique ID_SERIAL of the disk.

udevadm info --name=/dev/sdb | grep ID_SERIAL

E: ID_SERIAL=36d0946606d79f90025f3e09a0c1f9e81
E: ID_SERIAL_SHORT=6d0946606d79f90025f3e09a0c1f9e81

Note
If data directories are assigned to multiple disks, you need to execute the above command multiple times to record the unique identifiers of all disks.

4. Execute the following command to view the energy-saving strategy selected by the cpufreq module.

cpupower frequency-info --policy

analyzing CPU 0:
current policy: frequency should be within 1.20 GHz and 3.10 GHz.
              The governor "powersave" may decide which speed to use within this range.

Note that
The governor “powersave” indicates that the energy-saving strategy of cpufreq uses powersave, which needs to be adjusted to the performance strategy. If it is a virtual machine or cloud host, no adjustment is required, and the command output is usually Unable to determine current policy.

5. Configure system optimization parameters
Use tuned (recommended)
a. Execute the tuned-adm list command to view the tuned policy of the current operating system.

tuned-adm list

Available profiles:
- balanced                    - General non-specialized tuned profile
- desktop                     - Optimize for the desktop use-case
- hpc-compute                 - Optimize for HPC compute workloads
- latency-performance         - Optimize for deterministic performance at the cost of increased power consumption
- network-latency             - Optimize for deterministic performance at the cost of increased power consumption, focused on low latency network performance
- network-throughput          - Optimize for streaming network throughput, generally only necessary on older CPUs or 40G+ networks
- powersave                   - Optimize for low power consumption
- throughput-performance      - Broadly applicable tuning that provides excellent performance across a variety of common server workloads
- virtual-guest               - Optimize for running inside a virtual guest
- virtual-host                - Optimize for running KVM guests
Current active profile: balanced

Current active profile: balanced means that the tuned policy of the current operating system uses balanced, and it is recommended to add the optimized configuration of the operating system on the basis of the current policy.

b. Create a new tuned policy.

mkdir /etc/tuned/balanced-tidb-optimal/
vi /etc/tuned/balanced-tidb-optimal/tuned.conf

[main]
include=balanced

[cpu]
governor=performance

[vm]
transparent_hugepages=never

[disk]
devices_udev_regex=(ID_SERIAL=36d0946606d79f90025f3e09a0c1fc035)|(ID_SERIAL=36d0946606d79f90025f3e09a0c1f9e81)
elevator=noop

include=balanced means to add operating system optimization configuration on the basis of the existing balanced policy.

c. Apply the new tuned policy.

tuned-adm profile balanced-tidb-optimal

6. Execute the following command to verify the status of the transparent huge page

cat /sys/kernel/mm/transparent_hugepage/enabled

always madvise [never]

7. Execute the following command to verify the I/O scheduler of the disk where the data directory is located

cat /sys/block/sd[bc]/queue/scheduler

[noop] deadline cfq
[noop] deadline cfq

8. Execute the following command to view the energy saving strategy selected by the cpufreq module

cpupower frequency-info --policy

analyzing CPU 0:
current policy: frequency should be within 1.20 GHz and 3.10 GHz.
              The governor "performance" may decide which speed to use within this range.

9. Execute the following command to modify sysctl parameters

echo "fs.file-max = 1000000">> /etc/sysctl.conf
echo "net.core.somaxconn = 32768">> /etc/sysctl.conf
echo "net.ipv4.tcp_tw_recycle = 0">> /etc/sysctl.conf
echo "net.ipv4.tcp_syncookies = 0">> /etc/sysctl.conf
echo "vm.overcommit_memory = 1">> /etc/sysctl.conf
sysctl -p

10. Execute the following command to configure the user's limits.conf file

cat << EOF >>/etc/security/limits.conf
tidb           soft    nofile          1000000
tidb           hard    nofile          1000000
tidb           soft    stack          32768
tidb           hard    stack          32768
EOF

3. Manually configure SSH mutual trust and sudo without password

For the scenario where there is a need to manually configure the mutual trust between the central control computer and the target node, you can refer to this paragraph. It is generally recommended to use the TiUP deployment tool to automatically configure SSH mutual trust and password-free login, and this section can be ignored.

1. Log in to the deployment target machine as the root user to create a tidb user and set the login password

useradd tidb && \
passwd tidb

2. Execute the following command, add tidb ALL=(ALL) NOPASSWD: ALL to the end of the file, that is, configure sudo without password

visudo

tidb ALL=(ALL) NOPASSWD: ALL

3. Log in to the central control machine as the tidb user and execute the following commands. Replace 10.0.1.1 with the IP of your deployment target machine, and enter the tidb user password of the deployment target machine as prompted. After the execution is successful, SSH mutual trust is established, and the same is true for other machines. There is no .ssh directory under the newly created tidb user, you need to execute the command to generate the rsa key to generate the .ssh directory. If you want to deploy TiDB components on the central control computer, you need to configure mutual trust between the central control computer and the central control computer itself.

ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub 10.0.1.1

4. Log in to the central control machine as the tidb user, and log in to the IP of the target machine through ssh. If you do not need to enter a password and the login is successful, it means that the SSH mutual trust configuration is successful.

ssh 10.0.1.1

[tidb@10.0.1.1 ~]$

5. After logging in to the deployment target machine as the tidb user, execute the following command without entering a password and switch to the root user, indicating that the tidb user sudo password-free configuration is successful.

sudo -su root

[[email protected] tidb]#

Fourth, install the numactl tool

This section mainly introduces how to install NUMA tools. In the production environment, because the hardware machine configuration is often higher than the demand, in order to plan resources more reasonably, we will consider deploying TiDB or TiKV on a single machine with multiple instances. The use of NUMA core binding tools is mainly to prevent CPU resource competition and cause performance degradation.

Notice:

• NUMA core binding is a method for isolating CPU resources and is suitable for deploying multiple instances in a high-configuration physical machine environment.
• After the deployment operation is completed through tiup cluster deploy, cluster-level management can be performed through the exec command.

There are two ways to install NUMA tools:

Method 1: Log in to the target node to install (take CentOS Linux release 7.7.1908 (Core) as an example).

sudo yum -y install numactl

Method 2: Install NUMA in batches on the cluster through tiup cluster exec.

Use TiUP to install the TiDB cluster, and complete the deployment of the tidb-test cluster by referring to Deploying a TiDB Cluster Using TiUP. If you already have a local cluster, you can skip this step.

tiup cluster deploy tidb-test v6.1.0 ./topology.yaml --user root [-p] [-i /home/root/.ssh/gcp_rsa]

Execute the tiup cluster exec command to install NUMA on all target hosts in the tidb-test cluster with sudo authority.

tiup cluster exec tidb-test --sudo --command "yum -y install numactl"