TiUP: A must-have tool for TiDBAer

TiUP is a daily essential tool for TiDBer, so this article is classified as a series of "Reviewing the Old and Knowing the New". If you are new to TiDB, please refer to this article first: "From Horse Carriage to Electric Vehicle, TiDB Deployment Tool Metamorphosis" .

Environmental description

The environment and component version information involved in this article are as follows:

TiDB v5.4.0

TiUP v1.9.3 (2022-03-24 Release)

CentOS 7.9

Introduction to TiUP

In the installation management of various system software and application software, the package manager has a wide range of applications, and the appearance of the package management tool greatly simplifies the installation, upgrade and maintenance of the software. For example, almost all Linux using RPM will use Yum for package management, and Anaconda can be very convenient to manage the python environment and related packages.

Starting from TiDB 4.0, TiUP, as a new tool, assumes the role of package manager and manages many components in the TiDB ecosystem, such as TiDB, PD, TiKV, etc. When users want to run any component in the TiDB ecosystem, they only need to execute one line of TiUP command, which greatly reduces the difficulty of management compared to before.

Figure 1 - tiup source code contributors

Figure 2 - tiup source line number statistics (2022-03-24)

TiUP has been released for more than two years. The version has been iterated several times, and the total amount of code has doubled. It can be seen from the above figure that the code update has slowed down, and TiDBer can be safely used in the production environment.

On the TiUP component again

As a Ti weapon, TiUP is a must for daily work. Let's discuss the classic components and common commands of TiUP again. First, the important commands of tiup are listed below, and then we will focus on the discussion.

• blow
  • main/cmd/root
    • tiup env
    • blow status
    • tiup mirror
    • tiup list --all --verbose
    • tiup install hello
    • blow update
    • tiup playground
    • tiup client
    • tiup cluster
    • tiup bench ch(CH-benCHmark)/TPCC(TPC-C)/TPCH(TPC-H)/YCSB(Yahoo! Cloud Serving Benchmark)
    • tiup dm
    • tiup clean

tiup mirror

不是每个公司都会把数据库部在公有云上的，即便是在公有云，为了便于版本统一管理，大都会选择自建一个仓库，是为生产库基准版本管理。而对于金融业务更不用多说，那么，如何在内网快速、简洁、有效地搭建和维护仓库，下面做个简单的示例。

首先需要在能连接外网的机器安装 tiup，并克隆官方 tiup 仓库：

• 下载 tiup 文件，并添加环境变量

mkdir -pv ~/.tiup/bin
wget https://tiup-mirrors.pingcap.com/tiup-linux-amd64.tar.gz
tar zxf tiup-linux-amd64.tar.gz -C ~/.tiup/bin/
echo 'export PATH=~/.tiup/bin:$PATH' >> ~/.bash_profile
source ~/.bash_profile
tiup -v
复制代码

输出的 tiup 版本信息：

1.9.3 tiup
Go Version: go1.17.7
Git Ref: v1.9.3
GitHash: f523cd5e051d0001e25d5a8b2c0d5d3ff058a5d5
复制代码

• 克隆官方库

先将仓库镜像指向官方库：

tiup mirror set https://tiup-mirrors.pingcap.com
# 屏幕输出日志> Successfully set mirror to https://tiup-mirrors.pingcap.com
复制代码

只克隆适合当前操作系统的最新版本，这里只需指定TiDB v5.4.0版本，其他组件会自动识别最新版本，并下载。

tiup mirror clone ~/.tiup/package -a amd64 -o linux v5.4.0
复制代码

• 将 package 文件夹打包复制到内网机器：

# current server
tar zcf package.tgz package/
# new server
cd ~/.tiup
tar zxvf package.tgz
./package/local_install.sh
source ~/.bash_profile
tiup list
复制代码

此时，新的本地仓已建好，创建一个hello组件进行测试：

# test mirror
CMP_TMP_DIR=`mktemp -d -p ~/.tiup`
cat > $CMP_TMP_DIR/hello.sh << EOF
#! /bin/sh
echo -e "\033[0;36m<<< Hello, TiDB! >>>\033[0m"
EOF
chmod 755 $CMP_TMP_DIR/hello.sh
tar -C $CMP_TMP_DIR -czf $CMP_TMP_DIR/hello.tar.gz hello.sh
复制代码

• 将hello组件发布到本地仓：

tiup mirror genkey
tiup mirror grant admin
tiup mirror publish hello v0.0.1 $CMP_TMP_DIR/hello.tar.gz hello.sh --desc 'Hello, TiDB'
复制代码

查看已发布的组件，并运行组件：

tiup list hello
tiup hello
复制代码

图 3-hello 组件运行输出

此时的本地仓以可以管理自发布组件，但仍无法对外提供服务，下面用tiup server一键搭建私有库：

# 运行tiup server
tiup server ~/.tiup/package --addr 0.0.0.0:8000 --upstream=""
# 修改镜像指向
tiup mirror set 'http://127.0.0.1:8000'
复制代码

注：由于版本差异，当前版本已不建议使用环境变量TIUP_MIRRORS，而是建议使用命令tiup mirror set <mirror-addr>。

tiup playground

对于分布式数据库来说，如何在本地快速搭建原型，以进行基本功能验证、测试，这是作为 DBA 的基本能力。于是，tiup playground应运而生，一键搭建最小可用集群，并可以指定初始 TiDB 每个组件启动的个数，以及扩缩容。

例如，启动一个标签为mydb1的集群，包含一个 TiDB 实例，一个 TiKV 实例，一个 PD 实例，一个 TiFlash 实例，不启动监控组件：

$ tiup playground v5.4.0 --host 127.0.0.1 --tag mydb1 --db 1 --kv 1 --pd 1 --tiflash 1 --without-monitor
127.0.0.1:4000 ... Done
127.0.0.1:3930 ... Done
CLUSTER START SUCCESSFULLY, Enjoy it ^-^
To connect TiDB: mysql --comments --host 127.0.0.1 --port 4000 -u root -p (no password)
To view the dashboard: http://127.0.0.1:2379/dashboard
PD client endpoints: [127.0.0.1:2379]
复制代码

查看各组件进程 id:

$ tiup playground display
Pid   Role     Uptime
---   ----     ------
4321  pd       10m39.092616075s
4322  tikv     10m39.087748551s
4353  tidb     10m37.765844216s
4527  tiflash  9m50.16054123s
复制代码

连接 tidb server，并查询版本：

$ mysql -uroot -h127.1 -P4000 -e 'select version()\G'
*************************** 1. row ***************************
version(): 5.7.25-TiDB-v5.4.0
复制代码

再例如，启动一个标签为mydb2的集群，只启动 TiKV 实例，和 3 个 PD 实例：

$ tiup playground v5.4.0 --host 127.0.0.1 --tag mydb2 --mode tikv-slim --pd 3 --without-monitor
Playground Bootstrapping...
Start pd instance:v5.4.0
Start pd instance:v5.4.0
Start pd instance:v5.4.0
Start tikv instance:v5.4.0
PD client endpoints: [127.0.0.1:2379 127.0.0.1:2382 127.0.0.1:2384]
复制代码

通过 PD API 查看当前 PD 有几个成员，以及 TiKV 实例信息：

$ curl -s http://127.0.0.1:2379/pd/api/v1/members | jq .members[].name
"pd-1"
"pd-0"
"pd-2"

$ curl -s http://127.0.0.1:2379/pd/api/v1/stores | jq .stores[].store
{
  "id": 1,
  "address": "127.0.0.1:20160",
  "version": "5.4.0",
  "status_address": "127.0.0.1:20180",
  "git_hash": "b5262299604df88711d9ed4b84d43e9c507749a2",
  "start_timestamp": 1648110516,
  "deploy_path": "/data/tiup/components/tikv/v5.4.0",
  "last_heartbeat": 1648112496884914000,
  "state_name": "Up"
}
复制代码

Misc

性能测试也是必要环节，所以看到 tiup 已经集成了 tpcc,tpch,ycsh,ch 四种测试工具集。可通过下面命令进行一键测试。

tiup bench ch/tpcc/tpch/ycsb
复制代码

一键清理数据的命令如下：

tiup clean --all
复制代码

这里需要强调的是，在生产环境需要慎重执行下面的指令，除非你知道你在做什么：

tiup cluster clean mydb3 --all
tiup cluster destroy mydb3
tiup cluster prune mydb3
复制代码

图 4-查看所有可用组件

其他组件另行讨论，或请先参阅官方文档。

TiUP v1.9.3 Release

2022-03-24，TiUP 发布了 v1.9.3 版本，从变更日志中我们可以清晰地了解到，本次更新修复了 5 个 bug，做了 2 点改进。

修复：

• 修复了主机名存在-时，tiup-cluster的exec子命令无法使用的错误。 (#1794, @nexustar)
• 修复了使用tiup-cluster命令时，TiFlash 实例的端口（服务端口、代理端口、代理状态端口）冲突检测问题。 (#1805, @AstroProfundis)
• 修复了下一代监控(ng-monitor)在 Prometheus 不可用的问题。 (#1806, @nexustar)
• 修复了在主机只部署 Prometheus 的情况下，node_exporter 度量指标无法收集的问题。（与上个问题一同修复。） (#1806, @nexustar)
• 修复了tiup-playground在指定--host 0.0.0.0时，无法工作的问题。 (#1811, @nexustar)

改进：

• 支持tiup-cluster和tiup-dm命令清理审计日志。(#1780, @srstack)

tiup cluster audit cleanup
tiup dm audit cleanup
复制代码

• 在 Grafana 配置模板中增加匿名登陆示例。（这里需要确保 DM 组件版本在 v1.9.0 之上 => tiup dm -v） (#1785@sunzhaoyang)

引申思考

In the cloud database era, or the distributed database era, how should the DBA role adjust itself? Do we still need a DBA that can only operate and maintain a certain database, such as traditional relational databases such as db2, oracle, mysql, and postgresql, or an advanced general? Do we need a business-oriented DBA who understands business and has development skills? In fact, these are not outdated now, and should not be abandoned, but should be used as basic skills to become the underlying modules of the DBA knowledge structure. DBA needs to use this as a basis to evolve to a higher stage, just like the experience of your predecessors, write After obtaining the database source code and understanding what front-end customers really need, we can develop and adjust a high-performance database suitable for business scenarios, as well as a set of easy-to-use and easy-to-manage database ecological tools.

Needless to say, tiup is a tool that fits this profile. One-click to build a private library, one-click to run the smallest cluster, one-click to manage the entire TiDB cluster, and one-click expansion and contraction of the TiDB cluster. However, there are functional trade-offs behind the seemingly simple. For example, it tiup mirrorcan only be operated on the command line, and there is no interface tool like Nexus that can publish and delete packages in the browser. For another example, the most commonly used scenario of tiup is still running on ordinary machines. There are TiDB Operatortools for the k8s environment, but there are few functions or cases for batch operation of ECS. In short, I hope that tiup will be more powerful under the premise of maintaining practicality.

This article uses the latest version of tiup as a demonstration. When writing this article, I have collected the public information of tiup, and try to de-duplicate and remove the content, hoping to bring readers the latest and useful articles.

May the opening of the book be beneficial, thank you!