主机之间能够相互解析
[root@tidb1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.137.161 tidb1
192.168.137.162 tidb2
192.168.137.163 tidb3
192.168.137.164 tidb4
192.168.137.165 tidb5
192.168.137.166 tidb6
useradd tidb
passwd tidb
只需要在中控机创建用户即可
[root@tidb1 tidb-ansible]# cd /etc/yum.repos.d/
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
[root@tidb1 ~]# yum install -y ansible
vim /etc/sudoers 添加如下内容 ：
# visudo
tidb ALL=(ALL) NOPASSWD: ALL
su - tidb
ssh-keygen -t rsa
[tidb@tidb1 ~]$ git clone -b release-2.0 https://github.com/pingcap/tidb-ansible.git
cd tidb-ansible
[root@tidb1 ~]# cd /home/tidb/
[root@tidb1 ~]# chmod -R 777 tidb-ansible
参考文档
https://github.com/pingcap/docs-cn/blob/master/op-guide/ansible-deployment.md#如何使用-ansible-部署-ntp-服务
如何使用 Ansible 部署 NTP 服务
参照在中控机器上下载 TiDB-Ansible下载 TiDB-Ansible，将你的部署目标机器 IP 添加到 [servers] 区块下，ntp_server 变量的值 pool.ntp.org 可替换为其他 NTP server，在启动 NTP 服务前，系统会 ntpdate 该 NTP server，Ansible 安装的 NTP 服务使用安装包默认 server 列表，见配置文件 cat /etc/ntp.conf 中 server 参数。
$ vi hosts.ini
[servers]
192.168.137.161
192.168.137.162
192.168.137.163
192.168.137.164
192.168.137.165
192.168.137.166
[all:vars]
username = tidb
ntp_server = pool.ntp.org
执行以下命令，按提示输入部署目标机器 root 密码。
$ ansible-playbook -i hosts.ini deploy_ntp.yml -k 该命令中控机在root和 tidb 用户均可
$ansible-playbook -i hosts.ini create_users.yml -k 该命令只能中控机 tidb 用户执行
ssh localhost
ssh tidb1
ssh tidb2
ssh tidb3
ssh tidb4
ssh tidb5
ssh tidb6
[root@tidb1 tidb-ansible]# cat inventory.ini
## TiDB Cluster Part
[tidb_servers]
192.168.137.161
192.168.137.162
[tikv_servers]
192.168.137.164
192.168.137.165
192.168.137.166
[pd_servers]
192.168.137.161
192.168.137.162
192.168.137.163
[spark_master]
192.168.137.164
[spark_slaves]
192.168.137.165
192.168.137.166
## Monitoring Part
# prometheus and pushgateway servers
[monitoring_servers]
192.168.137.161
[grafana_servers]
192.168.137.161
# node_exporter and blackbox_exporter servers
[monitored_servers]
192.168.137.161
192.168.137.162
192.168.137.163
192.168.137.164
192.168.137.165
192.168.137.166
[alertmanager_servers]
192.168.137.161
## Binlog Part
[pump_servers:children]
tidb_servers
## Group variables
[pd_servers:vars]
# location_labels = ["zone","rack","host"]
## Global variables
[all:vars]
deploy_dir = /home/tidb/deploy
## Connection
# ssh via normal user
#ansible_user = tidb
# ssh via root:
ansible_user = root
ansible_become = true
ansible_become_user = tidb
cluster_name = test-cluster
tidb_version = v2.0.0-rc.3
# deployment methods, [binary, docker]
deployment_method = binary
# process supervision, [systemd, supervise]
process_supervision = systemd
# timezone of deployment region
timezone = Asia/Shanghai
set_timezone = True
enable_firewalld = False
# check NTP service
enable_ntpd = True
set_hostname = False
## binlog trigger
enable_binlog = False
# zookeeper address of kafka cluster, example:
# zookeeper_addrs = "192.168.0.11:2181,192.168.0.12:2181,192.168.0.13:2181"
zookeeper_addrs = ""
# store slow query log into seperate file
enable_slow_query_log = False
# enable TLS authentication in the TiDB cluster
enable_tls = False
# KV mode
deploy_without_tidb = False
# Optional. Set if you already have a alertmanager server.
# Format: alertmanager_host:alertmanager_port
alertmanager_target = ""
使用local_prepare.yml,联网下载 tidb binary 到中控机
ansible-playbook local_prepare.yml 该命令只能中控机 tidb 用户执行
初始话系统环境，修改内核参数
如果服务用户尚未建立，此初始化操作会自动创建该用户
ansible-playbook bootstrap.yml 该命令只能中控机 tidb 用户执行
如果ansible 使用root用户远程连接需要密码,使用 -k 参数 ，执行playbook ,同理：
ansible-playbook bootstrap.yml 该命令只能中控机 tidb 用户执行
因为是在 虚拟机的测试环境中搞的，
执行 ansible-playbook bootstrap.yml 的过程中 ，会报如下的错误
[tidb_servers]: Ansible UNREACHABLE! => playbook: bootstrap.yml; TASK: pre-ansible : disk space check - fail when disk is full; message: {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname tidb_servers: Name or service not known\r\n", "unreachable": true}
[tikv_servers]: Ansible UNREACHABLE! => playbook: bootstrap.yml; TASK: pre-ansible : disk space check - fail when disk is full; message: {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname tikv_servers: Name or service not known\r\n", "unreachable": true}
[pd_servers]: Ansible UNREACHABLE! => playbook: bootstrap.yml; TASK: pre-ansible : disk space check - fail when disk is full; message: {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname pd_servers: Name or service not known\r\n", "unreachable": true}
[spark_master]: Ansible UNREACHABLE! => playbook: bootstrap.yml; TASK: pre-ansible : disk space check - fail when disk is full; message: {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname spark_master: Name or service not known\r\n", "unreachable": true}
[spark_slaves]: Ansible UNREACHABLE! => playbook: bootstrap.yml; TASK: pre-ansible : disk space check - fail when disk is full; message: {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname spark_slaves: Name or service not known\r\n", "unreachable": true}
[192.168.137.161]: Ansible FAILED! => playbook: bootstrap.yml; TASK: check_system_optional : Preflight check - Check TiDB server's CPU; message: {"changed": false, "msg": "This machine does not have sufficient CPU to run TiDB, at least 8 cores."}
[192.168.137.162]: Ansible FAILED! => playbook: bootstrap.yml; TASK: check_system_optional : Preflight check - Check TiDB server's CPU; message: {"changed": false, "msg": "This machine does not have sufficient CPU to run TiDB, at least 8 cores."}
TASK [check_system_optional : Preflight check - Check TiDB server's CPU] ************************************************************************
fatal: [192.168.137.162]: FAILED! => {"changed": false, "msg": "This machine does not have sufficient CPU to run TiDB, at least 8 cores."}
NO MORE HOSTS LEFT ******************************************************************************************************************************
to retry, use: --limit @/root/tidb-ansible/retry_files/bootstrap.retry
PLAY RECAP **************************************************************************************************************************************
192.168.137.161 : ok=0 changed=0 unreachable=1 failed=0
192.168.137.162 : ok=30 changed=8 unreachable=0 failed=1
192.168.137.163 : ok=30 changed=8 unreachable=0 failed=0
192.168.137.164 : ok=33 changed=10 unreachable=0 failed=0
192.168.137.165 : ok=33 changed=10 unreachable=0 failed=0
192.168.137.166 : ok=33 changed=10 unreachable=0 failed=0
localhost : ok=1 changed=0 unreachable=0 failed=0
ERROR MESSAGE SUMMARY ***************************************************************************************************************************
[192.168.137.161]: Ansible UNREACHABLE! => playbook: bootstrap.yml; TASK: pre-ansible : disk space check - fail when disk is full; message: {"changed": false, "msg": "Failed to connect to the host via ssh: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r\n", "unreachable": true}
[192.168.137.162]: Ansible FAILED! => playbook: bootstrap.yml; TASK: check_system_optional : Preflight check - Check TiDB server's CPU; message: {"changed": false, "msg": "This machine does not have sufficient CPU to run TiDB, at least 8 cores."}
ERROR MESSAGE SUMMARY ***************************************************************************************************************************
[192.168.137.161]: Ansible UNREACHABLE! => playbook: bootstrap.yml; TASK: pre-ansible : disk space check - fail when disk is full; message: {"changed": false, "msg": "Failed to connect to the host via ssh: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r\n", "unreachable": true}
vim bootstrap.yml 修改该文件 并注释掉以下的内容
- name: check system
hosts: all
any_errors_fatal: true
roles:
- check_system_necessary
# - { role: check_system_optional, when: not dev_mode }
如果是非SSD 测试的话 ，最好将如下的内容注释掉
- name: tikv_servers machine benchmark
hosts: tikv_servers
gather_facts: false
roles:
# - { role: machine_benchmark, when: not dev_mode }
部署 TIDB 集群软件
ansible-playbook deploy.yml 该命令只能中控机 tidb 用户执行
启动TIDB 集群
ansible-playbook start.yml 该命令只能中控机 tidb 用户执行
关闭整个集群
ansible-playbook stop.yml 该命令只能中控机 tidb 用户执行
测试集群
测试连接 TiDB 集群，推荐在 TiDB 前配置负载均衡来对外统一提供 SQL 接口。
使用 MySQL 客户端连接测试，TCP 4000 端口是 TiDB 服务默认端口。
netstat -anp | grep 4000
mysql -u root -h 192.168.137.161 -P 4000
[root@tidb1 ~]# mysql -h 127.0.0.1 -uroot -P 4000 -D mysql
bash: mysql: command not found.
rpm -ivh mysql57-community-release-el7-11.noarch.rpm
yum install -y mysql-community-client.x86_64
mysql -h 127.0.0.1 -uroot -P 4000
mysql -h 127.0.0.1 -uroot -P 4000 -D mysql
mysql -h 192.168.137.161 -uroot -P 4000
mysql -h 192.168.137.161 -uroot -P 4000 -D mysql
[root@tidb1 ~]# mysql -h 127.0.0.1 -uroot -P 4000
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.7.10-TiDB-v2.0.0-rc.3 MySQL Community Server (Apache License 2.0)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| INFORMATION_SCHEMA |
| PERFORMANCE_SCHEMA |
| mysql |
| test |
+--------------------+
4 rows in set (0.00 sec)
[root@tidb1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.137.161 tidb1
192.168.137.162 tidb2
192.168.137.163 tidb3
192.168.137.164 tidb4
192.168.137.165 tidb5
192.168.137.166 tidb6
useradd tidb
passwd tidb
只需要在中控机创建用户即可
[root@tidb1 tidb-ansible]# cd /etc/yum.repos.d/
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
[root@tidb1 ~]# yum install -y ansible
vim /etc/sudoers 添加如下内容 ：
# visudo
tidb ALL=(ALL) NOPASSWD: ALL
su - tidb
ssh-keygen -t rsa
[tidb@tidb1 ~]$ git clone -b release-2.0 https://github.com/pingcap/tidb-ansible.git
cd tidb-ansible
[root@tidb1 ~]# cd /home/tidb/
[root@tidb1 ~]# chmod -R 777 tidb-ansible
参考文档
https://github.com/pingcap/docs-cn/blob/master/op-guide/ansible-deployment.md#如何使用-ansible-部署-ntp-服务
如何使用 Ansible 部署 NTP 服务
参照在中控机器上下载 TiDB-Ansible下载 TiDB-Ansible，将你的部署目标机器 IP 添加到 [servers] 区块下，ntp_server 变量的值 pool.ntp.org 可替换为其他 NTP server，在启动 NTP 服务前，系统会 ntpdate 该 NTP server，Ansible 安装的 NTP 服务使用安装包默认 server 列表，见配置文件 cat /etc/ntp.conf 中 server 参数。
$ vi hosts.ini
[servers]
192.168.137.161
192.168.137.162
192.168.137.163
192.168.137.164
192.168.137.165
192.168.137.166
[all:vars]
username = tidb
ntp_server = pool.ntp.org
执行以下命令，按提示输入部署目标机器 root 密码。
$ ansible-playbook -i hosts.ini deploy_ntp.yml -k 该命令中控机在root和 tidb 用户均可
$ansible-playbook -i hosts.ini create_users.yml -k 该命令只能中控机 tidb 用户执行
ssh localhost
ssh tidb1
ssh tidb2
ssh tidb3
ssh tidb4
ssh tidb5
ssh tidb6
[root@tidb1 tidb-ansible]# cat inventory.ini
## TiDB Cluster Part
[tidb_servers]
192.168.137.161
192.168.137.162
[tikv_servers]
192.168.137.164
192.168.137.165
192.168.137.166
[pd_servers]
192.168.137.161
192.168.137.162
192.168.137.163
[spark_master]
192.168.137.164
[spark_slaves]
192.168.137.165
192.168.137.166
## Monitoring Part
# prometheus and pushgateway servers
[monitoring_servers]
192.168.137.161
[grafana_servers]
192.168.137.161
# node_exporter and blackbox_exporter servers
[monitored_servers]
192.168.137.161
192.168.137.162
192.168.137.163
192.168.137.164
192.168.137.165
192.168.137.166
[alertmanager_servers]
192.168.137.161
## Binlog Part
[pump_servers:children]
tidb_servers
## Group variables
[pd_servers:vars]
# location_labels = ["zone","rack","host"]
## Global variables
[all:vars]
deploy_dir = /home/tidb/deploy
## Connection
# ssh via normal user
#ansible_user = tidb
# ssh via root:
ansible_user = root
ansible_become = true
ansible_become_user = tidb
cluster_name = test-cluster
tidb_version = v2.0.0-rc.3
# deployment methods, [binary, docker]
deployment_method = binary
# process supervision, [systemd, supervise]
process_supervision = systemd
# timezone of deployment region
timezone = Asia/Shanghai
set_timezone = True
enable_firewalld = False
# check NTP service
enable_ntpd = True
set_hostname = False
## binlog trigger
enable_binlog = False
# zookeeper address of kafka cluster, example:
# zookeeper_addrs = "192.168.0.11:2181,192.168.0.12:2181,192.168.0.13:2181"
zookeeper_addrs = ""
# store slow query log into seperate file
enable_slow_query_log = False
# enable TLS authentication in the TiDB cluster
enable_tls = False
# KV mode
deploy_without_tidb = False
# Optional. Set if you already have a alertmanager server.
# Format: alertmanager_host:alertmanager_port
alertmanager_target = ""
使用local_prepare.yml,联网下载 tidb binary 到中控机
ansible-playbook local_prepare.yml 该命令只能中控机 tidb 用户执行
初始话系统环境，修改内核参数
如果服务用户尚未建立，此初始化操作会自动创建该用户
ansible-playbook bootstrap.yml 该命令只能中控机 tidb 用户执行
如果ansible 使用root用户远程连接需要密码,使用 -k 参数 ，执行playbook ,同理：
ansible-playbook bootstrap.yml 该命令只能中控机 tidb 用户执行
因为是在 虚拟机的测试环境中搞的，
执行 ansible-playbook bootstrap.yml 的过程中 ，会报如下的错误
[tidb_servers]: Ansible UNREACHABLE! => playbook: bootstrap.yml; TASK: pre-ansible : disk space check - fail when disk is full; message: {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname tidb_servers: Name or service not known\r\n", "unreachable": true}
[tikv_servers]: Ansible UNREACHABLE! => playbook: bootstrap.yml; TASK: pre-ansible : disk space check - fail when disk is full; message: {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname tikv_servers: Name or service not known\r\n", "unreachable": true}
[pd_servers]: Ansible UNREACHABLE! => playbook: bootstrap.yml; TASK: pre-ansible : disk space check - fail when disk is full; message: {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname pd_servers: Name or service not known\r\n", "unreachable": true}
[spark_master]: Ansible UNREACHABLE! => playbook: bootstrap.yml; TASK: pre-ansible : disk space check - fail when disk is full; message: {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname spark_master: Name or service not known\r\n", "unreachable": true}
[spark_slaves]: Ansible UNREACHABLE! => playbook: bootstrap.yml; TASK: pre-ansible : disk space check - fail when disk is full; message: {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname spark_slaves: Name or service not known\r\n", "unreachable": true}
[192.168.137.161]: Ansible FAILED! => playbook: bootstrap.yml; TASK: check_system_optional : Preflight check - Check TiDB server's CPU; message: {"changed": false, "msg": "This machine does not have sufficient CPU to run TiDB, at least 8 cores."}
[192.168.137.162]: Ansible FAILED! => playbook: bootstrap.yml; TASK: check_system_optional : Preflight check - Check TiDB server's CPU; message: {"changed": false, "msg": "This machine does not have sufficient CPU to run TiDB, at least 8 cores."}
TASK [check_system_optional : Preflight check - Check TiDB server's CPU] ************************************************************************
fatal: [192.168.137.162]: FAILED! => {"changed": false, "msg": "This machine does not have sufficient CPU to run TiDB, at least 8 cores."}
NO MORE HOSTS LEFT ******************************************************************************************************************************
to retry, use: --limit @/root/tidb-ansible/retry_files/bootstrap.retry
PLAY RECAP **************************************************************************************************************************************
192.168.137.161 : ok=0 changed=0 unreachable=1 failed=0
192.168.137.162 : ok=30 changed=8 unreachable=0 failed=1
192.168.137.163 : ok=30 changed=8 unreachable=0 failed=0
192.168.137.164 : ok=33 changed=10 unreachable=0 failed=0
192.168.137.165 : ok=33 changed=10 unreachable=0 failed=0
192.168.137.166 : ok=33 changed=10 unreachable=0 failed=0
localhost : ok=1 changed=0 unreachable=0 failed=0
ERROR MESSAGE SUMMARY ***************************************************************************************************************************
[192.168.137.161]: Ansible UNREACHABLE! => playbook: bootstrap.yml; TASK: pre-ansible : disk space check - fail when disk is full; message: {"changed": false, "msg": "Failed to connect to the host via ssh: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r\n", "unreachable": true}
[192.168.137.162]: Ansible FAILED! => playbook: bootstrap.yml; TASK: check_system_optional : Preflight check - Check TiDB server's CPU; message: {"changed": false, "msg": "This machine does not have sufficient CPU to run TiDB, at least 8 cores."}
ERROR MESSAGE SUMMARY ***************************************************************************************************************************
[192.168.137.161]: Ansible UNREACHABLE! => playbook: bootstrap.yml; TASK: pre-ansible : disk space check - fail when disk is full; message: {"changed": false, "msg": "Failed to connect to the host via ssh: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r\n", "unreachable": true}
vim bootstrap.yml 修改该文件 并注释掉以下的内容
- name: check system
hosts: all
any_errors_fatal: true
roles:
- check_system_necessary
# - { role: check_system_optional, when: not dev_mode }
如果是非SSD 测试的话 ，最好将如下的内容注释掉
- name: tikv_servers machine benchmark
hosts: tikv_servers
gather_facts: false
roles:
# - { role: machine_benchmark, when: not dev_mode }
部署 TIDB 集群软件
ansible-playbook deploy.yml 该命令只能中控机 tidb 用户执行
启动TIDB 集群
ansible-playbook start.yml 该命令只能中控机 tidb 用户执行
关闭整个集群
ansible-playbook stop.yml 该命令只能中控机 tidb 用户执行
测试集群
测试连接 TiDB 集群，推荐在 TiDB 前配置负载均衡来对外统一提供 SQL 接口。
使用 MySQL 客户端连接测试，TCP 4000 端口是 TiDB 服务默认端口。
netstat -anp | grep 4000
mysql -u root -h 192.168.137.161 -P 4000
[root@tidb1 ~]# mysql -h 127.0.0.1 -uroot -P 4000 -D mysql
bash: mysql: command not found.
rpm -ivh mysql57-community-release-el7-11.noarch.rpm
yum install -y mysql-community-client.x86_64
mysql -h 127.0.0.1 -uroot -P 4000
mysql -h 127.0.0.1 -uroot -P 4000 -D mysql
mysql -h 192.168.137.161 -uroot -P 4000
mysql -h 192.168.137.161 -uroot -P 4000 -D mysql
[root@tidb1 ~]# mysql -h 127.0.0.1 -uroot -P 4000
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.7.10-TiDB-v2.0.0-rc.3 MySQL Community Server (Apache License 2.0)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| INFORMATION_SCHEMA |
| PERFORMANCE_SCHEMA |
| mysql |
| test |
+--------------------+
4 rows in set (0.00 sec)