satlstack之return与job管理
1. return组件
return组件可以理解为SaltStack系统对执行Minion返回后的数据进行存储或者返回给其他程序，它支持多种存储方式，比如用MySQL、MongoDB、Redis、Memcache等，通过return我们可以对SaltStack的每次操作进行记录，对以后日志审计提供了数据来源。目前官方已经支持30种return数据存储与接口，我们可以很方便的配置与使用它。当然也支持自己定义的return，自定义的return需由python来编写。在选择和配置好要使用的return后，只需在salt命令后面指定return即可。
1.1 在minion上查看yum源里有Mysql-python模块
[root@node02 ~]# yum list all |grep -i mysql
MySQL-python.x86_64 1.2.5-1.el7 myrepo
akonadi-mysql.x86_64 1.9.2-4.el7 myrepo
dovecot-mysql.x86_64 1:2.2.10-8.el7 myrepo
libdbi-dbd-mysql.x86_64 0.8.3-16.el7 myrepo
mysql-connector-java.noarch 1:5.1.25-3.el7 myrepo
mysql-connector-odbc.x86_64 5.2.5-6.el7 myrepo
pcp-pmda-mysql.x86_64 3.11.8-7.el7 myrepo
perl-DBD-MySQL.x86_64 4.023-5.el7 myrepo
php-mysql.x86_64 5.4.16-42.el7 myrepo
qt-mysql.i686 1:4.8.5-13.el7 myrepo
qt-mysql.x86_64 1:4.8.5-13.el7 myrepo
qt3-MySQL.i686 3.3.8b-51.el7 myrepo
qt3-MySQL.x86_64 3.3.8b-51.el7 myrepo
qt5-qtbase-mysql.i686 5.6.2-1.el7 myrepo
qt5-qtbase-mysql.x86_64 5.6.2-1.el7 myrepo
rsyslog-mysql.x86_64 8.24.0-12.el7 myrepo
1.2 然后在minion上安装Mysql-python模块
[root@node01 ~]# salt '*' cmd.run 'rpm -qa|grep MySQL-python'
node02:
MySQL-python-1.2.5-1.el7.x86_64
node01:
MySQL-python-1.2.5-1.el7.x86_64
1.3 查看minion上的MYSQL-python是否安装完成
[root@node01 ~]# salt '*' cmd.run 'rpm -qa|grep MySQL-python'
node02:
MySQL-python-1.2.5-1.el7.x86_64
node01:
MySQL-python-1.2.5-1.el7.x86_64
1.4 使用新的服务器部署mysql来用作存储服务器
[root@mysql yum.repos.d]# yum -y install mariadb mariadb-server
开启mysql服务
[root@mysql ~]# systemctl restart mariadb
[root@mysql ~]# systemctl enable mariadb
Created symlink from /etc/systemd/system/multi-user.target.wants/mariadb.service to /usr/lib/systemd/system/mariadb.service.
[root@mysql ~]# systemctl status mariadb
● mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: active (running) since 五 2020-02-28 20:47:18 CST; 16s ago
Main PID: 2803 (mysqld_safe)
CGroup: /system.slice/mariadb.service
├─2803 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
└─2965 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugi...
Hint: Some lines were ellipsized, use -l to show in full.
[root@mysql ~]# ss -antl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 50 *:3306 *:*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
进入mysql,创建数据库和表结构
[root@mysql ~]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 5
Server version: 5.5.56-MariaDB MariaDB Server
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> CREATE DATABASE `salt` DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> use 'salt';
Database changed
MariaDB [salt]> CREATE TABLE `jids` (
-> `jid` varchar(255) NOT NULL,
-> `load` mediumtext NOT NULL,
-> UNIQUE KEY `jid` (`jid`)
-> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.00 sec)
MariaDB [salt]> CREATE TABLE `salt_returns` (
-> `fun` varchar(50) NOT NULL,
-> `jid` varchar(255) NOT NULL,
-> `return` mediumtext NOT NULL,
-> `id` varchar(255) NOT NULL,
-> `success` varchar(10) NOT NULL,
-> `full_ret` mediumtext NOT NULL,
-> `alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-> KEY `id` (`id`),
-> KEY `jid` (`jid`),
-> KEY `fun` (`fun`)
-> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.00 sec)
MariaDB [salt]> CREATE TABLE `salt_events` (
-> `id` BIGINT NOT NULL AUTO_INCREMENT,
-> `tag` varchar(255) NOT NULL,
-> `data` mediumtext NOT NULL,
-> `alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-> `master_id` varchar(255) NOT NULL,
-> PRIMARY KEY (`id`),
-> KEY `tag` (`tag`)
-> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.01 sec)
MariaDB [salt]> show tables;
+----------------+
| Tables_in_salt |
+----------------+
| jids |
| salt_events |
| salt_returns |
+----------------+
3 rows in set (0.00 sec)
MariaDB [salt]> grant all on salt.* to salt@'%' identified by 'salt'; //授权访问
Query OK, 0 rows affected (0.00 sec)
MariaDB [salt]> flush privileges;
Query OK, 0 rows affected (0.00 sec)
MariaDB [salt]> quit
Bye
1.5 在minion上安装mariadb使用mysql命令来测试远程的mysql服务器是否可以连接
[root@node01 ~]# salt '*' pkg.install mariadb
node02:
----------
mariadb:
----------
new:
1:5.5.56-2.el7
old:
perl:
----------
new:
4:5.16.3-292.el7
old:
perl-Carp:
----------
new:
1.26-244.el7
old:
perl-Encode:
----------
new:
2.51-7.el7
old:
perl-Exporter:
----------
[root@node02 ~]# mysql -usalt -psalt -h192.168.100.130
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 6
Server version: 5.5.56-MariaDB MariaDB Server
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]>
MariaDB [(none)]>
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| salt |
| test |
+--------------------+
3 rows in set (0.00 sec)
MariaDB [(none)]>
1.6 配置minion
现在其中一台的minion上配置
[root@node01 ~]# vi /etc/salt/minion
[root@node01 ~]# head -893 /etc/salt/minion|tail -6
#return: mysql
mysql.host: '192.168.100.130'
mysql.user: 'salt'
mysql.pass: 'salt'
mysql.db: 'salt'
mysql.port: 3306
将配置好的配置文件复制到另一台minion上
[root@node01 ~]# salt-cp 'node02' /etc/salt/minion /etc/salt/
node02:
----------
/etc/salt/minion:
True
[root@node01 ~]# salt 'node02' cmd.run 'grep mysql /etc/salt/minion'
node02:
#return: mysql
mysql.user: 'salt'
mysql.pass: 'salt'
mysql.db: 'salt'
mysql.port: 3306
#return: mysql,slack,redis
# - mysql
1.7 重启minion上的minion服务
[root@node01 ~]# salt '*' service.restart salt-minion
[root@node01 ~]# salt '*' test.ping
node01:
True
node02:
True
1.8 在 Maste上测试存储到nysql中
[root@node01 ~]# salt '*' test.ping --return mysql
node01:
True
node02:
True
1.9 在数据库中查询数据已经成功写入进去
[root@mysql ~]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 19
Server version: 5.5.56-MariaDB MariaDB Server
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> use salt;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [salt]> show tables;
+----------------+
| Tables_in_salt |
+----------------+
| jids |
| salt_events |
| salt_returns |
+----------------+
MariaDB [salt]> select * from salt_returns\G;
*************************** 1. row ***************************
fun: test.ping
jid: 20200228214856929152
return: true
id: node02
success: 1
full_ret: {"fun_args": [], "jid": "20200228214856929152", "return": true, "retcode": 0, "success": true, "fun": "test.ping", "id": "node02"}
alter_time: 2020-02-28 21:48:57
*************************** 2. row ***************************
fun: test.ping
jid: 20200228214856929152
return: true
id: node01
success: 1
full_ret: {"fun_args": [], "jid": "20200228214856929152", "return": true, "retcode": 0, "success": true, "fun": "test.ping", "id": "node01"}
alter_time: 2020-02-28 21:48:57
2.job cache 组件
- return时是由Minion直接与存储服务器进行交互，因此需要在每台Minion上安装指定的存储方式的模块。
- job cache 可以直接在Master上就把返回的结果给存储到存储服务器 。意思是当Minion将结果返回给Master后，由Master将结果给缓存在本地，然后将缓存的结果给存储到指定的存储服务器，比如存储到mysql中。
2.1 开启master端的master_job_cache
[root@node01 ~]# vim /etc/salt/master
[root@node01 ~]# head -140 /etc/salt/master|tail -6
master_job_cache: mysql
mysql.host: '192.168.100.130'
mysql.user: 'salt'
mysql.pass: 'salt'
mysql.db: 'salt'
mysql.port: 3306
2.2在数据库服务器中清空之前生成的表内容
[root@mysql ~]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 22
Server version: 5.5.56-MariaDB MariaDB Server
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> delete from salt.salt_returns;
Query OK, 6 rows affected (0.01 sec)
MariaDB [(none)]> select '*' from salt.salt_returns;
Empty set (0.00 sec)
2.3 在master上再次测试是否能够够存储到数据库
[root@node01 ~]# salt '*' cmd.run 'ls /root'
node01:
123.sql
2019.sql
a
anaconda-ks.cfg
haha
kkkk
lala
ll
pass
node02:
a
anaconda-ks.cfg
haha
在数据库中查询
[root@mysql ~]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 28
Server version: 5.5.56-MariaDB MariaDB Server
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> use salt;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [salt]> select * from salt.salt_returns\G
*************************** 1. row ***************************
fun: cmd.run
jid: 20200229183605615414
return: "123.sql\n2019.sql\na\nanaconda-ks.cfg\nhaha\nkkkk\nlala\nll\npass"
id: node01
success: 1
full_ret: {"fun_args": ["ls /root"], "jid": "20200229183605615414", "return": "123.sql\n2019.sql\na\nanaconda-ks.cfg\nhaha\nkkkk\nlala\nll\npass", "retcode": 0, "success": true, "cmd": "_return", "_stamp": "2020-02-29T10:36:05.744654", "fun": "cmd.run", "id": "node01"}
alter_time: 2020-02-29 18:36:05
*************************** 2. row ***************************
fun: cmd.run
jid: 20200229183629089972
return: "123.sql\n2019.sql\na\nanaconda-ks.cfg\nhaha\nkkkk\nlala\nll\npass"
id: node01
success: 1
full_ret: {"fun_args": ["ls /root"], "jid": "20200229183629089972", "return": "123.sql\n2019.sql\na\nanaconda-ks.cfg\nhaha\nkkkk\nlala\nll\npass", "retcode": 0, "success": true, "cmd": "_return", "_stamp": "2020-02-29T10:36:29.199421", "fun": "cmd.run", "id": "node01"}
alter_time: 2020-02-29 18:36:29
*************************** 3. row ***************************
fun: cmd.run
jid: 20200229183629089972
return: "a\nanaconda-ks.cfg\nhaha"
id: node02
success: 1
full_ret: {"fun_args": ["ls /root"], "jid": "20200229183629089972", "return": "a\nanaconda-ks.cfg\nhaha", "retcode": 0, "success": true, "cmd": "_return", "_stamp": "2020-02-29T10:36:29.255025", "fun": "cmd.run", "id": "node02"}
alter_time: 2020-02-29 18:36:29
3 rows in set (0.00 sec)
2.4 job管理
获取任务的jid
[root@node01 ~]# salt '*' cmd.run 'uptime' -v
Executing job with jid 20200229183929818990
-------------------------------------------
node01:
18:39:29 up 9:14, 2 users, load average: 0.02, 0.08, 0.07
node02:
18:39:29 up 9:18, 2 users, load average: 0.00, 0.01, 0.05
通过jid来获取执行命令的返回结果，这里使用之前ls /root的jid来查询返回结果
[root@node01 ~]# salt-run jobs.lookup_jid 20200229183629089972
node01:
123.sql
2019.sql
a
anaconda-ks.cfg
haha
kkkk
lala
ll
pass
node02:
a
anaconda-ks.cfg
haha
