airflow 安装配置celery+rabbitmq celery+redis

AirFlow的安装可以参考：https://www.cnblogs.com/braveym/p/11378851.html

这里介绍的是AirFlow 安装配置celery+rabbitmq   和   celery+redis

 

一、airflow 安装配置celery+rabbitmq

1、安装celery和rabbitmq组件

sudo pip install apache-airflow[celery]

sudo pip install apache-airflow[rabbitmq]

2、系统安装rabbit

安装 erlang

sudo yum install erlang

安装 rabbitmq

sudo wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.6.15/rabbitmq-server-3.6.15-1.el6.noarch.rpm
sudo rpm --import https://www.rabbitmq.com/rabbitmq-release-signing-key.asc
sudo yum install rabbitmq-server-3.6.15-1.el6.noarch.rpm

扫描二维码关注公众号，回复： 7068628 查看本文章

配置rabbitmq（设置用户名root，密码rabbitmq，创建虚拟主机airflow-rabbitmq）

[bigdata@bigdata airflow]$ su
Password: 
[root@bigdata airflow]# rabbitmqctl add_user root rabbitmq
Creating user "root"
[root@bigdata airflow]# rabbitmqctl add_vhost airflow-rabbitmq
Creating vhost "airflow-rabbitmq"
[root@bigdata airflow]# rabbitmqctl set_user_tags root airflow-rabbitmq
Setting tags for user "root" to ['airflow-rabbitmq']
[root@bigdata airflow]# rabbitmqctl set_permissions -p airflow-rabbitmq root ".*" ".*" ".*"
Setting permissions for user "root" in vhost "airflow-rabbitmq"
[root@bigdata airflow]# rabbitmq-plugins enable rabbitmq_management
The following plugins have been enabled:
  amqp_client
  cowlib
  cowboy
  rabbitmq_web_dispatch
  rabbitmq_management_agent
  rabbitmq_management

Applying plugin configuration to rabbit@bigdata... started 6 plugins.
[root@bigdata airflow]# 

修改airflow配置

executor = CeleryExecutor

broker_url = amqp://root:rabbitmq@localhost:5672/airflow-rabbitmq

result_backend = amqp://root:rabbitmq@localhost:5672/airflow-rabbitmq

 

 

 在启动worker 时，提示不能用root用户权限启动。修改方法：
    echo  "export C_FORCE_ROOT="True"" >> /etc/profile
    source /etc/profile

二、airflow 安装配置celery+redis

 

安装redis

sudo yum -y install redis

修改配置文件

sudo vim /etc/redis.conf
#bind 127.0.0.1     // 注释掉，使redis允许远程访问
requirepass 密码  // 修改这行，redis登录密码

开启redis服务

sudo systemctl start redis

python下安装redis库

sudo pip install redis

安装celery

sudo pip install celery

配置airflow.cfg

executor = CeleryExecutor
broker_url = redis://127.0.0.1:6379/0
result_backend = redis://127.0.0.1:6379/0