Elastalert alarm frame installation and use

Elastalert is an alarm framework written by Yelp in python2.6, the github address is

 

Code: https://github.com/Yelp/elastaler

Documentation: http://elastalert.readthedocs.io/en/latest/

Reference: http://blog.csdn.net/gamer_gyt/article/details/52917116

An environment:

    Cent OS 7

    ElasticSearch 2.4.2

    Elastalert

 

 

Second installation

 

1) Install pip

    yum -y install epel-release

    yum -y install python-pip

 

2) Install dependencies

    pip install -r requirements.txt  

 

   (1) Question 1:

Collecting blist==1.3.6 (from -r requirements.txt (line 5))   Downloading blist-1.3.6.tar.gz (122kB)     100% |################################| 122kB 139kB/s     Complete output from command python setup.py egg_info:     The required version of setuptools (>=1.1.6) is not available,     and can't be installed while this script is running. Please     install a more recent version first, using     'easy_install -U setuptools'.       (Currently using setuptools 0.9.8 (/usr/lib/python2.7/site-packages))       ---------------------------------------- Command "python setup.py egg_info" failed with error code 2 in /tmp/pip-build-P4itjY/blist/ You are using pip version 8.1.2, however version 9.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.

    Require setuptools to be the latest new version otherwise an error will be reported

    solve:

        Download: https://pypi.python.org/packages/87/1a/33d3d05569e857c5c5cc3e90d197bf4d9696dc740a05f66a09599d66e5bd/setuptools-32.2.0.zip#md5=3909c3e2941028cad881d81dc4a35e6

        Install:

                python setup.py install

                easy_install -U setuptools  

           Note: easy_install is not executed, the system always installs the setuptools that pip depends on

 

   (2) Question 2:

unable to execute gcc: No such file or directory     error: command 'gcc' failed with exit status 1

    

    solve:

            yum install -y gcc    

 

    (3) Question 3:

gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DBLIST_FLOAT_RADIX_SORT=1 -I/usr/include/python2.7 -c blist/_blist.c -o build/temp.linux-x86_64-2.7/blist/_blist.o     blist/_blist.c:38:20: fatal error: Python.h: No such file or directory      #include <Python.h>                         ^     compilation terminated.     error: command 'gcc' failed with exit status 1       ---------------------------------------- Command "/usr/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-afOL6v/blist/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-oB3kYl-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-afOL6v/blist/

 

    solve:

         yum install -y python-devel

 

success log:

Running setup.py install for blist ... done   Running setup.py install for httplib2 ... done   Running setup.py install for twilio ... done   Running setup.py install for configparser ... done   Running setup.py install for croniter ... done   Running setup.py install for urllib3 ... done   Running setup.py install for oauthlib ... done   Running setup.py install for tlslite ... done   Running setup.py install for jira ... done   Running setup.py install for jsonschema ... done   Running setup.py install for mock ... done   Running setup.py install for simplejson ... done   Running setup.py install for supervisor ... done   Running setup.py install for texttable ... done   Running setup.py install for stomp.py ... done Successfully installed blist-1.3.6 boto-2.34.0 botocore-1.4.5 configparser-3.5.0 croniter-0.3.8 docutils-0.13.1 elasticsearch-1.3.0 httplib2-0.9.2 jira-0.32 jmespath-0.9.0 jsonschema-2.2.0 meld3-1.0.2 mock-1.0.0 oauthlib-0.7.2 python-dateutil-2.4.0 pytz-2016.10 requests-oauthlib-0.4.2 simplejson-3.3.0 stomp.py-4.1.11 supervisor-3.1.2 texttable-0.8.4 tlslite-0.4.8 twilio-5.6.0 unittest2-0.8.0 urllib3-1.8.2

 

3）安装 elastalert

python setup.py install

这里只列出重要日志

#python setup.py install running install 。。。。。。 Adding elastalert 0.1.4 to easy-install.pth file Installing elastalert-create-index script to /usr/bin Installing elastalert script to /usr/bin Installing elastalert-rule-from-kibana script to /usr/bin Installing elastalert-test-rule script to /usr/bin 。。。。。。 Installed /usr/lib/python2.7/site-packages/elastalert-0.1.4-py2.7.egg Finished processing dependencies for elastalert==0.1.4

 

4）安装ElasticSearch 略

 

三使用

 

1）elastalert-create-index：ElastAlert会把执行记录存放到一个ES 索引中，该命令就是用来 创建这个索引的，默认情况下，索引名叫elastalert_status。其中有4个 _type，都有 自己的@timestamp字段，所以同样也可以用kibana，来查看这个索引的日志记录情况。

2）elastalert-rule-from-kibana：从Kibana3已保存的仪表盘中读取Filtering设置，帮助生成config.yaml里的配置。不过注意，它只会读取filtering，不包括queries。

3）elastalert-test-rule：测试自定义配置中的rule设置。

 

4) 启动：python -m elastalert.elastalert --debug --rule /etc/elastalsert/ test_rule.yaml >> /var/log/test_rule.log

 

四 测试

1）创建rule文件：

vi test_rule.yaml

 

# (Optional) # Elasticsearch host es_host: 127.0.0.1   # (Optional) # Elasticsearch port http es_port: 9200   # (Optional) basic-auth username and password for elasticsearch #es_username: someusername #es_password: somepassword   # (Required) # Rule name, must be unique name: example.rule   # (Required) # Type of alert. # the change rule will alert when a certain field changes in two documents within a timeframe type: any   # (Required) # Index to search, wildcard supported index: test-errorlog   # (Required) # A list of Elasticsearch filters used for find events # These filters are joined with AND and nested in a filtered query # For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html filter: - query:     query_string:       query: "level: error"   # (Required) # The alert is use when a match is found alert: - "command"   command: ['date']

 

说明：扫描 elasticsearch （127.0.0.1:9200）的 index : test-errorlog，type ：* 的所有日志，过滤 level 为error的记录，如果发现，报警，这里报警时command方式，只是打印个日期

 

2）向es 中发一条用于测试的error记录

curl -XPOST 'http://127.0.0.1:9200/test-errorlog/logs/1' -d ' {   "@timestamp": "2016-12-22T18:28:56.365Z",   "level": "ERROR",   "message": "test error message " }'

 

注意：时间戳一定要再测试范围内，默认是1天

 

3）执行测试

    elastalert-test-rule example_rules/test_rule.yaml

 

# elastalert-test-rule example_rules/test_rule.yaml --alert Successfully loaded example.rule   INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1 Got 1 hits from the last 1 day   Available terms in first hit:      @timestamp      message      level   INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1   。。。。。。（此处省略）   INFO:elastalert:Queried rule example.rule from 2016-12-23 02:22 CST to 2016-12-23 03:07 CST: 1 / 1 hits   。。。。。。（此处省略）   Fri Dec 23 13:37:22 CST 2016 （报警）   Would have written the following documents to writeback index (default is elastalert_status):   silence - {'rule_name': 'example.rule', '@timestamp': datetime.datetime(2016, 12, 23, 5, 37, 22, 990531, tzinfo=tzutc()), 'exponent': 0, 'until': datetime.datetime(2016, 12, 23, 5, 38, 22, 990523, tzinfo=tzutc())}   elastalert - {'match_body': {'_type': u'logs', '_index': u'test-errorlog', u'@timestamp': '2016-12-22T18:28:56.365Z', u'level': u'ERROR', u'message': u'test error message ', '_id': u'1'}, 'rule_name': 'example.rule', 'alert_time': datetime.datetime(2016, 12, 23, 5, 37, 22, 990845, tzinfo=tzutc()), 'alert_sent': True, 'alert_info': {'command': 'date', 'type': 'command'}}   elastalert_status - {'hits': 1, 'matches': 1, '@timestamp': datetime.datetime(2016, 12, 23, 5, 37, 22, 997762, tzinfo=tzutc()), 'rule_name': 'example.rule', 'starttime': datetime.datetime(2016, 12, 22, 5, 37, 22, 854558, tzinfo=tzutc()), 'endtime': datetime.datetime(2016, 12, 23, 5, 37, 22, 854558, tzinfo=tzutc()), 'time_taken': 0.14146089553833008}

 

 

附录

（一）config.ymal中的配置项

<!--?xml version="1.0" encoding="UTF-8"?-->

• Rules_folder：用来加载下一阶段rule的设置，默认是example_rules
• Run_every：用来设置定时向elasticsearch发送请求
• Buffer_time：用来设置请求里时间字段的范围，默认是45分钟
• Es_host：elasticsearch的host地址
• Es_port：elasticsearch 对应的端口号
• Use_ssl：可选的，选择是否用SSL连接es，true或者false
• Verify_certs：可选的，是否验证TLS证书，设置为true或者false，默认为true
• Es_username：es认证的username
• Es_password：es认证的password
• Es_url_prefix：可选的，es的url前缀（我的理解是https或者http）
• Es_send_get_body_as：可选的，查询es的方式，默认的是GET
• Writeback_index：elastalert产生的日志在elasticsearch中的创建的索引
• Alert_time_limit：失败重试的时间限制

（二）Elastalert的rule规则

• name：配置，每个rule需要有自己独立的name，一旦重复，进程将无法启动。
• type：配置，选择某一种数据验证方式。
• index：配置，从某类索引里读取数据，目前已经支持Ymd格式，需要先设置 use_strftime_index:true，然后匹配索引，配置形如：index: logstash-es-test%Y.%m.%d，表示匹配logstash-es-test名称开头，以年月日作为索引后缀的index。
• filter：配置，设置向ES请求的过滤条件。
• timeframe：配置，累积触发报警的时长。
• alert：配置，设置触发报警时执行哪些报警手段。不同的type还有自己独特的配置选项。目前ElastAlert 有以下几种自带ruletype：

• any：只要有匹配就报警；
• blacklist：compare_key字段的内容匹配上 blacklist数组里任意内容；
• whitelist：compare_key字段的内容一个都没能匹配上whitelist数组里内容；
• change：在相同query_key条件下，compare_key字段的内容，在 timeframe范围内 发送变化；
• frequency：在相同 query_key条件下，timeframe 范围内有num_events个被过滤出 来的异常；
• spike：在相同query_key条件下，前后两个timeframe范围内数据量相差比例超过spike_height。其中可以通过spike_type设置具体涨跌方向是up,down,both 。还可以通过threshold_ref设置要求上一个周期数据量的下限，threshold_cur设置要求当前周期数据量的下限，如果数据量不到下限，也不触发；
• flatline：timeframe 范围内，数据量小于threshold 阈值；
• new_term：fields字段新出现之前terms_window_size(默认30天)范围内最多的terms_size (默认50)个结果以外的数据；
• cardinality：在相同 query_key条件下，timeframe范围内cardinality_field的值超过 max_cardinality 或者低于min_cardinality