将公开的威胁情报数据采集做数据分析和内网安全问题 排查。情报获取只要可以提供全量数据的接口
情报一：
http://www.freebuf.com/sectool/159648.html
踩坑：
critical-stack-intel list
critical-stack 14:19:19 [INFO] Pulling feed list from the Intel Marketplace.
ID | NAME | LAST UPDATED | INDICATOR COUNT
+------+-----------------------------------------------------------------------------+--------------+-----------------+
217 | volexity.com-Wekby-Adobe-Flash-Exploit-IOCs | - | 0
216 | volexity.com-Virtual-Private-Keylogging-report-IOCs | - | 0
215 | team-cymru.org-Poseidon-IOCs | - | 0
214 | talos-Angler-Exposed-2015-10-06-IOCs | - | 0
213 | symantec.com-Black-Vine-IOCs | - | 0
212 | symantec.com-Cadelspy-and-Remexi-Backdoor-IOCs | - | 0
211 | shadowserver.org-The-Italian-Connection-IOCs | - | 0
210 | secureworks.com-Point-of-Sale-Malware-Threats | - | 0
207 | pan-unit42-Piano-Gh0st-IOCs | - | 0
206 | pan-unit42-Lotus-Blossom-IOCs | - | 0
205 | pan-unit42-Dridex-2015-10-01-IOCs | - | 0
204 | morphick.com-BernhardPOS-IOCs | - | 0
203 | checkpoint.com-Rocket-Kitten:-A-Campaign-with-9-Lives-(2015-11-09) | - | 0
202 | anubisnetworks.com-Dridex-Botnet-Report-(2015-10-15) | - | 0
201 | volexity.com-Wekby-Adobe-Flash-Exploit-IOCs | - | 0
200 | volexity.com-Virtual-Private-Keylogging-report-IOCs | - | 0
199 | team-cymru.org-Poseidon-IOCs | - | 0
198 | talos-Angler-Exposed-2015-10-06-IOCs | - | 0
197 | symantec.com-Black-Vine-IOCs | - | 0
196 | symantec.com-Cadelspy-and-Remexi-Backdoor-IOCs | - | 0
195 | shadowserver.org-The-Italian-Connection-IOCs | - | 0
194 | secureworks.com-Point-of-Sale-Malware-Threats | - | 0
192 | pan-unit42-Piano-Gh0st-IOCs | - | 0
191 | pan-unit42-Lotus-Blossom-IOCs | - | 0
190 | pan-unit42-Dridex-2015-10-01-IOCs | - | 0
189 | morphick.com-BernhardPOS-IOCs | - | 0
188 | checkpoint.com-Rocket-Kitten:-A-Campaign-with-9-Lives-(2015-11-09) | - | 0
187 | anubisnetworks.com-Dridex-Botnet-Report-(2015-10-15) | - | 0
186 | volexity.com-Wekby-Adobe-Flash-Exploit-IOCs | - | 0
185 | volexity.com-Virtual-Private-Keylogging-report-IOCs | - | 0
184 | team-cymru.org-Poseidon-IOCs | - | 0
183 | talos-Angler-Exposed-2015-10-06-IOCs | - | 0
182 | symantec.com-Black-Vine-IOCs | - | 0 critical-stack-intel pull
critical-stack 14:19:40 [ERROR] --- NOTICE ----------
critical-stack 14:19:40 [ERROR] Unable to locate bro or configure permissions properly.
critical-stack 14:19:40 [ERROR] Unable to add sudoers access for bro binary.
critical-stack 14:19:40 [INFO] If you have a custom setup you can add your paths manually.
critical-stack 14:19:40 [INFO] $ sudo critical-stack-intel config --set bro.path=/my/path/bro
critical-stack 14:19:40 [INFO] $ sudo critical-stack-intel config --set bro.include.path=/my/path/local.bro
critical-stack 14:19:40 [INFO] $ sudo critical-stack-intel config --set bro.broctl.path=/my/path/broctl
critical-stack 14:19:40 [ERROR] --- NOTICE ----------
很明显的数据并没有采集到，还报了一堆错，解决无果，后续厂商给答复了再更新。
情报二：
http://threatcrowd.blogspot.gr/2016/02/crowdsourced-feeds-from-threatcrowd.html 每小时 更新一次
获取方式：
https://www.threatcrowd.org/feeds/domains.txt
https://www.threatcrowd.org/feeds/ips.txt
https://www.threatcrowd.org/feeds/hashes.txt
页面做了简单的防DDOS,直接爬虫没法跳转。
selenium简单处理了一下:拿到数据，入库
# -*- coding: utf-8 -*-
from selenium import webdriver
from time import sleep
import sys
reload(sys)
sys.setdefaultencoding('utf8')
options = webdriver.ChromeOptions()
browser = webdriver.Chrome(executable_path='C:\Python27\Scripts\chromedriver.exe',chrome_options= options) #mac系统的话chromedriver()放到 usr/local/bin/ 下面就可以 不需要禁用sip
def read(iplist):
xx=iplist.split("\n")
for i in range(len(xx)):
if i==0:
xxxx=xx[i].split(">")[-1]
print xxxx
elif i ==len(xx)-1:
xxx = xx[i].split("<")[0]
print xxx
else:
print i
print xx[i]
def main():
browser.get('https://www.threatcrowd.org/feeds/ips.txt')
sleep(10)
htmlstart1 = browser.page_source
read(htmlstart1)
if __name__ == '__main__':
main()
最后入库搞定
数据源三：
来源：https://github.com/scu-igroup/Ti_Collector
直接使用吧。后续持续更新。