python初探，爬了点彩票数据

最近网球群的群友痴迷高频彩票 11选5。中奖频率颇高，眼馋却胆颤，爬点数据找找规律，看看是不是真的随机。

1、依赖的pip package

certifi (2018.1.18)
chardet (3.0.4)
DateTime (4.2)
flake8 (3.5.0)
idna (2.6)
mccabe (0.6.1)
pew (1.1.2)
pip (9.0.1)
pipenv (9.0.3)
pycodestyle (2.3.1)
pyflakes (1.6.0)
pytz (2018.3)
requests (2.18.4)
setuptools (28.8.0)
urllib3 (1.22)
virtualenv (15.1.0)
virtualenv-clone (0.2.6)
zope.interface (4.4.3)

python的包与nodejs的还是不同，首先python可以多个版本共存，不同版本可以下载不同的package。然后nodejs默认的是装在本地目录的，而python默认是全局安装的，我这里用的是python3 and pip3。

2、source

引用块内容

import requests
import json
import xml.etree.ElementTree as ET
import os
from datetime import date
import datetime
import re
import string
end = date.today();
start = date(2015,10,20);
f = open('update.txt', 'r')
dateStamp = f.read();
f.close()
y = int('20'+dateStamp[0:2])
m = int(dateStamp[2:4])
d = int(dateStamp[4:6])
start = date(y,m,d);
print('\033[1;32;40m')
print('Requesting data from ',start);
print('Requesting data to ',end);
print('\033[0m')
currentPath = os.getcwd()
dataPath = os.path.join(currentPath, 'data')
os.chdir(dataPath)
while start < end:
    temp = start.strftime("%y%m%d")
    start =start +datetime.timedelta(days=1)
    # 先在api接口进行请求
    url = 'http://kaijiang.500.com/static/info/kaijiang/xml/hebsyxw/20'+temp+'.xml?_A=TGZNWVQO1518789888290'
    r = requests.get(url)
    print(temp,"'s status is ",r.status_code,"in kaijiang' api")
    if r.status_code == 200:
        xmlcontent = r.text.replace("gb2312", "utf-8")
        doc = {}
        root = ET.fromstring(xmlcontent)
        for item in root:
            doc[item.attrib['expect']] = item.attrib['opencode']
        json.dumps(doc)
        fileName = temp+'.json'
        content = open(fileName,"w+")
        tempStr = str(doc).replace("'",r'"')
        content.write(tempStr)
        content.close()
else:
        # api请求失败了，去爬另一个网页
        content = open('error.log',"a+")
        content.write(temp+"'s data is missing in kaijiang's api\n")
        content.close()
        url = 'https://zst.cjcp.com.cn/cjw11x5_qs/view/11x5_jiben-5-hb11x5-11-1-20'+temp+'01-20'+temp+'85-9.html'
        r = requests.get(url)
        print(temp,"'s status is ",r.status_code,"in zst's webpage")
        if r.status_code == 200:
            regex = re.compile(r"<td class='z_bg_05'>\d{10}</td><td class='z_bg_13'>\d{2}\s\d{2}\s\d{2}\s\d{2}\s\d{2}</td>")
            infoList = regex.findall(r.text)
            if len(infoList) ==0:
                # 判断网页上是否有对应的数据
                content = open('error.log',"a+")
                content.write(url+"'s data is something wrong, I can't find it in the Internet\n")
                content.close()
                continue
            doc = {}
            for item in infoList:
                key = re.findall(r"20\d{8}", item)
                value = re.findall(r"\d{2}\s\d{2}\s\d{2}\s\d{2}\s\d{2}",item)
                if len(key)>0 and len(value):
                    valuewithdot = ','.join(value[0].split(" "))
                    doc[key[0][2:]] = valuewithdot
            doc = str(doc).replace("'",r'"')
            json.dumps(doc)
            fileName = temp+'.json'
            content = open(fileName,"w+")
            tempStr = str(doc).replace("'",r'"')
            content.write(tempStr)
            content.close()
        else:
            content = open('error.log',"a+")
            content.write(temp+"'s data is missing in zst's webpage\n")
            content.close()
 # 切换到外层路径
os.chdir(currentPath)
content = open('update.txt',"w+")
content.write(end.strftime("%y%m%d"))
content.close()

代码中第一个是一个api接口，传入日期参数即可返回当天的11选5这种彩票的xml数据，转成json即可。但是很不幸，数据不全，有问题的数据，只好去另外一个网站去爬，根据正则过滤出需要的时间和开奖结果数据即可。

3、文件结构

这是文件夹里应该有py脚本（e.g. 11x5.py），update.txt，data文件夹（里边有生成的json和error.log），Pipfile ，Pipfile.lock。
因为我想让他每天自动去爬数据，爬取结束后，将结束日期写在update.txt里，下次获取时间从update.txt里的日期开始即可。
这里写图片描述

菜鸡初探python，如有不对，恳请指正。下一步就是分析一下这数据，看看能否从中获取某种猫腻，帮群友提高一丢丢中奖率。FIGHTING