订单导出的预发和线上的自动化对比工具

问题与背景

订单导出需要将交易数据通过报表的形式导出并提供下载給商家，供商家发货、对账等。由于交易的场景非常多，承接多个业务（微商城、零售单店、零售连锁版、餐饮），订单类型很多，新老报表的字段覆盖交易、支付、会员、优惠、发货、退款、特定业务等，合计多达120个。每次代码变更（尤其是比较大的改动），如果想要手工验证指定时间段内的绝大多数场景下绝大多数订单类型的所有字段都没有问题，在前端页面点击下载报表，然后手工对比，将是非常大的工作量。因此，迫切需要一个自动化的对比工具，对比变更分支与线上分支的导出报表，找出和分析差异，修复问题。

为什么选择要在预发而不在QA进行呢？ 因为订单导出的准确性不仅包含导出和下载功能（20%），更重要的是数据的准确性(80%)。而QA的数据不一定准确，且涵盖面不广，不准确的数据会导致错误的对比结果，对变更的影响造成很大的干扰，延误时间。 因此，这里直接选择用线上的数据来做对比，有时也会意外发现线上数据的一点问题。

整体思路

先做出一个假定：如果master分支的线上逻辑是没有问题的，那么预发的branch分支导出的结果，应该跟线上保持一致； 如果线上的逻辑有问题，那么预发 branch 分支导出的结果，应该有部分跟线上不一致，且不一致的地方根据推断应该仅跟改动部分有关。 分两种情况：

• 系统代码优化与重构：逻辑没有改动，那么预发和线上的导出结果应该完全一致。如果有不一致的情况发生，那么需要分析不一致的原因，决定是否可以接受和取舍。
• 业务逻辑优化：比如在某个场景下，“订单类型”字段原来输出“分销买家订单”，现在需要输出“分销买家订单/拼团订单”，那么导出结果的不一致应该限于“订单类型”。当然，如果有其他报表字段的输出也依赖于“订单类型”字段，那么可能其他字段也会不一致，这时候需要进一步分析。

整体思路如下：

• 使用 Python 来完成该任务，因为 Python 非常简洁实用 ，适合做质量要求不是非常高的接口测试工具；
• 分别往预发和线上发送相同的请求，然后通过导出ID拿到预发请求的文件和线上请求的文件，然后读取并逐字段对比，打印出差异；
• 将对比结果保存在 /tmp/cmp_export.txt ， 发送邮件保存。
• 不同店铺的不同业务配置的导出测试用例通过一个单独的配置文件来给出，测试用例配置与请求测试功能分离。

源代码

test.py : 主测试程序。 只要运行 python test.py 即可。然后看看是否有 diff 。如果没有 diff ，那就说明预发和线上导出结果一致； 如果有 diff ，就需要仔细分析 diff ，找出原因并解决。

#!/usr/bin/env python
# -*- coding: utf-8 -*-
#-------------------------------------------------------------------------------
# Name:        test.py
# Purpose:     test if exports from pre is consistent with exports from production
# USAGE:       python test.py
# When:        before deploy to production
#                  STEP1: login in bc-pre-order-export0 and vim test.py, cases.py in your directory ,
#                             enter :set paste ,  copy this script and save ;
#                  STEP2: run python test.py
#
# Author:      qin.shuq
#
# Created:     12/22/2017
# Copyright:   (c) qin.shuq 2017
# Licence:     <your licence>
#-------------------------------------------------------------------------------
import requests
import os
import json
import time
import math
import urllib2
import traceback

import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.header import Header


from cases import *

import sys
import codecs
import locale
sys.stdout = codecs.getwriter(locale.getpreferredencoding())(sys.stdout)

preUrl = 'http://pre-host:7001/api/general/export'
prodUrl = 'http://prod-host:7001/api/general/export'
queryUrl = 'http://prod-host:7001/api/general/queryRecords'

filedir = './files/'
resultfile = '/tmp/export_cmp.txt'

def extractFields(fieldsStr):
    return map(lambda x: x.strip(), fieldsStr.split(','))

templateIdFieldsMap = { \
    "1": extractFields("OrderNo,ExpressCompany,ExpressNo,OrderState,BuyerName,BuyerSex,BuyerProvince,BuyerCity,IsFans,ShouldPay,Postage,TotalPay,RealPay,BuyWay,SKU,SKUCode,GoodsCode,FeedBackInfo,ReceiverName,ReceiverProvince,ReceiverCity,ReceiverCounty,ReceiverDetailAddress,PostCode,ExpressWay,SelfFetchAddress,SelfFetcher,SelfFetchPhone,SelfFetchTime,Phone,BookTime,PayTime,Verificator,VerificateTime,Title,GoodsPrice,OrderRemark,GoodsNum,ShopId,TeamName,GoodsMsg,OrderMsg,OuterTransactionNo,InnerTransactionNo,Star,Remark,FenxiaoOrderNo,FenxiaoPay,GroupNo,StoreId,StoreName,ItemRealPay,ItemRefundPay,ItemExpressTime, DeliveryTime,SuccessTime,PeriodInfo,OrderType") , \
    "2": extractFields("TeamName,OrderNo,OrderType,OrderState,OrderMsg,OrderRemark,AllGoodsTotalNum,AllGoodsOriginTotalPrice, AllGoodsPromotionTotalPrice,FoodBoxFee,DeliveryFee,Coupon,ManLiMinus,TotalPay,RealPay,OrderRefundFee,BuyWay,BookTime,PayTime,SuccessTime"), \
    "3": extractFields("TeamName,OrderNo,OrderType,OrderState,OrderMsg,OrderRemark,ReceiverName,Phone,ReceiverDetailAddress,AllGoodsTotalNum,AllGoodsOriginTotalPrice, AllGoodsPromotionTotalPrice,FoodBoxFee,DeliveryFee,Coupon,ManLiMinus,TotalPay,RealPay,OrderRefundFee,BuyWay,BookTime,PayTime,SuccessTime"), \
    "4": extractFields("OrderNo, Saleway, OrderType, OrderState, OrderSource, BookTime, PayTime, SuccessTime, BuyWay, InnerTransactionNo, OrderTotalPrice, OrderPostage, TotalPromotion, ShouldPay, RealPay, CashReturn, PromotionDetail, AllGoodsTitle, AllGoodsKinds, DeliveryType, DeliveryTime, ReceiverProvince, ReceiverCity, ReceiverCounty, ReceiverDetail, Receiver, ReceiverTel, OrderMsg, OrderStar, FansName, IsMember, FansTel, StoreName, Cashier, OrderRefundState, OrderRefundFee, OrderRemark, PeriodInfo, IDCard"), \
    "5": extractFields("OrderId, SimpleOrderState, GoodsTitle, GoodsType, GoodsKind, GoodsSaleway, GoodsSku, GoodsSkuCode, GoodsBizCode, GoodsOriginPrice, GoodsPromotionDetail, GoodsActivityPrice, GoodsTotalNum, GoodsUnit, GoodsTotalPrice, GoodsSharedShopPromotion, GoodsActualDealPay, GoodsPointsPrice, GoodsRemark, GoodsExpressState, GoodsExpressWay, GoodsExpressCorp, GoodsExpressNo, GoodsExpressPerson, GoodsExpressTime, GoodsRefundState, GoodsRefundFee"), \
    "7": extractFields("OrderID, OrderType, OrderState, OrderSource, BookTime, PayTime, SuccessTime, BuyWay, OuterTransactionNo, InnerTransactionNo, OrderTotalPrice, OrderPostage, TotalPromotion, ShouldPay, RealPay, CashReturn, PromotionDetail, AllGoodsTitle, AllGoodsKinds, DeliveryType, AppointmentTime, Receiver, ReceiverTel, ReceiverProvince, ReceiverCity, ReceiverCounty, ReceiverDetail, OrderMsg, OrderStar, FansName, IsMember, FansTel, BookStoreName, OrderRefundState, OrderRefundFee, OrderRemark, PeriodInfo, IDCard"), \
    "8": extractFields("OrderId, OuterTransactionNo, SimpleOrderState, SuccessTime, GoodsTitle, GoodsType, GoodsCategory, GoodsSku, GoodsSkuCode, GoodsBizCode, GoodsOriginPrice, GoodsPromotionDetail, GoodsActivityPrice, GoodsTotalNum, GoodsTotalPrice, GoodsSharedShopPromotion, GoodsActualDealPay, GoodsPointsPrice, GoodsRemark, Receiver, ReceiverTel, ReceiverProvince, ReceiverCity, ReceiverCounty, ReceiverDetail, OrderMsg, GoodsExpressState, GoodsExpressWay, GoodsExpressCorp, GoodsExpressNo, WscGoodsExpressPerson, WscGoodsExpressTime, GoodsRefundState, GoodsRefundFee") \
    }

def mkdir(filedir):
    isExists=os.path.exists(filedir)
    if not isExists:
        os.makedirs(filedir)
        return True
    else:
        return False

def sendRequest(url, query):
    r = requests.post(url, data=query, headers={"Content-type":"application/json"})
    return r.json()

def getData(url, query):
    try:
        resp = sendRequest(url, query)
        if resp['result'] and resp['data']['success']:
            return resp['data']['data']
        return None
    except:
        return None

def download(url, tag, exportId):
    f = urllib2.urlopen(url)
    data = f.read()
    filename = filedir + tag + "_" + str(exportId) + ".csv"
    with open(filename, "w") as csvFile:
        csvFile.write(data)
    return filename

def getFileLines(filename):
    with open(filename, 'r') as f:
        lines = f.readlines()
    return (filename,lines)

def getExportId(url, query):
    exportId = getData(url,query)
    if not exportId:
        return 0
    return int(exportId)

def getExportRecord(exportId):
    rec = {}
    while len(rec) == 0:
        time.sleep(5)
        queryReq = '{"export_id": %d, "page": 1, "size": 1, "export_state": 10}' % (exportId)
        resp = getData(queryUrl, queryReq)
        if resp['total'] > 0:
            rec = resp['list'][0]
    return rec

def getExportedFile(url, query, tag):
    exportId = getExportId(url, query)
    return getByExportId(exportId, tag)

def getExportedFileWithExportId(url, query, tag, exportId):
    getExportId(url, query)
    return getByExportId(exportId, tag)

def getByExportId(exportId, tag):
    exportRecord = getExportRecord(exportId)
    print 'tag=%s, exportId=%s, url=%s' % (tag, exportId, exportRecord['url'])
    csvFile = download(exportRecord['url'], tag, exportId)
    return csvFile

def cmpByOldExportReq(oldExportReq):
    preOldExportUrl = 'http://pre-host:7001/api/order/export'
    prodOldExportUrl = 'http://prod-host:7001/api/order/export'
    query = oldExportReq
    exportId = int(json.loads(oldExportReq)['export_id'])
    updateRecord(exportId)
    preFile = getExportedFileWithExportId(preOldExportUrl, query, 'pre', exportId)
    updateRecord(exportId)
    prodFile = getExportedFileWithExportId(prodOldExportUrl, query, 'prod', exportId)
    cmpExportFile(preFile, prodFile, query, "1")

def updateRecord(exportId):
    updateParam = '{"source":"AVeryComplexSecretTestHacker", "export_id": %s, "url":"", "export_state": 1, "token": "0"}' % (exportId)
    updateUrl = 'http://pre-host:7001/api/general/update'
    updateResult = getData(updateUrl, updateParam)
    if updateResult:
        print 'update record to init success !'

def cmplines(prodLines, preLines, fields, keyIndex=0):
    print 'length: online=%d, pre=%d' % (len(prodLines), len(preLines))
    try:
        for i in range(len(prodLines)):
            online = prodLines[i].strip().split(',')
            preline = preLines[i].strip().split(',')
            for t in range(len(online)):
                try:
                    if online[t] != preline[t]:
                        print 'diff: field=%s, online=%s, pre=%s, orderNo=%s' % (fields[t], online[t].decode('gb18030'), preline[t].decode('gb18030'), online[keyIndex])
                except:
                    print 'compare failed. field=%s orderNo=%s %s' % (fields[t], online[keyIndex], traceback.format_exc())
        print 'passed.'
    except:
        print 'compare failed. %s' % traceback.format_exc()

def cmpExport(exportReq):
    preFile = getExportedFile(preUrl, exportReq, 'pre')
    prodFile = getExportedFile(prodUrl, exportReq, 'prod')
    templateId = json.loads(exportReq)['template_id']
    cmpExportFile(preFile, prodFile, exportReq, templateId)

def cmpExportFile(preFile, prodFile, exportReq, templateId="1"):
    fields = templateIdFieldsMap[templateId]
    keyIndex = 0
    if templateId == "2" or templateId == "3":  #餐饮的报表,订单号在第二列
        keyIndex = 1

    # 按照订单号排序，因为下单时间可能相同，对比时有不必要的不一致
    preSortedFile = getSortedFile(preFile, keyIndex)
    prodSortedFile = getSortedFile(prodFile, keyIndex)
    preSorted = getFileLines(preSortedFile)[1]
    prodSorted = getFileLines(prodSortedFile)[1]
    print 'exportReq=[ %s ], prodFile=%s, preFile=%s' % (exportReq, prodSortedFile, preSortedFile)
    cmplines(prodSorted, preSorted, fields, keyIndex)

def getSortedFile(originFile, index):

    filename = originFile.rsplit('.',1)[0]
    sortedfilename = filename + "_sorted.csv"
    cmd = 'sort -k %d %s > %s' % (index+1, originFile, sortedfilename)
    os.system(cmd)
    return sortedfilename


def sendmail(text):
    sender = '[email protected]'
    receivers = ['[email protected]']

    message = MIMEMultipart()
    message['From'] = Header("导出对比工具", 'utf-8')
    message['To'] =  Header("导出对比工具", 'utf-8')
    subject = '导出对比结果'
    message['Subject'] = Header(subject, 'utf-8')

    message.attach(MIMEText('导出对比结果如附件所示', 'plain', 'utf-8'))

    att1 = MIMEText(open(resultfile, 'rb').read(), 'base64', 'utf-8')
    att1["Content-Type"] = 'application/octet-stream'
    att1["Content-Disposition"] = 'attachment; filename="export_cmp_result.txt"'
    message.attach(att1)

    try:
        smtpObj = smtplib.SMTP('smtp.exmail.qq.com', 25)
        smtpObj.login('[email protected]', 'YxtkETvis7Ck3UYZ')
        smtpObj.sendmail(sender, receivers, message.as_string())
        print "Email Send success!"
    except smtplib.SMTPException:
        print "Email Send failed!"


if __name__ == '__main__':

    savedStdout = sys.stdout

    mkdir(filedir)
    f_result = open(resultfile, 'w')
    sys.stdout = f_result

    allreqs = []
    for func in caseGenerateFuncs:
        allreqs.extend(func(startTimeParam, endTimeParam))
    for req in allreqs:
        cmpExport(req)
        print '\n'

    allOldReqs = []
    for func in oldExportInstGenerateFuncs:
        allOldReqs.extend(func(startTimeParam, endTimeParam))
    for req in allOldReqs:
        #cmpByOldExportReq(req)
        print '\n'

    print 'success done !'

    sendmail('export cmp result')

    f_result.close()

    sys.stdout = savedStdout

cases.py : 导出对比的测试用例配置

#!/usr/bin/env python
# -*- coding: utf-8 -*-
#-------------------------------------------------------------------------------
# Name:        cases.py
# Purpose:     Provides cases of exports
#
# Author:      qin.shuq
#
# Created:     12/22/2017
# Copyright:   (c) qin.shuq 2017
# Licence:     <your licence>
#-------------------------------------------------------------------------------

import time
import math
import json

kdtId = 63077
parts = 2

endTime = math.floor(time.time()) - 300
startTime = endTime - 600

baseExportReqStr = '{"biz_type":"order", "export_type":"default", "request_id": "123456", "source":"testsource", "order_by":"book_time", "order":"desc", "order_search_param": {"kdt_id":%d, "start_time":%d, "end_time":%d, "must_not": {"is_visible":0}}, "template_id":1}' % (kdtId, startTime, endTime)

def divideNParts(total, N):
    '''
       divide [0, total) into N parts:
        return [(0, total/N), (total/N, 2*total/N), ((N-1)*total/N, total)]
    '''

    each = total / N
    parts = []
    for index in range(N):
        begin = index*each
        if index == N-1:
            end = total
        else:
            end = begin + each
        parts.append((begin, end))
    return parts

def commonGenerateReqByTime(startTime, endTime, kdtId=xxx, templateId=1):
    def generateReqByTimeInner(startTime, endTime):
        totalInterval = endTime-startTime
        timeparts = divideNParts(totalInterval, parts)
        timeparts = map(lambda t: (t[0]+startTime, t[1]+startTime), timeparts)
        reqs = []
        for time in timeparts:
            baseReq = buildReq(baseExportReqStr, time[0], time[1], kdtId, templateId)
            reqs.append(json.dumps(baseReq))
        return reqs
    return generateReqByTimeInner

def commonGenerator(startTime, endTime, kdtId=xxx, templateId=1, field='', values=[]):
    def generateReqInner(startTime, endTime):
        reqs = []
        for val in values:
            baseReq = buildReq(baseExportReqStr, startTime, endTime, kdtId, templateId, field, val)
            reqs.append(json.dumps(baseReq))
        return reqs
    return generateReqInner

def buildReq(baseExportReqTemplate, startTime, endTime, kdtId=xxx, templateId=1, field=None, value=None):
    requestId = str(startTime) + "_" + str(endTime) + "_" + str(kdtId) + "_" + str(templateId)
    baseReq = json.loads(baseExportReqTemplate)
    baseReq['order_search_param']['start_time'] = startTime
    baseReq['order_search_param']['end_time'] = endTime
    if field and value:
        baseReq['order_search_param'][field] = value
    baseReq['order_search_param']['kdt_id'] = kdtId
    #baseReq['order_search_param']['order_nos'] = ['E2017**********00001', 'E2017**********0887']
    baseReq['request_id'] = requestId
    baseReq['template_id'] = templateId
    return baseReq

def generateGenerators(startTime, endTime, configs):
    gvars = globals()
    for (templateId,kdtId) in kdtIdTemplateIdMap.iteritems():
        if len(configs) == 0:
            funcName = 'generateReqByTime_' + str(kdtId) + "_" + str(templateId)
            gvars[funcName] = commonGenerateReqByTime(startTime, endTime, kdtId, templateId)
        else:
            for (field, values) in configs.iteritems():
                funcName = 'generateReqBy_' + str(kdtId) + "_" + str(templateId) + "_" + field
                gvars[funcName] = commonGenerator(startTime, endTime, kdtId, templateId, field, values)

#templateId=1,7,8 wsc export ;  =2,3 canyin ; =4,5 retail
kdtIdTemplateIdMap = {"1": xxx, "2":yyy, "3":yyy, "4":zzz, "5":zzz, "7": xxx, "8": xxx}
#kdtIdTemplateIdMap = {"1": xxx}

#如果改动了搜索相关，则需要测试订单搜索，使用该配置
searchconfigs = {"order_state": [[],["TOPAY"], ["CONFIRM"], ["PAID"], ["SENT"], ["SUCCESS"], ["CLOSE"]], \
                 "order_type": [[],["NORMAL"], ["GROUP"], ["HOTEL"]], \
                 "express_type": [[],["EXPRESS"], ["SELF_FETCH"], ["LOCAL_DELIVERY"]], \
                 "feedback": [[],["SAFE_NEW"], ["SAFE_FINISHED"]], \
                 "buy_way":[[],["WXPAY", "WXPAY_DAIXIAO", "WXPAY_SHOULI", "WX_APPPAY", "WX_WAPPAY"], ["ALIWAP", "ALIPAY"], ["UMPAY", "UNIONPAY", "UNIONWAP", "BAIDUWAP", "YZPAY"]]
                 }

# 如果只改动了详情，不需要测试订单搜索，只需要按照时间段来导出预发线上数据进行比较即可。
detailconfigs = {}

def buildOldExportInst(kdtId,startTime,endTime):
    def buildOldExportInstInner(startTime, endTime):
        totalInterval = endTime-startTime
        timeparts = divideNParts(totalInterval, parts)
        timeparts = map(lambda t: (t[0]+startTime, t[1]+startTime), timeparts)
        reqs = []
        for time in timeparts:
            req = {}
            req['kdt_id']=kdtId
            req['start_time']=time[0]
            req['end_time']=time[1]
            req['export_type']='default';
            req['biz_type']='order';
            req['export_id']= 'xxxxxx';   # 借壳生蛋
            req['param_hash']= '7295**********************e3c1';
            reqs.append(json.dumps(req))
        return reqs
    return buildOldExportInstInner

vipKdtIdConfigs = [xxx]

def buildAllOldExportInstGenerator(startTime, endTime, vipKdtIdConfigs):
    gvars = globals()
    for config in vipKdtIdConfigs:
        funcName = 'buildOldExportInstInner_' + str(config)
        gvars[funcName] = buildOldExportInst(config, startTime, endTime)

def getGenerateFuncs():
    gvars = globals()
    caseGenerators = [ gvars[var] for var in gvars if var.startswith('generateReq')  ]
    print 'case generators: ', [ var for var in gvars if var.startswith('generateReq') ]
    return caseGenerators

def getOldExportGenerateFuncs():
    gvars = globals()
    oldExportCaseGenerators = [ gvars[var] for var in gvars if var.startswith('buildOldExportInstInner')  ]
    print 'old export case generators: ', [ var for var in gvars if var.startswith('buildOldExportInstInner') ]

    return oldExportCaseGenerators

generateGenerators(startTime, endTime, detailconfigs)
caseGenerateFuncs = getGenerateFuncs()

buildAllOldExportInstGenerator(startTime, endTime, vipKdtIdConfigs)
oldExportInstGenerateFuncs = getOldExportGenerateFuncs()

# 5.14 - 5.22 has dirty data, should be excluded
startTimeParam = 1506787200
endTimeParam = 1530288000

小结

无论大改还是小改，通过运行这个预发和线上对比工具，很大程度上增强了成功发布的信心。可见，预发和线上的自动化对比工具，确实是发布前的最后一道防线。