Getting started learning python (excel import mysql)

Background note:

Jira design a set of mechanisms developed KPI assessment, mainly in the form of task + hours, divided into workload + + standard quality evaluation system based on.
Jira derived from the data, after calculation rules, each generated data and the corresponding evaluation value.
Now is the pre-exploration, it may be the middle of algorithms or data source has changed, encoding is not for a fixed algorithm, so now the first use of excel pivot table functions.
The basic data on A.xlsx, B.xlsx pivot table in the data source assigned to A. Thus after the base update data A, B can direct refresh PivotTable becomes the latest data.
The question now is employees want to know their basic data, less convenient query in excel, so I want to automatically import the database, allows employees to view their own BI system by analyzing the data.

Pre-conditions:

Jira and BI are existing systems, there is no R & D, A data base developed by the assistant handled manually. I imagine the easiest way is to let the assistant to upload to the server through the web directly, write a script on the server parses excel and inserted into the corresponding data in the table, BI can provide a direct connection to the database query interface.
shell script parsing and database operations more difficult, so we put his sights on python body.

Hardware environment:

• Operating System: CentOS 6.5
• Database: Mysql 5.6.41
• Programming languages: Python 2.7.3 Python developers annual survey, Python3 adoption rate of over 84% recommended Python3
• Coding tools: Vim (only a small section of the script directly on the server with vim written, but linux default color rendering is very good, as)

Points encoding process

1. introducing head

import MySQLdb as mdb # mysql
import os
import sys
import re
import xlrd # 读取 excel

References to be effective, you need to install the corresponding package
python package offers a lot of external calls to solve the problem, our focus is to excel in this myslq and so need to reference MySQLdb and xlrd, the system command line to execute the following command:

pip install MySQL-python
pip install xlrd

This will install the mysql support and excel in

2. excel read

wb = xlrd.open_workbook('%s%s' %(filepath,excel.split(',')[0]))
s01 = wb.sheet_by_name('S01-本月workpoint原始数据'.decode('utf-8'))
s02 = wb.sheet_by_name('S02-上月跨本月工时原始数据'.decode('utf-8'))
s03 = wb.sheet_by_name('S03-本月跨下月工时原始数据'.decode('utf-8'))
s04 = wb.sheet_by_name('S04-在线问题工时原始数据'.decode('utf-8'))

Xlrd.open_workbook line of code directly open excel, wb.sheet_by_name directly to the ranks of the data array read, no nonsense.

3. mysql write

mysql more conventional operation, divided into two blocks, connected mdb.connect (), the cursor connect.cursor ()
Connect responsible for establishing a connection channel, responsible for the specific operation performed Cursor

# 建立连接
conn = mdb.connect(host='127.0.0.1', port=3306, user='root', passwd='root', db='dbname', charset='utf8')

# 使用cursor()方法获取操作游标
cursor = conn.cursor()

# 执行插入语句
sSql = 'insert into z_kpi_s01 (k_month,col_1,col_2,col_3,col_4,col_5,col_6,col_7,col_8,col_9,col_10,col_11,col_12) values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)'
cursor.executemany(sSql,values)

4. Overall Code

Overall code is also used to:

• for loop
• if the judge
• Regular judge
• Print text with parameters

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import MySQLdb as mdb
import os
import sys
import re
import xlrd

conn = mdb.connect(host='127.0.0.1', port=3306, user='root', passwd='root', db='dbname', charset='utf8')

# 使用cursor()方法获取操作游标
cursor = conn.cursor()
# 因该模块底层其实是调用CAPI的，所以，需要先得到当前指向数据库的指针。

try:
    # 读取Excel文件，并且确认是否需要导入
    needImportXlsx = []

    filepath = '/data/'
    pathDir = os.listdir(filepath)
    print 'list size is : %s' % len(pathDir)
    for allDir in pathDir:
        fnCount = cursor.execute('select filename from fileattachment where id=%s' % allDir)
        filename = cursor.fetchone()[0]

        canMatch = re.match(r'\d{6}', filename)
        if not (canMatch):
            continue

        tmpCount = cursor.execute('select * from z_kpi_s01 where k_month=%s' % filename)
        print '%s tmp count is : %s ' % (filename, tmpCount)
        if tmpCount == 0:
            needImportXlsx.append(allDir + ',' + filename)

    if len(needImportXlsx) == 0:
        print '!!!no file need import!!!'
        sys.exit()

    print 'all need import xlsx are : %s ' % needImportXlsx

    for excel in needImportXlsx:
        wb = xlrd.open_workbook('%s%s' % (filepath, excel.split(',')[0]))
        s01 = wb.sheet_by_name('S01-本月workpoint原始数据'.decode('utf-8'))
        s02 = wb.sheet_by_name('S02-上月跨本月工时原始数据'.decode('utf-8'))

        print 's01 has col %d row %d \n s02 has col %d row %d ' % (
            s01.ncols, s01.nrows, s02.ncols, s02.nrows)

        # s01 insert mysql
        values = []

        for i in range(s01.nrows):
            if i == 0:
                continue
            value = s01.row_values(i)[:12]
            value.insert(0, excel.split(',')[1])
            values.append(value)

        sSql = 'insert into z_kpi_s01 (k_month,col_1,col_2,col_3,col_4,col_5,col_6,col_7,col_8,col_9,col_10,col_11,col_12) values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)'
        cursor.executemany(sSql, values)

        # s02 insert mysql
        values = []

        for i in range(s02.nrows):
            if i == 0:
                continue
            value = s02.row_values(i)[:31]
            value.insert(0, excel.split(',')[1])
            values.append(value)

        sSql = 'insert into z_kpi_s02 (k_month,col_1,col_2,col_3,col_4,col_5,col_6,col_7,col_8,col_9,col_10,col_11,col_12,col_13,col_14,col_15,col_16,col_17,col_18,col_19,col_20,col_21,col_22,col_23,col_24,col_25,col_26,col_27,col_28,col_29,col_30,col_31) values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)'
        cursor.executemany(sSql, values)

    # 如果没有设置自动提交事务，则这里需要手动提交一次
    conn.commit()
except:
    import traceback

    traceback.print_exc()
    # 发生错误时会滚
    conn.rollback()
finally:
    # 关闭游标连接
    cursor.close()
    # 关闭数据库连接
    conn.close()

to sum up

Before this had never written a Python, I write the code as a whole probably spent about two hours, the time spent on other servers installed above and some of the permissions. The overall feeling Python's advantages are obvious, as a kind of glue code, perform the functions of some of Shell scripts more difficult to achieve, it is very convenient . The syntax is very simple, but for me, Java developers and sometimes habitual and {}; on the contrary is wrong .

reference

The main reference to two articles:
Python connect to the MySQL database
python script to achieve import and export excel and mysql database table