Python import Excel to a database

Python import Excel to a database

Python as a scripting language easy to learn, when dealing with system operation and maintenance work, also helped by the great. To know CentOS package management tool is through Python.

In actual operation, the basic data provided by the user are often based on finishing Excel, Excel data into the database is a very common requirement. This article describes how to achieve read Excel data based Python3, and imported into the database.

This procedure will be applied to the two packages can be installed by pip

pip install openpyxl pyodbc

Program structure as shown in FIG.

Inlet module executed by Run. It relies SqlHelper and ExcelHelper. SqlHelper responsible for operating the database read, ExcelHelper responsible Excel read operation. ExcelHelper by reading the Excel file, parsed as a Model type of array. Model SqlHelper by reading the array to perform database operations.

run.py inlet module code is as follows:

#通过from ... import引入其他模块
from excelHelper import ExcelHelper
from sqlHelper import SqlHelper

def run():
    importxls('xls/xxx.xlsx')

def importxls(path):
    # 读取excel文件，获取model列表
    excelHelper = ExcelHelper(path,'Sheet1')
    modellist = excelHelper.readExcelForModels()
    # 通过sqlHelper传入读取到的model列表插入数据库
    sqlHelper = SqlHelper()
    sqlHelper.insertModels(modellist)

#执行
run()

model.py Model is the data model to define our needs after reading excel data obtained. Custom fields they need on the line:

class Model:
    def __init__(self,code,name):
        self.code = code
        self.name = name
    
    #通过定义str，帮助在调试时，通过print()函数打印数据
    def __str__(self):
        str = "code:%s  name:%s  tenantId:%s  targetRemark1:%s" % (self.code, self.name) 
        return str

excelHelper.py read excel help class:

#引入openpyxl以读取excel文件
import openpyxl
#引入模型定义
from model import Model

class ExcelHelper:
    def __init__(self, path, sheetname):
        #列数定义，Index从1开始计算
        self.codeIndex = 2
        self.nameIndex = 3
        #读取workbook。data_only=True，以避免Excel公式值读取的问题
        wb = openpyxl.load_workbook(path, data_only=True)
        #读取标签
        self.sheet = wb[sheetname]
        #读取excel中的行数
        self.max_row = self.sheet.max_row

    def readExcelForModels(self):
        list = []
        #循环读取excel，行数从1开始计算
        for rowIndex in range(2, self.max_row):
            cellCode = self.sheet.cell(row=rowIndex, column=self.codeIndex).value
            if(cellCode == None):
                continue
            #读取所需数据
            code = self.sheet.cell(row=rowIndex, column=self.codeIndex).value
            name = self.sheet.cell(row=rowIndex, column=self.nameIndex).value
            #设置model
            data = Model(code, name) 
            #插入数组
            list.append(data)
        return list

appsettings.json define the database connection string odbc

{
  "ConnectionStrings": {
    "Default": "DRIVER={SQL SERVER};Server=xx; Database=xx; UID=xx;PWD=xx;"
  }
}

configReader.py read the database connection string:

import json

def getConnectString(name):
    config = open("appsettings.json")
    setting = json.load(config)
    connectstring = setting['ConnectionStrings'][name]
    return connectstring

sqlHelper.py database operations help class:

#sqlHelper.py
#pyodbc通过odbc读取数据库
import pyodbc
#configReader来读取appsettings.json
import configReader
#textwrap用于处理sql字符串
import textwrap

class SqlHelper:
    def __init__(self):
        #获取数据库连接字符串
        conn_info = configReader.getConnectString('Default')
        #连接数据库
        self.mssql_conn = pyodbc.connect(conn_info)
        #获取游标
        self.mssql_cur = self.mssql_conn.cursor()
    
    #校验字段是否为空
    def isEmpty(self, v):
        if(v == None or v.isspace()):
            return True
        else:
            return False
    #插入数据
    def insertModels(self, modelList):
        for model in modelList:
            if(self.isEmpty(item.code) or self.isEmpty(item.name)):
                continue
            #通过textwrap构建sql字符串，?代表预留参数位
            sql = textwrap.dedent("""
            INSERT INTO [dbo].[Model]
            ([Code]
            ,[Name]
            )
            VALUES
            (?
            ,?
            )
            """)
            #循环插入数据
            self.mssql_cur.execute(sql, model.code, model.name)
        #数据库事务提交
        self.mssql_conn.commit()
    
    #展示如何进行数据库查询，假设存在一个可能为null的字段tennantId
    def getIdByCode(self, code, tennantId):
        #由于tennantId可能为null，因此查询语句会有is null的差别
        if(tennantId == None):
            #通过fetchval()来读取数据
            return self.mssql_cur.execute("select Id from Model where TenantId is null and Code = ?", code).fetchval()
        else:
            return self.mssql_cur.execute("select Id from Model where TenantId = ? and Code = ?", tennantId, code).fetchval()

Python import excel database operations carried out, in general, it is very simple. Especially pyodbc for the operation of the database have a good package for the modification operations, providing default transaction commits. And supports input parameters by replacing the parameters of the way. python is indeed an easy to use language, other languages ​​do a similar thing, a lot of programming efficiency weaker.

Hope this help you.