Benchmarking and implementation of the configuration layer implementation method of Python automated testing | JD Cloud technical team

Yuan Chuanghui, offline restart! July 1, 2023 Shenzhen Station - Basic software technology interview! Early bird tickets are available for a limited time only!

What is a configuration file in Python, how to use the configuration file, what are the supported configuration files, etc., without further ado, let's take a look~

1 What is a configuration file?

The configuration file is a file used to configure the parameters and initialization settings of the computer program. Without these configuration programs, it may not be able to run or affect the operation (running speed, convenience, etc.). The advantage of using the configuration file is that some content and environment runtime You only need to modify the parameter content of the configuration file, instead of searching and modifying it in the code, which improves convenience and maintainability.

2 What kinds of configuration files are there?

There are four main forms of configuration:

1. The first type is YAML, JSON, XML, TOML, INI, and .bashrc in Linux systems, which are mainly used in the field of software testing. In the field of software testing, most companies use the latest YAML form as configuration files , such as the storage of database addresses, use case data, etc., and a small number of companies still use the old INI configuration form
2. The second is the form of an excel form. In the excel form, there will be a fixed title representing the meaning of each field column. There are multiple columns for configuration. It is mostly used in the game field, and a large number of excel forms are used in the game industry. Form is already a norm.
3. The third is the py file. The py file is very convenient for a pure Python project. It does not need to read data, but only needs to be imported. The bad point is that it is not as flexible as YAML. , YAML configuration files are supported in Python, Java and other languages, and support many data types, while py configuration files can only be used in python, which has certain limitations.
4. The fourth is the txt text format, which recognizes the txt text content by reading. Generally speaking, it is a simple tool made by testers or test engineers, and it is used by testers at the business level, which reduces the configuration of YAML. It is difficult to understand and avoid the slowness of opening excel. It is a good choice to use lightweight txt instead.

2.1 this

The ini .ini file that comes with python3
is the abbreviation of Initialization File, that is, the initialization file, which is the storage format used by the windows system configuration file, and manages the various configurations of windows

2.1.1 Definition of ini file

.ini files usually consist of sections, keys, and values. The specific form is as follows:

db.ini
[mysql]
host = 127.0.0.1
port = 3306
user = root
password = 123456
database = test

2.1.2 python read ini file

Use python's built-in configparser standard library to parse the ini file.
read() reads the contents of the file items() gets all the key-value pairs of the specified section

# -*- coding: utf-8 -*-
'''
 * @Author       : wxy
 * @Date         : 2022-08-24 11:11:06
 * @Description  : 读取ini文件
 * @LastEditTime : 2022-08-24 11:11:06
'''
from configparser import ConfigParser
from pprint import pprint
import pymysql


# ini文件路径
ini_file = './db.ini'
# 读取ini的节（Section）
db_name = 'mysql'

# configparser实例化
text = ConfigParser()

# 读取ini文件内容
text.read(ini_file)

# text.items()返回list，元素为tuple,元组格式为 key,value
db_tuple = text.items(db_name)
print(db_tuple)

# 将元组转换成dict
db_dict = dict(text.items(db_name))
print(db_dict)

2.2 json

JSON (JavaScript Object Notation,) is a lightweight data exchange format.

2.2.1 Definition of json file

simple json example

  {
        "mysql": {
            "host": "127.0.0.1",
            "port": 3306,
            "user": "root",
            "password": "123456",
            "database": "test"
        }
    }

2.2.2 python read json file

load() Read json format data from json file
loads() Convert string type data to json format data
dump() Save json format data to file
dumps() Save json format data as string type

# -*- coding: utf-8 -*-
'''
 * @Author       : wxy
 * @Date         : 2022-8-24 11:39:44
 * @Description  : 读取json文件
 * @LastEditTime : 2022-8-24 11:39:44
'''

import json
from pprint import pprint
import pymysql

json_file = "./db.json"
db_name = "mysql"
web = "web"

with open(json_file) as f:
    cfg = json.load(f)[db_name]
    print(cfg)

with open(json_file) as f:
    cfg = json.load(f)[web]
    print(cfg['user'])

2.3 toml

TOML is a configuration file format proposed by Github co-founder Tom Preston-Werner. It is a small-scale, easy-to-use semantic configuration file format designed to be transformed without ambiguity. for a hash table.

2.3.1 Define the toml file

Syntax:
TOML's syntax is broadly composed of key = "value", [section name], # comment.
The following data types are supported: String, Integer, Float, Boolean, DateTime, Array, and Chart.

 # db.toml
[mysql]
    [mysql.config]
    host     = "127.0.0.1"
    user     = "root"
    port     = 3306
    password = "123456"
    database = "test"

    [mysql.parameters]
    pool_size = 5
    charset   = "utf8"

    [mysql.fields]
    course_cols = ["cno", "cname", "ccredit", "cdept"]

2.3.2 python read toml file

Use the external library toml to parse the toml file

# -*- coding: utf-8 -*-
'''
 * @Description  : 读取toml文件
 * @LastEditTime : 2022-08-14 11:31:07
'''
import toml
from pprint import pprint
import pymysql

toml_file = "./db.toml"

cfg = toml.load(toml_file)['mysql']

pprint(cfg)

2.4 yaml

YAML (YAML Ain't a Markup Language", YAML is not a markup language) format is currently a more popular configuration file, it was proposed by a person named Clark Evans as early as 2001; it is also widely used configuration file type.

2.4.1 Define yaml file

# db.yaml
mysql:
  config:
    host: "127.0.0.1"
    port: 3306
    user: "root"
    password: ""
    database: "stu_sys"

  parameters:
    pool_size: 5
    charset: "utf8"

  fileds:
    course_cols:
      - cno
      - cname
      - ccredit
      - cdept

2.4.2 Python reads yaml files

Use the external library pyyaml ​​to parse toml files.

# -*- coding: utf-8 -*-
'''
 * @Author       : wxy
 * @Date         : 2022-8-24 11:34:37
 * @Description  : 读取yaml文件
 * @LastEditTime : 2022-8-24 11:34:37
'''
import yaml
from pprint import pprint
import pymysql

yaml_file = "./db.yaml"

with open(yaml_file, 'r') as f:
    cfg = yaml.safe_load(f)

print(cfg)

2.5 Python xlrd read operation Excel

Python Excel library comparison

This time, we mainly focus on python xlrd read operation excel

2.5.1 Introduction to xlrd module

1. What is the xlrd module?
The python operation excel mainly uses the two libraries of xlrd and xlwt, that is, xlrd is the library for reading excel, and xlwt is the library for writing excel.

2. Why use the xlrd module?
Data maintenance is a core in UI automation or interface automation, so this module is very practical.
The xlrd module can be used to read Excel data, the speed is very fast, it is recommended to use!
Official documentation: https://xlrd.readthedocs.io/en/latest/

2.5.2 Install the xlrd module

Go to the python official website to download http://pypi.python.org/pypi/xlrd module installation, the premise is that the python environment has been installed.
Or pip install xlrd in the cmd window.
The latest xlrd does not support reading Excel xlsx files. So you need to install the old version pip install xlrd==1.2.0

2.5.3 Introduction to use

1. The data type of commonly used cells

• empty
• string（text）
• number
• date
• boolean
• error
• blank (blank form)

2. Import module

import xlrd

3. Open the Excel file to read the data

data = xlrd.open_workbook(filename)#文件名以及路径，如果路径或者文件名有中文给前面加一个 r

4. The most important method in the commonly used function excel is the operation of book and sheet

• Get a worksheet in the book (excel file)

table = data.sheets()[0]             #通过索引顺序获取
table = data.sheet_by_index(sheet_indx)  #通过索引顺序获取
table = data.sheet_by_name(sheet_name)  #通过名称获取

# 以上三个函数都会返回一个xlrd.sheet.Sheet()对象

names = data.sheet_names()        #返回book中所有工作表的名字
data.sheet_loaded(sheet_name or indx)    # 检查某个sheet是否导入完毕

• row operation

nrows = table.nrows
    # 获取该sheet中的行数，注，这里table.nrows后面不带().
table.row(rowx)
    # 返回由该行中所有的单元格对象组成的列表,这与tabel.raw()方法并没有区别。
table.row_slice(rowx)
    # 返回由该行中所有的单元格对象组成的列表
table.row_types(rowx, start_colx=0, end_colx=None)
    # 返回由该行中所有单元格的数据类型组成的列表；    
    # 返回值为逻辑值列表，若类型为empy则为0，否则为1
table.row_values(rowx, start_colx=0, end_colx=None)
    # 返回由该行中所有单元格的数据组成的列表
table.row_len(rowx)
    # 返回该行的有效单元格长度，即这一行有多少个数据

• Column (colnum) operations

ncols = table.ncols
    # 获取列表的有效列数
table.col(colx, start_rowx=0, end_rowx=None)
    # 返回由该列中所有的单元格对象组成的列表
table.col_slice(colx, start_rowx=0, end_rowx=None)
    # 返回由该列中所有的单元格对象组成的列表
table.col_types(colx, start_rowx=0, end_rowx=None)
    # 返回由该列中所有单元格的数据类型组成的列表
table.col_values(colx, start_rowx=0, end_rowx=None)
    # 返回由该列中所有单元格的数据组成的列表

• Cell operations

table.cell(rowx,colx)
    # 返回单元格对象
table.cell_type(rowx,colx)
    # 返回对应位置单元格中的数据类型
table.cell_value(rowx,colx)
    # 返回对应位置单元格中的数据

2.5.4 Practical training

Use the xlrd module to read:

import xlrd

xlsx = xlrd.open_workbook('./3_1 xlrd 读取 操作练习.xlsx')

# 通过sheet名查找：xlsx.sheet_by_name("sheet1")
# 通过索引查找：xlsx.sheet_by_index(3)
table = xlsx.sheet_by_index(0)

# 获取单个表格值 (2,1)表示获取第3行第2列单元格的值
value = table.cell_value(2, 1)
print("第3行2列值为",value)

# 获取表格行数
nrows = table.nrows
print("表格一共有",nrows,"行")

# 获取第4列所有值（列表生成式）
name_list = [str(table.cell_value(i, 3)) for i in range(1, nrows)]
print("第4列所有的值：",name_list)

2.6 Python xlwt write operation Excel (xls format only!)

xlwt can be used to write a new Excel table or modify the original table, the speed is also very fast, it is recommended to use!
Official documentation: https://xlwt.readthedocs.io/en/latest/

2.6.1 pip install xlwt

pip install xlwt

2.6.2 Use xlwt to create a new table and write

Write the xlwt new table writer:

# 3.2.2 使用xlwt创建新表格并写入
def fun3_2_2():
    # 创建新的workbook（其实就是创建新的excel）
    workbook = xlwt.Workbook(encoding= 'ascii')

    # 创建新的sheet表
    worksheet = workbook.add_sheet("My new Sheet")

    # 往表格写入内容
    worksheet.write(0,0, "内容1")
    worksheet.write(2,1, "内容2")

    # 保存
    workbook.save("新创建的表格.xls")

2.6.3 xlwt set font format

Program example:

# 3.2.3 xlwt设置字体格式
def fun3_2_3():
    # 创建新的workbook（其实就是创建新的excel）
    workbook = xlwt.Workbook(encoding= 'ascii')

    # 创建新的sheet表
    worksheet = workbook.add_sheet("My new Sheet")

    # 初始化样式
    style = xlwt.XFStyle()

    # 创建字体样式
    font = xlwt.Font()
    font.name = 'Times New Roman'   #字体
    font.bold = True                #加粗
    font.underline = True           #下划线
    font.italic = True              #斜体

    # 设置样式
    style.font = font
    # 往表格写入内容
    worksheet.write(0,0, "内容1")
    worksheet.write(2,1, "内容2",style)
    # 保存
    workbook.save("新创建的表格.xls")
    # 设置列宽
    worksheet.col(0).width = 256*20
     # 设置行高
    style = xlwt.easyxf('font:height 360;')  # 18pt,类型小初的字号
    row = worksheet.row(0)
    row.set_style(style)
     # 合并 第1行到第2行 的 第0列到第3列
    worksheet.write_merge(1, 2, 0, 3, 'Merge Test')
      # 设置边框样式
    borders = xlwt.Borders()  # Create Borders
    borders.left = xlwt.Borders.DASHED
    borders.right = xlwt.Borders.DASHED
    borders.top = xlwt.Borders.DASHED
    borders.bottom = xlwt.Borders.DASHED
    borders.left_colour = 0x40
    borders.right_colour = 0x40
    borders.top_colour = 0x40
  borders.bottom_colour = 0x40

2.7 Python xlutils modify Excel

xlutils can be used to copy the original excel or modify it on the basis of the original excel and save it;
official document: https://xlutils.readthedocs.io/en/latest/

2.7.1 pip install xlutils

pip install xlutils

2.7.2 xlutils copy source files (need to be used with xlrd)

Program example:

# 3.3.2 拷贝源文件
def fun3_3_2():
    workbook = xlrd.open_workbook('3_3 xlutils 修改操作练习.xlsx')  # 打开工作簿
    new_workbook = copy(workbook)  # 将xlrd对象拷贝转化为xlwt对象
    new_workbook.save("new_test.xls")  # 保存工作簿

2.7.3 xlutils reads and writes (that is, modifies) Excel table information

Program example:

# 3.3.3 xlutils读取 写入 Excel 表格信息
def fun3_3_3():
    # file_path：文件路径，包含文件的全名称
    # formatting_info=True：保留Excel的原格式（使用与xlsx文件）
    workbook = xlrd.open_workbook('3_3 xlutils 修改操作练习.xlsx')

    new_workbook = copy(workbook)  # 将xlrd对象拷贝转化为xlwt对象

    # 读取表格信息
    sheet = workbook.sheet_by_index(0)
    col2 = sheet.col_values(1)  # 取出第二列
    cel_value = sheet.cell_value(1, 1)
    print(col2)
    print(cel_value)

    # 写入表格信息
    write_save = new_workbook.get_sheet(0)
    write_save.write(0, 0, "xlutils写入！")

    new_workbook.save("new_test.xls")  # 保存工作簿

2.8 Python xlwings read write modify operation Excel

2.8.1 pip install xlwings

pip install xlwings

2.8.2 Basic Operation

Introduce the library
import xlwings as xw
(1) Open an existing Excel document

# 导入xlwings模块
import xlwings as xw

# 打开Excel程序，默认设置：程序可见，只打开不新建工作薄，屏幕更新关闭
app=xw.App(visible=True,add_book=False)
app.display_alerts=False
app.screen_updating=False

# 文件位置：filepath，打开test文档，然后保存，关闭，结束程序
filepath=r'g:\Python Scripts\test.xlsx'
wb=app.books.open(filepath)
wb.save()
wb.close()
app.quit()

(2) Create a new Excel document, name it test.xlsx, and save it in the D drive

import xlwings as xw

app=xw.App(visible=True,add_book=False)
wb=app.books.add()
wb.save(r'd:\test.xlsx')
wb.close()
app.quit()

(3) xlwings reads and writes Excel
to create a new test.xlsx, enter "life" in the first cell of sheet1, then save and close, and exit the Excel program.

def fun3_4_4():
    # 新建Excle 默认设置：程序可见，只打开不新建工作薄，屏幕更新关闭
    app = xw.App(visible=True, add_book=False)
    app.display_alerts = False
    app.screen_updating = False

    # 打开已存在的Excel文件
    wb=app.books.open('./3_4 xlwings 修改操作练习.xlsx')

    # 获取sheet对象
    print(wb.sheets)
    sheet = wb.sheets[0]
    # sheet = wb.sheets["sheet1"]

    # 读取Excel信息
    cellB1_value = sheet.range('B1').value
    print("单元格B1内容为：",cellB1_value)

    # 清除单元格内容和格式
    sheet.range('A1').clear()

    # 写入单元格
    sheet.range('A1').value = "xlwings写入"

    # 保存工作簿
    wb.save('example_3.xlsx')

    # 退出工作簿
    wb.close()

    # 退出Excel
    app.quit()l

2.9 Python openpyxl read write modify operation Excel

In openpyxl, three concepts are mainly used: Workbooks, Sheets, and Cells.
Workbook is an excel worksheet;
Sheet is a table page in the worksheet;
Cell is a simple grid.
openpyxl revolves around these three concepts, regardless of reading and writing are "three tricks": open the Workbook, locate the Sheet, and operate the Cell.
Official document: https://openpyxl.readthedocs.io/en/stable/
1. Install
pip install openpyxl
2. Open the file
(1) New

from  openpyxl import  Workbook 
# 实例化
wb = Workbook()
# 激活 worksheet
ws = wb.active

(2) Open the existing

from openpyxl  import load_workbook

wb = load_workbook('文件名称.xlsx')

3. Write data

# 方式一：数据可以直接分配到单元格中(可以输入公式)
ws['A1'] = 42
# 方式二：可以附加行，从第一列开始附加(从最下方空白处，最左开始)(可以输入多行)
ws.append([1, 2, 3])
# 方式三：Python 类型会被自动转换
ws['A3'] = datetime.datetime.now().strftime("%Y-%m-%d")

4. Create a table (sheet)

# 方式一：插入到最后(default)
ws1 = wb.create_sheet("Mysheet") 
# 方式二：插入到最开始的位置
ws2 = wb.create_sheet("Mysheet", 0)

5. Select the table (sheet)

# sheet 名称可以作为 key 进行索引
>>> ws3 = wb["New Title"]
>>> ws4 = wb.get_sheet_by_name("New Title")
>>> ws is ws3 is ws4
True

6. View the table name (sheet)

# 显示所有表名
>>> print(wb.sheetnames)
['Sheet2', 'New Title',  'Sheet1']
# 遍历所有表
>>> for sheet in  wb:
...     print(sheet.title)

7. Save data

wb.save('文件名称.xlsx')

8. Others
(1) Change the color of the sheet label button

ws.sheet_properties.tabColor = "1072BA" # 色值为RGB16进制值

(2) Get the largest row, the largest column

# 获得最大列和最大行
print(sheet.max_row)
print(sheet.max_column)

(3) Get each row and each column
sheet.rows as a generator, which contains the data of each row, and each row is wrapped by a tuple.
sheet.columns is similar, but inside each tuple is the cell of each column.

# 因为按行，所以返回A1, B1, C1这样的顺序
for row in sheet.rows:
    for cell in row:
        print(cell.value)

# A1, A2, A3这样的顺序
for column in sheet.columns:
    for cell in column:
        print(cell.value)

(4) Get letters from numbers and numbers from letters

from openpyxl.utils import get_column_letter, column_index_from_string

# 根据列的数字返回字母
print(get_column_letter(2))  # B
# 根据字母返回列的数字
print(column_index_from_string('D'))  # 4

(5) Delete the worksheet

# 方式一
wb.remove(sheet)
# 方式二
del wb[sheet]

Project practice——Practice project
requirements in UI automation: business is written into the picking container, used once and cannot be used for the second time, and the script reads a fixed position

import openpyxl
from openpyxl.cell.cell import ILLEGAL_CHARACTERS_RE
from openpyxl.utils import get_column_letter, column_index_from_string

# 向sheetobj中的columnname列从start_row开始写入listdata
def insert_listdata_to_column(sheetobj,listdata,column_name,start_row=3):
    # 根据列名获取列索引
    colindex = column_index_from_string(column_name)
    print('colindex为{}'.format(colindex))
    # 循环从开始行数到数据写入后最后一行
    for rowindex in range(start_row, start_row + len(listdata)):
        # 写入list数值根据索引取值，从0开始
        val = listdata[rowindex - start_row]
        print('val{}'.format(val))
        print('rowindex{}'.format(rowindex))
        try:
            sheetobj.cell(row = rowindex,column = colindex,value = val)
        except:
            # 出现非法字符时，可以将字符串的非法字符替换掉
            val = ILLEGAL_CHARACTERS_RE.sub(r'',val)
            sheetobj.cell(row = rowindex,column = colindex,value = val)
    delrow = start_row + len(listdata)
    print('*********{}'.format(delrow))
    sheetobj.delete_rows(delrow)


def del_excel():

    xlsx = xlrd.open_workbook(r'D:\pytest\inbound_data.xlsx')
    table =xlsx.sheet_by_index(2)
    # 获取第2列所有值
    cel_value =table.col_values(1)
    cel_value=cel_value[3::]

    wb = openpyxl.load_workbook(r'D:\pytest\inbound_data.xlsx')
    sheet = wb["B2B出库"]
    print(sheet)
    insert_listdata_to_column(sheet,cel_value,'B',3)
    wb.save(r'D:\pytest\inbound_data.xlsx')

del_excel()

3 summary

In this article, several configuration files and usage are briefly introduced. Depending on the use case, complex tools/frameworks are not always better than simple packages. But no matter which one you choose, you should always consider readability, maintainability, and how to catch bugs early. In fact, it could be argued that configuration files are just another type of code. You can use it flexibly according to your own project framework~

Author: Jingdong Logistics Wang Xiaoyun

Source: JD Cloud Developer Community

{{o.name}}

{{m.name}}