Part1Background
Internet applications provide services and interact with users' operations. The traffic generated by their actions contains many excavable values, such as performance analysis, data comparison, behavior analysis, etc. User traffic is like oil in the real world, and there are many evolutionary uses that we can extract from it.
This series develops application design for testers' traffic from the perspective of software testing, forming a traffic management platform that integrates traffic recording, traffic playback, real-time comparison, traffic use case editing, and user management to improve testing work efficiency and provide multi-faceted Verify software product quality.
Part2 Architecture Design
Data flow diagram of traffic recording function
Part3 implementation effect
Record task creation, agent and monitoring logs:
Generate use cases based on traffic:
Part4 functional design
The main functions of the traffic recording module are:
-
Task information list
-
Create a recording task
-
Modify task information
-
perform tasks
-
View traffic logs in real time
-
View detailed information
-
Customize traffic use cases and store them
-
Use case list
-
Use case details, edit
All lists and detailed data in the system are inseparable from the four interfaces of "add, delete, modify, and check". System deletions are currently logical deletions, and the display is controlled by setting status values to prepare for subsequent data recovery and problems. track.
One of the more special features is the real-time traffic log display function. Due to the asynchronous nature of the http interface, it is difficult to ensure the real-time and serialization of the log data. Therefore, the double guarantee adopted here is mainly websocket long connection and supplemented by http interface. Also benefiting from the open source of the mitmproxy project, I combined the system API and conducted a second custom development of its interface method to adapt to the data needs of this system and maintain the accuracy of the data.
Part5 Problems and Solutions
1. How to isolate different configuration requirements of multitasking?
Solution: Before the system is started for the first time, the encapsulated DockerFile needs to be executed to create the basic image we need. The system uses the docker service to create a docker container based on the task configuration information (host, agent, etc.). In fact, the docker image at this time has encapsulated the tools and basic configuration required for container runtime. Each container is created and mounted on its own independent port to achieve isolation.
2How to obtain real-time traffic logs?
Solution: Mitmproxy in the container has multiple startup methods, each with different usage scenarios. Among them, the proxy started in mitmweb mode will have a socket service. Through custom development of this interface, it can support our cross-domain connection. And parse the message information, and then through front-end rendering, you can achieve the real-time log display function.
3. What should I do if the interface gets stuck due to too many logs?
Solution: It is best to limit the size of the log content according to the actual usage scenario. At the same time, the front end uses a virtual list, which only renders the visible area and a small amount of DOM data that will be visible, and calculates the height of the container based on the data length. You can write your own components to create and destroy Dom. This system uses the virtualizedtableforantd4 component that combines react-virtualized and antd to achieve stability and performance optimization of the log display module.
4. How to deal with a large amount of data when generating use cases for traffic?
Solution: For recording use cases with a large amount of data, when generating the use case, a dump file that can be used for playback will be generated and saved by requesting a value passed to the custom dump interface based on the checked traffic ID. At the same time, in order to facilitate front-end display and subsequent editing, the traffic needs to be stored in the use case table. When the amount of data exceeds the limit, the background will discard some redundant field values and slice them into the database to ensure the stability of the system.
5How to handle https traffic?
Solution: https traffic needs to be parsed and obtained by installing the middleman certificate of mitmproxy. There is a point to note here. Android mobile devices after 6.0 cannot parse https through forged certificates. The simple way is to get the server certificate and use openssl to generate a pem certificate. Configure the pem certificate for the project in the system. When the proxy of the project starts, use this certificate to parse the https request of the specified domain name. This will resolve the https data requested by all sources.
Part6 core code implementation
The core part of this functional module is around the container management script for data preparation, configuration management and container management.
Front-end virtual table component design
import React, { useRef, useEffect, useMemo } from 'react';
import { Table } from 'antd';
import { useVT } from 'virtualizedtableforantd4';
const MyRow = React.forwardRef((props, ref) => {
const { children, ...rest } = props;
return <tr {...rest} ref={ref}>{children}</tr>;
});
function CustomRowsHooks(props) {
const columns = useRef(props.columns);
const [VT, setVT] = useVT(() => ({ scroll: { y: 600 }, debug: true }));
useMemo(() => setVT({ body: { row: MyRow } }), [setVT]);
return (
<Table
{...props}
components={VT}
scroll={
{ y: 600 }}
dataSource={props.dataSource}
columns={columns.current}
/>
);
}
export default CustomRowsHooks;
Server core script
The complete script code is as follows:
#-*-coding:utf-8-*-
__author__="orion-c"
def buildContainer(taskInfo, mountPort, wsMountPort, projectDir):
for i in range(5):
if len(os.listdir(projectDir)) > 0:
break
time.sleep(1)
if sys.platform.startswith('win'):
projectDir = "/taskFile/"+projectDir.split('taskFile\\')[1]
formatHost = {}
try:
if taskInfo:
if taskInfo['hosts']:
hosts = taskInfo['hosts'].split('\n')
for host in hosts:
host = ' '.join(host.split())
host = host.split(' ')
if len(host) == 2:
formatHost[host[1]] = host[0]
try:
container = docker_client.containers.get(str(mountPort))
if container.status == 'exited':
container.remove()
logger.info('移除容器成功：{}'.format(str(mountPort)))
except:
logger.info('can build')
docker_client.containers.run(
name=mountPort,
image='myproxy:0.0.13',
command='/bin/bash -c "sh /root/script/start_sync.sh"',
volumes={
projectDir: {'bind': '/root/script', 'mode': 'rw'}
},
ports={'8080': mountPort, '8081': wsMountPort},
extra_hosts=formatHost,
stderr=True,
detach=True
)
logger.info('任务 {} 创建容器 {} 成功，ws端口 {}'.format(taskInfo['id'], mountPort, wsMountPort))
return str(mountPort)
except Exception as e:
logger.error(e)
logger.error('创建容器：{} 失败'.format(mountPort))
logger.error('请检查你的docker服务')
def copyDumpFile(projectDir, content):
dumpFilePath = content['suiteInfo']['dumpFile']
dumpFilePath = '../../' + dumpFilePath if os.path.isfile('../../' + dumpFilePath) else dumpFilePath
replayDumpFile = projectDir + '/replayDumpFile'
shutil.copyfile(dumpFilePath, replayDumpFile)
def makeSyncReplayShell(projectDir, mountPort, content):
shellPath = '../../templte/start_sync.sh' if os.path.isfile('../../templte/start_sync.sh') else 'templte/start_sync.sh'
with open(shellPath, 'r', encoding='utf-8') as shellFile:
shellTemplte = shellFile.read()
cmd = "mitmdump -nC /root/script/replayDumpFile -s /root/script/getReplayResponse.py --ssl-insecure --set ssl_version_client=all --set ssl_version_server=all"
if content['proxySetting']:
cmd = cmd + " --set http2=false --set mode=upstream:{proxySetting}".format(proxySetting=content['proxySetting'])
if content['cerPem'] and content['domain']:
cmd = cmd + " --certs {domain}=/root/script/cer.pem".format(domain= content['domain'])
shellConfig = shellTemplte.format(cmd= cmd)
with open(projectDir + '/start_sync.sh', 'w', encoding='utf-8') as file:
file.write(shellConfig)
def makePem(content,projectDir):
if content['cerPem'] and content['domain']:
with open(projectDir + '/cer.pem', 'w', encoding='utf-8') as file:
file.write(content['cerPem'])
def makeMiddlareScript(content,projectDir):
responsePath = '../../templte/getReplayResponse.py' if os.path.isfile('../../templte/getReplayResponse.py') else 'templte/getReplayResponse.py'
with open(responsePath, 'r', encoding='utf-8') as middlareFile:
middlareTemplte = middlareFile.read()
middlareConfig = middlareTemplte.format(appHost= app.config['HOST'],taskId=content['id'])
with open(projectDir + '/getReplayResponse.py', 'w', encoding='utf-8') as file:
file.write(middlareConfig)
def makeConfig(projectDir, mountPort, content):
copyDumpFile(projectDir,content)
makeSyncReplayShell(projectDir, mountPort,content)
makePem(content, projectDir)
makeMiddlareScript(content, projectDir)
@manager.option('-i','--taskId',dest='taskId',default='')
def runScript(taskId):
now = datetime.now().strftime('%Y-%m-%d_%H_%M_%S')
taskRootPath = encrypt_name(now)
content = getTaskInfo(taskId)
for i in range(2):
if i == 0:
proxyMountPort = freeport.get()
else:
wsMountPort = freeport.get()
projectDir = createProjectDir(taskRootPath)
makeConfig(projectDir, proxyMountPort, content)
setTaskStatus(taskId, 2, taskRootPath)
containName = buildContainer(content, proxyMountPort, wsMountPort, projectDir)
if containName:
setTaskStatus(taskId, 3, taskRootPath, containName, wsMountPort)
else:
setTaskStatus(taskId, 6)
This system adopts a front-end and back-end separation architecture. Due to the complexity of the system architecture, this article only analyzes and explains the design and implementation of the traffic recording module. Other modules will be discussed separately in subsequent articles.
Finally: The complete software testing video tutorial below has been compiled and uploaded. Friends who need it can get it by themselves [guaranteed 100% free]
Software Testing Interview Document
We must study to find a high-paying job. The following interview questions are the latest interview materials from first-tier Internet companies such as Alibaba, Tencent, Byte, etc., and some Byte bosses have given authoritative answers. After finishing this set I believe everyone can find a satisfactory job based on the interview information.
