[Test joint debugging] How to elegantly construct abnormal scenarios during front-end and back-end test joint debugging

The current applications all use a separate front-end and back-end architecture, and the front-end and back-end systems need to cooperate to achieve various functions. Backend systems are usually responsible for handling business logic, data storage, and interaction with other services, while frontends are responsible for user interface and user interaction. In the process of front-end and back-end data interaction, various exceptions and errors may occur. It is a very important part of test verification to confirm whether the processing of the front-end and back-end systems is reasonable when an exception occurs.

In the last blog, I introduced how to use test stubs to isolate the dependence on the environment. This time, let's take a look at how to use exception injection to deal with abnormal scenarios in joint debugging.

Implemented using iptables

Among system exceptions, database connection failures and third-party service unavailability are typical scenarios. A common verification method is often that the front-end students inform the background students to enable network isolation, and then perform verification.

Use to `iptables`discard an ip packet

Use -L to list all rules

Specific operation:

$  iptables     -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

IP Connectivity Communication Test

#  检查发现能 是否能正常 ping通
$  ping {数据库/后端地址IP}
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.*: icmp_seq=1 ttl=61 time=0.704 ms
64 bytes from 1.1.1.*: icmp_seq=2 ttl=61 time=0.802 ms           
^C
--- 1.1.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.704/0.753/0.802/0.049 ms

Insert a rule to discard all protocol requests for this ip

$  iptables  -I   INPUT   -p   all   -s {数据库/后端地址IP}   -j   DROP

list all rules

$ iptables  -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  1.1.1.*        anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Test IP connectivity within drop rules

$ ping 1.1.1.*
PING 1.1.1.1 (1.1.1.*) 56(84) bytes of data.
^C
--- 1.1.1.1 ping statistics ---
85 packets transmitted, 0 received, 100% packet loss, time 84312ms

Clear the limit of the rule list

$  iptables     -F

Simulate ip to deal with 50% packet loss.

iptables -I INPUT -s {后端IP} -m statistic --mode random --probability 0.5 -j DROP

The above method can achieve related joint debugging, but there are two points that can be improved:

The biggest limitation of this method is that it will affect all testers who call this system.
Human intervention is required, human operation is required every time

So is there a less intrusive and more elegant way to verify exception scenarios? The answer is yes.

mysql proxy proxy

A more elegant way: write a mysql proxyproxy, let the backend svr directly connect to the proxy, and the proxy connects to the real mysql.

When a normal request passes through the proxy, the proxy forwards it directly to the real mysql, and returns the mysql reply packet to the caller normally.
When a certain keyword ( timeout) is received, the TCP connection is directly disconnected, and the connection DB timeout scenario is simulated

proxy code

import asyncio

# 真实mysqlIP端口
mysql_host = settings.MYSQL_HOST
mysql_port = settings.MYSQL_PORT

# 处理客户端连接
async def handle_connection(client_reader, client_writer):
    # 连接到实际的 MySQL 服务器
    mysql_reader, mysql_writer = await asyncio.open_connection(mysql_host, mysql_port)

    # 转发握手包
    handshake_packet = await mysql_reader.read(4096)
    client_writer.write(handshake_packet)
    await client_writer.drain()

    # 处理客户端认证请求
    auth_packet = await client_reader.read(4096)
    mysql_writer.write(auth_packet)
    await mysql_writer.drain()

    # 转发认证响应
    auth_response = await mysql_reader.read(4096)
    client_writer.write(auth_response)
    await client_writer.drain()

    # 转发请求和响应
    while True:
        data = await client_reader.read(4096)
        if not data:
            break
        sql_query = data[5:].decode('utf-8')
        if "timeout" in sql_query:  # sql 包含timeout 关键字时，直接关闭连接
            await asyncio.sleep(1)
            break
        else:
            mysql_writer.write(data)
            await mysql_writer.drain()
            response = await mysql_reader.read(4096)
            client_writer.write(response)
            await client_writer.drain()
    # 关闭连接
    client_writer.close()
    mysql_writer.close()

async def main(host, port):
    server = await asyncio.start_server(handle_connection, host, port)
    async with server:
        await server.serve_forever()

# 使用示例
if __name__ == "__main__":
    # 本地监听 3307， svr 连接到3307
    host = "0.0.0.0"
    port = 3307

    asyncio.run(main(host, port))

Test directly with pymysql

The following code times out on the second select statement:

import pymysql

connection = pymysql.connect(
    host="192.168.31.76",
    port=8899,
    # 真实mysql 账户信息
    user="{user}",
    password="{password}",
    database="mydb",

)

curs = connection.cursor()
curs.execute("select * from user where name='bingo';")
print(curs.fetchall())

curs.execute("insert into user set name='bingtime';")
connection.commit()

curs.execute("select * from user where name='timeoutbingo';")
print(curs.fetchall())
curs.close()
connection.close()

Python version lower than 3.7

The lower version of Python does not have asyncio.run and server.serve_forever()needs to modify the main function.

# 主函数，启动服务器并监听连接
async def main(host, port):
    server = await asyncio.start_server(handle_connection, host, port)
    try:
        # Python 3.6.5 中没有 server.serve_forever() 方法，所以需要使用一个无限循环来保持服务器运行。
        # 我们使用 asyncio.sleep() 来避免阻塞事件循环，使其可以处理其他任务，如新连接。
        while True:
            await asyncio.sleep(3600)  # 1 hour
    except KeyboardInterrupt:
        pass
    finally:
        server.close()
        await server.wait_closed()


# 使用示例
if __name__ == "__main__":
    host = "0.0.0.0"
    port = 3307

    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(main(host, port))
    except KeyboardInterrupt:
        pass
    finally:
        loop.close()

other extensions

Write a proxy to monitor the incoming and outgoing SQL statements:

import pymysql
import socket
import threading
# 配置
SERVER_ADDRESS = (settings.MYSQL_HOST, settings.MYSQL_PORT)  # 真实MySQL 服务器地址
PROXY_ADDRESS = ('0.0.0.0', 8899)  # 监听代理服务器地址

def print_query(data):
    try:
        command = data[4]
        if command == pymysql.constants.COMMAND.COM_QUERY:
            query = data[5:].decode("utf-8")
            print(f"SQL: {query}")
    except Exception as e:
        print(f"Error parsing packet: {e}")

def forward_data(src_socket, dst_socket, parser=None):
    while True:
        try:
            data = src_socket.recv(4096)
            if not data:
                break
            if parser:
                parser(data)
            dst_socket.sendall(data)
        except OSError as e:
            if e.errno == 9:
                break
            else:
                raise

def main():
    # 创建代理服务器套接字
    proxy_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    proxy_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    proxy_socket.bind(PROXY_ADDRESS)
    proxy_socket.listen(5)
    print(f"Proxy server listening on {PROXY_ADDRESS}")

    while True:
        client_socket, client_address = proxy_socket.accept()
        print(f"Client {client_address} connected")

        # 连接到 MySQL 服务器
        server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        server_socket.connect(SERVER_ADDRESS)

        try:
            # 创建线程转发客户端数据到服务器
            client_to_server = threading.Thread(target=forward_data, args=(client_socket, server_socket, print_query))
            client_to_server.start()

            # 创建线程转发服务器数据到客户端
            server_to_client = threading.Thread(target=forward_data, args=(server_socket, client_socket))
            server_to_client.start()

            # 等待线程完成
            client_to_server.join()
            server_to_client.join()

        finally:
            client_socket.close()
            server_socket.close()
            print(f"Client {client_address} disconnected")

if __name__ == "__main__":
    main()

If executed using the above pymysql test code, the following information will be printed:

Client ('127.0.0.1', 57184) connected
SQL: SET AUTOCOMMIT = 0
SQL: select * from user where name='bingo';
SQL: insert into user set name='bing21211';
SQL: COMMIT
SQL: select * from user where name='bingo';
SQL: select * from user where name='bingo';
SQL: select * from user where name='timeoutbingo';
Client ('127.0.0.1', 57184) disconnected

Summarize

By implementing appropriate exception handling mechanisms, you can ensure that users get useful feedback when encountering problems, and verifying these handling mechanisms can improve system stability and security. iptables It is powerful but requires manual operation. mysql proxyThe proxy function is direct, but the application scenarios are relatively limited. You can choose according to the actual situation.

Data acquisition method

【Message 777】

Friends who want to get source code and other tutorial materials, please like + comment + bookmark , triple!

After three times in a row , I will send you private messages one by one in the comment area~