[ChatGPT-Application] Preliminary exploration of the chatGPT coverage test process | JD Logistics Technical Team

1 Introduction

By the end of 2022, ChatGPT has become popular both inside and outside the industry. Simply put, it is a chat robot based on a natural language generative AI model. It is the latest work launched by OpenAI on November 30 for free testing by the public. It can imitate human-like conversations based on user prompts, which is very different from ordinary intelligent robots and very real. We talked to him about various things, such as writing code, summarizing weekly reports, writing emails, writing poems, checking encyclopedias, etc. ChatGPT answered everything fluently, making it a breeze.

The industry began to try to use chatGPT to improve work efficiency, such as VSCode, and the IDE plug-in market quickly launched ChatGPT. Many masters have also built WeChat, browser plug-ins, clients, etc. based on chatGPT.

With such a popular trend, as a tester, I am also quite curious about this. What simple human-computer dialogue can help us in our testing work? This article mainly talks about the application of chatGPT from a testing perspective and combined with the testing process.

2. Introduction to testing process

Combined with the team’s existing testing process, it is as follows:

We follow the testing process and break down specific tasks. What content do testers need to output in each link?

• Analyze requirements, write test plans, clarify test strategies, including security, performance, etc., and output test schedules;
• Extract test points and split test cases specifically by function points, including smoke use cases, process use cases, exception use cases, joint call cases, etc.;
• According to the interface definition, prepare automated scripts in advance, and add assertions and script tuning according to the business process during the test phase;

Next, refine the test plan and refine the functions. These must be customized one by one by testers through their own analysis of PRD and familiarity with the business. Since GPT is an artificial intelligence dialogue, then I will summarize the "fixed requirements" for him. Ask questions and see what he produces.

3. GPT answer evaluation

Write test cases

According to the extracted test points, communicate with him in concise and concise language.

Next, since ChatGPT has a memory function, it can know what we asked it to do just now.

Give him some more guidance and let him continue

Change a GPT client and try again.

As shown in the picture above, his answers basically belong to the basic common sense of testing. They are still far from the use case requirements in real work, and some of his answers are even wrong. The boundary values, equivalence classes, decision tables, etc. I asked about were not answered in place. It seemed that he was still rummaging through the libraries he had learned, patting his head and randomly writing use cases, and adding new ones to the self-operated vehicles that he originally asked about. The interface seems to have no specific connection. But it is very good to use as a supplement and reference. Use it as a search engine. The results are highly aggregated. You don’t need to search multiple pages. You can summarize the template answers yourself.

Write automation scripts

In the second step, as the pre-test continues to advance, we expect that the proportion of automation in smoke use cases will gradually increase. So how can we improve the speed of writing automated scripts? Let's see if GPT can help us?

Let’s try interface automation first.

Then ask about the UI in the same scenario.

As mentioned above, there will be occasional interruptions when answering questions. We need to be more patient and continue to ask questions.

GPT above gives a simple linear script. His answer cannot be copied directly. Maybe his knowledge base does not cover enough business scenarios?

So let’s just skip the business and ask for a basic login example to see what we can use?

It’s inconvenient to take long screenshots, so I’ll just copy the code:

from seLenium import webdriver
import time
#设置测览器塑动路径
driver = webdriver.Chrome(executable_path='/path/to/chromedriver')

# 打开网址
driver.get('http:/example/")

# 验证用户名和密码输入框是否存在
assert driver.find_element_by_name('username')
assert driver.find_element_by_name('password')

# 测试无效用户名和密码
invalid_username ='fakeuser'
invalid_password ='fakepass'
driver.find_element_by_name('username').send_keys(invalid_username)
driver.find_element_by_name('password').send_keys(invalid_password)
driver.find_element_by_css_selector('button[type="submit"]').click()

# 检查错误消息是香正碑量示
error_msg= driver.find_element_by_css_selector(',alert-danger').text
assert "用户名或密码不正确" in error_msg

# 测试正确凭证
valid_username = 'testuser'
valid_password = 'testpass'

driver.find_element_by_name('username').clear()
driver.find_element_by_name('username').send_keys(vaLid_username)
driver.find_element_by_name('password').clear()
driver.find_element_by_name('password').send_keys(valid_password)
driver.find_element_by_name("button[type="submit"]') .click()

# 验证是否成功跳转到主页
time.sleep(1) #等待页面刷新
assert driver.current_url == 'http://example.cn/home/'

# 验证是显示用户信息
user_info = driver.find_element_by_css_selector('userinfo').text
assert valid_username in user_info

# 退出登录
driver.find_element_by_link_text("退出"),click()

# 关闭浏览器
driver.quit()

There is no big problem in starting and ending the selenium automation script; then verify whether the username and password input box exists, and if it does not exist, an error will be reported; then log in once with a wrong username and password, verify whether the prompt is correct; and then log in again Enter the correct username and password once to verify whether you can jump to the homepage; then verify whether the username display element on the homepage contains the username; finally click the exit button and close the browser.

Analyze the above script:

Advantages: The code written by GPT basically has no grammatical problems, and supports scanning the positioning of several key elements in the page being tested; I think the code comments are particularly good, and it is a good code specification that can be used for reference in our daily life;

Disadvantages: The correct username, password, homepage, homepage username display, homepage exit button, etc. are all guessed by GPT without explanation in advance. People who do not understand scripts will be fooled if they see it;

There are too few use case scripts, and they do not come together with the large number of use cases above; the use cases do not introduce any standard framework, which is relatively backward; Selenium’s positioning writing method is also relatively backward, and in the latest version, this writing method will report an error;

Scripts have no layering concept and cannot be directly reused into existing mature frameworks.

Summary: The code is not yet fully implemented and still needs people to improve it, but it is still a good start.

I then went ahead and updated the question to be rewritten using unittest in order to test the limits of what GPT can do.

import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

class LoginTestCase(unittest.TestCase):

    def setup(self):
        self.driver = webdriver.executable_path='/path/to/chromedriver')
        self.driver.get('http://Logout/')
    
    def tearDown(self):
        self.driver.quit()
    
    def test_invalid_credentials(self):
        # 测试无效用户名和密码
        invalid_usernane = "fakeuser"
        invalid_password = "fakepass"
        username_input = self.driver.find_element_by_name('username')
        password_input = self.driver.find_element_by_name('password')
        submit_button = self.driver.find_element_by_css_selector("button[type='submit']")
        username_input.send_keys(invalid_username)
        password_input.send_keys(invalid_password)
        submit_button.click()
        
        # 检查错误消息是否正确显示
        error_msg = self.driver.find_element_by_css_selector('.alert-danger').text
        self.assertIn("用户名或密码不正确'，error_msg）


    def test_valid_credentials(self):
        # 测试正确数据
        valid_username='testuser'
        valid_password='testpass'
    
        username_input = self.driver.find_element_by_name('username')
        password_input = self.driver.find_element_by_name('password')
        submit_button = self.driver.find_element_by_css_selector('button[type="subnit"]')
        username_input.clear()
        username_input.send_keys(valid_username)
        password_input.clear()
        password_input.send_keys(valid_password)
        submit_button.click()
        
        # 验证是否成功跳转到主页
        self.assertEqual(self.driver.current_url,"http://example/")
        
        # 验证是否显示用户信息
        user_info = self.driver.find_element_by_css_selector('.user-info').text
        self.assertIn(valid_username, user_info)
     def test_remember_me(self):
        # 测试是否记住用户名和密码
        valid_username ='testuser'
        valid_password='testpass'
        username_input = setf.driver.find_element_by_name('username')
        password_input = self.driver.find_element_by_name('password')
        remember_me_checkbox = self.driver.find_eLement_by_id('remember_me')
        submit_button = self.driver.find_element_by_css_selector("button[type='submit']")

        username_input.clear()
        username_input.send_keys(valid_username)
        password_input.clear()
        password_input.send_keys(valid_password)
        remember_me_checkbox.cLick()
        submit_button.click()

        # 验证是否已记住用户名和密码
        self.driver.get('http://example/")
        username_input = setf.driver.find_element_by_name('username')
        password_input = self.driver.find_element_by_name('password')

        self.assertEqual(usernane_input.get_attribute("value"),valid_usernane)
        self.assertEqual(password_input.get_attribute("value"),valid_password)

Let’s analyze the above script again. Advantages: The unitest framework is used; a test case is added to remember; the initialization code is written in the setup to ensure low coupling of each use case; disadvantages: the number, scenarios, etc. are still far from enough, and there is no Use any advanced writing method without considering abnormal risks such as time waiting. Then I raised the level of inquiry again and looked at the level of the code. The code was relatively repetitive and I intercepted some key points and new parts:

import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
class LoginTestCase(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Chrome(executable_path='/path/to/chromedriver')
        self.driver.get('http:/example/')
    def tearDown(self):
        self.driver.quit()
    def test_invalid_credentials(self):
        # 测试无效用户名和密码
    def test_valid_credentials(self):
        # 测试正确凭据
    def test_remember_me(self):
        # 测试是否记住用户名和密码
    def test_empty_credentials(self):
        # 测试空用户名和密码
    def test_long_credentials(self):
        # 测试过长的用户名和密码
    def test_lockout_after_failed_attempts(self):
        # 测试多次失败登录后账户锁定
        invalid_username = 'testuser'
        invalid_password = 'wrongpass'
        username_input, password_input, submit_button = self._get_login_elements()
        for _ in range(3):
            ...
        # 检查账户是否被锁定
        error_msg = self.driver.find_element_by_css_selector('.alert-danger').text
        self.assertIn('您的账户已被锁定', error_msg)

    def _get_login_elements(self):
        # 获取登录表单元素
        username_input = self.driver.find_element_by_name('username')
        password_input = self.driver.find_element_by_name('password')
        remember_me_checkbox = self.driver.find_element_by_id('remember-me')
        submit_button = self.driver.find_element_by_css_selector('button[type="submit"]')
        return username_input, password_input, remember_me_checkbox, submit_button
if __name__ == '__main__':
    unittest.main()

Continue to comment:

Advantages: The number of use cases has increased, and if you continue to ask, it should be possible to increase; with the shadow of the primary page-object mode, element positioning becomes independent;

Disadvantages: Code redundancy, use case procedures and concurrent threads are not fully utilized, and a lot of execution time is wasted. It still does not take into account the handling of abnormal risks such as network delays and lags. There is no concept of data-driven and keyword-driven yet.

Then we continue to upgrade the question copy and let it be sent concurrently:

def test_login(self):
    #使用多个数据组合测试登录功能
    test_data=
    [
        ('username':'testuser','password':"testpass",'expected_uri':"http://example/",'expected_text':"欢迎testuser"},
        ('username':"fakeuser','password':"fakepass",'expected_error':"用户名或密码不正确"},
        ('username':'','password':"",'expected_error':"请输入用户名"}
    [
    threads = []
    for data in test_data:
        thread = threading.Thread(target=self._perform_login, args=(data,))
        thread.start()
        threads.append(thread)
    for thread in threads:
        thread.join()
def _perform_login(self,data):
    #执行单次登录操作
    username_input,password_input,submit_button=self._get_login_elements()
    
    username_input.clear()
    username_input.send_keys(data['username'])
    password_input.clear()
    password_input.send_keys(data['password'])
    submit_button.click()
    
    if 'expected_error' in data:
        error_msg = self.driver.find_element_by_css_selector('.alert-danger').text
        self.assertIn(data['expected_error'],error_msg)
    else:
        self.assertEqual(data['expected_url'],self.driver.current_url)
        self.assertIn(data['expected_text'],self.driver.page_source)

Comments:

Advantages: The above code uses the thread library for multi-thread concurrency; it cleverly uses the concept of data-driven, not only the user name and password are different, but also the prompts and URLs are asserted each time; it uses keywords initially Driver to allow each thread to automatically determine its own assertion purpose. And the shadow of the po mode still exists;

Disadvantages: The application of multi-threading is not encapsulated in the system, and there are greater risks and limitations; the content structure of the use case is relatively simple, and previously complex tasks such as testing the browser to remember user passwords cannot participate in this concurrency; threads are used as functions based on use cases. The basic unit cannot be implemented for multiple use cases that require context association (so it is better to create a new middle layer between threads and use cases as a unit); data separation is still not implemented.

Continue to optimize the questions and see the results: (the code is still simplified, leaving only the key lines)

class LoginPage:
    def __init__(self, driver):
        self.driver = driver
    def load(self):
        self.driver.get('http:/t/')
    def login(self, username, password, remember_me):
        # 输入用户名和密码
        username_input, password_input, remember_me_checkbox, submit_button = self._get_login_elements()
        ...
        # 点击“登录”按钮
    def get_error_message(self):

        return error_msg
    def get_user_info(self):
        return user_info
    def _get_login_elements(self):
        # 获取登录表单元素
        return username_input, password_input, remember_me_checkbox, submit_button

class LoginTest(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Chrome(executable_path='/path/to/chromedriver')
        self.login_page = LoginPage(self.driver)
    def tearDown(self):
        self.driver.quit()
    def test_valid_login(self):
        # 测试正确凭据登录
        test_data = {'username': 'testuser', 'password': 'testpass', 'remember_me': True}
        self.login_page.load()
        self.login_page.login(test_data['username'], test_data['password'], test_data['remember_me'])
        # 验证是否成功跳转到主页并显示用户信息
    def test_invalid_login(self):
        # 测试无效凭据登录
    def test_empty_username(self):
        # 测试空用户名登录

class LoginKeywordTest(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Chrome(executable_path='/path/to/chromedriver')
    def tearDown(self):
        self.driver.quit()
    def test_valid_login(self):
        # 测试正确凭据登录
        test_data = {'username': 'testuser', 'password': 'testpass', 'remember_me': True}
        self._perform_login(test_data)
        # 验证是否成功跳转到主页并显示用户信息
    def test_invalid_login(self):
        # 测试无效凭据登录
        test_data = {'username': 'fakeuser', 'password': 'fakepass', 'expected_error': '用户名或密码不正确'}
        self._perform_login(test_data)
        # 检查错误消息是否正确显示
    def test_empty_username(self):
        # 测试空用户名登录
         ......
        

Comments:

Advantages: Added keyword-driven, data-driven, po mode, concurrency and other advanced usage.

Disadvantages: Various advanced usages are seriously separated and are not integrated together for implementation. Instead, several independent class demos are simply written.

Summary: There will still be many problems when the code is run directly, requiring a lot of manual correction. Also in the continuous dialogue, it is also necessary to test the students' own understanding of the code and framework construction in order to dig deeper into more value of GPT. Otherwise, the answers to the first questions cannot provide more reference at present.

functional testing phase

Prepare the test cases in the early stage, start executing the tests now, and what are the three key points of daily work testing to verify expectations? Look at the page, flip through the log, and check the database.

So, I continued to ask him to write SQL and check the logs.

Add query conditions and try again

As for the integration into the execution test phase, the above questions feel like having a dragon-slaying knife, but not yet mastering the corresponding skills. The answer he gave us was just to use the dragon-slaying knife to chop down trees and organize the time to think about the problem, so as to just get started. All test students are capable enough to write basic SQL.

4. Summary

If you want chatGPT to produce effective answers instead of saying sorry to you, you need to follow the following four principles:

• Ask questions clearly: describe the problem as clearly and completely as possible
• Be concise and concise: Try to use simple language and concise sentences to express the problem. Currently, many free plug-ins have restrictions on the number of words in the description.
• Single question: please ask one by one instead of putting all the questions in one question
• Do not provide sensitive information: Do not provide any sensitive personal information in the question

As can be seen from the above practices, GPT will help you implement some details and bottom layers. During the development process, we need to pay more attention to surface applications and interactive use. We have good ideas and designs in our daily life, but there is not enough time to implement them personally. A lot of Repeatedly building code is a waste of your own energy. With GPT, you can theoretically achieve explosive output in a short time. Regarding the application of the testing process, I am still in the preliminary exploration stage. Whether it can improve the efficiency of team testing work requires further research. It will be added in the future. Everyone is welcome to communicate online and offline at any time.

Author: JD Logistics Liu Hongyan

Source: JD Cloud Developer Community Ziyuanqishuo Tech Please indicate the source when reprinting