What is TestAgent?
TestAgent aims to build an "intelligent agent" in the testing field, integrate large models and engineering technology in the quality field, and promote the upgrading of quality technology generations. We look forward to working with community members to create innovative solutions in the testing field and build a 24-hour online testing assistant service to make testing as smooth as silk.
We are pleased to announce that we have open sourced TestAgent, a large model and tool in the testing industry for the first time in China. This version includes the most powerful 7B test field large model, as well as the supporting local model rapid release and experience engineering framework. Welcome to experience and pay attention!
project address:
GitHub - codefuse-ai/Test-Agent
Local Mac M1 experience effect
Magic experience effect
Experience address:TestGPT Test Assistant Demo
Features of this issue
- Model: In this issue, we open source the test domain model TestGPT-7B. The model is based on CodeLlama-7B and has been fine-tuned on related downstream tasks:
-
- Multi-language test case generation (Java/Python/Javascript) has always been an area of great concern to academia and industry. In recent years, new products or tools have been hatched. , such as EvoSuite, Randoop, SmartUnit, etc. However, traditional use case generation has pain points that are difficult to solve. Test case generation based on large models is superior to traditional use case generation tools in terms of test case readability, test scenario completeness, and multi-language support. This time, we focus on supporting multi-language test case generation. Our open source version first includes test case generation capabilities for Java, Python, and Javascript. In the next version, languages such as Go and C++ will gradually be opened.
- Test case Assert completion When analyzing and exploring the current status of test cases, we found that there is a certain proportion of existing test cases in the code warehouse that do not contain Assert. Although the test cases without Assert can be executed and passed during the regression process, the problem cannot be found. Therefore, we expanded the scenario of automatic completion of test case Assert. Through this model capability, combined with certain engineering support, it is possible to automatically complete batches of test cases in the entire library and intelligently improve the quality of the project.
- Engineering framework:Local model rapid release and experience engineering framework.
-
- ChatBot page
- Model quick start
- Private deployment, localized GPT large model interacts with your data and environment, no risk of data leakage, 100% safe
We will continue to iterate our models and engineering capabilities in the future:
- Continuously add more exciting test domain application scenarios, such as domain knowledge Q&A, test scenario analysis, etc.
- Support the openness of the copilot engineering framework for test scenarios, such as intelligent embedding of test domain knowledge, general test tool API system, intelligent test agent, etc., so stay tuned!
- Based on 7B, it will gradually expand to 13B and 34B models. Welcome to pay attention!
The most powerful large model in the 7B test field
Currently in TestAgent, we use the TestGPT-7B model by default. Compared with the current open source models, the TestGPT-7B model is at the leading level in the industry in terms of use case execution pass rate (pass@1) and use case scenario coverage (average number of test scenarios).
The evaluation results of the core capabilities of the TestGPT-7B model are as follows:
- Multilingual test case generation
For the three languages supported by the model: Java, Python, and Javascript, the Pass@1 evaluation results are as follows:
| Model |
Java pass@1 |
Java Average number of test scenarios |
Python pass@1 |
Python Average number of test scenarios |
Javascript pass@1 |
Javascript Average number of test scenarios |
| TestGPT-7B |
48.6% |
4.37 |
35.67% |
3.56 |
36% |
2.76 |
| CodeLlama-13B-Instruct |
40.54% |
1.08 |
30.57% |
1.65 |
31.7% |
3.13 |
| Qwen-14B-Chat |
10.81% |
2.78 |
15.9% |
1.32 |
9.15% |
4.22 |
| Baichuan2-13B-Chat |
13.5% |
2.24 |
12.7% |
2.12 |
6.1% |
3.31 |
- Test case Assert completion
The current model supports Assert completion for Java use cases. The Pass@1 evaluation results are as follows:
| Model |
pass@1 |
Strong verification ratio |
| Codefuse-TestGPT-7B |
71.1% |
100% |
Engineering architecture
The horn of large models has sounded, and large models in the testing field are also constantly evolving. Through the rich world knowledge accumulated in the pre-training process, they have demonstrated extraordinary reasoning and decision-making capabilities in complex interactive environments.
Although the basic model has achieved remarkable results in the testing field, there are still some limitations, and domain-specific testing tasks usually require specialized tools or domain knowledge to solve. For example, the basic model can complete tasks such as single test code generation and test text generation through pre-training knowledge, but when dealing with complex integrated use case generation, domain-specific use case generation, and test process pipeline interaction, more professional tools and fields are needed. Knowledge.
Therefore, integrating specialized tools with basic models can give full play to their respective advantages. Specialized tools can solve the problem of insufficient model timeliness, enhance professional knowledge, improve interpretability and robustness. The basic model has human-like reasoning and planning capabilities, can understand complex data and scenarios, and interact with the real world.
Based on the open model engineering deployment and ChatBot in this issue, we will continue to invest deeply in the field of testing open source. Collaborate with like-minded developers in the community to create the most advanced Tools engineering system, intelligent testing assistants and open source testing projects in the testing field!
Quick to use
Preparation
Model download
You can get it onmodelscope or huggingface Model details and download model files.
Environment installation
git clone https://github.com/codefuse-ai/Test-Agent
cd Test-Agent
pip install -r requirements.txtBefore starting to run the TestGPT-7B model, please ensure that your execution environment has approximately 14GB of video memory.
Start service
The project provides the ability to quickly build UI on the web page to more intuitively display model interactions and effects. We can use a few simple commands to wake up the front-end page and call the model capabilities in real time. In the project directory, start the following services in sequence:
Start controller
python3 -m chat.server.controller
Start model worker
python3 -m chat.server.model_worker --model-path models/testgpt --device mps
For the startup method, you can choose the following configuration options as needed:
- --device mps option to enable GPU acceleration on Mac computers (Apple Silicon or AMD GPUs);
- --device xpu option to enable acceleration on Intel XPU (Intel Data Center and Arc A-Series GPUs);
-
- Intel Extension for PyTorch Intel Extension for PyTorch
- Set OneAPI environment variables: source /opt/intel/oneapi/setvars.sh
- --device npu is the option to enable acceleration on Huawei AI processor;
-
- Requires installationAscend PyTorch Adapter
- Set CANN environment variables: source /usr/local/Ascend/ascend-toolkit/set_env.sh
- --device cpu option to run on CPU alone, no GPU required;
- --num-gpus 2 specifies options for concurrent gpu running.
Start web service
python3 -m chat.server.gradio_testgpt
After the service is ready, we can open the locally started web service address (http://0.0.0.0:7860), You can see the complete front-end page.
contact us
Welcome to the codefuse official website:codefuse