Exploring the prompt coding paradigm: how to elegantly build test codes to generate prompt words?

From April until now, we have continued to write a series of functions for AutoDev. Despite being in development for over three months, we've been continuously thinking about and refactoring the way we manage prompts. In the upcoming AutoDev 0.8, we further improved the existing context construction method, rethought and designed a new context engineering system in a modular way.

And test generation is our first new pilot, is it more convenient to explore the new prompt mode? Here is a video based on the new prompt coding paradigm:

See detailed code: https://github.com/unit-mesh/auto-dev

AutoDev's prompt evolution

In that " Prompt Writing Mode: How to Give a Thinking Framework to a Machine ", I summarized how to better write prmopt. So, in ArchGuard Co-mate, we coded these patterns:

interface BaseTemplate {
    fun getRole(): String = ""
    fun getInstruction(): String = ""
    fun getRequirements(): String = ""
    fun getSample(): String = ""
    fun getExtendData(): String = ""
}

In the course of developing AutoDev, we found that it matched what we understand as a programming paradigm: schematization. However, after the introduction of standardized code generation, some prompts become configurations to support different teams to configure their own prompts:

{
    "spec": {
    "controller": "- 在 Controller 中使用 BeanUtils.copyProperties 进行 DTO 转换 Entity\n- 禁止使用 Autowired\n-使用 Swagger Annotation 表明 API 含义\n-Controller 方法应该捕获并处理业务异常，不应该抛出系统异常。",
    "service": "- Service 层应该使用构造函数注入或者 setter 注入，不要使用 @Autowired 注解注入。",
    "entity": "- Entity 类应该使用 JPA 注解进行数据库映射\n- 实体类名应该与对应的数据库表名相同。实体类应该使用注解标记主键和表名，例如：@Id、@GeneratedValue、@Table 等。",
    "repository": "- Repository 接口应该继承 JpaRepository 接口，以获得基本的 CRUD 操作",
    "ddl": "-  字段应该使用 NOT NULL 约束，确保数据的完整性"
    }
}

With the further evolution of requirements, we added the required code context, technical framework, etc. based on import and similarity.

such as:

when (action) {
  ...
    ChatActionType.CODE_COMPLETE -> {
        val codeComplete = customPromptConfig?.autoComplete
        if (codeComplete?.instruction?.isNotEmpty() == true) {
            prompt = codeComplete.instruction
        }
        when {
            MvcUtil.isController(fileName, lang) -> {
                val spec = CustomPromptConfig.load().spec["controller"]
                if (!spec.isNullOrEmpty()) {
                    additionContext = "requirements: \n$spec"
                }
                additionContext += mvcContextService.controllerPrompt(file)
            }
          ...
         }
    }
}

And as we further iterate our functionality, code similar to the above will become more complex. In addition, as our support for multiple languages ​​​​is getting better, we can no longer create a complex prompt based on each language. So, we need to think about some new paradigms.

AutoDev new prompt paradigm

After combining the previously referenced design ideas of JetBrains AI Assistant, we redesigned part of the prompts in writing the prompts generated by the automatic test. A new prompt will consist of the following parts:

• ChatActionType. That is, the initial instruction (instruction), such as writing tests, interpreting code, etc.

• Specific scene requirements. Based on specific scenarios, such as different layered writing modes under MVC.

• technology stack context. Specific prompts generated according to different languages ​​and technology stacks.

• code context. Refined code information is generated as comments.

• <code>

• Command start prompt word. That is, it is used to more clearly prompt AI, the return format that humans expect.

Thus, a final prompt example is as follows ([xxx] is for explanation only):

【ChatActionType】
Write unit test for following code.
【特定场景要求】
You MUST return code only, not explain.
You MUST use given-when-then style.
You MUST use should_xx style for test method name.
When testing controller, you MUST use MockMvc and test API only.
【技术栈上下文】
You are working on a project that uses Spring MVC,Spring WebFlux,JDBC to build RESTful APIs.
【代码上下文】
// class name: BookMeetingRoomRequest
// class fields: meetingRoomId
// ...
// class name: BookMeetingRoomResponse
// class fields: bookingId meetingRoomId userId startTime endTime
// ...
【代码】
```java
@PostMapping("/{meetingRoomId}/book")
    public ResponseEntity<BookMeetingRoomResponse> bookMeetingRoom(@PathVariable String meetingRoomId, @RequestBody BookMeetingRoomRequest request) {
        BookMeetingRoomResponse response = new BookMeetingRoomResponse();
        return new ResponseEntity<>(response, HttpStatus.CREATED);
    }
```
【指令起始提示词】
Start with `import` syntax here

Simply put, a complex prompt will complete the context through a series of dependency injections (Intellij extension points).

ChatActionType

That is, instructions created according to the user's intent, such as interpreting code, refactoring code, writing tests, etc.:

when (this) {
    EXPLAIN -> "Explain selected $lang code"
    REVIEW -> "Code Review for following $lang code"
    REFACTOR -> "Refactor the following $lang code"
    CODE_COMPLETE -> "Complete $lang  code, return rest code, no explaining"
    WRITE_TEST -> "Write unit test for following $lang code"
    FIX_ISSUE -> "Help me fix this issue"
    GEN_COMMIT_MESSAGE -> """suggest 10 commit messages based on the following diff:..."""
    CREATE_DDL -> "create ddl"
    CREATE_CHANGELOG -> "generate release note"
    CHAT -> ""
}

Then, according to different types, add corresponding instruction requirements, such as how to write code, command style, etc. under the MVC layer.

Specific scene requirements

According to different scenario requirements, such as when writing Java Controller tests, we expect to use MockMVC as the API test framework to generate tests, and when writing Service tests, we expect to use Mockito as the Mock framework to generate tests. So, a corresponding code is:

class JavaTestContextProvider : ChatContextProvider {
override fun isApplicable(project: Project, creationContext: ChatCreationContext): Boolean {
    return creationContext.action == ChatActionType.WRITE_TEST
}
override fun collect(project: Project, creationContext: ChatCreationContext): List<ChatContextItem> {
   ...
}

Then, provide corresponding prompts according to different types, such as Controller and Service.

technology stack context

Similarly, in order to generate this prompt  You are working on a project that uses Spring MVC,Spring WebFlux,JDBC to build RESTful APIs. , we need to obtain the project framework from dependency management tools/build tools (such as Gradle, Package.json), and only list key technology stacks to generate code that conforms to the project technology stack. That is, in the above prompt:  Spring MVC,Spring WebFlux,JDBC .

So, the code would look something like:

override fun isApplicable(project: Project, creationContext: ChatCreationContext): Boolean {
    return hasProjectLibraries(project)
}
private fun hasProjectLibraries(project: Project): Boolean {
    prepareLibraryData(project)?.forEach {
        if (it.groupId?.contains("org.springframework") == true) {
            return true
        }
    }
    return false
}

See the relevant code for details.

code context

That is, the code related to the current code. In order to reduce the impact of useless prompts, we did not completely adopt the same  Jaccard Similarity method as GitHub Copilot to build, but through two strategies.

• In key scenarios, such as CRUD code scenarios, the reference code of the import + selection code is presented, and a UML-like way is used to reduce the context.

• In other scenarios, refer to (copy) the similar chunk implementation of JetBrains AI Assistant as part of the code.

Everyone is familiar with this part. The only difference is how to support multiple languages. You can see the implementation in the code yourself.

command prompt word

When targeting large new features, Java programmers usually create a new Controller, etc.; when optimizing function points, we usually modify existing functions or add new functions.

Therefore, in AutoDev's automatic test generation, in order to make LLM more understandable, we added a trailing hint word.

if (!testContext.isNewFile) {
    "Start test code with `@Test` syntax here:  \n"
} else {
    "Start ${testContext.testClassName} with `import` syntax here:  \n"
}

Simply put, it is to let GPT know what to do next, so as to avoid losing focus due to various prompts in the process.

Summarize

Summarized by ChatGPT

The AutoDev 0.8 project continues to evolve through continuous development, with a special focus on prompt optimization and refactoring. The new context engineering system is redesigned in a modular way, realizing smarter and more humanized code prompts and generation. The new prompt paradigm combines elements such as ChatActionType, specific scenario requirements, technology stack context, code context, and command start prompts, enabling AutoDev to better understand the developer's intentions and generate code that meets expectations based on specific scenarios and technology stacks . This improvement not only improves development efficiency and reduces errors, but also enhances the intelligence and practicality of AutoDev.