In the first two articles of this series, " LLM-Enabled R&D Effectiveness " and " LLM-Enabled Software Architecture ", we introduced a series of experiments that we combined with BizDevOps and software architecture in LLM. Focusing on these two types of exploration, we built two open source LLM tools: AI-assisted programming tool AutoDev (IDEA plug-in), and architecture governance tool ArchGuard Co-mate.
Take a look first, based on our understanding of LLM + software development AutoDev automatic CRUD video:
Introduction: How to use the limited ability of LLM to solve complex problems?
Let's look at a few different LLM usage scenarios:
Format-specific article summary
As a developer who writes article summaries every week, I found a fact in my past article [AI Summary]: LLM's ability to summarize my articles is extremely poor, and most of the time it is very tasteless. When it is long enough, there is a higher chance of summarizing wrongly.
However, my articles often look at the headlines, first-level titles, and second-level titles to get a preliminary impression. So if only a few levels of headings are handed over to LLM to write, the effect is far more efficient than LLM reading the full text.
DSL as language interface
As a developer who has explored combining LLM applications, I assume you have also tried using a DSL (Domain Specific Language) as an API for LLM. Such as using JSON as the return result of the LLM API, or an internal DSL similar to Co-mate, or a streaming DSL method similar to LangChain:
Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [introduce_system]
Action Input: the input to the action
...Either way, we're exploring API returns based on specific data structures.
LLM in processes and procedures
We all know that the contextual ability of LLM (not only the token limit, but also the ability to understand complex problems) is limited, it cannot solve complex problems at one time - we need to do a good job of Tasking (dismantling steps), and then hand it over to different LLMs to complete.
In the case of unclear procedures or processes, we can build a method similar to AutoGPT to automatically complete the task. However, due to the uncertainty of the process itself and the accuracy of the process results, the results are often unreliable. For example, suppose our problem needs to be disassembled into 6 steps, and the accuracy rate generated by LLM is 80%, and 0.8^6 = 0.262144, that is, only 1/4 of the possibility will generate what we want.
In the field of software engineering, when the process is sufficiently standardized, abstraction can be used to solve the problem of too long context. Therefore, when we explore developer tools, we don’t need a tool like LangChain to dynamically build context without considering the limited capabilities of the vector database, which will waste my token (money).
Process refers to the various links and steps in the development process, such as requirements analysis, design, coding, testing, etc. Integrating processes into AIGC touchpoints means that LLM and other tools can be used to aid development in each process.
Simply put, how humans solve complex problems requires how AI solves complex problems.
Element 1: Coding software processes to expand AIGC touchpoints
Software development is a complex group intellectual activity. Just like our previous exploration of combining BizDevOps in LLM, we need to open and look at the existing software development process, and refine the subtasks as much as possible, and combine them with AI to see obvious results.
That is, in the past, it was through the establishment of standardized software engineering and development processes that we got rid of the bazaar model and moved towards the cathedral model. And in a standardized process/assembly line, it is easy to be automated.
Coding of standardized processes
In AutoDev, in order to automate the code writing for CRUD, it is divided into five steps from the abstract level. That is, as described by CrudFlowAction in the code:
interface CrudFlowAction {
fun getOrCreateStoryDetail(id: String): String
fun updateOrCreateDtoAndEntity(storyDetail: String)
fun fetchSuggestEndpoint(storyDetail: String): TargetEndpoint
fun updateOrCreateEndpointCode(target: TargetEndpoint, storyDetail: String)
fun updateOrCreateServiceAndRepository()
}The same is to write code, we need a complete coding process of the simulated human, split to a sufficiently fine granularity:
Update or create new DTOs as needed
According to the requirements, find a suitable Controller as the modification entry (Endpoint)
Update the Controller code or create a new Controller as required
According to the Controller, create or update the corresponding Service and Repository codes.
And the most tricky point of this is that if the existing process lacks a link, it will cause too many problems in the generated results. For example, if we lack the generation of DTO, the Controller will not be generated correctly, and the wrong DTO method will be called.
Expand AIGC contacts in process
In the AIGC combined software development process, we can simply delineate three important stages: requirements, design, coding (including automated testing). In order to realize the aforementioned AutoDev automation, we need two key inputs: instantiation requirements and API interface design.
Requirements by example refers to transforming requirements descriptions into specific examples, use cases or scenarios according to the requirements and specifications of specific projects, so as to facilitate subsequent development and testing.
API interface design refers to defining and planning the interface of the system according to the requirements and system design, including the input, output, parameters and other information of the interface.
For these two key inputs, the usability of the generated code will be higher. And the premise is that there is a human intervention in generating the results. That is, when generating instantiation requirements, there needs to be a mechanism to check whether the results are accurate; when generating API interface design, there needs to be a mechanism to check whether the API is reasonable. And this mechanism can be checked by humans, or it can be checked by AIGC combined with specifications.
And once our AIGC stops at the code stage and there is no requirement stage and design stage, then humans may not give instantiation requirements or API interfaces because there are only 24 hours in a day. As a result, AIGC cannot be generated further downwards, so the improved results are very limited.
Element 2: Design around the co-pilot to improve the developer experience
Going back to the AutoDev video we introduced at the beginning, we will get the issue on GitHub in the initial stage. According to the issue ID such as #1, the content is as follows:
用户故事：新建博客
作为一位博客作者
我想在InnoRev网站上新建博客
以便于我可以发布我的文章
AC 1: 新建博客成功
假设用户已经登录，并且填写了正确的博客信息
当用户点击“新建博客”按钮
于是系统创建一个新的博客，并返回博客的id
AC 2: 博客标题过长
假设用户已经登录，并且填写了超过50个字符的博客标题
当用户点击“新建博客”按钮
于是系统返回“博客标题过长”的错误信息
AC 3:...
AC 4:...
// {API Method} POST {API URI} /blogs
// CreateBlogRequest: { "title": "My First Blog", "content": "Hello world!" }
// CreateBlogResponse: { "id": 1, "title": "My First Blog", "content": "Hello world!" }The comments consist of two parts, user story and API design. With the API design and response as the input and output, and the existing code as the context, no matter who is handed over to design the code, as long as the input and output meet the requirements, the AIGC is qualified.
Designed around the Copilot
Whether it is the IDE used by developers, or the tools for product managers, architects, and quality testing, we all need to design around the daily life of these developers.
In GitHub Copilot, we will be accustomed to the inlays mode it provides to provide intelligent code filling. And only this core capability is not enough. In daily development, we spend a lot of time on debugging or other issues, and we need to design around these links.
Therefore, a series of AI capabilities are provided in GitHub Copilot X, or the new version of JetBrains IDE: New chat using selection, Write documentation, Explain code, Suggest refactoring, Find potential problems, etc.
Of course, we also provide corresponding capabilities in AutoDev, and there are more powerful auxiliary CRUD functions.
Perfect developer experience
Many existing LLM model manufacturers are irresponsible, they only provide the coding capability of the model itself, but not supporting coding tools. Here, LLM itself is a weak point, and companies still need to build IDE plug-ins to improve the developer experience.
And if we can design IDE tools ourselves, then we should try to add some features that enhance the developer experience. For example, directly select the error log of development and runtime to ask LLM to quickly fix this problem.
Or in AutoDev, you can let LLM generate suggestions based on code changes when writing commit messages. The corresponding prompt is as follows:
suggest 10 commit messages based on the following diff:
commit messages should:
- follow conventional commits
- message format should be: <type>[scope]: <description>
examples:
- fix(authentication): add password regex pattern
- feat(storage): add new test casesAfter we provide basic specifications and examples, the commit messages it generates are more and more standard.
Element 3: Built-in development specifications to improve software quality
As we all know, GitHub Copilot will generate new code based on our current coding habits. This may lead to: because the original code is non-standard, the generated code is also non-standard. Therefore, in the "LLM-enabled Software Architecture", we mentioned that architecture and coding standards should be built into AIGC tools. Based on this, the AIGC quality of software development can be improved and the usability of the code can be enhanced.
In the previous " Context Engineering: Real-time Capability Analysis and Thinking Based on Github Copilot ", we shared the basic strategy of GitHub context in realizing automatic filling:
defaultPriorities.json = [
"BeforeCursor",
"SimilarFile",
"ImportedFile",
"PathMarker",
"LanguageMarker"
]Therefore, it will only calculate based on the user's behavior, and bring relevant codes based on the results. Similarly, we can also getMostRecentFiles calculate the relevant files through , pass through tokenLevelJaccardSimilarity, and finally generate SimilarChunkContext .
And in order to write the code, it is also standardized, we need to add some specific rules, such as when programming for business systems, you can:
when {
isController -> {
val spec = PromptConfig.load().spec["controller"]
additionContext += mvcContextService.controllerPrompt(file)
}
isService -> {
val spec = PromptConfig.load().spec["service"]
additionContext += mvcContextService.servicePrompt(file)
}
}And the specification itself should be configurable:
{
"spec": {
"controller": "- 在 Controller 中使用 BeanUtils.copyProperties 进行 DTO 转换 Entity\n- 禁止使用 Autowired\n-使用 Swagger Annotation 表明 API 含义\n-Controller 方法应该捕获并处理业务异常，不应该抛出系统异常。",
"service": "- Service 层应该使用构造函数注入或者 setter 注入，不要使用 @Autowired 注解注入。",
"entity": "- Entity 类应该使用 JPA 注解进行数据库映射\n- 实体类名应该与对应的数据库表名相同。实体类应该使用注解标记主键和表名，例如：@Id、@GeneratedValue、@Table 等。",
"repository": "- Repository 接口应该继承 JpaRepository 接口，以获得基本的 CRUD 操作",
"ddl": "- 字段应该使用 NOT NULL 约束，确保数据的完整性"
}
}Different teams slightly modify their standard configurations according to their business scenarios and needs.
summary
Summarized by ChatGPT:
The R&D tools empowered by LLM improve the efficiency and quality of software development by coding software processes, improving developer experience and built-in development specifications. They cover the end-to-end link and assist developers in completing tasks in an automated and intelligent manner. By refining the process, providing intelligent code completion and hints, and built-in specifications, these tools enable developers to write code more efficiently and ensure that the generated code conforms to the specification. The R&D tools empowered by LLM combine artificial intelligence with the software development process to provide developers with a better development experience and higher-quality code.