In the software development team, Code Review is a very important quality assurance environment. Good Code Review can promote team growth, while poor Code Review is like running water. After the introduction of LLM, things have undergone some subtle changes:
As code production increases, more code needs to be reviewed.
A lot of AI code that hasn’t been read carefully. That is, the working code generated by GitHub Copilot and ChatGPT may not have been carefully reviewed by its pair (human).
LLM can also help us review the code.
In the past, a small development team (6 people) spent an average of 30 minutes a day on code review (CodeReview). Therefore, no matter from the perspective of efficiency improvement, more code review time will be needed in the future, and it is urgent to improve Code review.
TL;DR version:
Update the AutoDev plug-in to 1.3.0 to experience the preliminary version of Code Review
Use DevOpsGenius to experience tool testing: https://github.com/unit-mesh/devops-genius
Thinking: Why does Code Review need collaborative combination?
First, let’s review: What are good technical practices—such as how to write good commit messages?
1. Requirements related code changes!
In writing commit messages, or in the middle school of DevOps standardization, the industry's best practices are: keeping commit messages and user stories, Requirements association.
For example, when we use the JIRA tool, we need to combine the JIRA ID in the submission information; when we use GitHub, we use the form # + issue id, as shown below: feat(devops): init first review command #8 . Based on this, we will keep the requirement information consistent with the code changes to facilitate review.
The question is, what if the developer is unwilling to write the commit message? ——Of course it is automatically generated using AIGC.
2. AIGC cannot replace quality management tools!
After understanding the basic principles of AIGC, most people will focus on the security of the code generated by AIGC. As for code security issues, before AIGC existed, we would use tools such as Sonarqube, JavaLint, ArchGuard, Feakin, etc. to monitor and improve the system's code, architectural quality, etc.
Considering that the results of AIGC are affected by prompts and lose focus under long prompts, it is not as stable as static code analysis in terms of reliability.
3. Are we concerned about business or code issues?
In fact, depending on the different code review formats, such as time (daily, weekly, each release), node (before release), the content that everyone will pay attention to is very different. For example, when I am in different stages of the team:
Junior development focuses on whether it conforms to code specifications, that is, other people's improvement opinions.
Intermediate development, focusing on whether you can get off work on time~~.
Advanced development focuses on whether what others write is reasonable and deviates from the business.
The technical leader, while reviewing, focuses on whether it will affect the overall progress - such as whether it can be completed on time.
Therefore, judging from the content of the review, it will mainly focus on the business and technology parts. Different people focus on different things. The part that should be left to traditional tools should still be left to traditional tools.
Design: AIGC-assisted Code Review
When it comes to using AIGC to assist Code Review, there are already various solutions on the Internet, and ours is just one of them.
Threshold: Good code submission practices
Going back to our previous experience in building AutoDev, when it comes to Code Review, the difficulty lies not in the design of the review point, but in how to handle the context. Unlike other fields, if you do not submit in small steps, it will be very difficult during review.
To put it simply:If the code submission practice is not done well, then AIGC's Code review will be very useless. Too divergent intentions prevent us from paying attention to every little point.
Simulate real code review designs
When designing our tool, we roughly analyzed various prompts on the Internet. For this reason, we found that prompt is not the difficulty. The difficulty of combining AIGC to conduct code review is: how to simulate real code review. Only by understanding and in-depth understanding of real Code Review practices can we design good tools.
In daily Code Review, first of all, we need to know the corresponding business background; secondly, ignore a large number of irrelevant non-code changes, such as document updates, test data updates, etc.; finally, if the changes are complex scenarios, skip the patch. View the code directly.
Based on this, we constructed the Prompt evolution route of Code Review. The first version is simpler and revolves around optimizing the prompt length to include more context:
Combine with DevOps platform to obtain business information.
Filter unnecessary files.
Filter out unnecessary content in the patch.
Optimize the generated results.
That is, through practice + configuration, we will highlight the key points and core content, while ignoring the irrelevant things.
Implementation: AutoDev + DevOpsGenius
Whether on the IDE side or the platform side, in the current first version, the two processing logics are similar. The main process is as follows (generated by ChatGPT):
Get commit messages: Get a series of commit messages from the version control system.
Parse commit messages: Parse commit messages into a processable format.
Filter commits that require review: Decide which commits require code review based on some conditions.
Get related user story information: Extract the issue identifier in the commit message, and then get the related user story title.
Generate diffs: Generate diffs between commits for review.
Send a review request: Send a review request to the review tool and wait for the review results.
The big difference is that the two Prompts are for different scenarios. The IDE is for developers who want to know about business changes, while the platform side is for checking syntax, etc.
Prompt design
Let’s first look at the context in Prompt (PS: You can read the simple-review.open-ai.vm template file in the code for more details).
.....
Business Context: ${context.businessContext}
Commit Message: ${context.fullMessage}
Code Changes: ${context.changes}simply put:
Business Context refers to the business context obtained from the Kanban and requirements systems.
Commit Message refers to the commit message written by the user.
Code Changes is based on the patch file before submission.
It should be noted that since most of the large coding model corpus and data are trained based on patch, its effect will be better. Therefore, the ideal changes form needs to be selected based on different models.
Combine submission information with filtering configuration
The following is an example submission format for Conventional Commits :
parses out the information we need type, references and other information as part of the filtering conditions:
// 从配置文件中读取，并过滤掉不需要 review 的 commit，诸如 chore、ci, docs 等
// 如果没有配置，则全部需要 review
val filterCommits = parsedMsgs.filter {
if (it.meta.containsKey("type")) {
val type = it.meta["type"] as String
project.commitLog?.isIgnoreType(type) ?: true
} else {
true
}
}Then only the key context is presented, reducing the length of the prompt. For more details, you can read the code and documentation of AutoDev and DevOps Genius: https://devops.unitmesh.cc/code-review.
Prompt differences in IDE and DevOpsGenius platform
In Code Review combined with IDE, we often choose multiple submissions of our own to explain the corresponding business functions, implementation, etc. And this also means that in the IDE scenario, it is easier for AI to summarize beyond the context limits. Therefore, more review methods need to be added to support it.
On the platform, it is more suitable to combine single submission or Pull Request. However, the basic patch optimization strategies are similar.
shortcomings
In fact, if you participate in a real team review, you will find that even under the same department, different teams have different code review methods and their focus. How to organically combine concerns and prompts is what we need to consider next.
The future: evolution
Combining LLM to reshape DevOps practices in software development is not a simple matter. If you are interested, welcome to explore together: https://github.com/unit-mesh/devops-genius.