#current situation
Generally do database-related development, unless learning, otherwise few people are willing to use JDBC directly. Originally Java code is more verbose, and the verbosity of writing code directly with JDBC is maddening! So in the actual development process, we usually use some frameworks/libraries to help us operate the database. And there are many choices in the open source market, as far as I have come into contact with: Hibernate, MyBatis, JdbcTemplate, DbUtils, ActiveRecord, JavaLite and so on. These frameworks can greatly improve the development efficiency. For some basic CRUD operations, although there are differences, they are basically sufficient.
However, for a slightly more complex data query, it is inevitable to write SQL code manually, and even to dynamically splicing SQL according to parameters. All kinds of frameworks basically have a set of solutions for splicing dynamic SQL, and they can easily convert the queried data into objects (DTO).
But so far, although these frameworks can easily help us complete the data mapping, these DTOs still need to be written manually one by one.
#existing problem
Usually, after writing the SQL query code, we need to have a corresponding DTO to map the data queried in the database to the DTO, so that the calling program can better use the data. Of course, in order to save trouble, sometimes the data is directly stored in a data structure like Map. However, although the Map method is very portable, it will bring several more important potential problems:
- The caller needs to remember the name of each key in the Map, which will bring some so-called memory burden to the programmer
- Excessive memory burden will lead to complex logic of the system, difficult to understand, and more difficult to maintain
- After the SQL change causes the key to change, it is difficult to find the problem, and the programmer needs to deal with these changes very carefully
If we want to avoid these problems caused by Map, we need to write a separate DTO for each SQL query. Although it is not difficult to write these DTOs, they are very tedious, especially when there are many fields; and if the fields of the SQL query are changed, remember to come back and modify the DTO. Although writing DTOs alone alleviates some of the problems caused by Maps, it also adds new workloads.
It would be perfect if there was a way to automatically do the following two points after the SQL code (including dynamically spliced SQL) was written:
- According to the SQL code, directly generate the corresponding DTO
- Change the SQL code and automatically modify the corresponding DTO
In this way, on the one hand, the trouble of manually writing DTOs is solved; on the other hand, when a field is changed due to the modification of SQL, since the automatically generated DTOs are also modified synchronously, in those places that refer to this field, the compiler will Immediately give an error message! Make the problem can be found immediately as soon as it occurs, which can avoid many potential problems.
This article is trying to solve the problem of how to automatically generate DTO according to SQL code, save the trouble of manual writing, and improve the development efficiency of programmers.
#Solution ideas
Ideal is always beautiful, reality is always cruel!
So, whether this idea can be realized, let's first analyze the feasibility of automatically generating DTO:
To achieve automatic DTO generation, the core is to get each column name and data type corresponding to the SQL query. With the column names and data types, it's easy to write a method to generate a DTO.
We know that in general, after the SQL query is written, including calling stored procedures and those SQL that are dynamically spliced according to the call parameters, although the final running SQL may be different, the field part of the query result is relatively fixed. of.
Of course, there are also rare cases where queries with uncertain fields are encountered, but in this extreme case, even if you can't write DTO manually, it is more suitable to use Map, which we won't discuss here.
So, how can I get the column name and type?
One solution is to analyze the fields in the SELECT part of the SQL code, but its limitations are relatively large:
- For the spliced SQL code, the analysis is more difficult
- The type of the field is also difficult to determine
- SELECT * ...; CALL statement such a common query method is also very difficult to analyze
The above scheme seems to be somewhat feasible for the way of writing SQL using configuration files (xml) like Mybatis. I have not tested it specifically, but it is estimated that there will be no less difficulties.
Another solution is to find a way to run this code containing SQL directly:
We know that JDBC executes a SQL query and returns a ResultSet object. Through the method getMetaData() in this object, we can get some metadata of this query: such as column name, column type, and the table name where the column is located, etc. These are The information is enough for us to generate the class we need.
So, how can we run the code that contains SQL?
It is a little easier to say about those fixed SQL statements. We get this fixed SQL and call JDBC to get MetaData, and then we can easily generate DTO based on this information. However, for those complex SQL queries that need to be dynamically generated according to a series of parameters, they cannot be run directly before the parameters are set, and MetaData cannot be obtained. Without MetaData, we cannot generate DTO.
How to do?
As discussed above, even in dynamic SQL, no matter what parameters are input, although the executed SQL statement may be different, the final result column is fixed. Isn't the problem we need to solve now to get these column information? That being the case, we construct a series of default parameter values. These parameters are of no practical use, just to allow us to edit the SQL code to run normally in order to get the required MetaData, it doesn't matter whether the data can be queried or not.
Usually the SQL code we write has two forms: one is directly in the Java code, and the other is placed in the configuration file. Which form is better is not discussed here, I will find a separate place to discuss it later. The main discussion here is the SQL spliced in the Java code, how to implement a code generator to automatically generate these DTOs:
To solve this problem fully automatically, let's take a look at some of the challenges this code generator faces and how to deal with it:
- How to identify a piece of SQL code that needs to generate DTOs
First, we need to identify this code so that the code generator can run this code that needs to generate DTOs. Usually, our data interface is at the method level, so we can annotate the method and use the annotation to identify the method to return a DTO object is a good choice.
- How to define the class name of DTO
A method that is easy to think of is to automatically combine a name by the class name + method name where the SQL code is located. Of course, sometimes for flexible control, the programmer should be allowed to specify a name.
- How to execute code
The key to executing the code is to construct a batch of suitable parameters that can invoke the annotated method. Of course, it is first necessary to perform code analysis on the annotated method to extract the method parameter name and type. Code analysis can use tools like JavaCC, or some syntax analyzers, which will not be discussed in detail here. The following mainly discusses the construction of default parameters:
To simplify things, by default we can construct as follows:
数字型参数，默认为：0, 例如：public Object find(int arg){...} 构造 int arg=0;
字符串参数，默认为："", 构造 String arg="";
布尔型参数，默认为：false, 构造 boolean arg=false;
数组型参数，默认为：类型[0], 构造 int[] arg=new int[0];
对象型参数，默认为：new 类型()， 例如：public Object find(User arg){...} 构造 User arg=new User();
Of course, for some simple parameters, the above construction rules can basically work. However, for some parameters: for example, the parameter is an interface, or a table name that needs to be dynamically connected, or the logic of the SQL splicing code requires that the parameter must be some special value, etc., the parameter constructed by default will cause the program to Unable to execute.
But how can we make our code generator continue to execute? It seems that there is really no way to deal with it automatically, so we have to leave this problem to the programmer, and let the programmer help the code generator to complete the initialization of the parameters.
We can provide a parameter on the annotation, which mainly completes the setting of parameters that cannot be initialized under the default rules. Of course, the initialization code in this parameter can also override the default rules, so that we can test and execute different SQL processes during the editing phase.
- How to generate DTOs
After the above series of processing, we can finally automatically run the method containing the SQL query code. However, right now we haven't got the MetaData we want, and we haven't been able to generate DTOs yet.
A possible way is to wrap a JDBC to intercept the SQL query executed when this method is called, but the problem is that it will be troublesome if there are multiple queries in the method.
The other method depends on the support of the framework. It can intercept the return statement of the method and obtain the SQL statement executed by it. With the SQL statement, it is not difficult to generate DTO.
- How to modify the code
In order to minimize the programmer's work, our code generator needs to automatically modify the return value of the method to this DTO class after generating the DTO.
- How to handle SQL changes
The simple way is: once a certain SQL code changes, regenerate all DTOs according to the previous method. However, it is clear that when there are many query methods, the process of DTO code generation will be unbearably slow.
Another more reasonable approach is: we add a fingerprint field when generating DTO, the value of which can be generated with the information contained in the SQL code, for example: code length + code hashCode. The code generator decides whether it needs to be processed Before this method, first calculate the fingerprint of this method and compare it with the fingerprint existing in the DTO. If they are the same, skip it. Otherwise, it is considered that the SQL of this method has changed, and the DTO needs to be updated.
# specific implementation
So far, basically the main obstacles of DTO code generators have corresponding solutions. Finally, we use a concrete implementation as a simple example.
2 items need to be introduced here:
- monalisa-orm: https://github.com/11039850/monalisa-orm
This is a powerful and very easy-to-use ORM framework that introduces the database through the @DB(jdbc_url, username, password) annotation.
- monalisa-eclipse: https://github.com/11039850/monalisa-eclipse
Here is a corresponding Eclipse plugin that:
- The interface annotated by @DB, when the file is saved, automatically generates the CRUD operation of the table
- The method annotated with @Select automatically generates DTO when the file is saved
- Easily write multi-line strings
Plugin installation and settings can refer to: https://github.com/11039850/monalisa-orm/wiki/Code-Generator
The following is an example of automatically generating DTO based on dynamic SQL. The complete example project can be referred to: https://github.com/11039850/monalisa-example
package test.dao;
public class UserBlogDao {
//@Select 注解指示该方法需自动生成DTO
//默认类名: Result + 方法名， 默认包名：数据访问类的包名+"."+数据访问类的名称(小写)
//可选参数：name 指定生成结果类的名称，如果未指定该参数，则采用默认类名
//可选参数：build 初始化调用参数的Java片段代码，替换默认的参数构造规则
@Select(name="test.result.UserBlogs")
//!!! 保存后会自动修改该函数的返回值为： List -> List<UserBlogs>
//第一次编写时，由于结果类还不存在, 为了保证能够编译正常,
//函数的返回值 和 查询结果要用 泛值 替代, 保存后，插件会自动修改.
//函数的返回值 和 查询结果 泛值的对应关系分三类如下：
//1. List查询
//public DataTable method_name(...){... return Query.getList(); } 或
//public List method_name(...){... return Query.getList(); }
//
//2. Page查询
//public Page method_name(...){... return Query.Page(); }
//
//3. 单条记录
//public Object method_name(...){... return Query.getResult(); }
//
public List selectUserBlogs(int user_id){
Query q=TestDB.DB.createQuery();
q.add(""/**~{
SELECT a.id,a.name,b.title, b.content,b.create_time
FROM user a, blog b
WHERE a.id=b.user_id AND a.id=?
}*/, user_id);
return q.getList();
}
}
After the above code is saved, the plugin will automatically generate a DTO class: test.result.UserBlogs, and automatically modify the method to the following declaration:
public List<UserBlogs> selectUserBlogs(int user_id){
...
return q.getList(UserBlogs.class);
}
Of course, if you make any changes to the selectUserBlogs method (including just adding a space), the plugin will automatically update UserBlogs after saving the file.
At the same time, in order to facilitate our debugging, the plug-in will also output information similar to the following in the Eclipse console window:
2016-06-27 17:00:31 [I] ****** Starting generate result classes from: test.dao.UserBlogDao ******
2016-06-27 17:00:31 [I] Create class: test.result.UserBlogs, from: [selectUserBlogs(int)]
SELECT a.id,a.name,b.title, b.content,b.create_time
FROM user a, blog b
WHERE a.id=b.user_id AND a.id=0
Just to add:
One of the annoying things about writing SQL in Java code is the concatenation of strings in the Java language. It makes a lot of line breaks/escape symbols inserted in the middle of a large SQL code, which is troublesome to write and uncomfortable to look at. The monalisa-eclipse plugin also solves the problem of writing multi-line strings.
E.g:
System.out.println(""/**~{
SELECT *
FROM user
WHERE name="zzg"
}*/);
will output:
SELECT *
FROM user
WHERE name="zzg"
Of course, for fast writing, the syntax of multi-line strings can be set as a code template in Eclipse. For more details on multi-line syntax, please refer to: https://github.com/11039850/monalisa-orm/wiki/Multiple-line-syntax
At this point, the ideas and implementation examples of automatic DTO generation from dynamic SQL code have basically been introduced. You are welcome to put forward various reasonable and unreasonable opinions, discuss and make progress together, thank you!