background
Data import and export are common requirements of application systems, and excel is the main auxiliary tool, especially data import. In the data export scene, excel is relatively the most used. Other export formats, such as word and pdf, are usually used for offline printing or electronic delivery, and report tools are used for more processing.
The excel component library for operating in the java field is mainly EasyPoi, which has relatively complete functions, but it is cumbersome to use and requires some additional customized development work, and it is said that there are several BUGs and performance problems.
Ali made secondary packaging and optimization on the basis of EasyPoi, and launched the EasyExcel open source project.
Official website: https://easyexcel.opensource.alibaba.com/docs/current/
The following information comes from the official website of easyexcel
The well-known frameworks for Java parsing and generating Excel include Apache poi and jxl. But they all have a serious problem that consumes a lot of memory. POI has a SAX mode API that can solve some memory overflow problems to a certain extent, but POI still has some defects, such as Excel version 07 decompression and storage after decompression. It is done in memory, and the memory consumption is still very large.
easyexcel rewrites poi's analysis of Excel version 07. A 3M excel with POI sax analysis still needs about 100M of memory. Using easyexcel can reduce it to a few M, and no matter how big the excel is, there will be no memory overflow; version 03 depends on The sax mode of POI has encapsulated the model conversion on the upper layer, making it easier and more convenient for users
16M memory reads 75M (46W rows and 25 columns) Excel (version 3.2.1+) in 23 seconds.
Of course, there is also a speed mode that can be faster, but the memory usage will be a little more than 100M
Key Design Questions to Consider
Core Function Design
1. First, you must have a template, read and write excel directly, and the table style is ugly
2. The import function requires data verification 3.
Data conversion is required. When importing, convert text into codes and logos, and convert codes into text when exporting
Should file upload and data import be divided into two actions?
From a functional point of view, these are indeed two actions. But from a business point of view, the main purpose is to import data. Uploading files is only an auxiliary work or one of the steps. Data import failures are often due to data problems, which need to be modified and uploaded again. If split into two actions, temporary storage and timely cleanup. From the perspective of user experience, click the import button, select the excel file, start uploading, and import data, which is more intuitive and convenient.
How data transformations are handled
Theoretically, excel importing data is similar to user entering data through a form, but in fact there are still differences. Form entry is supported by controls, such as a data dictionary, and the code is directly passed to the background instead of the name. In excel, the user can only enter the name, and the system needs to convert the name into an encoded repository.
In order to facilitate data conversion, there are some flexible methods, such as preprocessing on the excel template for data import, setting the drop-down list in advance, and using the splicing of code and name as the drop-down item (for example: the user status is expressed as "normal | NORMAL") , can be directly intercepted during processing. The advantage of this method is that no additional query and conversion are required, and it is suitable for data dictionaries and flat basic data with a small amount of data. The disadvantage is that it needs to be preprocessed and synchronized with the system. For example, when the system operation and maintenance personnel adjust the data dictionary or basic data, they need to adjust the excel template synchronously, which is cumbersome and error-prone, and is also prone to inconsistencies caused by forgetting to adjust.
Moreover, this solution is difficult to implement for the following two situations:
1. For the tree structure, such as setting the organization structure when importing users, there are many drop-down options on the one hand, making it difficult to choose, and on the other hand, the tree structure cannot be intuitively reflected Come.
2. The large amount of data, such as materials, hundreds, thousands or even tens of thousands, is not suitable for creating a copy of the data in the database for selection in excel.
Considering the above factors comprehensively, the solution of excel template preprocessing is not adopted in the end, but the data conversion is performed by the platform after the data is imported and before storage.
How data validation is handled
The first thing to say is that importing data from excel also requires strict data verification, which is a point that is easily overlooked. I have seen a lot of systems, which are heavily verified through the system form entry, while the data import link is relaxed, causing problematic data to enter the system, causing a series of subsequent problems. The most common is that a certain field cannot be empty, no verification is performed during import, and a null pointer appears in the subsequent link of using data. Another situation is that the imported data has unexpected values, which do not correspond to the back-end data dictionary or basic data, and the consequences are obvious.
Let's talk about the implementation plan. For form validation, it is implemented with the Hibernate validator component, specifically the mode of annotating the vo object, which is simple, flexible and versatile. Excel imports and reuses this technical solution. The EasyExcel functional component can map the read excel row records to a java object, and then call the Hibernate validator component to trigger validation.
The next question is, can the vo object originally implemented for form entry be reused? After a rough thought, it should be reused, and after in-depth consideration, there is a problem. Form entry has UI control processing and conversion, and Excel import can only be native text. For example, for an organization, the field of organization type is mapped to the type field in the form entry, and the front-end is processed by UI controls, and the returned code is directly coded COMPANY (company) or DEPARTMENT (department), etc., and the back-end vo verification is The type attribute is not empty. In Excel, the type column can only be a Chinese description of the company/department, which is mapped to the typeName field. That is to say, for form entry, it is necessary to verify that type cannot be empty; for Excel import, it is necessary to verify that typeName cannot be empty, which is different.
For this difference, the actual Hibernate validator also takes into account, and there is a corresponding solution, that is, to verify the group. The writing method is rather strange. Define two empty interfaces inside the class, and use the interface name as the group name, and then specify which group or groups need to be used in the groups attribute of the verification annotation. In the early platform development version, this scheme was adopted , the sample code is as follows:
package tech.popsoft.platform.core.modules.system.vo;
import com.alibaba.excel.annotation.ExcelIgnore;
import com.alibaba.excel.annotation.ExcelProperty;
import io.swagger.annotations.ApiModel;
import io.swagger.annotations.ApiModelProperty;
import lombok.Data;
import lombok.EqualsAndHashCode;
import tech.popsoft.platform.common.base.BaseVO;
import javax.validation.constraints.NotBlank;
/**
* 组织机构 视图对象
* @author wqliu
*/
@Data
@EqualsAndHashCode(callSuper = true)
@ApiModel(value = "Organization对象", description = "组织机构")
public class OrganizationVO extends BaseVO {
private static final long serialVersionUID = 1L;
/**
* 表单录入
*/
public interface FormInput{
}
/**
* excel导入
*/
public interface ExcelImport{
}
@ApiModelProperty(value = "父标识")
private String parentId;
@ApiModelProperty(value = "名称")
@NotBlank(message = "名称不能为空",groups = {
tech.popsoft.platform.core.modules.system.vo.UserVO.FormInput.class, tech.popsoft.platform.core.modules.system.vo.UserVO.ExcelImport.class})
@ExcelProperty("名称")
private String name;
@ApiModelProperty(value = "编码")
@ExcelProperty("编码")
private String code;
@ApiModelProperty(value = "类型")
@NotBlank(message = "请选择类型",groups = {
tech.popsoft.platform.core.modules.system.vo.UserVO.FormInput.class})
@ExcelIgnore
private String type;
@ApiModelProperty(value = "备注")
@ExcelProperty("备注")
private String remark;
@ApiModelProperty(value = "状态")
@ExcelIgnore
private String status;
@ApiModelProperty(value = "排序号")
@ExcelProperty("排序号")
private String orderNo;
@ApiModelProperty(value = "类型名称")
@NotBlank(message = "类型不能为空",groups = {
tech.popsoft.platform.core.modules.system.vo.UserVO.ExcelImport.class})
@ExcelProperty("类型")
private String typeName;
@ApiModelProperty(value = "状态名称")
@NotBlank(message = "状态不能为空",groups = {
tech.popsoft.platform.core.modules.system.vo.UserVO.ExcelImport.class})
@ExcelProperty("状态")
private String statusName;
@ApiModelProperty(value = "上级名称")
@NotBlank(message = "上级不能为空",groups = {
tech.popsoft.platform.core.modules.system.vo.UserVO.ExcelImport.class})
@ExcelProperty("上级名称")
private String parentName;
@ApiModelProperty(value = "忽略上级")
@ExcelIgnore
private Boolean ignoreParent;
}
Define the group name as follows.
/**
* 表单录入
*/
public interface FormInput{
}
/**
* excel导入
*/
public interface ExcelImport{
}
In the groups attribute of the attribute annotation, set which/groups the field should be assigned to.
@NotBlank(message = "名称不能为空",groups = {
tech.popsoft.platform.core.modules.system.vo.UserVO.FormInput.class, tech.popsoft.platform.core.modules.system.vo.UserVO.ExcelImport.class})
@ExcelProperty("名称")
private String name;
@ApiModelProperty(value = "类型")
@NotBlank(message = "请选择类型",groups = {
tech.popsoft.platform.core.modules.system.vo.UserVO.FormInput.class})
@ExcelIgnore
private String type;
@ApiModelProperty(value = "类型名称")
@NotBlank(message = "类型不能为空",groups = {
tech.popsoft.platform.core.modules.system.vo.UserVO.ExcelImport.class})
@ExcelProperty("类型")
private String typeName;
Where data validation is required, such as in the controller, a validation group needs to be specified.
/**
* 新增
*/
@ApiOperation(value = "新增")
@PostMapping("/")
@SystemLog(value = "组织机构-新增")
@PreAuthorize("hasPermission(null,'system:organization:add')")
public ResponseEntity<Result> add(@Validated(OrganizationVO.FormInput.class) @RequestBody OrganizationVO vo) {
Organization entity = convert2Entity(vo);
organizationService.add(entity);
OrganizationVO newVO = convert2VO(entity);
return ResultUtil.success(newVO);
}
When importing excel, the grouping is also specified
@PostMapping("/importExcel")
@Override
@PreAuthorize("hasPermission(null,'system:organization:import')")
@SystemLog(value = "导入excel", logRequestParam = false)
public ResponseEntity<Result> importExcel(MultipartFile file) {
SimpleReadDataListener readListener = new SimpleReadDataListener<OrganizationVO, Organization, OrganizationVO.ExcelImport>(organizationService) {
@Override
public Organization convertData(OrganizationVO vo) {
return convert2EntityForExcel(vo);
}
};
super.setReadListener(readListener);
return super.importExcel(file);
}
Looking at it now, the method of building verification groups and reusing vo objects is actually not good. It violates the single responsibility and the principle of opening and closing, especially when the system has only a small number of functions that require Excel import, and then superimpose the code developed by low code Generative factors, better suited for separation.
That is, keep the original VO unchanged, as the data object for the front-end and back-end interactions of the system.
If the function requires Excel import, create a new VO, such as OrganizationForImportVO, and define import-related proprietary columns in it, which is more elegant than the case where form input and Excel import share a VO.
Where should the import related methods be implemented?
New methods related to Excel import need to be placed in the controller layer, such as downloading templates, uploading files, importing data, data conversion, etc. Although they can be added to the corresponding entity classes, the existing problems are similar to the reuse of VO objects mentioned above . A more elegant solution is to separate out this part of the function and implement it separately. That is to create a new OrganizationExcelImportExtensionController. The exposed interface and permission control code are consistent with the OrganizationController. This is transparent and insensitive to the front-end, and the function is separated for the back-end.
Moreover, this part of the functions also has some common features. Extracting this part of the common features and implementing a parent class ExcelImportExtension to achieve reuse is also based on the practice of the template method design pattern.
System implementation
Let's talk about the overall work first, and then introduce the import and export separately.
Introduce dependencies
The first step, of course, is to introduce dependencies. The latest version is selected here, which was updated on May 6, 2023. It can be said that it is freshly released~
Add the following dependencies to the pom file of the platform-common module.
<!--阿里EasyExcel组件-->
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>easyexcel</artifactId>
<version>3.3.1</version>
</dependency>
create base class
Create package extension, and ExcelImportExtension class for public processing
package tech.abc.platform.common.extension;
import com.alibaba.excel.EasyExcel;
import org.apache.commons.io.IOUtils;
import org.springframework.core.io.ClassPathResource;
import org.springframework.http.ResponseEntity;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.multipart.MultipartFile;
import tech.abc.platform.common.base.BaseController;
import tech.abc.platform.common.component.easyexcel.ExcelExceptionEnum;
import tech.abc.platform.common.component.easyexcel.ReadDataListener;
import tech.abc.platform.common.exception.CustomException;
import tech.abc.platform.common.exception.FileException;
import tech.abc.platform.common.utils.ResultUtil;
import tech.abc.platform.common.vo.Result;
import javax.servlet.http.HttpServletResponse;
import java.io.InputStream;
import java.io.OutputStream;
import java.lang.reflect.ParameterizedType;
import java.net.URLEncoder;
/**
* excel导入功能扩展
*
* @author wqliu
* @date 2023-05-17
*/
public class ExcelImportExtension<V, E> extends BaseController {
/**
* 数据监听器
*/
private ReadDataListener readListener;
/**
* 导入模板
*/
private String importTemplate;
/**
* 设置数据监听器
*
* @param readListener
*/
public void setReadListener(ReadDataListener readListener) {
this.readListener = readListener;
}
/**
* 设置导入模板
*/
protected void setImportTemplate(String importTemplate) {
this.importTemplate = importTemplate;
}
/**
* 下载导入模板
*/
public void downloadImportTemplate(HttpServletResponse response) {
ClassPathResource classPathResource = new ClassPathResource(importTemplate);
try (InputStream inputStream = classPathResource.getInputStream();
OutputStream outputStream = response.getOutputStream()) {
// 设置响应信息
response.setContentType("application/vnd.ms-excel");
response.setCharacterEncoding("utf-8");
// 这里URLEncoder.encode可以防止中文乱码
String fileName = URLEncoder.encode("导入模板.xlsx", "UTF-8");
response.setHeader("Content-disposition", "attachment;filename=" + fileName);
IOUtils.copy(inputStream, outputStream);
} catch (Exception exception) {
throw new CustomException(FileException.DOWNLOAD_FAILURE, exception.getMessage());
}
}
/**
* excel上传
*/
@Transactional(rollbackFor = Exception.class)
public ResponseEntity<Result> importExcel(MultipartFile file) {
if (file == null) {
throw new CustomException(FileException.UPLOAD_IS_NULL);
}
ResponseEntity<Result> result = null;
try {
EasyExcel.read(file.getInputStream(), this.getClazz(), this.readListener).sheet().doRead();
return ResultUtil.success();
} catch (Exception exception) {
long currentRowNo = this.readListener.getCurrentRowNo();
Throwable throwable = exception;
while (throwable.getCause() != null) {
throwable = throwable.getCause();
}
throw new CustomException(FileException.EXCEL_IMPORT_FAILURE, currentRowNo, throwable.getMessage());
}
}
/**
* 数据转换
*/
protected E convert2EntityForExcel(V vo) {
throw new CustomException(ExcelExceptionEnum.EXPORT_METHOD_UNIMPLEMENTED);
}
/**
* 通过父类获取运行时泛型类型
*
* @return
*/
private Class<V> getClazz() {
// 获得当前类型的带有泛型类型的父类
Class subclass;
ParameterizedType pd = (ParameterizedType) this.getClass().getGenericSuperclass();
Class<V> clazz = (Class) pd.getActualTypeArguments()[0];
return clazz;
}
}
Create a data listener
EasyExcel uses a listener mechanism to read and parse data when importing. The core is the listener. We have implemented the ReadListener interface ourselves, and the listener needs to be reused by multiple entities and uses generics.
At the same time, it should be noted that the listener is stateful and cannot be managed by spring. It must be new every time the excel is read, and then the service class used in it is passed in through the construction method.
package tech.abc.platform.common.component.easyexcel;
import com.alibaba.excel.context.AnalysisContext;
import com.alibaba.excel.read.listener.ReadListener;
import com.alibaba.fastjson.JSON;
import lombok.extern.slf4j.Slf4j;
import tech.abc.platform.common.base.BaseService;
import tech.abc.platform.common.exception.CommonException;
import tech.abc.platform.common.exception.CustomException;
import javax.validation.ConstraintViolation;
import javax.validation.Validation;
import javax.validation.Validator;
import javax.validation.ValidatorFactory;
import java.util.Set;
/**
* 读取数据监听器
*
* @author wqliu
* @date 2023-05-17
*/
@Slf4j
public class ReadDataListener<V, E> implements ReadListener<V> {
/**
* 当前处理行号
*/
private long currentRowNo = 0;
/**
* 服务
*/
private BaseService service;
/**
* 获取当前行号
*
* @return long 行号
*/
public long getCurrentRowNo() {
return currentRowNo;
}
/**
* 构造方法
* 每次创建Listener的时候需要把spring管理的类传进来
*
* @param service
*/
public ReadDataListener(BaseService service) {
this.service = service;
}
/**
* 解析数据
*
* @param data 单行记录
* @param context
*/
@Override
public void invoke(V data, AnalysisContext context) {
currentRowNo++;
log.info("解析到一条数据:{}", JSON.toJSONString(data));
// 处理数据
E entity = handleData(data);
// 保存数据
service.add(entity);
}
/**
* 所有数据解析完成了 都会来调用
*
* @param context
*/
@Override
public void doAfterAllAnalysed(AnalysisContext context) {
log.info("所有数据解析完成！");
}
/**
* 数据预处理，可转化实体类中的字典数据，也可以设置默认字段
*
* @param data
*/
public E handleData(V vo) {
// 数据校验
validateData(vo);
// 设置默认值
setDefaultValue(vo);
// 数据转换
return convertData(vo);
}
private void validateData(V vo) {
// 进行数据验证
ValidatorFactory vf = Validation.buildDefaultValidatorFactory();
Validator validator = vf.getValidator();
Set<ConstraintViolation<V>> set = validator.validate(vo);
for (ConstraintViolation<V> constraintViolation : set) {
throw new CustomException(CommonException.DATA_VALIDATE_FAILURE, constraintViolation.getMessage());
}
}
/**
* 设置默认值
*
* @param vo 视图对象
*/
protected void setDefaultValue(V vo) {
// 如无需设置，则该方法可为空
}
/**
* 转换数据
*
* @param vo 视图对象
* @return {@link E}
*/
protected E convertData(V vo) {
throw new CustomException(ExcelExceptionEnum.EXPORT_METHOD_UNIMPLEMENTED);
}
}
Single processing or batch processing
The example provided by EasyExcel is to analyze the data and put them into a collection. When the specified amount of data is reached, such as 300, batch storage is performed, which is more performant from a technical point of view, but the user organizes it through excel. The data is usually not standardized, and the system needs to give a friendly error prompt, specifically which attribute of which row has a problem. In addition, the system framework itself has processing logic when creating a business entity (check whether it is empty, whether it is repeated, whether it exists , trigger the processing of related objects after adding new ones, etc.), and what users import through excel is usually a small batch of master data, within a few dozen, at most a few hundred, tens of thousands of rare, so it is adjusted to only process each time 1 piece of data.
Create import controller class
Including the processing of downloading templates and importing Excel data, the code of the common part is still extracted to the parent class ExcelImportExtension, which is mainly used for overwriting, specifying the path of the template, and the core lies in the reading and processing of data.
For organizations, there may be duplicate names. For example, there is a department named "Workshop No. 1" under the two departments of Production 1 and Production 2. At this time, it is impossible to determine the data through the name of the superior. Which department to import to, so the parent code is introduced. The code is unique, and the system will look for the code first when processing. If the code is empty, it will be processed by the name. If multiple departments with the same name are still found, they will be placed in unallocated and transferred to manual processing.
package tech.abc.platform.system.controller.extension;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.BeanUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;
import tech.abc.platform.common.annotation.SystemLog;
import tech.abc.platform.common.component.easyexcel.ReadDataListener;
import tech.abc.platform.common.exception.CustomException;
import tech.abc.platform.common.extension.ExcelImportExtension;
import tech.abc.platform.common.vo.Result;
import tech.abc.platform.system.constant.SystemConstant;
import tech.abc.platform.system.entity.Organization;
import tech.abc.platform.system.exception.OrganizationExceptionEnum;
import tech.abc.platform.system.service.OrganizationService;
import tech.abc.platform.system.vo.OrganizationForImportVO;
import javax.servlet.http.HttpServletResponse;
import java.util.List;
/**
* 实现Excel导入功能的组织机构控制器
*
* @author wqliu
* @date 2023-05-17
*/
@RestController
@RequestMapping("/system/organization")
@Slf4j
public class OrganizationExcelImportExtensionController extends ExcelImportExtension<OrganizationForImportVO, Organization> {
@Autowired
private OrganizationService organizationService;
@GetMapping("/downloadImportTemplate")
@Override
@PreAuthorize("hasPermission(null,'system:organization:downloadImportTemplate')")
@SystemLog(value = "下载excel模板", logResponseData = false)
public void downloadImportTemplate(HttpServletResponse response) {
super.setImportTemplate("/template/system/organization/import.xlsx");
super.downloadImportTemplate(response);
}
@PostMapping("/importExcel")
@Override
@PreAuthorize("hasPermission(null,'system:organization:import')")
@SystemLog(value = "导入excel", logRequestParam = false)
@Transactional
public ResponseEntity<Result> importExcel(MultipartFile file) {
ReadDataListener readListener = new ReadDataListener<OrganizationForImportVO, Organization>(organizationService) {
@Override
public Organization convertData(OrganizationForImportVO vo) {
return convert2EntityForExcel(vo);
}
};
super.setReadListener(readListener);
return super.importExcel(file);
}
@Override
protected Organization convert2EntityForExcel(OrganizationForImportVO vo) {
Organization entity = organizationService.init();
BeanUtils.copyProperties(vo, entity);
entity.setType(dictionaryUtil.getCodeByName("OrganizationType", vo.getTypeName()));
// 处理上级
if (StringUtils.isNotBlank(vo.getParentCode())) {
// 优先判断上级编码是否存在，若存在，则根据编码找上级
List<Organization> organizationList = organizationService.lambdaQuery()
.eq(Organization::getCode, vo.getParentCode()).list();
if (organizationList.size() == 1) {
// 找到数据，设置父级标识
entity.setOrganization(organizationList.get(0).getId());
} else {
// 未找到，抛出异常
throw new CustomException(OrganizationExceptionEnum.CODE_NOT_FOUND);
}
} else if (StringUtils.isNotBlank(vo.getParentName())) {
// 将上级名称转换为标识
List<Organization> organizationList = organizationService.lambdaQuery()
.eq(Organization::getName, vo.getParentName()).list();
if (organizationList.size() == 1) {
// 根据名称找到唯一的部门，设置部门标识
entity.setOrganization(organizationList.get(0).getId());
} else if (organizationList.size() == 0) {
throw new CustomException(OrganizationExceptionEnum.NAME_NOT_FOUND);
} else {
// 找到多个同名部门，不抛异常，统一设置到预定义的未分配部门，转人工处理
entity.setOrganization(SystemConstant.UNSIGNED_ORGANIZATION_ID);
}
} else {
// 上级名称和编码都为空，数据不合法
throw new CustomException(OrganizationExceptionEnum.PARENT_NAME_AND_CODE_CANOT_NULL);
}
return entity;
}
}
Create data import view object class
EasyExcel provides two modes to map data, one is column subscript, the other is column name. The subscript of the column will change due to the addition, deletion, and movement of the column, which is unreliable. It is more appropriate to use the column name. It is necessary to add the annotation @ExcelProperty("name") to the property of the vo object.
If you use a dedicated VO, you only need to create corresponding attributes for the limited columns that appear in excel. If you use a common VO scheme for inputting the form, you have to use the @ExcelIgnore annotation to ignore the unnecessary fields. Otherwise, the import link will go wrong.
There is a problem here that needs attention, you cannot add @Accessors(chain = true). This annotation will make the return value of the set method of the attribute be the object itself, and the EasyExcel component parses and processes the mapping using BeanMap to copy from Map to Bean. The Key of the Map needs to be consistent with the variable name of the Bean, and there is a corresponding set method, and the set method If it is void, the copy can be successful.
package tech.abc.platform.system.vo;
import com.alibaba.excel.annotation.ExcelProperty;
import lombok.Data;
import javax.validation.constraints.NotBlank;
/**
* 组织机构导入 视图对象类
*
* @author wqliu
* @date 2023-05-18
*/
@Data
public class OrganizationForImportVO {
/**
* 名称
*/
@NotBlank(message = "【名称】不能为空")
@ExcelProperty("名称")
private String name;
/**
* 编码
*/
@ExcelProperty("编码")
private String code;
/**
* 类型
*/
@NotBlank(message = "【类型】不能为空")
@ExcelProperty("类型")
private String typeName;
/**
* 上级名称
*/
@ExcelProperty("上级名称")
private String parentName;
/**
* 上级编码
*/
@ExcelProperty("上级编码")
private String parentCode;
/**
* 排序
*/
@ExcelProperty("排序")
private String orderNo;
/**
* 备注
*/
@ExcelProperty("备注")
private String remark;
}
Configure import templates
There are many advantages to pre-configuring import templates. In addition to being beautiful, the main thing is that it is easy to use and avoids mistakes. In addition, you can also do some simple configurations in the preset templates, such as the format of cells (such as text, amount, date), and the drop-down selection of simple data sources (such as whether), and put some sample data, etc.
We put the template in /template/system/organization/import.xlsx under the resources directory of the corresponding module. Note that the module code here must not be less, otherwise it is easy to cause the problem of the same entity name under different modules, resulting in the template coverage problem during packaging.
pom package, remember to add **/*.xlsx, and package the excel template as a resource.
<build>
<finalName>abc-platform-system</finalName>
<plugins>
<!--编译插件 -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<!--指定JDK编译版本 -->
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<!-- 测试插件 -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.2</version>
<configuration>
<!-- 跳过测试 -->
<skipTests>true</skipTests>
</configuration>
</plugin>
</plugins>
<resources>
<!--处理mybatis的mapper.xml文件-->
<resource>
<directory>src/main/java</directory>
<includes>
<include>**/*.xml</include>
</includes>
</resource>
<!--处理其他资源文件-->
<resource>
<directory>src/main/resources</directory>
<includes>
<!--系统配置文件-->
<include>*.yml</include>
<!--excel模板-->
<include>**/*.xlsx</include>
<!--其他配置文件-->
<include>*.xml</include>
</includes>
</resource>
</resources>
</build>
Configure permission items
There is nothing to say about this, use the permission item configuration function under the platform system management module, and add configuration items according to the specifications and agreements.
Front-end list view add button
<el-button
v-permission="pageCode + 'downloadImportTemplate'"
type="primary"
icon="Download"
@click="downloadImportTemplate"
>下载模板</el-button
>
<el-upload
ref="uploader"
:limit="1"
:http-request="importData"
action="/"
:show-file-list="false"
:before-upload="onBeforeUpload"
class="uploader"
>
<el-button v-permission="pageCode + 'import'" type="primary" icon="List"
>批量导入
</el-button>
</el-upload>
</div>
There is a pitfall here. The el-upload control will start a new line by default.
Adding display: inline; is not enough. It will look like this,
aligned horizontally with the button, and the final added style is as follows:
<style scoped>
:deep(.el-upload) {
margin-left: 12px;
display: inline;
text-align: center;
cursor: pointer;
outline: 0;
}
:deep(.uploader) {
display: inline;
}
</style>
The final effect is as follows, perfect alignment.
front-end call
Vue page calls, corresponding to button events, as follows
// 下载导入模板
downloadImportTemplate() {
this.api.downloadImportTemplate()
},
// 导入
importData(file) {
const formData = new FormData()
formData.append('file', file.file)
this.api.import(formData).finally(() => {
this.clearFile()
})
}
The system module uniformly encapsulates the api, receives the vue call, and then calls the axios service instance, as follows
// 下载导入模板
downloadImportTemplate() {
return request.download({
url: this.serveUrl + 'downloadImportTemplate' })
},
// 导入
import(formData) {
return request.upload({
url: this.serveUrl + 'importExcel', data: formData })
}
The download is relatively simple, there is nothing to say.
There is a point that needs special attention here, : http-request="importData" bound event, the incoming parameter encapsulates the file with another layer, if written as formData.append('file', file), the backend The file parameter received by SpringMVC is always null, and the attribute file of the parameter file needs to be used, that is, lformData.append('file', file.file).
The problem encountered in the implementation, using the uploader control of element plus, the upload file can only be executed once, upload again (data verification fails, re-import after modification or import multiple times), because of the cache problem, there will be no operation response (not the browser suspended animation , but does not trigger the operation), the file cleanup operation needs to be performed in both cases of successful and failed uploads.
clearFile() {
// 上传成功之后清除历史记录,否则再次上传浏览器无响应
this.$refs.uploader.clearFiles()
}
Extend Axios to realize upload and download
The path is src\config\axios\index.ts. Get, put, post and delete have been implemented before. Although get is used for download and post is used for upload, but because the file processing is special, there is additional processing, so separate encapsulation.
To download, the responseType attribute must be specified as 'blob'.
To upload, headersType must be specified as 'multipart/form-data'
download: (option: any) => {
request({
method: 'get', responseType: 'blob', ...option })
.then((res) => {
const {
data, headers } = res
const fileName = headers['content-disposition'].replace(/\w+;filename=(.*)/, '$1')
// 此处当返回json文件时需要先对data进行JSON.stringify处理，其他类型文件不用做处理
const blob = new Blob([data], {
type: headers['content-type'] })
const dom = document.createElement('a')
const url = window.URL.createObjectURL(blob)
dom.href = url
dom.download = decodeURI(fileName)
dom.style.display = 'none'
document.body.appendChild(dom)
dom.click()
dom.parentNode.removeChild(dom)
window.URL.revokeObjectURL(url)
})
.catch((err) => {
reject(err)
})
},
upload: (option: any) => {
return new Promise((resolve, reject) => {
option.headersType = 'multipart/form-data'
request({
method: 'post', ...option })
.then((res) => {
// 明确设置为true时，显示提示
if (option.showInfo === true) {
ElMessage.info(res.data.message)
}
resolve(res.data)
})
.catch((err) => {
reject(err)
})
})
}
Data output
Although EasyExcel provides an API to directly export Excel, there are two key issues. One is the format of Excel, which is quite cumbersome to set in the program, and it is not beautiful enough. If it involves compound headers and merged cells, it is not normal. On the other hand, it is necessary to add a lot of annotations to the view model class to control whether to display and the necessary format conversion. The initialization work and adjustment work of these two points are too large, and the flexibility is too poor.
From a design point of view, data and presentation should be separated and should not be coupled together. Therefore, using EasyExcel's filling api, that is, first manually edit the excel template, set the style, and occupy the data. The specific data is determined by the application. The backend of the program is dynamically generated to fill, each performing its own duties, providing flexibility and scalability.
How to deal with data export
In different scenarios, the amount of exported data is different, such as the system's master data organization, personnel, etc., the amount of data is limited, it can be read into memory at one time, and written to Excel at one time, but not Avoid the situation of exporting a large number of business documents, such as tens of thousands or even hundreds of thousands of data. At this time, you should read and write in batches to avoid a large amount of application server memory and reduce the number of full garbage collections to make the application run more stable.
There are two options. One is that the platform encapsulates two methods, which are one-time processing and batch processing. When developing business functions, it is up to the estimated data volume to decide which one to call; the other is that the platform only provides one method, which is internally based on The size and configuration of the data volume determine whether to process at one time or in batches.
After consideration, plan 2 is adopted, so that the user does not need to pay too much attention to details, just call and export, and some business documents may be in the millions, but after the user selects filter conditions such as time period, the data volume may be only Hundreds of thousands of pieces are also more suitable for one-time processing.
In the implementation stage, the scheme is optimized. At the controller layer, it is difficult for the parent class to obtain the amount of data (technically, it can be obtained, but each subclass needs to realize the method of obtaining the total amount), so a flexible processing method is adopted, that is, unified use In the batch processing mode, set a larger value for the amount of data processed each time, such as 10000, so that the export of small data volumes will be processed once, while large data volumes will still be processed in batches for multiple times.
The controller corresponding to the business entity that needs to export the function of excel only needs to inherit the parent class of the controller and override a method for obtaining paging data.
And you can control the amount of data processed at a time by yourself (such as too many columns, complex logical processing, etc.)
The specific design and implementation ideas of data export are highly similar to those of import. Only key codes are released here. For details, see the open source code library.
Excel export function public base class
package tech.abc.platform.common.extension;
import com.alibaba.excel.EasyExcel;
import com.alibaba.excel.ExcelWriter;
import com.alibaba.excel.write.metadata.WriteSheet;
import com.baomidou.mybatisplus.core.metadata.IPage;
import org.springframework.core.io.ClassPathResource;
import tech.abc.platform.common.base.BaseController;
import tech.abc.platform.common.component.easyexcel.ExcelExceptionEnum;
import tech.abc.platform.common.exception.CustomException;
import tech.abc.platform.common.exception.FileException;
import javax.servlet.http.HttpServletResponse;
import java.io.InputStream;
import java.io.OutputStream;
import java.lang.reflect.ParameterizedType;
import java.net.URLEncoder;
/**
* excel导出功能扩展
*
* @author wqliu
* @date 2023-05-19
*/
public class ExcelExportExtension<V, E> extends BaseController {
/**
* 导出数据分页大小
* TODO：可配置化
*/
private static final long EXPORT_DATA_PAGE_SIZE = 200;
/**
* 导出模板
*/
private String exportTemplate;
/**
* 设置导出模板
*/
public void setExportTemplate(String exportTemplate) {
this.exportTemplate = exportTemplate;
}
/**
* 导出到excel
*
* @param queryParam
* @param response
* @throws Exception
*/
public void exportExcel(V queryParam, HttpServletResponse response) {
ClassPathResource classPathResource = new ClassPathResource(this.exportTemplate);
try (InputStream inputStream = classPathResource.getInputStream();
OutputStream outputStream = response.getOutputStream()) {
// 设置响应信息
response.setContentType("application/vnd.ms-excel");
response.setCharacterEncoding("utf-8");
// 这里URLEncoder.encode可以防止中文乱码
String fileName = URLEncoder.encode("导出数据.xlsx", "UTF-8");
response.setHeader("Content-disposition", "attachment;filename=" + fileName);
// excel导出处理
ExcelWriter excelWriter = EasyExcel.write(outputStream, this.getClazz())
.withTemplate(inputStream).build();
WriteSheet writeSheet = EasyExcel.writerSheet().build();
// 此处并没有获取数据总量后自行计算分页，而是利用了已有的分页查询功能
long pageSize = EXPORT_DATA_PAGE_SIZE;
// 开始第一次查询，并获取分页总数
IPage<V> pagedResult = getExportData(queryParam, pageSize, 1);
excelWriter.fill(pagedResult.getRecords(), writeSheet);
// 读取后续数据
for (int i = 2; i <= pagedResult.getPages(); i++) {
// 分页去数据库查询数据 这里可以去数据库查询每一页的数据
pagedResult = getExportData(queryParam, pageSize, i);
excelWriter.fill(pagedResult.getRecords(), writeSheet);
}
// 关闭流
excelWriter.finish();
} catch (Exception exception) {
throw new CustomException(FileException.EXCEL_EXPORT_FAILURE, exception.getMessage());
}
}
/**
* 获取导出数据
*
* @return
* @throws Exception
*/
public IPage<V> getExportData(V queryParam, long pageSize, long pageNum) {
throw new CustomException(ExcelExceptionEnum.EXPORT_METHOD_UNIMPLEMENTED);
}
/**
* 通过父类获取运行时泛型类型
*
* @return
*/
private Class<V> getClazz() {
// 获得当前类型的带有泛型类型的父类
Class subclass;
ParameterizedType pd = (ParameterizedType) this.getClass().getGenericSuperclass();
Class<V> clazz = (Class) pd.getActualTypeArguments()[0];
return clazz;
}
}
Concrete export controller class
package tech.abc.platform.system.controller.extension;
import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
import com.baomidou.mybatisplus.core.metadata.IPage;
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import tech.abc.platform.common.annotation.SystemLog;
import tech.abc.platform.common.extension.ExcelExportExtension;
import tech.abc.platform.common.query.QueryGenerator;
import tech.abc.platform.system.controller.OrganizationController;
import tech.abc.platform.system.entity.Organization;
import tech.abc.platform.system.service.OrganizationService;
import tech.abc.platform.system.vo.OrganizationVO;
import javax.servlet.http.HttpServletResponse;
import java.util.List;
/**
* 实现Excel导出功能的组织机构控制器
*
* @author wqliu
* @date 2023-05-19
*/
@RestController
@RequestMapping("/system/organization")
@Slf4j
public class OrganizationExcelExportExtensionController extends ExcelExportExtension<OrganizationVO, Organization> {
@Autowired
private OrganizationService organizationService;
@Autowired
protected OrganizationController organizationController;
@GetMapping("/exportExcel")
@Override
@PreAuthorize("hasPermission(null,'system:organization:export')")
@SystemLog(value = "导出excel", logResponseData = false)
public void exportExcel(OrganizationVO vo, HttpServletResponse response) {
// 设置模板名称
super.setExportTemplate("/template/system/organization/export.xlsx");
// 当勾选查询所有复选框时，查询所有数据
if (vo.getIgnoreParent() != null && vo.getIgnoreParent()) {
vo.setOrganization(null);
}
// 导出到excel
super.exportExcel(vo, response);
}
@Override
public IPage<OrganizationVO> getExportData(OrganizationVO queryParam, long pageSize, long pageNum) {
// 构造分页对象
IPage<Organization> page = new Page<Organization>(pageNum, pageSize);
// 构造查询条件
QueryWrapper<Organization> queryWrapper = QueryGenerator.generateQueryWrapper(Organization.class, queryParam);
// 排序
queryWrapper.lambda().orderByAsc(Organization::getOrganization).orderByAsc(Organization::getOrderNo);
organizationService.page(page, queryWrapper);
// 转换vo
IPage<OrganizationVO> pageVO = mapperFacade.map(page, IPage.class);
List<OrganizationVO> organizationVOList = organizationController.convert2VO(page.getRecords());
pageVO.setRecords(organizationVOList);
return pageVO;
}
}
Data export effect
Development platform information
Platform name: One Two Three Development Platform
Introduction: Enterprise-level general development platform
Design information: csdn column
Open source address: Gitee
open source protocol: MIT
Favorites, likes, and comments are welcome. Your support is the driving force for me to move forward.