On one of our classes that implement and index cache tools index basic operations, this section we begin to implement load the full amount of index data prior to loading the full amount of index data, we need to export to a file in the first table data in the database. Let's code.
1. First define a constant class, used to store the export file storage directory and file name
Because we need to use the exported file in the search service, so we will name the file & directory and information derived object in writing
mscx-ad-commomproject.
public class FileConstant {
public static final String DATA_ROOT_DIR = "/Users/xxx/Documents/promotion/data/mysql/";
//各个表数据的存储文件名
public static final String AD_PLAN = "ad_plan.data";
public static final String AD_UNIT = "ad_unit.data";
public static final String AD_CREATIVE = "ad_creative.data";
public static final String AD_CREATIVE_RELARION_UNIT = "ad_creative_relation_unit.data";
public static final String AD_UNIT_HOBBY = "ad_unit_hobby.data";
public static final String AD_UNIT_DISTRICT = "ad_unit_district.data";
public static final String AD_UNIT_KEYWORD = "ad_unit_keyword.data";
}2. Define the index object exported fields of information, still with Ad_Planan example.
/**
* AdPlanTable for 需要导出的表字段信息 => 是搜索索引字段一一对应
*
* @author <a href="mailto:[email protected]">Isaac.Zhang | 若初</a>
*/
@Data
@AllArgsConstructor
@NoArgsConstructor
public class AdPlanTable {
private Long planId;
private Long userId;
private Integer planStatus;
private Date startDate;
private Date endDate;
}3. Export file service implementation
Likewise, the best way to achieve that is to export the service as a sub-project to be run independently, directly realize I am here in the
mscx-ad-dbproject
- Define an empty interface, in order to comply with our coding standards
/**
* IExportDataService for 导出数据库广告索引初始化数据
*
* @author <a href="mailto:[email protected]">Isaac.Zhang | 若初</a>
*/
public interface IExportDataService {
}- Achieve service
@Slf4j
@Service
public class ExportDataServiceImpl implements IExportDataService {
@Autowired
private AdPlanRepository planRepository;
/**
* 导出 {@code AdPlan} from DB to File
*
* @param fileName 文件名称
*/
public void exportAdPlanTable(String fileName) {
List<AdPlan> planList = planRepository.findAllByPlanStatus(CommonStatus.VALID.getStatus());
if (CollectionUtils.isEmpty(planList)) {
return;
}
List<AdPlanTable> planTables = new ArrayList<>();
planList.forEach(item -> planTables.add(
new AdPlanTable(
item.getPlanId(),
item.getUserId(),
item.getPlanStatus(),
item.getStartDate(),
item.getEndDate()
)
));
//将数据写入文件
Path path = Paths.get(fileName);
try (BufferedWriter writer = Files.newBufferedWriter(path)) {
for (AdPlanTable adPlanTable : planTables) {
writer.write(JSON.toJSONString(adPlanTable));
writer.newLine();
}
writer.close();
} catch (IOException e) {
e.printStackTrace();
log.error("export AdPlanTable Exception!");
}
}
}- Implemented Controller, providing Operator
@Slf4j
@Controller
@RequestMapping("/export")
public class ExportDataController {
private final ExportDataServiceImpl exportDataService;
@Autowired
public ExportDataController(ExportDataServiceImpl exportDataService) {
this.exportDataService = exportDataService;
}
@GetMapping("/export-plan")
public CommonResponse exportAdPlans() {
exportDataService.exportAdPlanTable(String.format("%s%s", FileConstant.DATA_ROOT_DIR, FileConstant.AD_PLAN));
return new CommonResponse();
}
}- The results document reads as follows, each row represents a promotion plan
{"endDate":1561438800000,"planId":10,"planStatus":1,"startDate":1561438800000,"userId":10}
{"endDate":1561438800000,"planId":11,"planStatus":1,"startDate":1561438800000,"userId":10}Build an index based on the file content
When we write indexing service before, created some entity object class index you want to use, such as building promotion plan index when the need to use physical objects com.sxzhongf.ad.index.adplan.AdPlanIndexObject, but then, we are on a time to achieve an index derived entity Object and common package com.sxzhongf.ad.common.export.table.AdPlanTable, read out the data in the file can only deserialize JSON.parseObject(p, AdPlanTable.class), we need to do two objects with each other to create an index mapping information.
1. First, we define an enumeration type of operation on our behalf every type of operation ( also needs to correspond to the type of operation late binlog listening )
public enum OperationTypeEnum {
ADD,
UPDATE,
DELETE,
OTHER;
public static OperationTypeEnum convert(EventType type) {
switch (type) {
case EXT_WRITE_ROWS:
return ADD;
case EXT_UPDATE_ROWS:
return UPDATE;
case EXT_DELETE_ROWS:
return DELETE;
default:
return OTHER;
}
}
}2. Because of the nature of the load and index the full amount of incremental indexing load is the same, the whole amount of the index is actually a special kind of incremental indexing, to reusable code, we create a unified class to manipulate the index.
/**
* AdLevelDataHandler for 通用处理索引类
* 1. 索引之间存在层级划分，也就是相互之间拥有依赖关系的划分
* 2. 加载全量索引其实是增量索引 "添加"的一种特殊实现
*
* @author <a href="mailto:[email protected]">Isaac.Zhang | 若初</a>
*/
@Slf4j
public class AdLevelDataHandler {
/**
* 实现广告推广计划的第二层级索引实现。
* （第一级为用户层级，但是用户层级不参与索引，所以从level 2开始）
* 第二层级的索引是表示 不依赖于其他索引，但是可被其他索引所依赖
*/
public static void handleLevel2Index(AdPlanTable adPlanTable, OperationTypeEnum type) {
// 对象转换
AdPlanIndexObject planIndexObject = new AdPlanIndexObject(
adPlanTable.getPlanId(),
adPlanTable.getUserId(),
adPlanTable.getPlanStatus(),
adPlanTable.getStartDate(),
adPlanTable.getEndDate()
);
//调用通用方法处理，使用IndexDataTableUtils#of来获取索引的实现类bean
handleBinlogEvent(
// 在前一节我们实现了一个索引工具类，来获取注入的bean对象
IndexDataTableUtils.of(AdPlanIndexAwareImpl.class),
planIndexObject.getPlanId(),
planIndexObject,
type
);
}
/**
* 处理全量索引和增量索引的通用处理方式
* K,V代表索引的键和值
*
* @param index 索引实现代理类父级
* @param key 键
* @param value 值
* @param type 操作类型
*/
private static <K, V> void handleBinlogEvent(IIndexAware<K, V> index, K key, V value, OperationTypeEnum type) {
switch (type) {
case ADD:
index.add(key, value);
break;
case UPDATE:
index.update(key, value);
break;
case DELETE:
index.delete(key, value);
break;
default:
break;
}
}
}3. Read the index file for full amount of load.
Because need to rely on another component before we file is loaded, which is our indexing tools, you need to add the
@DependsOn("indexDataTableUtils")full amount of the index will need to be loaded at system boot time, we need to add@PostConstructto realize initialization load, the@PostConstructmodified method will be loaded on the server Servlet running time, and will only be called once the server.
@Component
@DependsOn("indexDataTableUtils")
public class IndexFileLoader {
/**
* 服务启动时，执行全量索引加载
*/
@PostConstruct
public void init() {
//加载 推广计划
List<String> adPlanStrings = loadExportedData(String.format("%s%s",
FileConstant.DATA_ROOT_DIR, FileConstant.AD_PLAN
));
adPlanStrings.forEach(p -> AdLevelDataHandler.handleLevel2Index(
JSON.parseObject(p, AdPlanTable.class), OperationTypeEnum.ADD
));
}
/**
* <h3>读取全量索引加载需要的文件</h3>
*
* @param fileName 文件名称
* @return 文件行数据
*/
private List<String> loadExportedData(String fileName) {
try (BufferedReader reader = Files.newBufferedReader(Paths.get(fileName))) {
return reader.lines().collect(Collectors.toList());
} catch (IOException e) {
throw new RuntimeException(e.getMessage());
}
}
}Tips
In the process of implementing the initialization load full amount of the index, we must ensure that the order of data loading problems, because there are different data may exist interdependent relationship, once the order was wrong, will cause the program error problem.