Disclaimer: This article is a blogger original article, shall not be reproduced without the bloggers allowed. https://blog.csdn.net/eumenides_/article/details/82587412
Sub-table background
Our project has a very important function is to bulk SMS, then set before me was a big brother in charge of the (now leave), text messaging business to complain about slow query log details, I looked under the big brother actually took a sms_log with mysql keep a log table, with the rapid development of business, remember daily log more and more, I went online mysql looked at, as well as fast data 2000w, and it has a lot of field log table, I see faint egg pain Not to. I've seen remember a single mysql table data should not exceed 500w, 300w generally maintained at below to see where specific already do not know. . .
Sub-table program
- Nosql choose to save the log database mongodb by month, every month is a collection, each log is a document
- Since the state does not have a regular platform for the return of the update report, so in order to avoid frequent crud with a set of ideas on the use of separate read and write, create two collections a month with pre-log and log the full amount, the main logic is:
save the log to pre log -> receive status reports pre-query log -> update pre logs -> save log full amount -> delete pre log
doing so we can guarantee a steady collection of pre-log reduction in the total amount of log collection steadily increase - Data synchronization, mysql also need to do these things inside the old data in sync to mongodb
Here does not show specific logic and data synchronization code, and how to save the log herein, this only describes how the press about my score sheet after month paged query.
Paging points table query rules
- Due to the migration from a single mysql table custom field ordering mongodb more than a collection of original support it can not support, of course, have to support it is not impossible, but fortunately only in accordance with the main business creation time SMS flashback only, so this can be written in the dead of sql
- Paging query is a core function, indispensable, but due to the search condition creation time, so the time paging query parameters will bring a greater degree of difficulty. Here the current limit with one year of data only check, New Year's Eve does not support queries, however will produce the same month, give birth this month, three cases data plan period, so the specific query logic parameters need to be analyzed in time to make a specific inquiry and stitching.
- Colleagues mentioned can check all the data is good, well into memory and then ordering the interception of part of the data, this approach outset, I was denied, so the amount of data is too large, processing speed is too slow, and total memory, with the turn the greater the amount of data page, probably out of memory, if I remember correctly mycat it did. So how I do it, I was under the page number in which to locate the data set, if after the set number is not enough to take the weight of a continued and then take it from the next collection, of course, extreme cases multiple collections have taken enough , we need to meet the conditions set in accordance with the sequentially traversing up or down to get enough to take to completion.
Since logic we straightened around, they begin to look at the code it!
public Map<String, Object> listByParamsFromMongodb(SendmsgLog sendmsgLog, Integer page, Integer rows, String beginTime, String endTime,Integer logType) throws ParseException {
Map<String, Object> resultMap = new HashMap<>();
List<SendmsgLog> sendmsgLogList = new ArrayList<>();
int total = 0;
if (StringUtils.isBlank(beginTime) || StringUtils.isBlank(endTime) || logType == null){
resultMap.put("sendmsgLogList",sendmsgLogList);
resultMap.put("total",total);
return resultMap;
}
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
Date startDate = sdf.parse(beginTime);
Date endDate = sdf.parse(endTime);
Calendar cs = Calendar.getInstance();
Calendar ce = Calendar.getInstance();
cs.setTime(startDate);
ce.setTime(endDate);
int startYear = cs.get(Calendar.YEAR);
int startMonth = cs.get(Calendar.MONTH) + 1;
int endYear = ce.get(Calendar.YEAR);
int endMonth = ce.get(Calendar.MONTH) + 1;
int difYear = endYear - startYear;
int difMonth = endMonth - startMonth;
String baseCollectionName = logType == 0 ? MarketConstant.WAIT_STATUS_REPORT_COLLECTION_NAME : MarketConstant.SEND_SMS_LOG_COLLECTION_NAME;
if(difYear == 0 && difMonth == 0){
//同年同月份查询
String collectionName = startYear + "_" + baseCollectionName + "_" + (startMonth < 10 ? "0" + startMonth : startMonth);
sendmsgLogList = searchSameMonthFromMongoDb(sendmsgLog,(page - 1) * rows,rows,beginTime,endTime,collectionName);
total = getCollectionEleCount(sendmsgLog,beginTime,endTime,collectionName);
}else if((difYear == 0 && difMonth == 1) || (difYear == 1 && difMonth == -11)){
//同年相邻月份查询 || 跨年相邻月份查询
String startCollectionName = startYear + "_" + baseCollectionName + "_" + (startMonth < 10 ? "0" + startMonth : startMonth);
String endCollectionName = endYear + "_" + baseCollectionName + "_" + (endMonth < 10 ? "0" + endMonth : endMonth);
sendmsgLogList = searchAdjoinMonthFromMongoDb(sendmsgLog,page,rows,beginTime,endTime,startCollectionName,endCollectionName);
int startTotal = getCollectionEleCount(sendmsgLog,beginTime,endTime,startCollectionName);
int endTotal = getCollectionEleCount(sendmsgLog,beginTime,endTime,endCollectionName);
total = startTotal + endTotal;
}else if (difYear == 0 && difMonth > 1){
//同年跨多月份查询
sendmsgLogList = searchMultiMonthFromMongoDb(sendmsgLog,page,rows,beginTime,endTime,startYear,startMonth,endMonth,baseCollectionName);
String currentCollectionName;
for (int i = endMonth; i>=startMonth; i--){
currentCollectionName = startYear + "_" + baseCollectionName + "_" + (i < 10 ? "0" + i : i);
total += getCollectionEleCount(sendmsgLog,beginTime,endTime,currentCollectionName);
}
}else{
//跨年跨多月份查询[暂不支持,返回空结果]
resultMap.put("sendmsgLogList",sendmsgLogList);
resultMap.put("total",total);
return resultMap;
}
resultMap.put("sendmsgLogList",sendmsgLogList);
resultMap.put("total",total);
return resultMap;
}
/**
* 查询同年同月数据
* @param sendmsgLog
* @param skipCount
* @param rows
* @param beginTime
* @param endTime
* @param collectionName
* @return
* @throws ParseException
*/
public List<SendmsgLog> searchSameMonthFromMongoDb(SendmsgLog sendmsgLog, Integer skipCount, Integer rows, String beginTime, String endTime,String collectionName) throws ParseException {
Criteria criteria = buildSearchParams(sendmsgLog,beginTime,endTime);
Query query = new Query();
query.addCriteria(criteria).with(new Sort(new Sort.Order(Sort.Direction.DESC,"createTime"))).skip(skipCount).limit(rows);
return mongoTemplate.find(query,SendmsgLog.class,collectionName);
}
/**
* 获得集合中过滤后的元素个数
* @param sendmsgLog
* @param beginTime
* @param endTime
* @param collectionName
* @return
* @throws ParseException
*/
public int getCollectionEleCount(SendmsgLog sendmsgLog,String beginTime,String endTime,String collectionName) throws ParseException {
Criteria criteria = buildSearchParams(sendmsgLog,beginTime,endTime);
Query query = new Query();
query.addCriteria(criteria).with(new Sort(new Sort.Order(Sort.Direction.DESC,"createTime")));
return (int) mongoTemplate.count(query,collectionName);
}
/**
* 查询同年相邻月数据
* @param sendmsgLog
* @param page
* @param rows
* @param beginTime
* @param endTime
* @param startCollectionName
* @param endCollectionName
* @return
* @throws ParseException
*/
public List<SendmsgLog> searchAdjoinMonthFromMongoDb(SendmsgLog sendmsgLog, Integer page, Integer rows, String beginTime, String endTime,String startCollectionName,String endCollectionName) throws ParseException {
long endMonthCount = getCollectionEleCount(sendmsgLog,beginTime,endTime,endCollectionName);
int skipCount = (page - 1) * rows;
if (endMonthCount - skipCount >= rows){
//当前页数据全部处于结束月集合中
return searchSameMonthFromMongoDb(sendmsgLog,skipCount,rows,beginTime,endTime,endCollectionName);
}else{
//当前页数据全部处于开始月集合中 || 处于结束月和开始月集合中
//从结束月中获取数据
List<SendmsgLog> endMonthList = searchSameMonthFromMongoDb(sendmsgLog,page,rows,beginTime,endTime,endCollectionName);
//结束月中无数据
if (CollectionUtils.isEmpty(endMonthList)){
//当前页数据全部处于开始月集合中
skipCount = (page - 1) * rows - (int)endMonthCount;
return searchSameMonthFromMongoDb(sendmsgLog,skipCount,rows,beginTime,endTime,startCollectionName);
}else{
//当前页数据处于结束月和开始月集合中
if (rows - endMonthList.size() >= 0){
skipCount = 0;
rows = rows - endMonthList.size();
List<SendmsgLog> startMonthList = searchSameMonthFromMongoDb(sendmsgLog,skipCount,rows,beginTime,endTime,startCollectionName);
endMonthList.addAll(startMonthList);
return endMonthList;
}else{
//rows - endMonthList.size() = 0,此时有新数据插入endMonth刚好够rows数量
return endMonthList;
}
}
}
}
/**
* 同年跨多月份查询
* @param year
* @param startMonth
* @param endMonth
* @param baseCollectionName
* @return
*/
public List<SendmsgLog> searchMultiMonthFromMongoDb(SendmsgLog sendmsgLog, Integer page, Integer rows, String beginTime, String endTime, int year,int startMonth,int endMonth,String baseCollectionName) throws ParseException {
int skipCount = (page - 1) * rows;
int currentCollectionEleCount;
int sumCollectionEleCount = 0;
String currentCollectionName;
List<String> waitFindCollectionNames = new ArrayList<>();
for (int i = endMonth; i>=startMonth; i--){
currentCollectionName = year + "_" + baseCollectionName + "_" + (i < 10 ? "0" + i : i);
currentCollectionEleCount = getCollectionEleCount(sendmsgLog,beginTime,endTime,currentCollectionName);
sumCollectionEleCount += currentCollectionEleCount;
if (currentCollectionEleCount >= skipCount){
waitFindCollectionNames.add(currentCollectionName);
}
if (sumCollectionEleCount >= (skipCount + rows)){
break;
}
}
List<SendmsgLog> logList;
List<SendmsgLog> resultList = new ArrayList<>();
int size = rows;
for (String collectionName : waitFindCollectionNames){
if (resultList.size() >= size) {
break;
}else{
logList = searchSameMonthFromMongoDb(sendmsgLog,skipCount,rows,beginTime,endTime,collectionName);
resultList.addAll(logList);
}
skipCount = 0;
rows = size - resultList.size();
}
return resultList;
}
public Criteria buildSearchParams(SendmsgLog sendmsgLog, String beginTime, String endTime) throws ParseException{
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
Criteria criteria = new Criteria();
//发送时间开始和发送时间结束
if (StringUtils.isNotBlank(beginTime) && StringUtils.isNotBlank(endTime)) {
criteria.and("createTime").gte(sdf.parse(beginTime)).lte(sdf.parse(endTime));
}
//发送时间开始
if (StringUtils.isNotBlank(beginTime) && StringUtils.isBlank(endTime)) {
criteria.and("createTime").gte(sdf.parse(beginTime));
}
//发送时间结束
if (StringUtils.isBlank(beginTime) && StringUtils.isNotBlank(endTime)) {
criteria.and("createTime").lte(sdf.parse(endTime));
}
//任务编号
if (StringUtils.isNotBlank(sendmsgLog.getBatchNum())){
criteria.and("batchNum").is(sendmsgLog.getBatchNum());
}
//登录名
if (StringUtils.isNotBlank(sendmsgLog.getLoginName())){
criteria.and("loginName").is(sendmsgLog.getLoginName());
}
//通道简称
if (StringUtils.isNotBlank(sendmsgLog.getSmswayName()) && !sendmsgLog.getSmswayName().equals("-1")) {
criteria.and("smswayId").is(sendmsgLog.getSmswayName());
}
//发送方式
if (StringUtils.isNotBlank(sendmsgLog.getType()) && !sendmsgLog.getType().equals("-1")){
criteria.and("type").is(sendmsgLog.getType());
}
//短信类型
if (StringUtils.isNotBlank(sendmsgLog.getSmsType()) && !sendmsgLog.getSmsType().equals("-1")){
criteria.and("smsType").is(sendmsgLog.getSmsType());
}
if ((StringUtils.isNotBlank(sendmsgLog.getState()) && !sendmsgLog.getState().equals("-1")) && (StringUtils.isNotBlank(sendmsgLog.getSentStatus()) && !sendmsgLog.getSentStatus().equals("-1"))) {
if (sendmsgLog.getSentStatus().equals("1") && sendmsgLog.getState().equals("1")){
criteria.orOperator(Criteria.where("state").is(sendmsgLog.getState()),Criteria.where("sentStatus").is(sendmsgLog.getSentStatus()));
}else {
criteria.and("state").is(sendmsgLog.getState());
criteria.and("sentStatus").is(sendmsgLog.getSentStatus());
}
}else if (StringUtils.isNotBlank(sendmsgLog.getState()) && !sendmsgLog.getState().equals("-1")) {
//发送状态
criteria.and("state").is(sendmsgLog.getState());
}else if (StringUtils.isNotBlank(sendmsgLog.getSentStatus()) && !sendmsgLog.getSentStatus().equals("-1")) {
//下发状态
criteria.and("sentStatus").is(sendmsgLog.getSentStatus());
}else{
//均为空的情况前面拦截掉了
}
return criteria;
}