Recently received a demand: order data cleaning, data volume 10W +.
It is cumbersome and complex to implement logic:
circulating all users get invitation ID number -> get all of the corresponding ID number of the invitation ID invitees cycle -> The invitee user to get token ID -> user All token found orders -> loop checks the status of all orders (to determine whether a refund) -> If you cancel the refund corresponding to invite people incentives red envelope -> after deducting cash reward ordinary envelopes, to recalculate the number of persons invited to produce red ladder.
Business code is indeed very complicated very complex, the most important thing is to invite people to the invitees are many relationships, people are invited to take the order details are to-many relationship, in which various doping judgment and triple for the cycle, may want to with knowledge of the cleaning data interface on the server running on top of it, the impact is significant.
Therefore, I call online here split into the line interface Interface relieve production pressure. Specific steps are as follows:
1: In line with local each way to get passed in a production interface invitees ID, which is the production environment for each cleaning process only one invitee.
2: Due to the different development environment and production environment database, so the production environment to invite all people to export a JSON file.
3: Local development environment to read JSON file line, each person ID pass an invitation to the production of clean interfaces.
4: very long because the logic, use a transmit delay interval of four seconds. (Test interface 3 to 5 seconds)
Well now the code, ignoring the business code, here is a record of learning it himself:
Libraries invite people to start producing real production environments ID export JSON.
Into a casual place, then node reads the JSON file.
Pit:
(1) to read the file must write an absolute path, relative path to the file can not be found!
(2) fs reads out of type String, do not believe the words of their own plus a look at the Type of data. JSON is that JSON not appear to be the type.
(3) which is attached in the callback function value is useless, unless forced into synchronization. Do not believe you can own various places console.log () to look at.
// 先引入
const fs = require('fs');
fs.readFile(
// 注意这里一定要写绝对路径，相对路径无法识别。
'/Users/XXXX/Desktop/XXXXX_project/XXXXapi/app/controller/data.json',
'utf8',
async function(err, data) {
if (err) {
console.log(err);
}
// 一定要JSON转了才是JSON格式，不然读取都是String。
const dataList = JSON.parse(data);
},
);
Because there is no JAVA node of sleep () thread to sleep, so to allow the process to achieve their own sleep (). times is their definition passed in the number of seconds.
function sleep(times) {
let now = new Date();
const exit = now.getTime() + times;
while (true) {
now = new Date();
if (now.getTime() > exit) {
return;
}
}
}
Finally, look at the complete code:
Pit:
(1) setTimeout that are not suitable for use in a for loop delay task, a start I was Sike setTimeOut; but think carefully:
A: If for setTimeOut on the inside, then that is the moment to finish second delay N for, reach each data delayed a few seconds and then won the next data.
B: If for on the outside setTimeOut, it will keep repeating several times the same output as a set of elements (hard to describe, you can own experience)
forced resignation achieve their own Sleep ().
async test() {
fs.readFile(
'/Users/自己文件存放的位置/data.json',
'utf8',
async function(err, data) {
if (err) {
console.log(err);
}
const dataList = JSON.parse(data);
for (const item of dataList.record) {
sleep(4000); // 停滞四秒
const res = await axios.get(
`https://线上接口的域名+路径?invite_user_id=${item.invite_user_id}`,
);
console.log(item.invite_user_id);
console.log(res.data);
}
},
);
function sleep(times) {
let now = new Date();
const exit = now.getTime() + times;
while (true) {
now = new Date();
if (now.getTime() > exit) {
return;
}
}
}
this.ctx.success();
}
This part does not have to careful study! ! ! Online tedious cleaning business codes, lengthy cleaning process do not read. According to business on the line, these business code no reference value. It is also because it is lengthy, before taking this design pattern.
/**
* 清洗退费用户接口
*/
async cleanRefundUser() {
// 首先根据邀请者ID拿到全部被邀请者ID，再用被邀请者ID，也就是newcomer ID去users表拿accesstoken，再用token去查用户订单。
const inviteUserId = this.ctx.query.invite_user_id;
let accessToken;
let noRefund = 0;
let refund = 0;
const userIdList = await this.inviteVacationClass.getAllUserIdByInviteUserId(
inviteUserId,
);
logger.info(LOG_CAT, {
msg: '清洗数据开始',
data: {
inviteUserIds: inviteUserId,
userIdLists: userIdList,
},
});
let ordersActivity;
for (const id of userIdList) {
// 拿被邀请者的token
const userAccessToken = await this.inviteVacationClass.getAccessTokenByUserId(
id.newcomer_id,
);
if (userAccessToken) {
accessToken = userAccessToken.access_token;
}
// 用token去查询订单，注意这里会返回多个订单！
const res = await axios.get(
`${this.app.config.get_teacher_info_api}/api/1/pay/get_order_list?access_token=${accessToken}`,
);
if (res && res.data.data.finish_sub_orders[0]) {
for (const item of res.data.data.finish_sub_orders) {
ordersActivity = item.activity;
// 判断是否老带新活动
if (
// 过滤掉不是老带新活动的订单
ordersActivity === '2019grow50yuanldx' ||
ordersActivity === '2019grow50yuanldx_bzr_xiaoxue' ||
ordersActivity === '2019grow50yuanldx_bzr_chuzhong' ||
ordersActivity === '2019grow50yuanldx_xiaodi' ||
ordersActivity === '2019grow50yuanldx_chuzhong' ||
ordersActivity === '2019grow50yuanhanjia_jiesuo_50'
) {
// 订单状态判断：1 已支付 2 退费 0 未付款 3 取消
let orderStatus;
orderStatus = item.status;
if (orderStatus === 1) {
noRefund = noRefund + 1;
} else if (orderStatus === 2) {
// 再根据newcommer id 和 invite user id拿到对应哪个红包。
const redPocket = await this.ctx.app.dao.inviteNewcomerV2.inviteRedpocket.getRedPocketById(
inviteUserId,
id.newcomer_id,
);
if (redPocket[0][0]) {
// 把该普通固定奖励红包设置为退费状态，不可领取
await this.ctx.app.dao.inviteNewcomerV2.inviteRedpocket.cancelRedPocketById(
redPocket[0][0].id,
new Date(),
);
}
logger.info(LOG_CAT, {
msg: '用户退费信息',
data: {
newcomer_id: id.newcomer_id,
inviteUserIds: inviteUserId,
redPocketId: redPocket[0][0].id,
redPockets: redPocket[0][0],
},
});
refund = refund + 1;
}
}
}
}
}
logger.info(LOG_CAT, {
msg: '清洗普通红包结束，开始统计剩余人数',
data: {
inviteUserIds: inviteUserId,
noRefunds: noRefund,
refunds: refund,
},
});
// 假如未退费好友小于10人，要扣除100元阶梯红包，其它实物奖励不用管。
if (noRefund < 10) {
const redpocketType = 25;
await this.ctx.app.dao.inviteNewcomerV2.inviteRedpocket.cancelExtraRedPocket(
inviteUserId,
new Date(),
redpocketType,
);
}
this.ctx.success('未退费人数：' + noRefund, '退费人数：' + refund);
}
Well, now began to run it and see hundreds of thousands of orders can not be cleaned and accurate deduction corresponding reward it.
can. Well, let it quietly call the local production run for several hours on the line.
This is their actual share good ideas encountered in the development or design process for some special scenes, recorded share. bow!
