background
The customer has a jewelry ERP that has been used for many years, which contains product information and accurate inventory. Their original ordering system can no longer meet the needs of business development, and will be switched to a PC version and a small program version with more advanced and complete functions. Jewelry ordering system - Premium Ordering .
Statement of needs
Customers hope that ERP product and inventory information can be automatically connected with the ordering system to reduce operational workload and maintain data synchronization.
Implementation plan
Originally, ERP only provided a data query interface that supported paging. The query interface supported searches by three fields: product barcode, product name, and product creation time.
I analyzed it and found that based on the current ERP interface, it is impossible to achieve real-time data synchronization because there are many products, and the configuration and bandwidth of the ERP server are not enough, the response speed is slow, and it cannot support too frequent queries. , so we communicated with the ERP side and asked them to add the "Last Update Time" field, and add query support based on the "Last Update Time" field interval in the query interface, and then the ordering system initiated every 15 minutes a query that had changed every 15 minutes. For product inventory query, if the result is obtained, the data will be synchronized. If the result is empty, it means that no product information has changed in this time interval. Mark this time interval as updated and wait for the next update. See the flow chart below for the logic.
Solution advantages
The logic is rigorous. Common network errors in data synchronization between the two systems will not cause data synchronization errors, because each page in each time interval must ensure that the synchronization is successful before the update log is written. In this way, when the network fails or one server has a problem, When a problem occurs, after returning to normal, the synchronization task can be restored from the last updated record and continue to update. Moreover, the entire synchronization process has visual output, which is very clear and easy to control. There are also detailed logs to track the problem.
flow chart
Database Design
Field description:
Core code reference (php)
This is the main method of the scheduled task, that is, the entrance
/**
* [定时任务入口]增量同步商品信息,主要是找到更新到哪个时间段的哪一页了
*/
public function additionalSync(){
Tools::realTimeOutputPrepare();
$page = 1;
$lastRecord = ErpSyncModel::open()->order('updateAt', 'desc')->first();
if($lastRecord){
if($lastRecord['status'] === 1){
//说明上次的更新已经完成
$startTime = $lastRecord['endTime']+1;
}else{
//如果未完成则继续更新
$startTime = $lastRecord['startTime'];
$page = $lastRecord['page'];
if(time() - $lastRecord['updateAt'] <= 60){
die('距离上次更新未超过60秒,暂不执行更新');
}
}
}else{
//没有记录则从头开始
$startTime = strtotime('-1 days');
}
$endTime = $startTime + 15*60;//每15分钟为一个周期
if($endTime > time()){
die('当前已经是最新数据了,请等待下一轮更新');
}
$this->pullData($startTime, $endTime, $page);
}
The following is to access the ERP interface to synchronize data and update synchronization records
/**
* 拉取数据
* @param $startTime
* @param $endTime
* @param $page
*/
public function pullData($startTime, $endTime, $page){
while(true){
$startTimeString = urlencode(date('Y-m-d H:i:s', $startTime));
$endTimeString = urlencode(date('Y-m-d H:i:s', $endTime));
Tools::realTimeOutput('正在获取['.date('Y-m-d H:i:s', $startTime).']至['.date('Y-m-d H:i:s', $endTime).']的数据,当前是第['.$page.']页');
$url = 'https://api.xxx.com?pageId='.$page.'&pageSize='.$this->pageSize.'&startCreateDate='.$startTimeString.'&endCreateDate='.$endTimeString;
$res = Tools::curlGet($url, 30);
if($res['success']){
//CURL成功
$responseData = Tools::jsonToArray($res['data']);
if(intval($responseData['status']) === 200){
//表示接口返回是成功的
$productList = $responseData['data'];//商品列表数据
$getProductCount = count($productList);
Tools::realTimeOutput('已获取到第['.$page.']页数据,共['.$getProductCount.']条记录');
$upsertData = [
'startTime' => $startTime,
'endTime' => $endTime,
'count' => $getProductCount,
'page' => $page,
'status' => 0//表示这个时间段的已经拉取完了
];
if($getProductCount === 0){
$upsertData['status'] = 1;//如果没有记录了就将记录状态改为完成
ErpSyncModel::open()->upsert($upsertData, $upsertData, 'startTime,endTime');
break;
}else{
//如果有记录就更新记录
ErpSyncModel::open()->upsert($upsertData, $upsertData, 'startTime,endTime');
try{
foreach($productList as $item){
$this->importOneByOne($item);
}
if($getProductCount >= $this->pageSize){
//说明还有下一页
++$page;
continue;
}else{
//说明没有下一页了
$upsertData['status'] = 1;
ErpSyncModel::open()->upsert($upsertData, $upsertData, 'startTime,endTime');
break;
}
}catch (Exception $e){
$upsertData['data'] = $productList;
$upsertData['message'] = $e->getMessage();
ErpSyncExceptionModel::open()->add($upsertData);
++$page;
}catch(\PDOException $pe){
$upsertData['data'] = $productList;
$upsertData['message'] = $pe->getMessage();
ErpSyncExceptionModel::open()->add($upsertData);
++$page;
}
}
}else{
Tools::realTimeOutput('CURL未获取到数据,5秒后将重试获取第'.$page.'页');
sleep(5);
}
}else{
//如果curl获取失败就睡5秒再试
Tools::realTimeOutput('CURL获取失败,5秒后将重试获取第'.$page.'页');
sleep(5);
}
}