1. Background
Jinli Platform, as an enterprise-level B2B2C e-commerce platform, serves both corporate customers and corporate employees. Therefore, it needs to follow the policies and regulations of corporate customers to ensure that the products in the mall comply with regulations and improve the employee shopping experience. However, this unique operating model has led to a more prominent problem of visible and unsaleable goods on the Jinli platform, which has had a greater negative impact on the end consumer's shopping experience and the platform's products and business.
2. Solution
As the title says, the reason why it is called a small trick is because we do not use some high-precision technology, but just combine a variety of mature technologies and add some algorithms.
The following are the 3 versions of the solution iterations we have experienced, which also represent the transformation of a technical person from technical thinking to business thinking.
Version 1.0 : We try to add a mask to unsellable items to mark the reason why they are unsaleable to prevent customers from misoperation. However, this approach does not completely solve the problem, as consumers may still be confused as to why certain items are not available for sale (such as why gold cannot be purchased on the Jinli platform, or why the items they see are blacklisted).
Version 2.0 : We strive to improve the efficiency of search, speed up the delivery of unsaleable products from the warehouse, and optimize the product synchronization mechanism to reduce the frequency of unsaleable products. However, with the expansion of Jinli platform’s unsaleable rules (such as pricing rules and price inversion restrictions, etc.), this customized approach was too complex for the search team.
Version 3.0 : As we deepen our understanding of consumer needs, we gradually realize that although our early methods have reduced the number of unsaleable products, even if there is only one unsaleable product on the platform, it may have an impact on the consumer's shopping experience. cause losses. Therefore, we need to use technical means to "hide" these commodities.
Plan overview
Figure 1 Secondary filtering tool
Plan preview
When we reached 3.0, we first deduced the following two options:
- Front-end preloading : Every time the front-end makes a request, it will request data from the back-end according to the number of displays per page, and after receiving the results returned by the back-end, the data will be filtered according to the established rules. If the amount of filtered data is insufficient, the next page of data will continue to be requested from the backend.
- Backend polling : The backend will continuously send requests to the downstream interface, and each request will increment the page number until the data results that meet the conditions are obtained.
However, the shortcomings of the above technical solutions are also obvious:
- Requests initiated by the front end and filtered will result in redundant front-end business logic, which is inconsistent with the positioning of the front-end view layer.
- If the request initiated by the front end does not receive data return, it will try to request again. If multiple requests have data incompleteness issues, this may cause the front-end waterfall flow to freeze, manifesting as loading speeds that are sometimes fast and sometimes slow, giving consumers the illusion of network freezes.
- The front end frequently initiates invalid requests, which will cause unnecessary consumption of network traffic.
- Backend requests are stateless every time. If not handled properly, it can easily cause repeated acquisition of data or confusion of page numbers.
- The depth of backend request polling cannot be set dynamically. If set incorrectly, it can easily cause the interface to time out.
We have made a series of optimizations to address the possibility that front-end preloading may complicate front-end business logic and the introduction of back-end polling may cause data synchronization problems.
We introduced two important components, registers and filters, to reasonably schedule and process original requests.
First of all, the register is used to temporarily store product data, so that the front end only needs to send a request from the front end to the register each time to obtain the latest product data, avoiding repeated requests and timeout issues with back-end polling.
Secondly, filters are used to filter and process product data. After obtaining the latest product data, the filter will filter the data according to certain rules, select product data that meets policy specifications and user needs, and display it to the user. At the same time, for product data that does not comply with policy specifications or contains illegal products, the filter hides or marks them as unsaleable to prevent users from mistakenly purchasing illegal products.
Through such optimization, we not only avoid the problem of complicating front-end business logic, but also solve the data synchronization problems that may arise in back-end polling. At the same time, the modified interface is still an ordinary REST interface, which can be easily understood and used by both the front end and the back end.
Figure 2 Composition of filter tools
The following is an introduction to the functions of each link
original request(orgin request)
There are three common list requests:
1. Query services provided by third parties, RPC paging requests, such as search product list RPC interface, recommended product list RPC interface
2. Database query, mybatis paging query, you can use the auto-incremented primary key id to make subsequent requests
3. ES search query, ES paging query, scrolling query
Input parameters: PageParams<T>, standard paging request parameters, T is the real paging request object
Output parameters: PageResult < R > , standard paging return parameters, R is the real paging request return object
The implementation of the SDK is based on generic development, and the caller needs to customize the method in accordance with the specifications.
filter(customer filter)
Input parameters: List< R > sourceData, original return result list
Output parameters: List< R > targetData, filter the filtered result list
The implementation of the SDK is based on generic development, and the caller needs to customize the method in accordance with the specifications.
register (storage register)
Implemented through caching middleware, query results are temporarily stored, and are obtained from the register first after the front-end request comes. The request parameters are compressed through the algorithm to ensure that the same request parameters can obtain the same value to determine whether it is a request with the same query condition.
The key structure of the register is as follows:
scroll_id: pin&activiivtyCode&uid&query input parameter MD5 compression algorithm,
In addition, query input parameters need to exclude page numbers and dynamically changing parameters.
Storage content: Cache completes the remaining data snapshot after paging, front-end request page number, actual back-end request page number
coordinator ( coordinate )
The coordinator implements data supplementation, query, deposit and withdrawal of register data snapshots, and updates of paging-related data.
1. The coordinator expands the step size based on the original front-end request and the actual back-end request to pull subsequent data.
2. The front and back ends make requests in fixed steps. For example, the front end requests 10 items each time and the back end requests 20 items each time. The step size ratio can be dynamically adjusted according to the actual situation.
3. The request depth and the front-end and back-end step size ratio are controlled through ducc. For example: the front end requests 10 pieces of data per page, and the step ratio is 3, then the back end requests 30 pieces of data per page each time; the request depth is 2, and if the request is made twice, it will be forced to stop if it still does not meet the requirements. Customer dimension customization request parameters
{
"DEFAULT": {
"deep": 2,
"multiple": 3
},
"营销测试2": {
"deep": 2,
"multiple": 5
},
"呼铁福利商城采购账号": {
"deep": 2,
"multiple": 4
}
}System principle
Figure 3 Data secondary filtering flow chart
We split the secondary filtering of data into the following four stages:
Query stage:
Receive the front-end request and determine the back-end interface request based on the size of the data in the register. The back-end performs RPC requests according to the front-end N times the step size. The hit result will create a priority queue snapshot in the cache and pass scroll_id (C dimension) cache key) points to it, lastPageNo points to the last visited page number, and realPageNo points to the real visited page number.
Filter stage:
Read the filtering rules and implement data filtering. Commonly used filtering rules, such as: not for sale, out of stock, price higher than the market price, etc., data that does not comply with the rules are directly removed.
Cache stage:
Fetch stage
Through the above two stages, we can obtain a data set that meets the front-end request step size, supplement other ancillary information, and return the paging data structure.
After the first search for data, find the corresponding snapshot through scroll_id in the Query stage, then use lastPageNo, realPageNo to add query conditions to the original query statement (pageNo=realPageNo + 1), and find data in the snapshot.
3. Sharing results
After the technical transformation plan was launched, the Mid-Autumn Festival traffic was intensively observed, and tens of millions of product filtering times were performed daily. On average, 3-4 products were filtered per request.
Figure 4 Effects before and after secondary filtering is turned on
Figure 5 Jinli platform traffic during the Mid-Autumn Festival
Figure 6 Number of product filters during the Mid-Autumn Festival
Figure 7 Interface performance after product search filtering
4. Thinking and understanding
In the iterative evolution of products, we firmly believe that technology can bring unlimited possibilities to our business. It can help us innovate, optimize processes and improve efficiency. At the same time, we also understand that technology is not a panacea. But it is this deep awe of technology that keeps us humble and open-minded, enabling us to always find new breakthroughs and surpass ourselves.
A small skill can also bring great value to the business! ! !
The author of the open source framework NanUI switched to selling steel, and the project was suspended. The first free list in the Apple App Store is the pornographic software TypeScript. It has just become popular, why do the big guys start to abandon it? TIOBE October list: Java has the biggest decline, C# is approaching Java Rust 1.73.0 Released A man was encouraged by his AI girlfriend to assassinate the Queen of England and was sentenced to nine years in prison Qt 6.6 officially released Reuters: RISC-V technology becomes the key to the Sino-US technology war New battlefield RISC-V: Not controlled by any single company or country, Lenovo plans to launch Android PCAuthor: JD Retail Mao Chenfei
Source: JD Cloud Developer Community Please indicate the source when reprinting