From data to large model application, on November 25th, Hangzhou Yuanchuang Conference was held to share development tips.

I. Overview

1.Project value and results

Using the group's unified point-of-sale collection capabilities and point-of-care platform, we completed the point-of-spot migration of Dada's 7 business lines and a total of 43 site applications, reducing the R&D investment and machine costs of self-developed collection tools and platforms, opening up data links, and creating More data analysis value. The specific cost reduction and efficiency improvement values are as follows:

1.1 Value of data analysis: Connect with JD.com’s traffic data, align data standards, and connect traffic in series

1) Information island: Kuaishou and JD traffic data are separated, database table access costs are high, and combined analysis cannot be performed

2) Data caliber: Alignment of buried point specifications: For example, hidden point reporting rules such as advertising space exposure, page views, etc. are aligned with JD.com’s logic to align the data caliber and improve accuracy.

3) User information concatenation: Through the concatenation of user basic information, such as device_id, user pin/supplier_id, etc., we can analyze the traffic link that the user enters from the JD entrance, the access depth of the JD traffic, etc.

1.2 Reduce iteration/maintenance labor costs: 0.5 people/year

1) Tianhe platform iteration cost

2) Traffic transmission link operation and maintenance costs

1.3 Save machine/middleware costs: 2w+ per year

1) Reduce 2 cloud hosts + estimated middleware costs of around 20,000/year

2.Project rhythm

2. Plan research and selection

1. Program measurement

In the early stage, we formulated a migration plan one. However, considering the huge migration cost, we formulated a plan two after a comprehensive evaluation. Based on comprehensive considerations such as minimizing the migration cost, we selected the plan one.

	Plan details	workload
Option One	Each business line of APP, mini program, PC&H5 will be transformed and migrated respectively.	Need to do SDK layer adaptation for 43 site applications
Option II	1) The APP is encapsulated in SDK, and other business lines only need to access it. 2) PC&H5 is adapted at the SDK layer, and the other 35 applications only need to access it.	3 SDK layer adaptations are required, and other business lines can be connected.

3. Project realization

1. Upstream and downstream migration process

2. Migrate technical architecture

Before joining the project, we need to consider two points

Stability : On the one hand, the project needs to run smoothly as a whole, and on the other hand, the data differences before and after migration must be kept within a reasonable error range.

Efficient : This migration involves dozens of projects such as clients, applets, h5, etc. Saving development costs is also key

(1) Plan selection

1. Direct replacement: Due to the large number of projects and the many points in each project, it is obvious that we cannot directly replace the buried points in the business. The workload will be too large and the risk of problems will be higher.

2. Aspect replacement: Unified replacement in a certain link of buried point reporting. The advantage of this is that it only needs to be modified once, the code is less intrusive, and the risks are controllable. The most important thing is that the workload is greatly reduced.

(2) Scheme design

Taking the client as an example, the old tracking framework includes three processes: tracking collection, storage, and reporting, which are independent of each other. We can easily intercept aspects between tracking collection and storage, and convert the original Convert the buried point data, as shown below

Old frame architecture diagram

(3) Plan implementation

After determining the specific plan, we began to plan the specific code implementation. Considering that there are many projects that need to be replaced (taking the client as an example, there are projects such as Dada Knight, Dada Express, Hongliu, and Kongming), each project must be implemented It is undoubtedly a waste of resources. We divide the project into Android, ios, applets, and h5 by end. In this way, the original dozens of projects are simplified into a set of solutions and four end codes. Each project only needs simple integration. In this way Save a lot of human resources

3.Technical highlights

1. Use the aspect interception and migration solution to save labor costs

2. Unified encapsulation and multi-terminal multiplexing unify buried point specifications

3. Data warehouse layer: Directly consume real-time topics from JD.com and drop them into the database. Make the data structures of JD.com and Dada compatible on the database surface layer to achieve "unconscious migration" at the downstream report layer, minimizing the impact of business reports and downstream migration costs.

4. The pits that have been stepped on

1. iOS does not support background reporting of hidden points.

Problem: Knights and merchants sometimes report hidden points when the app exits the foreground and is in the background mode. However, Meridian initially did not support reporting hidden points in the background for a long time.

Solution: Meridian added configuration to support background reporting and burying functions without time limit.

2. iOS network problems

Problem: When the app is installed for the first time and the user does not agree to the network permissions, the Meridian SDK will report DAU points. After the report fails, it will be retried three times and the failure will trigger the 2-minute limit. The points will not be reported within 2 minutes.

Solution: Meridian offers separate packages without the 2-minute limit.

3. APP reports policy issues

Problem: Meridian’s default reporting strategy is 15 seconds and 10 items, which results in some users not triggering the reporting conditions and being unable to report existing hidden points after exiting the app.

Solution: Meridian update configuration is 2s2.

4. The uid is empty, resulting in buried points not being included in the table.

Problem: New users will not obtain the user's device information before agreeing to the privacy agreement, resulting in the appUniqueID being passed empty and not falling into the Meridian offline table.

Solution: Before the new user agrees to the privacy agreement, randomly generate a string and encrypt it, and pass it to Meridian as the device ID to ensure that all buried points can be recorded.

5. Issues with the mini program reporting mechanism

Problem: The mini program Dada SDK is reported in batches, while the Meridian SDK is reported individually, resulting in data differences.

Solution: Meridian SDK supports batch reporting

6. H5 buried points will be discarded when their magnitude is too large.

Problem: In torrent applications, data will be discarded by the Meridian offline table after the session_id is greater than 10,000 times.

Solution: The data warehouse in the same city consumes data directly from the real-time topic and drops it into the offline table without adding session_id quantity limit.

Author: JD Retail Priscilla Zhou

Source: JD Cloud Developer Community Please indicate the source when reprinting

Dada buried point migration JD Meridian practice | JD Cloud technical team