Indispensable program buried point

Fill-in purpose

For the buried point of the data statistical analysis program modified this time, it is only for some errors that are not easy to find in the statistical data, and all the programs will actively run out. But as long as the statistical data is actively thrown out, it is a buried point.

Data burying is the preparatory work for data analysis. Only after burying the data can the data be obtained for analysis. After doing enough data burying, you can get the desired target data:

1. Behavioral data: time, place, person, interaction, content of interaction;
2. Quality data: browser loading status, error exceptions, etc.;
3. Environmental data: browser-related metadata, geography, operators, etc.;
4. Operational data: PV, UV, conversion rate, retention rate (very intuitive data)

After adjusting the standardization of the Python project as a whole, adding error data buried points belongs to the quality data part.

Buried program

Buried points are divided into front-end buried points and back-end buried points. Common buried point solutions include

1. Front-end code burying point: directly embed code in the front-end to collect target data
2. No front-end buried point: common front-end buried point technology. Collect all, select as needed
3. Back-end code burying point: directly embed code in the back-end to collect target data

This time, it mainly focuses on the back-end code burying of projects that do not use third-party data analysis platforms (applicable to the scenario of data statistics analysis monitoring program)

The design of back-end buried points must consider the following points:

1. High flexibility: you can manually embed points anytime, anywhere
2. Small impact: less intrusion into source code
3. Updated: When the buried point content changes, you only need to modify the buried point interface, instead of modifying one by one in the code

There are no buried points in some public libraries or encapsulated custom frameworks, and all buried points are in the business layer. For example, when the number of connections in the database connection pool overflows, the error needs to be processed, and then the relevant information is thrown to the business layer according to the custom unified style, so that the business layer can judge whether there is an error in the execution of the public library and the custom framework.

The specific implementation plan of the back-end burying point

Directly write buried point code to collect data at the specified location on the backend. Each method has some differences in sharing common information.

1. Import a piece of configuration information in each file that needs to be used to send messages, and then manually bury points according to the configuration information where needed.
2. Encapsulate message sending into a MixIn class, and use inheritance to mix it into each class. Every time you need to send a message to disconnect after establishing a connection.
3. Encapsulate the buried point information into a module, provide a global initialization function and a buried point message generation function, call the initialization function of the initialization module at the program entry for initialization, and then use the message generation function anywhere in the program. This method is similar to the method used when using a third-party data analysis platform to provide SDK.

Program selection:

• Since the project is written in python, the buried point data is sent to kafka, and the packaged third-party python module can collect all the information it needs. Therefore, the last method is adopted, written in the form of a pluggable third-party sdk, which can directly encapsulate some new methods, including some usable information, such as the number of lines where the error message is automatically collected.
• For non-python projects, provide python scripts for other language programs to call. For example, calling a Python script in a retweet stream service written in Erlang throws error messages and other data.

---

reference:

• Sensors data: buried deep routine, don't fall into the "pit"
• The next evolution of data analysis products: based on non-buried points with buried points
• Meituan Dianping front-end traceless burying practice

---

Remember to give me a thumbs up!

Carefully sort out the video courses and e-books in all directions of computer, from introductory, advanced, and practical, and classify them reasonably according to the catalog. You can always find the learning materials you need. What are you waiting for? Follow and download now! ! !

If you never forget, there will be echoes, friends, please give me a thumbs up, thank you very much.

I am a bright brother in the workplace, a YY senior software engineer with four years of work experience, and a slash programmer who refuses to be the leader.

Listen to me, a lot of progress, the life of the program is a shuttle

If you are lucky enough to be able to help you, please give me a [Like], give me a follow, and if you can comment to give encouragement, it will be very grateful.

Workplace Liangge article list: more articles

All my articles and answers have cooperated with the copyright protection platform, and the copyright belongs to Brother Liang in the workplace. Unauthorized reprinting will be punished!