Table of contents
Self-purchase link at the end of the article
The construction and improvement of an advertising platform is a long-term project. For example, Google launched the Google AdSense project through the acquisition of Applied Semantics as early as 2003, and until today, 20 years later, the Google display advertising platform is still innovating and improving. Advertising platforms are complex online platforms with revenue responsibilities, and any changes to them must be made with extreme care. At the same time, as the platform matures, it will become more difficult to improve the advertising effect. A complete, easy-to-use, and rigorous online experiment system is a key tool for the quality team to maintain high-efficiency output for a long time, and its importance is no less than the compass in the era of great navigation.
Every multinational Internet company will have an advertising system that contributes a steady stream of profits to the company. The development history of the Internet is also basically consistent with the development history of Internet advertising. Internet advertising can be said to be a perfect business model created by using modern information technology.
Advertising platforms are usually divided into three major areas: business (usually including delivery system, business database, etc.), system (also called engineering, infrastructure) and quality (also called algorithm, usually including data). The core competitiveness of the advertising platform is to use efficient and large-scale systems to improve the quality of advertising, thereby helping customers achieve business goals.
"Internet Advertising System: Architecture, Algorithm and Intelligence"
By Tang Xiliu
Combining the author's more than 10 years of advertising experience in Google and Tencent
Comprehensively explain the architecture, technology selection, landing methodology, implementation path and industry cases of the Internet advertising system
This book is written by the author based on his own rich industry knowledge and practical experience. It aims to help readers gain a deep understanding of the construction and improvement of advertising platforms, gain practical experience, and improve their skills and competitiveness. For example, data engineering is the core of a big data platform, and to give full play to the advantages of data, a strong system capability is required to support the realization of various algorithms for business goals. Chapter 5 of this book completely covers the elements of the data architecture, and introduces several types of common advertising targeting feature processing algorithms, including many practical experience sharing that is rare in the industry. Chapter 6 specifically elaborates various details of the experimental system construction.
This book is a "secret book" and an extremely detailed "line-level design diagram" of a large-scale advertising platform. I believe that whether you are a system engineer, algorithm researcher, product manager, or someone who is interested in joining the advertising industry, you can use this book to gain an in-depth understanding of the architecture and application of advanced advertising systems.
brief introduction
This is a book explaining the business value, product form, architecture design, technology selection, landing methodology, implementation path and industry cases of the Internet advertising system from the perspective of engineering practice. The author is engaged in advertising system architecture design and engineering at Google and Tencent. Summary of more than 10 years of experience in realization.
Through this book, you will master the following:
-
The ecology and product form of Internet advertising
first introduces major online advertising platforms such as Google, Meta, Amazon, and Microsoft; then introduces the main online advertising network and programmatic buying ecology; finally introduces the online advertising product form, including the delivery method of online advertising and billing method. The main purpose is to help readers establish a macroscopic understanding of the Internet advertising ecology -
The architecture and design of the advertising delivery system
combined with Facebook Marketing, Google Ads, Twitter Ads and other advertising platforms to explain the advertising delivery system. The main content includes the hierarchical structure of the advertisement, the characteristics and usage demonstration of the three advertising platform APIs, and the basic design principles of the API and design methodology. Readers can learn the design ideas of the delivery subsystem of the modern advertising system from this chapter. -
Design methodology of advertising system
The scale of Internet advertising systems is often relatively large, so this part introduces in detail the architecture design methodology of large-scale network systems represented by distributed systems, laying a foundation for readers to design advertising systems. -
The engineering structure of the advertising system
explains in detail the architectural design of the advertising playback system, the data architecture design of the advertising system, and the A/B testing in the advertising system. -
The architectural design of the advertising strategy system
first introduces the principle of advertising bidding and the design of the advertising strategy system; then explains the advertising system estimation subsystem, including estimation model data processing, commonly used model evaluation methods and indicators, commonly used model training methods, and New Ad CTR Estimates and Conversion Rate Estimates.
This book systematically expounds all aspects of Internet advertising system design, and has high reference value for the construction of advertising systems, recommendation systems, and big data systems.
About the Author
Tang Xiliu is a senior advertising technology expert and software architecture expert. He has successively worked for Google and Tencent. He has led the architecture design and development of Tencent's advertising retrieval system, massive data analysis system, and deep learning system, as well as the construction of Tencent's social advertising system from 0 to 1. He is now a senior architect of Mobvista.
Over 20 years in software architecture, working on projects ranging from small web services to large enterprise applications. Focus on deep learning system technology, retrieval system, distributed network service, language model, big data processing, etc., and have a deep understanding of concepts such as coding, design principles, databases, and software architecture. The job objective is to create robust, secure, and scalable software systems, with a passion for developing innovative solutions that solve complex problems and help organizations achieve their goals.
Experienced in leading development teams, managing resources, and collaborating with other teams, is an excellent communicator who is good at explaining complex technical concepts to non-technical people. Has a keen eye for detail and is able to identify potential risks and issues before they arise.
Readers
This book is a comprehensive summary of the advertising system, suitable for architects, algorithm engineers, and school teachers and students who want to understand large-scale Internet systems. In addition, marketers who want to understand the principles of the advertising system can also read it selectively, which should be beneficial.
Big coffee recommendation
The author of this book is the main person in charge of the design and development of Tencent's advertising engine, and has led the development of retrieval systems, massive data analysis systems, and deep learning systems. These systems have supported the rapid development of Tencent's advertising for ten years. During this period, the annual revenue of Tencent Advertising has increased from hundreds of millions of yuan to hundreds of billions of yuan. This book systematically introduces the Internet advertising system, covering architecture and algorithm design. The main content includes deep learning system technology, retrieval system, distributed network service, big data processing, etc. It is a practical and comprehensive guide for advertising technology practitioners .
——Lin Shifei, Director of Advertising Technology at Tencent
How is an advertising system capable of handling tens of billions of massive requests effectively constructed in engineering practice? This book gives detailed instructions. The author of this book and I have cooperated for many years in the process of building Tencent's social advertising system from 0 to 1. I admire his profound knowledge in system architecture as an architect, and also appreciate his spirit of taking the lead and writing code on the front line. I believe this book can bring great help to the front-line engineers of the advertising system.
——Jin Zhihui Xiliu, CEO of Education Technology
, is my old colleague when I was working at Google, and also a friend for many years. In my nearly 20 years of working life, I have met all kinds of great people, who have algorithms, engineering, and theory. Among these people, Xiliu is one of the few people who are good at applying algorithms to practice. master. Moreover, Xiliu can always achieve both breadth and depth in the fields he has set foot in. This book is a living proof of the above evaluation. In this book, Xiliu describes all aspects of current online advertising from the shallower to the deeper, from the ecological status of Internet advertising to advertising model algorithms, from advertising placement to advertising effect testing, from distributed cloud computing architecture to advertising system architecture, from Advertising system data architecture to bidding strategy. I believe that practitioners and enthusiasts of online advertising systems can benefit from this book.
——Liu Chao, CTO & Co-Founder of camect.com
book catalog
sequence
foreword
Chapter 1 Internet Advertising Ecosystem 1
1.1 Participants in the Internet Advertising Ecosystem 1
1.1.1 Major Online Advertising Platforms 2
1.1.2 Major Online Advertising Networks 8
1.1.3 Programmatic Buying Ecosystem 12
1.2 Forms of Online Advertising Products 15
1.2.1 Delivery methods of online advertisements 15
1.2.2 Billing methods of online advertising 18
1.3 Chapter Summary 19
Chapter 2 Ad Delivery System 20
2.1 Advertising Hierarchy 20
2.1.1 Facebook Marketing advertising hierarchy 20
2.1.2 Google Ads Advertising Hierarchy 22
2.1.3 Twitter Ads Advertising Hierarchy 25
2.2 Basic Design Principles of API 26
2.2.1 The value of API 27
2.2.2 Planning of API 28
2.2.3 API flexibility 30
2.2.4 API manageability 32
2.2.5 API supportability 34
2.3 Advertising platform API36
2.3.1 Google Ads API37
2.3.2 Facebook Marketing API40
2.3.3 Twitter Ads API50
2.4 Chapter Summary 57
Chapter 3 Large-Scale Network System Architecture
Design 58
3.1 Historical background of large-scale network systems 58
3.2 Distributed cluster management system 59
3.2.1 Introduction to Docker 61
3.2.2 Introduction to Microservice Technology 63
3.2.3 Introduction to Kubernetes 64
3.3 Distributed file system 66
3.3.1 GFS66
3.3.2 HDFS68
3.4 Distributed Storage 69
3.4.1 Introduction to distributed storage 69
3.4.2 Introduction to HBase 70
3.5 Distributed consensus service 72
3.5.1 Introduction to Distributed Consensus Algorithms 72
3.5.2 ZooKeeper usage scenarios 73
3.6 Load Balancing 76
3.6.1 Load balancing of front-end requests
3.6.2 Loads inside the data center
Balanced 77
3.7 Monitoring and warning system 77
3.8 Web Services Interface Specification 78
3.8.1 Introduction to RESTful 79
3.8.2 Introduction to GraphQL 79
3.8.3 Introduction to RPC 81
3.9 Chapter Summary 84
Chapter 4 Advertising System Architecture Design 85
4.1 Advertisement playback system architecture 85
4.2 Data ETL Module 86
4.2.1 Data Extraction 87
4.2.2 Data Transformation 91
4.2.3 Data loading 92
4.3 Retrieval Module 95
4.3.1 Text retrieval techniques 96
4.3.2 Boolean search 103
4.3.3 Nearest Neighbor Search 119
4.4 Chapter Summary 129
Chapter 5 Advertising System Data Architecture Design 130
5.1 Advertising System Data Architecture 130
5.2 Advertising system data types 131
5.2.1 Advertising metadata 131
5.2.2 Advertising log data 133
5.2.3 User portrait data 134
5.2.4 Advertising context data 147
5.2.5 Advertiser private data and
Remarketing 148
5.3 Data Management Platform 150
5.3.1 Introduction to Data Management Platform 150
5.3.2 Lookalike Audience Expansion 153
5.4 Feature Engineering Platform 157
5.4.1 Feature Production 157
5.4.2 Feature Supplementary Recording and Training Samples
data flow 159
5.4.3 Feature storage 161
5.4.4 Feature Processing Algorithms 162
5.4.5 Building a unified feature engineering platform
Necessity 165
5.5 Chapter Summary 165
Chapter 6 A/B Testing and Internet Advertising 166
6.1 Introduction to A/B Testing 166
6.2 Collecting experimental data 167
6.2.1 Traffic management 168
6.2.2 Traffic stratification 169
6.3 Analysis of experimental data 176
6.3.1 The law of large numbers and the central limit theorem 176
6.3.2 Sample size in A/B testing
estimated 179
6.3.3 Simpson's paradox 181
6.3.4 Mantel-Haenszel indicator 182
6.3.5 Bucketing and Jackknife Weight
Sample 186
6.4 Experiment Information Management 191
6.5 Advertising Application Scenarios of A/B Testing 194
6.6 Chapter Summary 194
Chapter 7 Advertising System Strategies 195
7.1 Ad Bidding 195
7.1.1 Fundamentals of game theory 197
7.1.2 Nash Equilibrium 200
7.1.3 Nash Equilibrium Existence Proof 202
7.1.4 Mechanism Design Theory 211
7.1.5 Generalized second-price bidding 231
7.1.6 VCG bidding 237
7.2 Advertising Strategy System Design 242
7.2.1 Rough layout design of advertising system 243
7.2.2 Advertising budget control system 245
7.2.3 Advertising price adjustment algorithm 247
7.3 Chapter summary 250
Chapter 8 Prediction Algorithms 251
8.1 Training data preparation and model offline
Assessment 252
8.1.1 Training data preparation 252
8.1.2 Model offline evaluation 254
8.2 Commonly used forecasting models 261
8.2.1 Logistic regression model and machine learning
Foundation 262
8.2.2 Models supporting automatic feature discovery
Method 288
8.2.3 Deep Learning Models 294
8.3 CTR Estimation for New Ads 314
8.3.1 The Thompson sampling algorithm 315
8.3.2 Monte Carlo sampling 318
8.3.3 Markov Chain Monte Carlo
Sample 325
8.3.4 Gibbs sampling 330
8.3.5 The Laplace approximation 331
8.4 Advertisement Conversion Rate Estimation 334
8.5 Chapter Summary 336
