In recent years, the amount of model calculations and parameters in applications such as video, image, text, and promotional search in the field of machine learning have far exceeded the growth rate of CPU Moore's Law. In this context, the development of GPU computing power coincides with the development of large models. Many companies are combining the development of GPU computing power to explore solutions to machine learning problems that are suitable for them. For example, Xiaohongshu will begin GPU-based transformation of the promotion search model in 2021 to improve inference performance and efficiency. During the migration process, we also faced some difficulties, such as how to smoothly migrate to heterogeneous hardware, how to develop our own solutions based on Xiaohongshu's business scenarios and online architecture, etc. Under the global trend of cost reduction and efficiency improvement, heterogeneous computing has become a promising direction, which can improve computing performance by combining different types of processors (such as CPU, GPU, FPGA, etc.) to achieve Better efficiency and lower costs.
Model services for major scenarios such as Xiaohongshu recommendation, advertising, and search are uniformly carried by the middle-end inference architecture. With the continuous development of Xiaohongshu's business, the scale of models for scenarios such as promotional search is also increasing. Taking the main model of refined recommendation scenarios as an example, since the beginning of 2020, the algorithm has launched full-interest modeling, and the average length of user historical behavior records has expanded by about 100 times. The model structure has also gone through multiple rounds of iterations from the initial muti-task, and the complexity of the model structure has also continued to increase. These changes have resulted in a 30-fold increase in the number of floating-point operations for model inference and an approximately 5-fold increase in model memory access.
Model features: Take Xiaohongshu’s recommendation main model at the end of 2022 as an example. This model has sufficient sparsity. Part of the structure is composed of continuous value features and matrix operations. There are also large-scale sparse parameters such as <note id * user city> , the sparse features of a single model are up to 1TB, but through relatively effective model structure optimization, the dense part is controlled within 10GB and can be placed in the video memory. Every time the user swipes Xiaohongshu, the total FLOPs calculated reach 40B, and the timeout is controlled within 300ms (excluding feature processing, with lookup).
Inference framework: Before 2020, Xiaohongshu adopted the TensorFlow Serving framework as the online service framework. After 2020, it gradually iterated into a self-developed Lambda Service based on TensorFlowCore. TensorFlow Serving performs a memory copy of TensorProto -> CTensor before entering the graph to ensure the correctness and reliability of model inference. However, as the business scale expands, memory copy operations will have an impact on model performance. Xiaohongshu's self-developed framework eliminates unnecessary copying through optimization, while retaining the pluggable features of runtime, graph scheduling capabilities, and optimization capabilities, and lays the foundation for the later use of different optimization frameworks such as TRT, BLADE, and TVM. . It now seems that choosing self-research at the right time is a wise choice. At the same time, in order to minimize the cost of data transmission, the inference framework also undertakes part of the implementation of feature extraction and transformation. Here Xiaohongshu is still estimating Self-developed edge storage is deployed on the near side of the service, which solves the cost problem of pulling data remotely.
Model characteristics: Xiaohongshu does not have its own computer room, and all machines are purchased from cloud vendors. Therefore, the decision to choose different models depends largely on what types of machines can be purchased. The calculation of model inference is not purely GPU calculation. To find a reasonable hardware ratio, in addition to considering GPU\CPU, it also involves bandwidth, memory bandwidth, cross-numa communication delay and other issues.
GPU features
GPU characteristics: Here, Xiaohongshu and other companies encounter the same problems. The execution of the GPU kernel can be divided into the following stages: data transmission, kernel startup, kernel calculation and result transmission. Among them, data transmission is to transfer data from the host memory to the GPU memory; kernel startup is to transfer the kernel code from the host side to the GPU side, and start the kernel on the GPU; kernel calculation is to actually execute the kernel code calculation result; result transmission is to Computational results are transferred from GPU memory back to host memory. If a large amount of time is spent on data transmission and kernel startup, and the work delivered to the kernel calculation is not heavy, and the actual calculation time is very short, the GPU utilization will not be improved, and even empty running will occur.
Estimated service framework
3.1 System optimization
3.1.1 Physical machine
In terms of physical machine optimization, some conventional optimization ideas can be adopted. The main purpose is to reduce the cost of other system overheads other than the GPU and reduce the price difference earned by virtualization middlemen. Generally speaking, a set of system optimization can improve performance by 1%-2%. From our practice, optimization needs to be combined with the actual capabilities of cloud vendors.
● Interrupt isolation: Isolate GPU interrupts to avoid interrupts from other devices affecting GPU computing performance.
● Kernel version upgrade: Improve system stability and security, and improve GPU driver compatibility and performance.
● Instruction transparent transmission: Transparently transmit GPU instructions directly to the physical device to accelerate the computing speed of the GPU.
3.1.2 Virtualization and Containers
In the case of multiple cards, a single pod is bound to a specific NUMA node to increase the speed of data transfer between the CPU and GPU.
● CPU NUMA Affinity, affinity refers to which memory accesses are faster and have lower latency from the CPU perspective. As mentioned before, local memory directly connected to the CPU is faster. Therefore, the operating system can allocate local memory according to the CPU where the task is located to improve access speed and performance. This is based on the consideration of CPU NUMA Affinity, and try to run the task in the local NUMA Node. In the Xiaohongshu scenario, the memory access overhead on the CPU is not small. Allowing the CPU to directly connect to local memory can save a lot of time spent on kernel execution on the CPU, thereby leaving enough space for the GPU.
● Controlling the CPU usage at 70% can reduce the delay from 200ms -> 150ms.
3.1.3 Mirroring
Compilation optimization. Different CPUs have different support capabilities for instruction levels, and the models purchased by different cloud vendors are also different. A relatively simple idea is to compile the image with different instruction sets in different hardware scenarios. When implementing operators, a large number of operators already have instructions such as AVX512. Taking Alibaba Cloud's Intel(R) Xeon(R) Platinum 8163 + 2 A10 model as an example, we compile, optimize and adjust the appropriate instruction set based on the characteristics of the model and the supported instruction set. Overall, compared to not performing With instruction optimization, the CPU throughput on this model is increased by 10%.
# Intel(R) Xeon(R) Platinum 8163 for ali intel
build:intel --copt=-march=skylake-avx512 --copt=-mmmx --copt=-mno-3dnow --copt=-msse
build:intel --copt=-msse2 --copt=-msse3 --copt=-mssse3 --copt=-mno-sse4a --copt=-mcx16
build:intel --copt=-msahf --copt=-mmovbe --copt=-maes --copt=-mno-sha --copt=-mpclmul
build:intel --copt=-mpopcnt --copt=-mabm --copt=-mno-lwp --copt=-mfma --copt=-mno-fma4
build:intel --copt=-mno-xop --copt=-mbmi --copt=-mno-sgx --copt=-mbmi2 --copt=-mno-pconfig
build:intel --copt=-mno-wbnoinvd --copt=-mno-tbm --copt=-mavx --copt=-mavx2 --copt=-msse4.2
build:intel --copt=-msse4.1 --copt=-mlzcnt --copt=-mrtm --copt=-mhle --copt=-mrdrnd --copt=-mf16c
build:intel --copt=-mfsgsbase --copt=-mrdseed --copt=-mprfchw --copt=-madx --copt=-mfxsr
build:intel --copt=-mxsave --copt=-mxsaveopt --copt=-mavx512f --copt=-mno-avx512er
build:intel --copt=-mavx512cd --copt=-mno-avx512pf --copt=-mno-prefetchwt1
build:intel --copt=-mno-clflushopt --copt=-mxsavec --copt=-mxsaves
build:intel --copt=-mavx512dq --copt=-mavx512bw --copt=-mavx512vl --copt=-mno-avx512ifma
build:intel --copt=-mno-avx512vbmi --copt=-mno-avx5124fmaps --copt=-mno-avx5124vnniw
build:intel --copt=-mno-clwb --copt=-mno-mwaitx --copt=-mno-clzero --copt=-mno-pku
build:intel --copt=-mno-rdpid --copt=-mno-gfni --copt=-mno-shstk --copt=-mno-avx512vbmi2
build:intel --copt=-mavx512vnni --copt=-mno-vaes --copt=-mno-vpclmulqdq --copt=-mno-avx512bitalg
build:intel --copt=-mno-movdiri --copt=-mno-movdir64b --copt=-mtune=skylake-avx512
3.2 Computational optimization
3.2.1 Make full use of computing power
● For computing optimization, you first need to fully understand the hardware performance and understand it thoroughly. In the Xiaohongshu scenario, as shown in the figure below, we encountered two core problems:
1. There are many memory accesses on the CPU and the frequency of memory page faults is high, resulting in a waste of CPU resources and high request latency.
2. In online inference services, calculations usually have two characteristics: the batch size of a single request is small, and the concurrency scale of a single service is large. A small batch size will cause the kernel to be unable to fully utilize the computing power of the GPU. The execution time of GPU kernel is generally short, which cannot fully cover the overhead of kernel launch, and even the time of kernel launch is longer than the kernel execution time. In TensorFlow, a single Cuda Stream launch kernel becomes a bottleneck, resulting in only 50% GPU utilization in inference scenarios. In addition, for small model scenarios (simple dense networks), it is not cost-effective to replace the CPU with a GPU, which limits the complexity of the model.
● To solve the above two problems, we have taken the following measures:
1. To solve the problem of high memory page fault frequency, we use the jemalloc library to optimize the memory recycling mechanism and enable the transparent large page function of the operating system. In addition, for the special memory access characteristics of lambda, we design a special data structure and optimize the memory allocation strategy to avoid memory fragmentation as much as possible. At the same time, we directly bypassed the tf_serving interface and directly called TensorFlow, reducing the serialization and deserialization of data. These optimizations have increased throughput by 10+% in homepage and in-stream fine-tuning scenarios, and reduced latency by 50% in most advertising scenarios.
Compatible with tensorflow::Tensor format, it is a zero copy before passing features to tensorflow::SessionRun
2. In response to the problem of TensorFlow's single Cuda Stream, we supported the functions of Multi Streams and Multi Contexts, avoiding the performance bottleneck caused by mutex locks, and successfully increased the GPU utilization to 90+%. At the same time, we use the Cuda MPS function provided by Nvidia to realize spatial division multiplexing of the GPU (supporting multiple kernel executions at the same time), further improving the utilization of the GPU. Based on this, Search's sorting model was successfully implemented on GPU. In addition, we have also successfully implemented it in other business lines, including home page layout, advertising, etc. The following table is an optimization situation in the search ranking scenario.
3. Op/Kernel fusion technology: Generate Tensorflow operators with higher performance through handwriting or graph compilation and optimization tools, making full use of the CPU's Cache and GPU's Shared Memory to improve system throughput.
In the internal flow scenario, the operators are fused, and you can see that a single call is 12ms -> 5ms
3.2.2 Avoid wasting computing power
1. There is room for optimization on the system link
a. Preliminary calculation: When processing related calculations on the user side, a large number of notes need to be calculated for preliminary sorting. For example, taking outflow as an example, about 5,000 notes need to be calculated, and lambda has slicing processing for them. In order to avoid repeated calculations, the user-side calculations of the initial row are moved in parallel with the recall phase, so that the calculation of the user vector is reduced from multiple repetitions to only one time, 40% of the machines are optimized in the rough row scenario.
2. In-graph training to inference process:
a. Calculation pre-processing: Part of the calculation can be processed in advance through graph freeze. When reasoning, there is no need to repeat calculations.
b. Output model freeze optimization: When the model is output, all parameters are generated together with the graph itself to generate a frozen graph (frozen graph) and perform preprocessing calculations. Many precomputed Variable operators can be converted into Const operators (GPU uses rate dropped by 12%)
c. Merged calculations in inference scenarios: Each batch contains only one user, that is, there are a large number of repeated calculations on the user side, and there is the possibility of merging.
d. CPU/GPU operator split: move all operators after lookup to GPU to avoid data copying between CPU and GPU
e. GPU to CPU data copy: package the data and copy it once
f. BilinearNet operator GPU cuda implementation: Accelerate calculation through GPU to improve performance
g. Partial operators are GPU-based: omitting CPU -> GPU copy
h. BatchNorm & MLP merger: By implementing a new MLP layer, according to a goal, reduce the number of times entering the GPU (N -> 1), increase the amount of calculation for one calculation (reuse the concurrency capability of the GPU small core)
3.2.3 Dynamic computing power throughout the day
● Dynamic calculation of downgrade improves resource usage efficiency throughout the day. Automatic negative feedback adjustment of lambda load is performed at the second level, eliminating the need for manual downgrade preparation before stress testing a single zone.
● Major business scenarios such as outbound refined sorting, outbound preliminary sorting, inbound refined sorting, internal inflow preliminary sorting, and search have all been launched.
● Solved the capacity problem in multiple business lines, effectively alleviated the linear increase in resources caused by business growth, and greatly improved the robustness of the system. In the business lines after the function was launched, there were no P3 or above accidents caused by a sharp drop in the instantaneous success rate.
● Significantly improve the efficiency of resource usage throughout the day. Taking instream fine-tuning as an example (as shown in the figure below), the number of CPU cores used during the three-day May Day holiday from 10:00 to 24:00 maintains a flat line of 50 cores (jitter) Corresponding to the release version)
3.2.4 Change to better hardware
● The performance of the A10 GPU is 1.5 times that of the T4 GPU. At the same time, the A10 model is equipped with a newer generation CPU (icelake, 10nm) than the T4 model (skylake, 14nm), and the price is only 1.2 times that of the T4 model. In the future, we will also consider using A30 and other models online.
3.3 Graph optimization
3.3.1 Automatic compilation optimization of DL stack
● BladeDISC is Alibaba’s latest open source dynamic shape deep learning compiler based on MLIR. Xiaohongshu’s automatic graph optimization part comes from this framework (Blade inference acceleration library is Apache 2.0 open source and can be used across any cloud, with no intellectual property risk ). This framework provides TF graph compilation optimization (including Dynamic Shape Compiler, sparse subgraph optimization), and can also superimpose our own operator customization optimization, which can better adapt to our business scenarios. In the stress test single-machine inference, QPS can be increased by 20%.
● Key technologies of this framework
(1) MLIR infrastructure
MLIR, or Multi-Level Intermediate Representation, is an open source project initiated by Google. Its purpose is to provide a flexible, extensible multi-layer IR infrastructure and compiler utility library, providing a unified framework for compiler and language tool developers.
The design of MLIR is influenced by LLVM, but unlike LLVM, MLIR mainly focuses on the design and extension of intermediate representation (IR). MLIR provides a multi-level IR design that can support the compilation process from high-level languages to low-level hardware, and provides rich infrastructure support and modular design architecture, allowing developers to easily expand the functions of MLIR. In addition, MLIR also has strong glue capabilities and can be integrated with different programming languages and tools. MLIR is a powerful compiler infrastructure and tool library that provides developers of compilers and language tools with a unified and flexible intermediate representation language that can facilitate compilation optimization and code generation.
(2) Dynamic shape compilation
The limitation of static shapes means that the shape of each input and output needs to be determined in advance when writing a deep learning model, and they cannot be changed at runtime. This limits the flexibility and scalability of deep learning models, so a deep learning compiler that supports dynamic shapes is needed.
3.3.2 Accuracy adjustment
● One of the ways to achieve quantization is to use FP16
FP16 calculation optimization: Replacing FP32 calculations with FP16 in the MLP layer can greatly reduce GPU usage (relative reduction of 13%)
In the process of adjusting FP16, choosing the white box method for precision optimization means that we can more finely control which layers use low-precision calculations, and can continuously adjust and optimize based on experience. This method requires a relatively in-depth understanding and analysis of the model structure, and targeted adjustments can be made according to the characteristics and calculation requirements of the model to achieve a higher cost performance.
In contrast, the black-box method is relatively simple and does not require understanding the internal structure of the model. It only needs to set a certain tolerance threshold to complete accuracy optimization. The advantage of this method is that it is simple to operate and has relatively low requirements on model students, but it may sacrifice certain performance and accuracy.
Therefore, whether to choose white box or black box method for accuracy optimization needs to be determined according to the specific situation. If you need to pursue higher performance and accuracy, and have sufficient experience and technical capabilities, then the white-box approach may be more suitable. If simplicity of operation and rapid iteration are more important, then the black box approach may be more practical.
From the beginning of 2021 to the end of 2022, through the optimization of this project, Xiaohongshu's inference computing power will increase by 30 times, key user indicators will increase by 10%+, and the cumulative cluster resources will be saved by 50%+. In our opinion, Xiaohongshu’s development path in AI technology should be oriented by business needs and balance the development of technology and business: while achieving technological innovation, cost, efficiency and sustainability must also be considered. The following are some thoughts during the optimization process:
Optimize algorithms and improve system performance. This is the core mission of Xiaohongshu’s machine learning team. Optimizing algorithms and improving systematization can better support business needs and improve user experience. However, when resources are limited, the team needs to clarify the focus of optimization and avoid over-optimization.
Build infrastructure and improve data processing capabilities. Infrastructure is critical to supporting AI applications. Xiaohongshu can consider further investing in infrastructure construction, including computing and storage capabilities, data centers and network architecture. In addition, it is also very important to improve data processing capabilities to better support machine learning and data science applications.
Improve team talent density and organizational structure. An excellent machine learning team needs talents with different skills and backgrounds, including data scientists, algorithm engineers, software engineers, etc.; optimizing the organizational structure can also help improve team efficiency and innovation capabilities.
Win-win cooperation and open innovation. Xiaohongshu continues to cooperate with other companies, academic institutions and open source communities to jointly promote the development of AI technology, which helps Xiaohongshu obtain more resources and knowledge and become a more open and innovative organization.
This solution brings Xiaohongshu’s machine learning architecture to the top level in the industry. In the future, we will continue to promote engine upgrades, reduce costs and increase efficiency, introduce new technologies to improve the productivity of Xiaohongshu's machine learning, and further integrate Xiaohongshu's actual business scenarios, upgrading from single-module optimization to full-system optimization, and further Introduce the personalized differential characteristics of business-side traffic to achieve the ultimate cost reduction and efficiency increase. We are looking forward to people with lofty ideals to join us!
Zhang Chulan (Du Zeyu) : Business Technology Department
Graduated from East China Normal University, head of the commercialization engine team, mainly responsible for building commercialized online services.
Lu Guang (Peng Peng) : Intelligent Distribution Department
Graduated from Shanghai Jiao Tong University, machine learning engine engineer, mainly responsible for Lambda GPU optimization.
Ian (Chen Jianxin) : Intelligent Distribution Department
Graduated from Beijing University of Posts and Telecommunications, machine learning engine engineer, mainly responsible for Lambda parameter server and GPU optimization.
Akabane (Liu Zhaoyu) : Intelligent Distribution Department
Graduated from Tsinghua University and is a machine learning engine engineer. He is mainly responsible for related research and exploration in the direction of feature engines.
Special thanks to: All students in the Intelligent Distribution Department