This article will share with you the development status, history and thoughts of Taobao IPv6.
introduction
In my personal impression, the mobile Internet started with the release of iPhone 3GS in China in 2009. It has been advancing rapidly for more than ten years, creating Taobao, an aircraft carrier-level APP. From the initial blindfolded running and brutal growth of "traffic" " is the king. In the current "reserve" era of intensive cultivation and experience priority, in the new stage, the "experience" proposition is closer to the C position than ever before. As various network protocols that carry services, their performance often determines the service experience. Since I started my career, the technical fields I have worked in are closely related to network protocols. Before joining Taobao, I was responsible for the optimization and implementation of protocols for cross-device media resource sharing and transmission at a mobile phone manufacturer. That is, everyone puts content on mobile phone players at home. When it comes to the scene on the TV screen, the protocols involved mainly include UPnP, Reliable UDP, Real-time UDP, etc. After joining Taobao, I continued to delve into gateway protocols, duplex long-connect message channels, Networking strategies and other closely related network protocols. Move PaaS domains, never down.
Taobao App’s changes in network protocols last year can be summed up as “ one drop and one rise . ” “One drop” refers to the offline SPDY protocol . Considering that the SPDY protocol is a transitional implementation of H2, it is currently in a state of abandonment, so last year We launched a special project to remove the SPDY protocol from Taobao, pushing the proportion of SPDY protocol in Taobao APP to converge to 95%+, and basically completed the removal of SPDY protocol from Taobao App; "One liter" is the increase in IPv6 protocol traffic, which was sensed through dual-stack single pass last year , composite connection racing, normalized operation construction and other technical upgrades have promoted the breakthrough of Taobao App IPv6 concentration index from excellent (80%) to excellent (90%), while ensuring the user experience in complex network environments.
The IPv6 protocol upgrade is the area where I have been working on Taobao’s network protocols for the longest time, have the largest impact, and think about it the most.
Taobao IPv6 latest achievements
The network is the water, electricity and coal of Internet applications, and the IP protocol is the lowest-level protocol of the network that our application layer faces in most cases. Therefore, we call the evolution from IPv4 to IPv6 "the upgrade of the cornerstone." This is a huge systematic project, especially for a company as large as Taobao, which involves a very large scope. This article returns to Taobao App, in one sentence To sum up, this year’s IPv6 work content is based on network library + AMDC product performance upgrade to support IPv6 experience upgrade . The result is “ precipitating a technical map, building an operation system, and achieving a set of target data .”
Precipitate a big technical picture : an end-to-end complete IPv6 solution based on Taobao terminal network library as the core:
Figure 1: Enhanced IPv6-inside Networking architecture
Facing the complexity of the mobile network environment, based on the IPv6 inside Networking architecture and technical capabilities enhanced by Taobao terminal network library, a complete end-to-end IPv6 solution is realized, ensuring the user experience under the large-scale application of Taobao IPv6, and exporting groups through mobile PaaS , supporting the rapid migration of a large number of second- and third-party apps, including network environment awareness, policy experience, integrated data products, and cloud scheduling services. At the same time, we continue to accumulate field data and lead industry standards.
Build an operation system : a normalized IPv6 operation system for compliance and app experience:
Figure 2: Normalized IPv6 operation system
Realize normalized IPv6 operation support, guarantee IPv6 high concentration water level and excellent experience, ensure that high IPv6 concentration will not fall back and user experience will not be degraded, and use the research and development process to avoid compliance risks and risks caused by daily business iteration and IPv6 orthogonality. Experiencing damage.
Achieve a set of target data : Current IPv6 large-scale deployment achievement indicators of Taobao App:
Figure 3: Taobao 2022 IPv6 large-scale application achievement indicators
Note: The time consumption in the table refers to the average time consumption of services relative to IPv4.
At present, Taobao IPv6 MAU has reached 1 billion+, with a maximum concentration of 95%+. The service request time is reduced by about 11.4% compared to IPv4, and the IPv6 network success rate reaches 3 nines. The concentration and experience improvement goals proposed at the beginning of FY23 were completed ahead of schedule. The main goals include: 1) FY23 IPv6 concentration monthly average of 80%+; 2) Achieving a leading (90%) breakthrough in concentration indicators during the year; 3) IPv6 experience indicators No degradation compared to IPv4. Through the mobile PaaS component, more than 200 apps within the group were exported.
Taobao IPv6 Networking Evolution
Since Apple’s IPv6-only review incident in 2016, Taobao’s IPv6 upgrade journey has been going on for six years. As one of the first people to start researching IPv6-related technologies at the group’s application layer, the author is fortunate to participate in the group’s IPv6 project. Group, and as the core owner of Taobao's IPv6 promotion, has accompanied the continuous evolution of Taobao's IPv6 technology through innovative iterations of terminal network technology, and has witnessed Taobao APP's IPv6 from zero users and zero traffic to now affecting 1 billion+ end users and covering For all business scenarios such as shopping guide/transaction/logistics/live broadcast, the concentration assessment is as high as 95%+, and it empowers the results of 200+ APPs across the group. This article will briefly review the technological evolution of Taobao IPv6 from the perspective of a Taobao terminal network developer. I divide Taobao’s IPv6 development in the past into five stages:
The first stage: Originated in 2016, solving the IPv6-only listing review problem in Apple App store;
Phase 2: Gathering momentum in 2017, the General Office of the CPC Central Committee released an action plan for large-scale deployment of IPv6 and began to evaluate the availability of IPv6 in domestic WANs;
The third stage: Development in 2018, the goal of IPv6 MAU reaching 100 million was proposed for the first time. By the beginning of 2019, Taobao APP MAU exceeded 100 million for the first time;
The fourth stage: deepening in 2020, starting from user scale to traffic proportion, focusing on increasing the actual application proportion of IPv6;
The fifth stage: Normalization 2022, the normal operation requirements of IPv6 under high concentration water levels, the IPv6 concentration of existing services must reach 85%, the full volume of incremental services must support IPv6, the concentration will not flow back, and the experience will not be degraded. The goal has been fully achieved and exceeded.
▐Origin 2016
Apple announced as early as WWDC2015 that it would support IPv6-only network services in iOS 9 and made it mandatory in 2016:
Apple officially issued a statement on May 4, 2016: Starting from June 1, 2016, the App Store policy requires that all iOS devices must include support for IPv6-only networks, and all application applications submitted to the Apple App Store must Compatible with IPv6-only networks.
The core of this problem is to solve how to determine on the end whether IPv6 supports single stack or dual stack. Packet detection will have a great impact on the user experience, so local quick judgment must be achieved. We achieved this through local UDP Binding. However, there is an error in the judgment of local UDP Binding, and the correction problem needs to be solved. We achieve local quick judgment through system native status notification, gateway address judgment and DNS address judgment. At the same time, after years of online practice, before this year’s Double Eleven, the DHCP-lag and dual-stack single-pass problems with dual-stack address allocation under Wi-Fi were gradually solved by improving multi-dimensional diagnostic capabilities.
Figure 4: Accurate identification of terminal local network stack type
The application of this technology makes Taobao APP the first App in the group to pass Apple's IPv6-only listing review, and it is exported to the whole group through mobile PaaS , and is accessed by multiple apps from Taobao/Tmall/Youku/Amap and other groups.
▐Building momentum for 2017
In 2017, the General Office of the CPC Central Committee and the General Office of the State Council jointly issued the "Action Plan for Promoting the Large-Scale Deployment of Internet Protocol Version 6 (IPv6)", officially kicking off the construction of IPv6 at the national level.
The "Action Plan" requires that it will take 5 to 10 years to form an independent technology system and industrial ecology for the next generation Internet, build the world's largest IPv6 commercial application network, realize the deep integration and application of the next generation Internet in various economic and social fields, and become a global next-generation Internet. An important leading force in the development of a generation of Internet.
As an application, we and operators face the old problem of "chicken and egg" in promoting IPv6. However, as a leading domestic Internet company, it is a policy requirement to unite ecological participants to face the problem head-on and promote the industry's ecological evolution. Our inner responsibility. From an application perspective, in the early stages of IPv6 deployment, unbalanced development of the basic network is the only way to go. Different paths and different operators have different IPv6 dual-stack coverage and continue to evolve and change. In addition, quality parameters such as network success rate and delay under different operators, different regions, and different network standards are variable and unstable. In order to ensure user experience, we must first solve the problem of measurable and observable pipeline quality, which is the basis for achieving high reliability at the application layer . In response to the above problems, we implemented large-scale active dial testing for pipelines.
Figure 5: Active quality dial test
On this basis, we established a panoramic IPv6 quality observation platform to obtain decision-making data basis to better support business scale in the future, so that the business layer can choose appropriate applications, regions, models, and users when IPv6 is ramping up. , equipment, etc. At the same time, we work with operators to quickly improve the basic quality of IPv6 in operator networks through large-scale application layer measurement feedback.
▐Development 2018
Our purpose of conducting dial tests and observations on operator networks is ultimately for service deployment. Theoretically, north-south terminals, pipes, and clouds support smooth evolution from IPv4 to IPv6, but relying solely on self-determination within the stack at each layer is still not enough. In extreme cases, it will be unavailable and the recovery cost will be high. We need a more efficient, more reliable, and more operable application layer network control plane . Therefore, we have refined the real-time IP scheduling system to realize the east-west solution and gradually realize the smooth evolution of the business. The system can provide fine control for different business scenarios, different device types and different network environments. For large-scale refined real-time scheduling, scheduling can be implemented based on refined dimensions such as device dimension, application dimension, user dimension, version dimension, and regional dimension.
Figure 6: Refined real-time IP scheduling completely decouples the east-west direction to achieve smooth system evolution.
Based on the above capabilities, we began to implement IPv6 large-scale transformation for the first time in 2018 for shopping guide & intelligent operation scenarios. In early 2019, we achieved Taobao APP IPv6 MAU exceeding 100 million for the first time. In the same year, we won the second prize of the National Communications Society Technology Progress Award and the internal 1 innovation and 1 patent .
▐Deepening 2020
After three years of IPv6 construction for user coverage, by the beginning of 2020, Taobao IPv6 MAU has reached 800 million+, but the application depth is not enough. At the same time, the Cyberspace Administration of China will start in 2020 to promote the IPv6 concentration of domestic leading application manufacturers. Taobao, as a key application, was selected as a model project for advancement. In 2021, the "Notice on Accelerating the Large-Scale Deployment and Application of Internet Protocol Version 6 (IPv6)" was officially issued.
The "Notice" makes it clear that we will unswervingly promote the large-scale deployment and application of IPv6, provide solid support for building a network power and digital China, and conduct a quantitative assessment of the IPv6 concentration of domestic leading applications.
IPv6 construction began to focus on increasing the proportion of actual traffic of IPv6 . This stage is mainly based on existing capabilities and continuing to expand the business scope of IPv6. During the evolution process, it will gradually and iteratively increase the priority of IPv6 and gradually increase the concentration of IPv6.
Figure 7: Taobao IPv6 strategy changes and large-scale application growth trends
The CDN core domain name has been upgraded to IPv6, and IPv6 support for static resource domain names such as images, js/css, etc. will gradually be enabled. By 2021, the Group's Taobao IPv6 traffic will account for >60%. And won the 2021 National IPv6 Scale Deployment and Application Excellent Case issued by the Ministry of Industry and Information Technology.
▐Normalization 2022
After five years of technological evolution, Taobao and the entire group have achieved north-south connectivity between terminals and clouds and built a large-scale dispatching system with east-west decoupling. On this basis, the IPv6 concentration used by Taobao has exceeded IPv4 under the cellular network, and IPv6 traffic has increased from Supplementation becomes mainstream. At the same time, the national competent authorities issued the "2022 Work Arrangement for Deeply Promoting IPv6 Scale Deployment and Application" in early 2022 to continue to vigorously promote the deployment of IPv6.
The "Work Arrangement" clarifies the work goals for 2022: by the end of 2022, the number of active IPv6 users will reach 700 million, and the proportion of IPv6 traffic on mobile networks will reach 45% (whether it is the number of active IPv6 or the proportion of IPv6 traffic, we have already reached this indicator . However, the Cyberspace Administration of China has separate assessment indicators for domestic leading Internet applications, which must not be lower than 80% during actual operation). The carrying capacity and service quality of network and application infrastructure continue to improve. IPv6 network performance indicators are equivalent to IPv4, and some indicators are better than IPv4.
When the proportion of IPv6 gradually exceeds that of IPv4, IPv6 under the new situation brings new pain points. For example, the high water level of IPv6 concentration monthly assessment indicators is a big challenge; the business is over-diversified and continues to iterate, and it is difficult to continue to ensure excellent experience; v4 stock There are many domain names, new domain names are not v6 by default, and complex domain names are not conducive to concentration improvement, etc. We summarize the pain points under the new situation into the following four categories of issues:
In summary, Taobao’s new proposition for IPv6 construction in 2022 is to build IPv6 technology and data capabilities in complex environments for compliance assurance and experience improvement, realize normalized IPv6 operations, and continue to lead IPv6 application innovation from excellence to leadership.
In response to the new propositions under the new situation, this year we will mainly carry out Taobao’s IPv6 construction through the following four directions:
As the terminal network library, it is the last line of defense to ensure user experience. Therefore, the IPv6 technological innovation of the terminal network library is the top priority of the entire Taobao IPv6 construction. This year, the core innovation in the network library is for complex wireless networks, realizing many The ability to connect multiple channels for racing:
Figure 8: Dual-stack racing and multi-channel optimization
In addition, we have improved the terminal network diagnosis capabilities under IPv6, added IPv6 environment/link detection for user-side diagnosis, and added CDN/IDC protocol checking for IPv6 dimensions on the R&D tool side:
Figure 9: Terminal network diagnostic capabilities supporting IPv6
In the new year, we have achieved the extreme concentration sprint of comprehensive IPv6. Through technological innovations such as dual-stack composite racing/multi-network card multi-channel, we have solved the experience problems in complex network environments such as Wi-Fi dual-stack single-pass. Achieved a breakthrough in IPv6 concentration from 60%+ to 95%+, and established an IPv6 normalization working mechanism.
Thoughts on the next generation of IPv6+ applications
So far, our upgrades on IPv6 have mainly focused on "large-scale deployment", and the essence is still seeking "quantitative change". During this process, we will continue to receive questions: What is the value of spending so much effort to upgrade IPv6? The country and the group continue to vigorously promote the upgrade of IPv6. For details on its value and significance, you can refer to the article "Alibaba IPv6 Large-Scale Deployment Sharing", and I will not go into details here. However, after receiving so many questions, it is obvious that the "value" we gain from IPv6 is not obvious to application developers or the majority of application developers. As an application developer or Internet application developer, those who have the strongest demands and experiences on the Internet will converge to two basic requirements:
Experience : How to be better? For the network, it means whether the performance parameters such as bandwidth, delay, jitter, and packet loss have been qualitatively improved;
Cost : How to be better? For the network, it means whether the cost of infrastructure investment such as bandwidth and network maintenance costs are significantly reduced without affecting the experience.
Currently, the IPv6 concentration ratio under cellular networks has reached 95%+ (under Wi-Fi networks, we will continue to follow up and invest in increasing the concentration). The large-scale deployment of IPv6 has reached a historic turning point. We must face the next generation of IPv6+ Technology changes from seeking "quantitative change" to seeking "qualitative change". This chapter is from the perspective of a terminal application network library developer, based on IPv6+ technology, the author thinks about the direction of our possible breakthroughs.
▐Deterministic network based on APN6
What is APN6
Internet applications have different requirements for network bandwidth, delay, jitter, packet loss rate, etc., and the decoupling of networks and applications results in the network being unable to effectively perceive the needs of applications, making it difficult to provide corresponding service quality SLA guarantees for applications. APN6, the application-aware network framework based on IPv6, stands for Application-aware IPv6 networking. By encapsulating application requirement information in data packets, the network can be aware of applications and their requirements, which facilitates network traffic scheduling and resource adjustment. The specific type is that the traditional IP (IPv4) header only contains the source address and destination address, and the network cannot identify the service type and service level requirements carried by the IP packet. APN6 uses the first 80 bits of the 128 bits of the IPv6 extensible header to mark the identity of the user, the identity of the App, and the level of service quality requirements, and uses the last 40 bits to define the bandwidth, jitter, and time of the channel for the business carried by the IP flow. Delay and packet loss rate requirements. The network can identify the QoS requirements of the IP flow on the channel based on the IPv6 address.
Figure 10: APN6 extension header
Application scenarios
In the author's opinion, the intergenerational development of computers is divided by the upgrade of human-computer interaction methods. The next generation of computers will be dominated by multi-modal + immersive interaction in scenes such as VR/AR and the game metaverse. In this In this interactive scenario, users have extremely high QoS requirements for the network, and a business application will have multiple channels of massive data input (video streams/massive sensor data/etc), but this does not mean that each channel of data input requires computing power. It has the same bandwidth, delay, and jitter requirements as the network. According to the definition of channel performance in the IPv6 extended address header, different channels can be selected for different services.
Figure 11: APN6 application scenario network topology
Summary of technical value
Network identification application, application-driven network: Traditional IP packet headers only contain basic information for routing and cannot identify the service type carried by IP packets. APN6 uses the application information carried in the IPv6 extension header to inform the network what kind of services it should provide. services, obtaining deterministic QoS guarantee based on uncertain networks;
Stateless business chain, accelerating business innovation: APN6 can achieve refined management of applications, accelerate the continuous innovation of new services, and save infrastructure investment such as bandwidth.
▐WAN multicast based on BIERv6
What is BIERv6
BIERv6 stands for Bit Index Explicit Replication IPv6 encapsulation. It is based on IPv6 bit index explicit replication multicast. It is a BIER multicast architecture implemented in Native IPv6 by using IPv6 extension headers, IPv6 address reachability and its programmable space. Provide better multicast deployment capabilities and the ability to expand support for subsequent Native IPv6 features. Its technical feature is to establish a BIER link state database through the entire network flood, and retain multicast status information at the entry and exit nodes of multicast traffic. Intermediate nodes do not perceive multicast flows, do not establish multicast forwarding trees, and do not maintain multicast forwarding. status to efficiently complete multicast packet distribution.
Figure 12: BIERv6 message format diagram
Application scenarios
As of December 2021, 930 million people in China watch short videos, 840 million people shop online, and 700 million people watch live broadcasts, accounting for 68.2% of the total netizens. Among them, the number of e-commerce live broadcast users is 460 million, accounting for 44.9% of the total Internet users. The overall number has increased by 75.79 million compared with December 2020, bringing an annual growth of 19.5%. With a population of 464 million and an annual growth of 19.5%, live streaming e-commerce has gradually become an increasingly "normal" and well-known trend - behind this is a market expected to exceed 3.5 trillion yuan in 2022 and the gradual Growing penetration of live streaming e-commerce. Bandwidth is the largest cost in live broadcast operations. According to estimates from Qianzhan.com, CDN expenses in the entire industry will exceed 30 billion yuan on December 11, 2022, and will be close to 100 billion yuan on December 11, 2022. Reducing bandwidth is an important part of cost control. A vital part. BIERv6 based on Native IPv6 makes it possible for us to implement cross-WAN multicast in live broadcast scenarios, significantly reducing CDN costs. As shown in the figure below, in a cross-regional multicast network, the head node arranges the multicast receivers in the form of bit strings and sends them out from the head node. The intermediate nodes send the data downward based on the address information in the message header. A node performs stateless forwarding. No additional multicast protocols and CDN services are required.
Figure 13: BIERv6-based multicast topology
Summary of technical value
Operator: Only configured on the head node, based on IPv6 encapsulation, it can easily traverse unsupported nodes and third-party networks to achieve cross-regional multicast services;
Internet applications: significantly reduce current CDN bandwidth costs and accelerate business innovation.
▐Multi -domain IPv6-only network response
China Telecom pointed out in the "Cloud Network Convergence 2030 Technical White Paper" that IP backbone networks will gradually adopt IPv6 single stack for routing and forwarding. Currently, the deployment of IPv6 is in the dual-stack stage, that is, IPv4 and IPv6 protocols run simultaneously on the online network. In the future, user terminal addressing will gradually be standardized and IPv6 addressing will be adopted. Operators will gradually cancel the allocation of IPv4 addresses starting from some scenarios. Therefore, we need to consider providing access to existing IPv4 services under the IPv6 single stack situation to ensure that the user experience is not affected.
Figure 14: Large-scale IPv6-only network scenario
In theory, entering the IPv6-only era, cross-domain (6 to 4 or 4 to 6) problems will be solved by the operator's network. However, the actual situation is not as simple as imagined. As mentioned in Section 3.3 above, Taobao is at the application layer. , has its own scheduling control plane, forming the application's own SDN network. The scheduling process will have three funnel links, resulting in the end-side obtaining an IPv4 address in the IPv6-only environment:
The end-side IP protocol stack type is incorrectly judged;
Scheduling system error, scheduling an IPv6-only terminal to an IPv4 address;
The scheduling system is configured incorrectly, or a certain service modification is omitted, resulting in a domain name having only IPv4 records in the scheduling system.
To sum up, terminal application App, as the last line of defense for business availability and user experience, needs to respond to this situation as much as possible. Effectively supporting the remaining IPv4 business bearers and ensuring that the user experience is not degraded is the key we must consider. question. Generally, an IPv6-only network will have a fixed synthetic prefix and a mapping rule. Different operators (or networks) have different prefixes and rules. The terminal needs to obtain these two parameters through pre-detection. , synthesize correctly accessible IPv6 mapping addresses to ensure the availability of remaining IPv4 services.
Summarize
Regarding IPv6 upgrades, looking back on the past, we have been driven by Networking technology breakthroughs such as dial testing/refined scheduling/multi-connection multi-channel racing, etc., and have supported IPv6 end-user experience upgrades around network library + AMDC product performance upgrades, leading mobile Taobao We have reached an important node in the "quantitative change" of IPv6 applications - the IPv6 traffic of Taobao App under cellular networks has basically increased. Based on the current situation, due to objective factors such as terminal equipment/home broadband routing, domestic IPv6 traffic has not yet been fully rolled out in Wi-Fi networks. The proportion of IPv6 traffic in Wi-Fi (14%) is far lower than that of cellular networks (80%+ ), we will continue to focus on the construction of IPv6 application quality standards, continue to promote the breakthrough of traffic bottlenecks under Wi-Fi, and complete the last step of IPv6's "quantitative change". Looking to the future, we will focus on IP6+ technology and think about the path to "qualitative change" in IPv6 applications. Although IPv6+ is still in the capacity building period, and some technical points are even in the standard construction period, we need to think and design in advance to cope with the new generation of VR/ Network requirements for high throughput and low latency in new business scenarios such as AR/live broadcast; at the same time, based on the fact that most of the current SRv6-based IPv6+ technologies are oriented to operator networks, terminal networks can try to quickly enable them through MobileSDN+ multi-connection and other methods In-App network slicing, etc., ultimately achieve network demands such as network capacity elasticity (elasticity), high reliability and low damage network experience (efficiency), differentiated SLAs for application network guarantees (perception), and re-optimization of network resource occupation costs (economy).
As the capability base of the entire mobile Taobao App, the terminal network’s performance directly affects the experience of almost all businesses on Taobao, and is also the last line of defense for terminal App user experience. IPv6 is the starting point and platform for the next generation of Internet innovation. Taobao's terminal network will be based on terminals and oriented towards cloud-network integration, connecting every user, every merchant, and every product.
team introduction
We are the terminal platform network technology team of Taotian Group, responsible for the construction of Taobao network technology/high-performance gateway technology, including but not limited to Tengine-Ingress high-performance gateway/XQUIC standardized protocol library/Taobao terminal network library, supporting mobile networks with hundreds of millions of traffic Access and gateway services. The team continues to iterate open source technology masterpieces such as XQUIC/Tengine-Ingress, publish many papers at top network academic conferences such as SIGCOMM/NSDI, and promote the MPQUIC RFC standard at the network standards organization IETF.
If you are interested in our work content, welcome to join the challenge and send your resume to [email protected].
¤Expand reading¤ _
3DXR technology | terminal technology | audio and video technology