In early 2017, the cold and windy outside the airport, Huawei OceanStor Dorado chief architect Zhang Peng is about one person alone went overseas. At the moment, he conceal his excitement, formal adoption OceanStor Dorado V6 all-flash project made him very excited; and excitement, Zhang Peng heart there is a trace of apprehension, because his team entrusted with Huawei to high-end storage memory Fengyun crown Pearl of the assault was launched on the task.
July 2019, Huawei has officially launched its new generation OceanStor Dorado V6 all-flash storage. As a lasted nearly three years to build high-end storage products from Huawei's global R & D team store, OceanStor Dorado V6 has a stable delay the industry's highest 20 million IOPS extreme performance, the industry's lowest 0.1ms, the groundbreaking high-end storage architecture enables SmartMatrix stability and reliability to a new level again, then let the chips into the AI OceanStor Dorado V6 leading the development trend of high-end storage intelligence.
Recalling the past two years many, Peng laments: "OceanStor Dorado V6 implements a number of groundbreaking stuff, which is very rare in the industry, I also feel pleased with that."
Decade of sword
Flash memory can be described as the largest field of technological innovation in recent years storage. Among them, all-flash storage represents an important direction of development of the market. Especially with the flash memory costs continue to decrease in recent years, as well as rising capacity flash memory to replace the traditional mechanical hard turning point has come, all-flash storage has become the commanding heights of the market competition.
"In fact, Huawei is stored in the all-flash storage areas has been more than ten years of accumulation. As early as 2009, Huawei is one of the industry's first to launch all-flash array vendors." Peng expressed said, "After a decade of accumulation and temper, we hope to seize the unprecedented opportunity to appear in the market today. "
The so-called unprecedented opportunity that entered the all-flash storage universal coverage period, mechanical hard disk storage system will accelerate the exit, but based on traditional storage architectures on the market of all-flash storage products market was leaving a huge space for innovation. Peng said: "On the occasion of the market and technological changes, the first to have a breakthrough product that will lay the absolute competitiveness of the industry."
Because of this, Huawei storage hopes to achieve a qualitative breakthrough in the OceanStor Dorado V6 generation products, has laid a solid foundation for their own high-end storage market. In Peng seems, a decade sword is a continuous process of accumulation, the industry needs to continue to attract more top talent to join. "A few years ago, to go overseas for the OceanStor Dorado V6 set up overseas R & D team, unfamiliar, just started really compare thrown off balance." However, Peng did not give up, one by one call to establish contact with the industry's top talent, to maintain communication and gradually gained confidence.
"Huawei's rapid growth in recent years storage, has occupied a certain position in the market. But at the time to convince top talent to store Huawei is still a difficult thing. The reason why ultimately successful, mainly Huawei storage provider stage and what hope did it, this is the main reason for Huawei to attract the industry's top talent to join. "
In fact, over the years, high-end storage market has long been held by the three major international giant, high-end storage very high technical threshold one hand to discourage competitors, on the other hand are to some extent tied the innovation and change their own products. "Huawei is stored want to achieve something, it is precisely these people want to realize the dream. It can be said, OceanStor Dorado V6 is a combination of the world's top talent wisdom of storage products, with a high ground-breaking."
Break the shackles of traditional architecture
In the era of the traditional mechanical disk, high-end storage architecture has been relatively stable, with Scale-Up longitudinal extension based. However, with the rapid growth in recent years the amount of data, and business applications passionate about performance, making Scale-Out scale high-end storage products became popular. However, Scale-Out architecture may help to expand, and have lost the traditional vertical architecture advantages in terms of performance and efficiency.
How to break the shackles of traditional architecture, has become the biggest challenge placed in front of Huawei's storage team. To this end, the OceanStor Dorado V6 start of a project, Huawei's storage team to determine the direction to overcome: the Scale-Up and Scale-Out fusion, to design a new architecture combines the advantages of both, this goal inspired the team members of the great power. "We have a number of overseas R & D team members from the market has long been famous companies, they challenge the new architecture can be described as motivated." As we all know, high-end storage architectures years move forward, to a large extent that historical burden is too heavy, the cost of innovation and the risk is too high. "It turns out that some of the overseas team members played an important role in product architecture, their years of innovative ideas in this generation can be achieved."
In the process of steady progress in architecture design, Huawei storage teams encountered a series of enormous challenges. Since architecture is a new design, the current market, many components of the product does not meet specific design requirements. After "After such components as interface cards, when we make specific product requirements to suppliers, after selecting the supplier tried to give up, because it is too difficult." This situation is similar to occur several times, so that Huawei's R & D team into storage among the confusion, but also stalled progress in the development stage.
"Why not try Hass?" At this point, it was suggested that the idea of using Hass chip.
In fact, Huawei Hass chip in a few years ago has entered the low-end product among Huawei storage, and played an important role. But Hass chip can Go On demanding requirements of high-end storage systems it? Huawei's storage team after several days of discussions, decided to look for other ways to incorporate Hass series chip architecture designs, to seek complete breakthrough architecture.
Hass chip verification and test series in OceanStor Dorado V6 architecture it is also a time, heavy task of work. As it relates to include five chip controller chip, SSD controller chip, AI chip, protocol processing chips and stable operation in OceanStor Dorado V6 architecture, Huawei's storage research and development team needs time to communicate closely with the needs of Hass chip team, to complete the adaptation chip architecture. "Team has a colleague, he was more than two years just focus on doing one thing, that is dedicated to specific communication needs and Hass chip team, from the demonstration, testing, adaptation, tuning and stable operation after that, this colleagues played an important role. "Zhang Peng introduced to.
自此，经过华为团队的不懈努力，OceanStor Dorado V6的SmartMatrix架构得以最终实现了高端存储开创性的计算和存储分离、前后端全互联架构，将计算型的存储控制器引擎和存储型的硬盘框完全分离，二者可以进行独立升级和扩展，这种设计架构具备良好的延续性和灵活性，可以很好地保护用户投资；此外，OceanStor Dorado V6实现了单系统最高可扩展到32控制器，控制器可以实现8坏7，实现了高达99.9999%的高可用性。
回顾整个架构从设计到实现的过程，张鹏感叹道：“正所谓是好事多磨。如果按照过去思路和友商的芯片产品，我们可以施展的事情就会被束缚住，这样研发出来的产品不会具有绝对竞争力，OceanStor Dorado V6的架构虽然经历了波折与坎坷，但是以当前数据增长和业务变化趋势来看，高端存储的架构必然需要变化，OceanStor Dorado V6率先走出了一条具有开创性的路，在市场中无疑是具有领先性的。”
极致性能如何炼成
华为OceanStor Dorado V6全闪存存储最高可达到2000万IOPS的性能，并且可以实现0.1ms的稳定时延。IOPS越高，意味着性能越强；而时延越低，则意味着性能越稳定。稳定和时延是一个螺旋上升的过程，中间需要反复的打磨与优化。
尤其是在架构设计获得稳步进展时，研发团队也开始对软件层面进行了优化，由于采用了鲲鹏920处理芯片，每个CPU都有48个核心，需要在软件架构层面对多CPU多核、高速网络进行优化。“最开始是为了实现2000万IOPS的性能。当实现之后，因为系统存在各种潜在的中断因素，使得IO经常会被打断，需要逐个梳理出其中的原因，并且进行修改和测试，这是个反复优化的过程。”张鹏介绍到。
“由于存储系统的处理器主要是做数据相关的处理，所以像鲲鹏920这样的多核ARM处理器反而更具优势，它可以专门划出资源来做像重删和压缩这些数据处理工作，而像通用X86处理器更加擅长的是运算类应用。”根据华为介绍，OceanStor Dorado V6同时打开重删和压缩这些功能，其性能可以超越同等高端存储产品50%。
在性能提升过程中，客户也发挥了意想不到的作用。“有时候，客户其实是最好的产品经理。OceanStor Dorado V6的出色和稳定的性能表现，离不开用户宝贵的建议。”张鹏感慨道。
曾经有这样一件记忆深刻的小事：一次恰逢欧洲某×××来华为拜访，于是张鹏团队向该客户展示了OceanStor Dorado V6样机，并演示了各项强大的存储功能。“当时向客户展示的时候，非常高兴，甚至有一点洋洋得意。”当展示结束之后，客户也对OceanStor Dorado V6的产品设计和强大功能所折服，不过客户也提出，如何解决万分之一概率的IO时延偏大问题，这是客户一直期待能够解决的难题。
张鹏直言，该客户所提出的建议促使了内部进行了激烈的讨论。有人认为解决这个小概率IO时延偏大问题，需要对之前原有系统设计进行改动，这将花费巨大的精力和投入。但是，经过多次讨论和研究之后，团队还是决定下决心解决潜在小概率IO时延偏大问题。张鹏表示：“解决这种长尾IO时延偏大的问题，的确会对原有设计产生一些冲击，需要进行任务的隔离和分开，会涉及到硬件、软件等多个层面。”
最终，华为存储团队经过多次的讨论和测试，重新对系统设计进行了修改，并且在硬件驱动、操作系统、软件等层面进行了反复的打磨与改进，长尾IO时延偏大的问题得以完美解决，OceanStor Dorado V6得以实现0.1ms的稳定时延。
“永远”有多远
高端存储之所以被誉为存储皇冠上的明珠，在于其拥有出色性能之外，也具备了极致的可靠性和稳定性。这些特性使得高端存储在众多行业中承载最为核心的关键业务。
“最近几年与客户频繁的接触，明显感觉到客户业务对于可靠性需求的提升。”张鹏介绍到，近年来华为存储团队经过与大量不同行业的用户接触，普遍对数据中心设备的多点故障感到焦虑，“随着数字化的步伐较快，很多用户的数据中心规模越来越大，设备也越来越多，多点故障成为用户未来不可逃避的挑战。”
华为OceanStor Dorado V6在可靠性和稳定性上可谓是下足了功夫，从部件级、产品级、方案级和云级四个层面打造出端到端的可靠性架构，可以承载全整合场景所需，保障业务高达99.9999%的高可用性，为高端存储的可靠性树立了新标杆。
首先，华为在OceanStor Dorado V6的闪存盘上采用全局磨损均衡技术，将业务负载均衡到所有SSD上，并且采用华为专利的反磨损均衡技术，避免多盘集体失效，在部件级构建了极高的可靠性；此外，OceanStor Dorado V6的SmartMatrix架构采用前后端全联接设计和智能多协议接口芯片，采用全对称的A-A控制器设计，LUN可以通过任意一个控制器访问应用服务器，当控制器故障出现之后，一秒就完成故障控制器的正常切换，并且可以实现控制器8坏7的极端情况。
张鹏直言：“数据中心规模变大和设备增多、客户潜在的误操作、以及内部软件升级等是造成多点故障的主要原因。像很多行业用户数据中心的软件极为复杂，软件失效率根本就算不出来，这不是物理失效，而需要做软硬件的隔离，保障软件的快速恢复。可以说，那些用双控节点去堆叠出来的高端存储产品，很容易出现规模越大、风险越高的情况。”
此外，在华为存储团队的努力下，OceanStor Dorado V6采用了备份容灾一体化设计，具备免网关的双活方案，减少了故障节点，降低了系统布置的复杂度；并且与公有云可以进行联动，实现备份容灾上云，云内分钟级业务恢复。
“OceanStor Dorado V6可以实现故障0感知、业务0影响、升级0影响，真正保障了业务永久在线。”张鹏自豪道。
智能赋予高端存储新生机
高端存储产品看似暮霭沉沉，其实却蕴含新生机。人工智能技术的崛起，赋予了高端存储新的生机。高端存储产品走向智能化成为必然的趋势。
“要让一个存在几十年历史的产品焕发活力，必须思考加入一些创新性的技术。”张鹏如是说。为此，华为在OceanStor Dorado V6中加入了大量的智能技术，“华为在AI算法方面投入很大。”比如，OceanStor Dorado V6的智能多协议接口芯片，可以承载通用CPU负责协议解析工作，智能完成协议的解析；而AI芯片则基于机器学习框架，主动分析并掌握多个应用模型的IO规律，让读缓存命中率提升50%。
“OceanStor Dorado V6这种全互联、全共享的架构，非常适合采用机器学习这些人工智能技术，对IO进行全局的学习和分析，实现存储操作更加智能化，从而提升系统的性能和效率。”张鹏补充道。
此外，OceanStor Dorado V6还基于AI芯片和算法实现了全生命周期的智能运维，包括资源规划、业务发放、系统调优、风险预测、故障定位等实现了全方位智能管理，使得性能容量趋势可以提前60天预判，提前14天发现故障盘、93%的问题可以即时给出方案。
“OceanStor Dorado V6整个系统架构就是智能化设计的，控制器和硬盘柜可以分别独立升级，确保10年内数据无需迁移。”张鹏介绍到。
写在最后
张鹏直言，OceanStor Dorado V6项目让他和他的团队在过去两年承受了巨大的压力，项目技术难度大、规模大，全新的硬件设计、全新的软件和操作系统，全都需要重新设计和实现，过去两年几乎每天都需要加班加点。“虽然很累，但是团队还是希望不断挑战自己，一步一步实现了OceanStor Dorado从设计到交付。”
与此同时，华为存储在全闪存市场刮起了一股旋风，不仅是国内全闪存市场第一，还是全球全闪存市场增速第一的厂商，远超其他竞争对手。这背后正是华为存储团队多年以来持续不懈的努力所铸就的。
On the same day OceanStor Dorado V6 release, the new image is also stunning debut Logo: one surf sea swordfish, vacated born, leaping out of the sea. As the saying goes, "The great ambition there, sail sea", Huawei OceanStor Dorado, is courage!