| Software-defined storage is the development trend of the industry, which can ensure that the storage access of the system can be managed more flexibly at a precise level. This article shares the database cloudification practice of software-defined storage, which comes from the sharing and arrangement of the four lectures last week. |
What is software-defined storage (SDS)
Software-defined storage (SDS) is a data storage method in which all storage-related control work is placed in external software relative to physical storage hardware. This software is not available as firmware on a storage device, but on a server or as part of the operating system (OS) or hypervisor.
Software-defined storage is a trend in the industry. It can ensure that the storage access of the system can be managed more flexibly at a precise level, abstracting software from hardware storage, which means that it can become a shared pool that is not limited by the physical system. in order to utilize resources more efficiently. Storage can also be deployed and provisioned through software and management, and its management procedures can be further simplified through automated policy-based management.
What are the characteristics of software-defined storage?
Scalable. The software-defined storage system can be expanded to a cluster size of hundreds or even thousands of units. Moreover, as the cluster size increases, the overall performance of the system increases linearly and at low cost
. The automatic fault tolerance and automatic load balancing mechanism of the software-defined storage system allows it to be built on ordinary X86 servers. In addition, the linear expansion capability also makes it very convenient to add and reduce machines, and can realize automatic operation and maintenance.
high performance. Whether it is for the entire cluster or a single server, the distributed storage system has high performance.
Traditional IT architecture model:
Traditionally, databases have been running on minicomputers and centralized SAN storage. With the rapid development of business, the transaction volume and data volume of business systems are increasing, and the requirements for the processing and storage capabilities of business systems are getting higher and higher. The traditional architecture has poor scalability, limited computing power, and a serious lack of IO capabilities. To achieve the corresponding processing capabilities, it is necessary to replace the existing minicomputers and storage with higher-configuration minicomputers and higher-end storage, which is technically complex. Expensive. At the same time, as equipment ages, equipment performance is increasingly insufficient to support business needs, and business system architecture transformation and upgrades are imperative.
Some disadvantages of traditional architecture:
Amount of data: M level –> G level –> T level –> Massive amount is in front of you
Data volume * number of users –> number of related users: tens of thousands –> hundreds of thousands –> millions –> the Internet is fully open
The pain of relational databases
The centralized storage system has become a bottleneck for I/O access:
1. Database nodes with traditional architecture require high-performance hosts and are costly. For a multi-node RAC database, since the communication bandwidth between nodes is usually 1Gb s, and the higher one is 10Gb s, the bandwidth of this order of magnitude makes the concurrent processing capabilities between nodes not fully utilized. At the same time, the traditional disk array is limited by the processing power of the controller and the bandwidth of the FC port. The IO throughput is usually only a few hundred MB/s. When the database requires a large number of disk reads and writes, the IO consumption time is too long.
2. Minicomputers are equipped with high-end storage, which is expensive, relatively closed, and poor in scalability: the storage expansion capability of traditional architecture is poor, and the performance does not improve correspondingly when the capacity increases. At the same time, the expansion cost is high and the expansion cycle is long.
3. The complex system brings the complexity of deployment, operation, operation and maintenance and management: Under the traditional architecture, it is necessary to understand multi-dimensional operation and maintenance knowledge and skills, such as: database, U ix, storage, etc., and the requirements for operation and maintenance personnel Extremely high, deployment and operation, operation and maintenance and management are extremely complex.
IT infrastructure development trends
The evolution of IT architecture-chimney to cloud
The two-tier computing-storage architecture has the following shortcomings:
Centralized storage system
IO access boundaries, high expansion costs
Complex deployment and operation and maintenance
Unable to quickly respond to IT staff resource needs
Advantages of cloud-based high-performance and high-availability architecture:
High
performance, elasticity, flexible and multi-dimensional dynamic expansion
, unified operation and maintenance management, and perfect data disaster recovery and protection
will bring about the improvement of IT infrastructure performance and x86 platform CPU performance. In the past 10 years, CPU processing speed has increased by 8-10 times; DRAM processing speed has increased by 7-9 times.
Storage and Network Development
In response to the development of IT infrastructure and IT infrastructure, zData, a distributed storage solution based on x86 servers, using flash memory cards and Infiniband switches that can enable Oracle RAC to achieve high processing capabilities and IO throughput, came into being.
zData software-defined storage is based on x86 PC server, while greatly improving system performance, maintaining system scalability and agile response capability, the cost is greatly reduced compared with traditional architecture. zData Light Storage distributed storage software can support thousands of nodes, and the storage pool capacity can reach 32PB. Supports storage resource pools composed of different storage media. Under the basic configuration (3 storage nodes), it reaches millions of IOPS and the throughput exceeds 20GB/s.
Implementation:
1. Users can purchase hardware themselves or recycle used hardware.
2. Yunhe Enmo provides zData software and services, including performance and capacity planning, zData best practices, database migration and optimization, and overall maintenance.
Solution value:
1. Standard x86 open architecture, the solution is flexible and convenient.
2. There is no need to use high-end minicomputers and high-end storage, and construction and expansion are extremely cost-effective.
Product structure:
zData Light Storage storage management software
zData Light Storage is a distributed storage management software independently developed by Yunhe Enmo. It is a set of flexible software-defined storage software that makes full use of internal direct-connected storage based on PC servers to create a virtual and scalable storage pool. , performance is vastly superior to external traditional Fiber Channel SANs, while cost and complexity are exponentially reduced. In zData, high-performance x86 servers can be used as storage nodes through the zData Light Storage storage management software, so that the internal disk resources of multiple storage nodes can be integrated to form a virtual SAN storage resource pool. On the database node (computing node) You can use the virtual SAN by installing the zData Light Storage client on it. zData Light Storage supports the RDMA protocol under the Infiniband network, which greatly reduces CPU usage and can significantly reduce IO access latency. It has unparalleled performance advantages. zData Light Storage supports mainstream storage media, including PCIe, NVMe, SAS, SATA and other interfaces. SSD disks and cards as well as traditional mechanical hard drives with SAS and SATA interfaces.
Infiniband Network
zData is based on a 40-100Gb Infiniband high-speed network and adopts a fully redundant network architecture. On the one hand, it can provide sufficient bandwidth to carry the data transmission of zData LightStorage storage management software, making its performance much better than the external traditional FC SAN, and its cost and complexity are doubled. Reduce; on the other hand, it provides guarantee for high-performance reading and writing of Oracle RAC cluster.
zData intelligent management tool zMan
According to the professional knowledge and strong team of Yunhe Enmo in the field of Oracle database, the user-friendly management and operation and maintenance tool zMan (zData Manager) has been developed. zMan provides intelligent and convenient one-key installation, deployment and management functions for zData, which is very convenient. It greatly reduces the cost of management and operation and maintenance and provides users with a simpler, more convenient and friendly operating experience in terms of management functions.
Yunhe Enmo zData software-defined storage has the following characteristics:
high performance
Storage nodes and computing nodes are interconnected through the high-speed 40~100Gb Infiniband network, and each connected link can transmit storage data. Through the zDataLight Storage storage management software, in the case of using a flash memory card, the IO throughput provided by a single storage node can reach more than 12GB/s, the IOPS (disk IO requests per second) can reach more than 400,000, and the disk access delay is 0.6 Within ms. The IO throughput on a single computing node can reach more than 12GB/s, the IOPS (disk IO requests per second) can reach more than 1 million, and the disk access latency is within 0.6ms. The three key capabilities of storage are more than 10 times that of traditional arrays.
Multi-dimensional dynamic expansion
In the zData software-defined storage architecture, computing nodes and storage nodes can be dynamically expanded according to business and application requirements, with linear growth in capacity and performance. Similarly, computing and storage nodes can be dynamically reduced according to business and application needs to ensure maximum availability of resources.
Resource pooling & cloudification
In the zData software-defined storage architecture, computing resources and storage resources are pooled, managed uniformly, and allocated and expanded on demand.
Architecture and software security and reliability
In zData software-defined storage, servers, Infiniband switches, Infiniband HCA cards and PCIe flash cards are all redundant. Failure of any device or node will not affect the availability of online services, and there is no single point of failure. At the same time, zData storage management software automatically manages and maintains storage resources. When a node in the distributed storage network is damaged, or storage nodes are moved or added, zData storage management software will reconstruct intact storage "blocks" on existing or intact nodes. ”, so that each storage “block” still has a mirror, ensuring data security. The reconstruction speed can reach 2TB per hour, and can be operated automatically online without manual intervention and without affecting online business.
Unify operation and maintenance management to improve resource utilization
The resources of the entire zData are managed uniformly through the zdata management platform, with automatic operation and maintenance, which greatly reduces the skill requirements for IT managers.
Cloud and Enmo zData software-defined storage unique intelligent caching tool zCache, provides the following functions:
• Supports writeback and passthrough modes
• Support online startup and shutdown of cache
• Support automatic/manual refresh of dirty data
• Supports dynamic control of refresh frequency and speed of dirty blocks
Application scenarios and advantages of zdata software-defined storage:
1. Data warehouse and reporting (OLAP) systems: The databases of this type of system have very high requirements for IO performance and storage capacity expansion, but they do not have high requirements for availability and disaster recovery. This type of system is a secondary system outside the core system in enterprise applications. If you buy the highest-end large-capacity centralized storage, the cost is very high. These types of systems are the best use cases for zData's software-defined storage solutions.
2. Online transaction processing (OLTP) systems: This type of system has high requirements for real-time performance and concurrency, frequent data operations, and correspondingly high requirements for CPU and IO. Since zData software-defined storage solutions can provide sufficient computing power and significantly reduce IO latency, this type of system is also the best use scenario for zData software-defined storage solutions.
3. Database cloudification and integration: Some large enterprises have a large number of databases and need to integrate many unimportant databases to reduce operation and maintenance complexity and costs. Use the zData software-defined distributed storage solution to build a database cloudization and integration platform to improve the performance of the integrated system and increase the flexibility of the system (facilitate expansion), and use the zMan software-defined storage management tool to make it easy for users to maintain the integrated system. . zData supports cloudification and management of OpenStack and VMware platforms.
4. Database backup and disaster recovery under the X86 architecture: It is suitable for many traditional X86 architecture database systems to establish a low-cost, high-efficiency backup and disaster recovery platform centered on zData software-defined distributed storage.