More data is being transferred and generated globally than ever before. Analysts at International Data Corporation (IDC) expect the global data layer to grow to 163zb by 2025. This is an increase of more than 1,000% from 2016’s figure of 16.1 ZB. There are many reasons for the massive increase in data:
Data is generated from many more sources and devices than ever before—embedded systems and devices are collecting data and transmitting it into big data applications and solutions for real-time analysis. The ongoing trend of using mobile devices, social media platforms, online shopping, and using various apps anytime and anywhere is generating huge amounts of data every day. Additionally, businesses are undergoing a transformation in delivering data to customers to meet their growing demand for never-before-seen news and real-time data.
According to the latest forecasts from Gartner, by 2020, more than half of major business processes and systems will incorporate some element of the Internet of Things (Internet of things) in their organizations. At the same time, the amount of data generated, transmitted and analyzed by big data applications (which will be stored internally or externally) will grow significantly.
Due to the need for storage, management and IT department representatives have significantly increased the number of solutions capable of processing and archiving more digital content than ever before.
However, from a hardware perspective, not only a larger number of storage devices - such as hard drives, SSDs or SSHDs - are now required, but a proper file system is also required to handle the results of this big data growth. This is because even if not all data is stored on the storage device, the most important data and analysis results will be stored on the storage device. This will result in increased storage space requirements. Additionally, most storage needs will be handled internally or through cloud services such as Amazon's S3 or Microsoft Azure.
The old storage concepts with file storage and block storage will not be suitable for future data growth, both for enterprises and cloud providers. The solution for storing these massive amounts of data is object storage (also known as object-based storage). But what are the differences between them compared to previous concepts? What makes object storage better suited to the data explosion?
To understand the benefits provided by object storage, one must first understand the old concepts of file storage and block storage, as there are significant differences between them.
Differences between file, block and object storage
File storage and block storage are methods of storing data on NAS and SAN storage systems.
On a NAS system, it exposes its storage as a network file system. When a device is attached to a NAS (Network Attached Storage) system, a mounted file system is displayed and users can access their files with appropriate access rights. Because NAS systems must manage user permissions, file locking, and other security measures so that multiple users can access files. Access to the NAS is handled via NFS and SMB/CIFS protocols. As with any server or storage solution, the file system is responsible for locating files in the NAS. This works great for hundreds of thousands or even millions of files, but not billions.
Block storage works similarly, but unlike file storage, which manages data at the file level, data is stored in blocks. Several blocks (for example in a SAN system) build a file. A block consists of an address, and if a SAN application makes a scsi request to this address, it will get the block. The storage application then determines whether the data block is stored in the system and on what specific disk or storage medium. Ultimately how these blocks are combined and how they are accessed determines the storage application. Blocks in a SAN have no metadata associated with the storage system or application. In other words: a block is a data segment that does not describe, relate and store the solution owner. Everything is handled and controlled by the SAN software. SAN and block storage are often used in applications that require performance, such as databases or transactions, because data can be accessed, modified, and saved.
Both methods of storing data have worked well for many years. So, why is another concept needed? This is because both concept solutions need to implement the functionality of user access rights in order to make changes to the data.
What we are seeing now is that most of the data produced is “fixed” or unstructured data. No further changes will be made to the content or materials. This is where object storage comes into play:
Objects in object storage are data "bound" (i.e. files) with corresponding metadata. This object gets a unique ID (identifier) calculated from the file content and metadata. The application identifies the object by this ID. Many objects in an object storage system are stored on a given storage disk. In a pure form of object storage, "only" one version of a file (object) can be saved. If the user makes changes, another version of the same file is stored as a new object. Therefore, object storage is a perfect solution for backup or archiving solutions. Or, for example, to store large amounts of videos or movies that can only be viewed and cannot be altered like videos on online movie streaming sites or YouTube.
The main difference between the other concepts is that objects are managed by the application itself which supports object storage. This means no real file system is required here. This layer is outdated. Applications that use object storage send storage queries to the location in the solution where the objects are stored. The object is then given an address in the huge storage space and saved there by the application itself.
Because data management is so simple—there is no real file system—object storage solutions are easier to scale than file storage or block storage-based systems. You just add some disks to the solution and no more big administration to get more storage space. This is a major benefit, especially in an era of exponential data growth.
Therefore, object storage is a perfect solution for handling large amounts of data and is therefore highly used by large cloud service providers such as Amazon and Google. But what about data protection and data recovery? We’ll provide answers to these questions in the second part of this article.
| This article: https://architect.pub/block-storage-vs-file-storage-vs-object-storage-pt-1 | ||
| Discussion: Knowledge Planet [Chief Architect Circle] or add WeChat account [ca_cto] or QQ group [792862318] | ||
| No public |
[jiagoushipro] [Architect Tavern] Excellent pictures and texts explain architecture methodology, architecture practice, technical principles, and technology trends in detail. We are waiting for you, please scan and follow us. |
 |
| WeChat account |
[ca_cea] Community of 50,000 people, discussing: enterprise architecture, cloud computing, big data, data science, Internet of Things, artificial intelligence, security, full-stack development, DevOps, digitalization . |
|
| QQ group |
【285069459】In-depth communication on enterprise architecture, business architecture, application architecture, data architecture, technical architecture, integration architecture, and security architecture. As well as various emerging technologies such as big data, cloud computing, Internet of Things, and artificial intelligence. Join the QQ group to share valuable reports and useful information. |
|
| Video number | [Architect Pub] Quickly understand the basic concepts, models, methods and experiences related to architecture in 1 minute. One minute a day, the structure will be familiar to you. |
|
| knowledge planet | [Chief Architect Circle] Ask questions to big names, get up close and personal, or get private information to share. |
|
| Himalayas | [Super Architect] Learn about the latest black technology information and architectural experience on the road or in the car. | [Smart Moment, Architecture Master will talk to you about black technology] |
| knowledge planet | Meet more friends and chat about work and technology. | Knowledge Planet [Workplace and Technology] |
| 【Architect Tavern】 | smart moment | |
| Bilibili | 【Architect Tavern】 |
|
| Tik Tok | 【cea_cio】Architect Tavern |
|
| quick worker | 【cea_cio_cto】Architect Tavern |
|
| little red book | 【cea_csa_cto】Architect Tavern |
|
| website | CIO(Chief Information Officer) | https://cio.ceo |
| website | CIO, CTO and CDO | https://cioctocdo.com |
| website | Architect’s practical sharing | https://architect.pub |
| website | Programmers cloud development sharing | https://pgmr.cloud |
| Official website | Xingtian Intelligent Technology Consulting Company | https://xingtian.ai |
| website | Developer Chat | https://blog.developer.chat |
| website | Chief Privacy Officer Internal Reference | https://cpo.work |
| website | Chief Security OfficerInternal Reference | https://cso.pub |
| website | CIOInformation | https://cio.cool |
| website | CDOInformation | https://cdo.fyi |
| website | CXOInformation | https://cxo.pub |
| website | Chief Architect Community | https://jiagoushi.pro |
Thank you all for following, forwarding, liking and watching.