Big data (1) Definition and characteristics
Contents of this article:
3.1. Volume characteristics of big data
3.2. Velocity characteristics of big data
3.3. Variety characteristics of big data
3.4. Value characteristics of big data
3.5. Veracity characteristics of big data
5. Data types involved in big data
6. Five core areas of big data
1. The words written in front
A week ago, I was watering the flowers on the roof of the building, and I ran into a neighbor in this building. She mentioned in a gossip that two people from a certain house on the 5th floor had left this year. Hearing this news, I felt very sad and couldn't help but shed tears. . .
In fact, I am not familiar with the family on the fifth floor, and the only few exchanges were more than ten years ago. At that time, I was still working, and when I came home, I could always meet the hostess of the house walking the dog near my house. The hostess is a teacher from the Propaganda Department of Huagong University. She has short hair and is very cheerful. The dog is a white long-haired Pekingese. The dog is a bit old. He would pant after running for two steps, and sometimes he would lie on the ground to rest. Whenever this time, the hostess always stood patiently on the road. While waiting for the dog, his eyes are very gentle and doting, as if looking at his own child. I like to tease the dog, touch its head, ask it if it is good today, and if it has a good meal. . .
The male owner is also a teacher of our Huagong University. I have no chance to meet him, but I know that he has done a lot of things for our building, such as handling a lot of elevator-related matters. Up to now, the owner of the elevator still writes this Teacher Wang's name.
Later, due to an accident, I stopped going to work and stayed at home to do housework. I basically never went out, and I didn't have much chance to meet the two neighbors on the fifth floor. I remember Teacher Wang's signature: Give some sunshine and it will shine, and give it a basket and you will lay eggs. . . A very humorous and optimistic person. After the epidemic began, Teacher Wang also posted funny pictures of the epidemic in the building group to make everyone happy. . .
Everything seems to be just yesterday, I didn't expect the two elders to leave so suddenly. . .
Good people will be rewarded, and the suffering and cultivation in this life will be exchanged for all the well-being in the next life. I would like to write this article to commemorate my old neighbors.
Sober in adversity
2023.8.24
2. Definition of big data
big data definition
Big Data refers to a collection of data that cannot be captured, managed and processed by conventional software tools within a certain period of time. It is a massive, high-growth and diverse information assets.
Big data refers to the collection of data with huge scale and various types, which cannot be efficiently processed by traditional data processing tools. It can be generated in a number of ways, including the internet, social media, sensors, financial transactions, etc.
Big data usually has three characteristics: large data scale, diverse data types, and fast data processing speed.
Applications of big data include business intelligence, finance, healthcare, energy, agriculture, transportation, and more. Using big data technology can obtain valuable information and knowledge from data to support decision making, predictive analysis, marketing and other aspects of work.
Big companies define "big data":
(1), the definition of Oracle
Oracle's big data definition:
In short, big data refers to very large and complex data sets, especially data sets from new data sources. Its scale is so large that traditional data processing software is helpless, but it can help us solve very difficult business problems in the past.
(2), research institution Gartner:
For the "big data" (Big data) research organization Gartner gives such a definition:
"Big data" requires a new processing model to have stronger decision-making power, insight and process optimization capabilities to adapt to massive, high-growth and diverse information assets.
(3) The definition given by the McKinsey Global Institute is:
McKinsey Global Institute's definition of big data:
A data set whose scale is so large that it exceeds the capabilities of traditional database software tools in terms of acquisition, storage, management, and analysis. It has four characteristics: massive data scale, fast data flow, diverse data types, and low value density.
3. Big data characteristics
Big Data Features :
♦ Volume: the size of the data determines the value and potential information of the considered data;
♦ Velocity: the speed at which data is obtained;
♦ Variety: the diversity of data types, there are many data types;
♦ Value (value): Reasonable use of big data to create high value at low cost.
♦ Veracity: The quality of the data.
3.1. Volume characteristics of big data
The massive nature of big data refers to the sheer volume of data. This data is often generated by data sources such as sensors, mobile devices, social media, financial data, medical records, etc.
A number of properties of big data include the following:
♦ Huge amount of data: the amount of data ranges from several gigabytes to hundreds of petabytes, and the scale is very huge.
♦ Rapid data growth: Data is growing exponentially, requiring constantly upgraded technologies and architectures to handle large-scale data.
♦ Diverse data sources: Data sources include sensors, mobile devices, social media, financial data, medical records and other fields.
♦ Various types of data: Data includes not only structured data, but also unstructured and semi-structured data, such as text, image, audio and video.
♦ High data complexity: Data is often highly complex and contains a large number of associations, interactions and changes, so efficient processing and analysis techniques are required.
A large amount of data brings great challenges to data processing and analysis, and advanced techniques and tools are required to process and analyze these data.
3.2. Velocity characteristics of big data
The high-speed (Velocity) characteristic of big data means that the speed of big data generation, transmission, storage and processing is very fast, and the amount of data is huge, often calculated in milliseconds or microseconds.
The high-speed characteristics of big data are mainly reflected in the following aspects:
♦ Data generated in real time: Big data is often generated in real time, such as user behavior on social media, sensor data generated by IoT devices, etc. These data need to be acquired and processed in real time.
♦ Fast data transmission speed: With the continuous improvement of network bandwidth and data transmission technology, a large amount of data can be quickly transmitted to the target system in a short time, such as cloud storage and data processing platform.
♦ Fast data storage: quickly write large client data into the database, process data in real time, etc.
♦ Fast data processing speed: Big data processing adopts distributed computing and parallel computing technology, which can quickly process a large amount of data, such as real-time data mining, real-time analysis and reporting, etc.
♦ Fast data update speed: Big data processing requires a very high data update speed, so as to ensure the real-time and accuracy of data.
To sum up, the high-speed characteristics of big data refer to the very fast speed of data generation, transmission, storage and processing, which can quickly respond to user needs and realize real-time data analysis and decision-making.
3.3. Variety characteristics of big data
The diversification of big data is mainly reflected in the following aspects:
♦ Diversified sources of data: Big data can come from a variety of sources, such as sensors, social media, logs, traditional databases, etc.
♦ Diversified data types: Big data types can be structured data (such as tabular data in relational databases), semi-structured data (such as XML files) and unstructured data (such as pictures, videos and sounds, etc.).
♦ Diversified data formats: Big data can be stored and transmitted in various standards and formats, such as CSV, JSON, XML, Avro, ORC, etc.
♦ Diversified data content: Big data can contain various types of information, such as text, numbers, images, audio, etc., and even intangible things, such as voice, emotion, opinion, etc.
♦ Diversified data scale: Big data can be massive, ultra-large, or even exponential data, which also brings great challenges to data analysis and processing.
3.4. Value characteristics of big data
The value characteristics of big data include the following aspects:
♦ Volume: Big data has a huge amount of data, which enables people to understand and predict the changing trend of things more comprehensively and accurately, so as to make better decisions.
♦ Diversity: Big data can come from various sources, such as sensors, social media, mobile devices, etc., and thus can contain different types of data, such as text, images, videos, etc. This diversity facilitates data integration and analysis , leading to more precise conclusions. Big data includes structured data (such as data in databases) and unstructured data (such as social media, logs, and pictures, etc.). These data come from different sources, types, and formats, providing a more comprehensive information basis for decision-making.
♦ Speed: The speed characteristic of big data means that the data processing speed and update speed are very fast, even in real time. Big data has the ability to process data at high speed, and can process a large amount of data in a short period of time, so as to quickly obtain information. Such data can help companies make decisions quickly, seize the market and gain a competitive advantage in the market.
♦ Scale: Big data is very large, containing billions or tens of billions of data points. This scale of data allows businesses to derive better information from larger data sets to better predict market and customer needs.
♦ Value: The true value of big data lies in extracting useful information from the data for analysis and application. This is very important for businesses as it can help them make better business decisions, improve products and services, optimize marketing, etc.
♦ Accuracy: The accuracy of big data refers to the accuracy and reliability of data. Ensuring the quality of data will help companies make better decisions and improve efficiency and effectiveness.
♦ Visualization: Data visualization allows people to better understand the data and thus discover patterns and trends in the data.
♦ Openness: Big data needs to be shared and accessed in an open manner so that more people can use and analyze the data.
Generally speaking, the comprehensive function of the value characteristics of big data is to help enterprises better understand their business, customers and markets, and formulate and execute strategies based on the results of data analysis, so as to obtain greater business value.
3.5. Veracity characteristics of big data
The veracity characteristic of big data refers to the accuracy and reliability of data. Since big data often comes from a variety of different sources and formats, they can have quality issues such as errors, deletions, duplications, ambiguities, etc. Therefore, for big data systems, it is very important to ensure the authenticity of the data to ensure the accuracy and reliability of the system.
In order to ensure the authenticity of the data, the following measures can be taken:
♦ Data cleaning: remove errors, repetitions and unnecessary information by cleaning the data, so as to improve the quality and accuracy of the data.
♦ Data Validation: Validate data to ensure it complies with business rules and standards, ensuring data correctness and reliability.
♦ Data monitoring: monitor data sources, discover and correct data quality problems in time to ensure data authenticity.
♦ Database management: manage the database, including backup, recovery and maintenance, to ensure data security and consistency.
♦ Data sharing: When sharing data externally, it is necessary to ensure the authenticity and security of the data to ensure that the data will not be tampered with or misused.
In short, the veracity feature of big data is a key element to ensure the quality and reliability of data and the correctness and reliability of big data systems.
4. Units of big data
The smallest basic unit is the bit , and all units are given in order: bit , Byte , KB , MB , GB , TB , PB , EB , ZB , YB , BB , NB , DB .
They are calculated according to the rate of 1024 ( 2 to the tenth power):
1 Byte =8 bit
1 KB = 1,024 Bytes = 8192 bit
1 MB = 1,024 KB = 1,048,576 Bytes
1 GB = 1,024 MB = 1,048,576 KB
1 TB = 1,024 GB = 1,048,576 MB
1 PB = 1,024 TB = 1,048,576 GB
1 EB = 1,024 PB = 1,048,576 TB
1 ZB = 1,024 EB = 1,048,576 PB
1 YB = 1,024 ZB = 1,048,576 EB
1 BB = 1,024 YB = 1,048,576 ZB
1 NB = 1,024 BB = 1,048,576 YB
1 DB = 1,024 NB = 1,048,576 BB
5. Data types involved in big data
| Data types involved in big data |
|||
| type of data |
concept |
Manifestations |
typical scene |
| structured data |
Also known as row data, it is data that has a unified structure and can be expressed and managed in a two-dimensional form of rows and columns, such as relational database data. |
database tables etc. |
Enterprise ERP, finance, HR database, etc. |
| semi-structured data |
It is a data model suitable for database integration, and it can also be a basic model of marking services for sharing information on the Web. |
Mail, HTML, reports, etc. |
Mail system, web page information, report system, etc. |
| unstructured data |
The data structure is irregular, and it is inconvenient to express data in two-dimensional form of rows and columns, such as pictures, texts, audio and video, etc. |
video, audio, etc. |
Online video content, audio content, graphic images, etc. |
6. Five core areas of big data
♦ Data storage and computing,
♦ Data management,
♦ Data circulation,
♦ Data application,
♦ Data security.
7. Big Data Trends
♦ Cloud computing: Cloud computing has become the preferred method for enterprises to store and process large amounts of data.
♦ Artificial intelligence and machine learning: Artificial intelligence and machine learning techniques are increasingly being applied to big data analysis and prediction.
♦ Blockchain: Blockchain technology can be used for data security and privacy protection.
♦ Data Science: Professionals in the field of data science are working with big data analysts to better understand and utilize big data.
♦ Data Quality Management: Data quality management has become an important area in big data management to ensure the accuracy and consistency of data.
♦ Data visualization: A large amount of data needs to be presented through data visualization tools in order to better understand and utilize the data.
♦ Edge computing: Edge computing technology can process large amounts of data on-site, thereby reducing data transmission and processing time.
Big data articles:
- Big data (1) Definition and characteristics
- Big data (2) Statistics related to big data industry
- Big data (3) Jobs related to big data
- Build big data visualization big screen based on Echarts
- Big Data (4) Mainstream Big Data Technology
Recommended reading:
|
|
|
|
| Tomcat11, tomcat10 installation configuration (Windows environment) (detailed graphics) |
Tomcat startup flashback problem solving set (eight categories in detail) |
|
