Big data architecture and technology - (1) Overview of big data

1.1 Big data era

1.1.1 The third wave of informatization

According to former IBM CEO Gerstner, the IT fieldEvery fifteen years there will be a major change

• On August 12, 1981, IBM launched its first personal computer, the IBM PC5150.
• Each IBM5150 computer weighs about 11.34 kilograms, and the keyboard alone weighs about 2.7 kilograms. It is equipped with a 16-bit, 4.77MHz Intel 8088 microprocessor and 16KB (can be expanded to up to 256KB) memory
  .
• Audio cassettes can be used to download and store data, and a 5.25-inch floppy disk drive is also available; Microsoft's Disk Operating System (X86-DOS), spreadsheet software Visicale, and text input software Easywriter are installed.
• IBM is the leader in the computer industry. Because its logo is blue, it is also called "Big Blue".
• In 1981, the IBM personal computer came out. It became very popular as soon as it was launched, occupying 3/4 of the personal computer market. IBM mistakenly believed that the most profitable part of personal computers was the hardware costing thousands of dollars, rather than the software costing dozens of dollars. In order to quickly launch personal computers, IBM had no time to develop its own operating system, so it solicited bids from other companies.
• At this time, Bill Gates saw an opportunity and sold the operating system to IBM. But the smart Bill Gates did not let IBM buy out the operating system. Instead, he collected a modest royalties from each IBM personal computer.
• While cooperating with IBM, Bill Gates made great efforts to develop a new operating system (Windows). Once Windows was launched, the Microsoft empire was formed.
• On April 20, 1994, the first 64K international dedicated line was connected to China. This was a memorable event for China's Internet because it was the beginning of China's entry into the international Internet as a country.
• In 1995, after the 64K international dedicated lines between Beijing and Shanghai were opened, China had full-featured access to the international Internet.

1.1.2 Information technology provides technical support for the big data era

1. Storage device capacity continues to increase
   
2. CPU processing power greatly improved
   
3. Network bandwidth continues to increase
   

1.1.3 Changes in data generation methods have led to the advent of the big data era

1.1.4 The development history of big data

1.2 Big data concept

1.2.1 Large amount of data

• According to estimates made by IDC, data has been growing at a rate of 50% per year, which means it doubles every two years (Moore's Law of Big Data)
• The amount of data humans have generated in the past two years is equivalent to the entire amount of data generated before
• It is estimated that by 2020, the world will have a total of 35ZB of data. Compared with 2010, the amount of data will increase nearly 30 times PB EB ZB

1.2.2 Various data types

• Big data is composed of structured and unstructured data
  • 10% of structured data is stored in
    the database
  • 90% of unstructured data is
    closely related to human information

1.2.3 Fast processing speed

• The time window from data generation to consumption is very small, leaving very little time available to generate decisions.
• 1 second rule: This is also fundamentally different from traditional data mining technology

1.3 Impact of big data

In 1998, Dr. Jim Gray, Turing Award winner and famous database expert, observed and concluded that since ancient times
, human beings have gone through four paradigms: experiment, theory, calculation and data in scientific research.

The era of big data has had a huge impact on the way humans think

• The first impact is
  
  after the big data era:
  
• The second impact
• The third impact
  

1.4 Application of big data

Big data is everywhere, and all walks of life, including finance, automobiles, retail, catering, telecommunications, energy, government affairs, medical care, sports, entertainment, etc., have been imprinted with big data.

1.5 Key technologies of big data

The core is:Data storage and management, data processing and analysis

The two core technologies of big data technology:

• Distributed storage: solving the storage problem of massive data
• Distributed processing: solving the problem of processing massive data

1.6 Big data computing model

• The first big data computing model:batch calculation
  

• The second big data computing model:Stream computing
  

• The third big data computing model:graph calculation
  
  

• The fourth big data computing model:Query analysis calculation
  

Big data computing models and their representative products:

1.7 Big data industry

The big data industry refers to the collection of all corporate economic activities related to supporting big data organization management and value discovery.

1.8 The relationship between big data, cloud computing and the Internet of Things

1.8.1 Cloud computing

Cloud computing solves the problem of distributed storage and distributed processing of massive data

• Typical characteristics of cloud computing
  • Virtualization
  • multi-tenant
• The concept of cloud computing: Cloud computing is to provide users with very cheap IT resources in the form of services through the network
• Advantages of cloud computing: Enterprises do not need to build their own IT infrastructure and can rent cloud resources
• Three modes of cloud computing:
  • Public cloud: Baidu Cloud - for all public
  • Hybrid cloud: part for yourself, part for the outside world
  • Private cloud: China Mobile, China Telecom - for internal enterprises
    
• Cloud computing key technologies
  • Virtualization
  • Distributed storage
  • Distributed Computing
  • multi-tenant
• Cloud computing data center
  • Data centers are the hotbeds of cloud computing
  • Various data and applications are not in the "cloud in the sky", but are located in data centers everywhere.
  • Data centers contain a large number of blade servers, which are compact stand-alone servers, each with its own CPU, memory and storage.
  • Data centers consume a lot of energy

1.8.2 Internet of Things

The Internet of Things is the Internet where things are connected and is an extension of the Internet.

• IoT layer architecture:
  

• Internet of Things Key Technologies

  • Identification technology
  • Perception technology

• Internet of Things Applications
  The Internet of Things has been widely used in smart transportation, smart medical care, smart homes, environmental monitoring, smart security, smart logistics, smart grids, smart agriculture, smart industry and other fields, and has played an important role in promoting the development of the national economy and society.
  

• Cloud computing, big data and the Internet of Things represent the latest technological development trends in the IT field. The three are both different and related.