Author: JD Retail Liu Huiqing
A history of database development
At first, the data management method is the file system, the data is stored in the file, and the data management and maintenance are completed by the programmer. Later, tree-structured and network-structured databases were developed, but both of them had problems of being difficult to expand and maintain. Until the 1970s, when the relational database theory was proposed, data was organized in the form of a table, and there was a relationship between the data, which had good structure and standardization characteristics, and became the mainstream database type.
Let’s take a look at a picture book of database development history:
With the advent of the era of high-concurrency big data, databases have been split and evolved in a more fine-grained manner according to various application scenarios. Typical representatives of the database subdivision field:
| type | product representative | Applicable scene |
|---|---|---|
| Hierarchical Database (NDB) | IMS/IDMS | Organize data in a tree structure, there is a parent-child relationship between the data, the query speed is fast, but it is difficult to expand and maintain |
| relational database (RDBMS) | Oracle/MySQL | Transaction Consistency Requirements Scenario |
| Key-Value Database (KVDB) | Redis/Memcached | For high-performance concurrent read and write scenarios |
| Document Database (DDB) | MongoDB/CouchDB | For massive and complex data access scenarios |
| Graph Database (GDB) | Neo4j | Use points and edges as the basic storage unit to efficiently store and query graph data scenarios |
| Time Series Database (TSDB) | InfluxDB/OpenTSDB | Scenarios such as persistence of time series data and multi-dimensional aggregation query |
| Object Database (ODB) | Db4O | Supports a complete object-oriented (OO) concept and control mechanism, currently there are few usage scenarios |
| search engine (SE) | ElasticSearch/Solr | Suitable for search-based business scenarios |
| Column Database (WCDB) | HBase/ClickHouse | Mass data storage and query scenarios of distributed storage |
| XML Database (NXD) | MarkLogic | Supports operation scenarios such as storing and querying XML format documents |
| Content repository (CDB) | Jackrabbit | Large-scale high-performance content warehouse |
Two database noun concepts
RDBS
In June 1970, Edgar Frank Codd, a researcher at IBM, published the famous paper "A Relational Model of Data for Large Shared Data Banks" , opened the prelude to the relational database (Relational DataBase Server) software revolution (previously it was mainly a hierarchical model and a network model database). Until now, relational databases are still one of the most important data storage methods in the field of basic software applications.
Relational databases are built on the basis of relational data models, and are databases that process data by means of mathematical concepts and methods such as set algebra. In a relational database, entities and their relationships are represented by a single structure type, and this logical structure is a two-dimensional table. A relational database stores data in the form of rows and columns. This series of rows and columns is called a table, and a group of tables makes up a database.
NoSQL
NoSQL (Not Only SQL) database is also a non-relational database. It was produced under the background of the era of big data. It is far beyond the capabilities of traditional relational databases. NoSQL database does not have a unified model. It sacrifices the transaction mechanism and strong consistency mechanism to obtain better distributed deployment and horizontal expansion capabilities, so that it has stronger control over specific business data in different application scenarios. Processing performance. Examples of commonly used data models are as follows:
| type | product representative | Application Scenario | data model | Advantages and disadvantages |
|---|---|---|---|---|
| key-value database | Redis/Memcached | Content caching, such as sessions, configuration files, etc.; applications with frequent reads and writes and simple data models; | Key-value pairs, implemented through a hash table | Advantages: Good scalability and flexibility, high performance; Disadvantages: The data is unstructured and can only be queried by key |
| column cluster database | HBase/ClickHouse | Distributed data storage management | Stored in column clusters, store the same column together | Advantages: Simple, strong scalability, fast query speed Disadvantages: Functional limitations, does not support strong consistency of transactions |
| document database | MongoDB/CouchDB | Web applications that store document-oriented or semi-structured data | Key-value pair, value is a JSON structure document | Advantages: Flexible data structure Disadvantages: Lack of unified query syntax |
| graph database | Neo4j/InfoGrid | Social networks, application monitoring, recommendation systems, etc. focus on building relationship graphs | graph structure | Advantages: Support complex graph algorithms Disadvantages: High complexity, limited support data size |
NewSQL
NewSQL is a new type of relational database, which is the abbreviation of various new scalable and high-performance databases. It not only has the storage and management capabilities of NoSQL databases for massive data, but also retains the ACID and SQL features supported by traditional databases. Typical representatives are TiDB and OceanBase.
OLTP
On-Line Transaction Processing (On-Line Transaction Processing): also known as transaction-oriented processing, its basic feature is that the user data received by the front desk can be immediately transmitted to the computing center for processing, and the processing results are given in a very short time , is one of the ways to quickly respond to user operations.
OLAP
On-Line Analytical Processing (On-Line Analytical Processing) is a processing process oriented to data analysis, which enables analysts to quickly, consistently, and interactively observe information from various aspects to achieve the purpose of in-depth understanding of data. It has the characteristics of FASMI (Fast Analysis of Shared Multidimensional Information), that is, the fast analysis of shared multidimensional information.
Regarding the difference between OLTP and OLAP, borrow a table for comparison as follows:
PAH
HTAP (Hybrid Transactional/Analytical Processing) hybrid database is based on a new computing and storage framework, which can support OLTP and OLAP scenarios at the same time, avoiding resource waste and conflicts caused by large amounts of data interaction in traditional architectures.
Three domain database
columnar database
Traditionally, data saved in the form of rows is mainly suitable for OLTP applications, and data saved in the form of columns is mainly used for query-based OLAP applications. In a columnar database, data is stored in columns, and the data types in each column are the same. This storage method enables columnar databases to process large amounts of data more efficiently, especially when large-scale data analysis and processing are required (such as finance, medical care, telecommunications, energy, logistics and other industries).
The difference between the two storage structures is as follows:
The main advantages of columnar databases:
• Higher compression ratio: Since the data types in each column are the same, the columnar database can use a more efficient compression algorithm to compress data (the compression ratio can reach 5-20 times), thereby reducing the use of storage space.
• Faster query speed: Columnar databases can only read the required columns instead of the entire row of data, thereby speeding up the query speed.
• Better scalability: Columnar databases can be more easily scaled horizontally, that is, adding more nodes and servers to handle larger-scale data.
• Better data analysis support: Since columnar databases can handle large-scale data, it can support more complex data analysis and processing operations, such as data mining, machine learning, etc.
The main disadvantages of columnar databases:
•Slower write speed: Because data is stored in columns, each write needs to write the entire column instead of a single row, so the write speed may be slower.
• More complex data model: Since data is stored in columns, the data model can be more complex than a row-based database, requiring more design and development work.
Application scenarios of columnar databases:
• Finance: Transaction data and market data in the financial industry, such as stock prices, foreign exchange rates, interest rates, etc. Columnar databases can process these data more quickly and support more complex data analysis and processing operations, such as risk management, investment analysis, etc.
•Medical: Medical record data, medical images and experimental data in the medical industry. Columnar databases can store and process these data more efficiently and support more complex medical research and analysis operations.
• Telecommunications: user data and communication data in the telecommunications industry, such as phone records, SMS records, network traffic, etc. Columnar databases can process these data more quickly and support more complex user behavior analysis and network optimization operations.
• Energy: sensor data, monitoring data and production data of the energy industry. Columnar databases can store and process this data more efficiently and support more complex energy management and control operations.
•Logistics: Transportation data, inventory data, order data, etc. of the logistics industry. Columnar databases can process this data more quickly and support more complex logistics management and optimization operations.
In short, a columnar database is a database management system that efficiently handles large-scale data, but it needs to weigh factors such as writing speed, data model complexity, and cost. With the continuous development of traditional relational databases and emerging distributed databases, columnar storage and row storage will continue to merge, and the database system presents a dual-mode data storage method.
time series database
The full name of time series database is time series database (Time Series Database), which is a specialized database for storing and managing time series data, and is a database optimized for ingesting, processing and storing time stamp data. Compared with the conventional relational database SQL, the biggest difference is that the time series database is a database recorded at regular time intervals indexed by time.
Time-series databases are widely used in scenarios such as the Internet of Things and Internet application monitoring (APM). Taking monitoring data collection as an example, if the data monitoring data collection interval is 1s, a monitoring item will generate 86,400 data points per day. If there are 10,000 monitoring items, 864,000,000 data points will be generated in one day. In the Internet of Things scenario, this number will be even greater. The scale of the entire data is terabytes or even petabytes.
Time series database development history:
Currently, the most common Kubernetes container management system is usually monitored with Prometheus. Prometheus is a combination of open source monitoring & alarm & time series database.
graph database
Graph Database is a new type of NoSQL database based on graph theory. Its data storage structure and data query methods are based on graph theory. The basic elements of a graph in graph theory are nodes and edges, which correspond to nodes and relationships in a graph database.
Graph databases can perform some very complex relational queries in scenarios such as anti-fraud multi-dimensional association analysis scenarios, social network graphs, and enterprise relationship graphs. This is because the graph data structure represents the entity connection itself, and it represents the essence of the connection between things in the real world. Its connection has been established when the node is created, so it can return the associated data in a quick path in the query, thereby expressing Very efficient query performance.
At present, the more popular graph database products on the market are as follows:
Compared with traditional relational databases, graph databases have the following advantages:
1. Faster query speed: The graph database can quickly traverse the graph data to find the associations and paths between nodes, so the query speed is faster.
2. Better scalability: Graph databases can easily scale to large-scale datasets because they can store and process data in a distributed manner.
3. Better data visualization: Graph databases can visualize data as graphs, making it easier for users to understand and analyze data.
4. Better data consistency: Graph databases can ensure data consistency because they can establish mandatory relationships between nodes and edges.
Four data structure design
The basic knowledge related to the database is briefly introduced above, and the application practices related to several common data structure design are introduced below: zipper table, bit operation and ring queue.
4.1 Zipper watch
The zipper table is a commonly used data model in data warehouses, which is used to record the change history of dimension data. Let's take a scenario of personnel change as an example. Suppose there is an employee information table, which contains information such as the employee's name, job number, position, department, and entry time. If you need to record the changes of employees, you can use the zipper table to achieve.
First, add two new fields on the basis of the employee information table: effective time and expiration time. When the employee information changes, instead of adding a new record, the expiration time of the original record is modified, and a new record is added at the same time. As shown in the table below:
| Name | Job number | Position | department | Entry Time | effective time | Expiration time |
|---|---|---|---|---|---|---|
| Zhang San | 001 | manager | technology | 2010-01-01 | 2010-01-01 | 2012-12-31 |
| Zhang San | 001 | Director | technology | 2013-01-01 | 2013-01-01 | 2015-12-31 |
| Zhang San | 001 | General manager | technology | 2016-01-01 | 2016-01-01 | 9999-12-31 |
The effective time here refers to the time when the record becomes effective, and the expiration time refers to the time when the record becomes invalid. For example, Zhang San was originally the manager of the technical department, the effective time was the entry time, and the expiration time was at the end of 2012. He was later promoted to the director of the technical department, the effective time was early 2013, and the expiration time was the end of 2015. Finally, he was promoted to the general manager of the technical department. The effective date is the beginning of 2016, and the expiration date is the end of 9999.
In this way, the historical information of employee changes can be recorded, and the employee information at a certain point in time can be easily queried. For example, if you need to query the position and department information of Zhang San in 2014, you only need to query the records whose effective time is less than 2014 and whose expiration time is greater than 2014.
A zipper table usually includes the following fields:
1.主键：唯一标识每个记录的字段，通常是一个或多个列的组合。
2.生效时间：记录的生效时间，即该记录开始生效的时间。
3.失效时间：记录的失效时间，即该记录失效的时间。
4.版本号：记录的版本号，用于标识该记录的版本。
5.其他维度属性：记录的其他维度属性，如客户名、产品名、员工名等。
When the dimension attribute of a record changes, no new record is added, but the expiration time of the original record is modified, and a new record is added at the same time. The effective time of the new record is the changed time, and the invalid time is the end of 9999. In this way, the historical change information of each dimension attribute can be recorded, and at the same time, the dimension attribute information at a certain point in time can be correctly obtained when querying.
Compared with the traditional flow meter, the main difference between the zipper watch is:
1. The data structure is different: the flow table is a table with only adding and updating operations. Each update will add a new record, which contains all historical information. The zipper table is a table with adding, updating, and deleting operations. Each record has an effective time period and an invalid time period. The recorded historical information is reflected by the change of the time period.
2. Different query methods: the query method of the flow meter is based on time point query, that is, to query the record information at a certain time point. The query method of the zipper table is based on the time period query, that is, to query the record information within a certain time period.
3. Different storage space: Since the flow meter needs to record all historical information, the storage space is relatively large. The zipper table only records the effective time period and the invalid time period, so the storage space is relatively small.
4. The data update method is different: the flow table only has new and update operations, and each update will add a new record, and will not modify the original record. The zipper table has add, update and delete operations, and each update will modify the expiration time of the original record and add a new record at the same time.
4.2 Clever use of bit operations
With the help of the characteristics of computer bit operations, some specific problems can be solved ingeniously, making the implementation more elegant, saving storage space, and improving operating efficiency. Typical application scenarios: compressed storage, bitmap indexing, data encryption, graphics processing And status judgment, etc., several typical cases are introduced below.
4.2.1 Bit operations
• Use bit operations to implement application scenarios such as switches and multi-option overlays (resource permissions). An int type has 32 bits, which can theoretically represent 32 switch states or business options; take the user's monthly check-in scenario as an example: use an int field to indicate the user's monthly check-in status, 0 means no check-in, 1 means check in. If you want to know whether you have signed in on a certain day, you only need to judge whether the corresponding bit is 1. To calculate how many times you have signed in in a month, you only need to count how many bits are 1. This ingeniously designed data storage structure will be introduced in more detail in the following bitmap (BitMap).
• Use bit operations to implement business priority calculations:
public abstract class PriorityManager {
// 定义业务优先级常量
public static final int PRIORITY_LOW = 1; // 二进制：001
public static final int PRIORITY_NORMAL = 2; // 二进制：010
public static final int PRIORITY_HIGH = 4; // 二进制：100
// 定义用户权限常量
public static final int PERMISSION_READ = 1; // 二进制：001
public static final int PERMISSION_WRITE = 2; // 二进制：010
public static final int PERMISSION_DELETE = 4;// 二进制：100
// 定义用户权限和业务优先级的组合值
public static final int PERMISSION_LOW_PRIORITY = PRIORITY_LOW | PERMISSION_READ; // 二进制：001 | 001 = 001
public static final int PERMISSION_NORMAL_PRIORITY = PRIORITY_NORMAL | PERMISSION_READ | PERMISSION_WRITE; // 二进制：010 | 001 | 010 = 011
public static final int PERMISSION_HIGH_PRIORITY = PRIORITY_HIGH | PERMISSION_READ | PERMISSION_WRITE | PERMISSION_DELETE; // 二进制：100 | 001 | 010 | 100 = 111
// 判断用户权限是否满足业务优先级要求
public static boolean checkPermission(int permission, int priority) {
return (permission & priority) == priority;
}
}
•Other typical scenarios using bit operations: the design of the queue length in HashMap and the use of the AtomicInteger field ctl in the thread pool ThreadPoolExcutor to store the current thread pool status and the number of threads (the upper 3 bits indicate the status of the current thread, and the lower 29 bits indicate the thread status quantity).
4.2.2 BitMap
Bitmap (BitMap) is a commonly used data structure, which is widely used in indexing and data compression. The basic idea is to use a bit to mark the Value corresponding to an element, and the Key is the element. Since the data is stored in the unit of Bit, the storage space can be greatly saved. It is a rare data structure that can guarantee both the storage space and the search speed (without having to exchange space for time).
For example, suppose there is such a requirement: find out whether a certain number m exists among 2 billion random integers, and assume a 32-bit operating system, 4G memory, in Java, int occupies 4 bytes, 1 byte = 8 bits (1 byte = 8 bits).
• If each number is stored in int, that is 2 billion ints, so the occupied space is about (2000000000*4/1024/1024/1024)≈7.45G
• If it is stored in bits, it will be different. 2 billion numbers are 2 billion bits, and the occupied space is about (2000000000/8/1024/1024/1024)≈0.233G
The storage space can be compressed and saved by 31 times! So how does it achieve digital marking through binary bits? The principle is to use each binary bit (subscript) to represent a real number, 0 means that it does not exist, and 1 means that it exists, so that we can easily represent the numbers {1,2,4,6}:
The smallest unit of computer memory allocation is a byte, which is 8 bits, so what if you want to represent {12,13,15}? You can apply for another byte b[1]:
A two-dimensional array is used to realize the superposition of digits, and one int occupies 32 bits, then we only need to apply for an int array whose length is int index[1+N/32] to store, where N represents the number to be stored The maximum value of:
index[0]: can represent 0~31
index[1]: can represent 32~63
index[2]: can represent 64~95
And so on... In this way, given any integer M, then M/32 will get the subscript, and M%32 will know where it is in this subscript.
The BitMap data structure is usually used in the following scenarios:
1. Compress and store a large number of Boolean values: BitMap can effectively compress a large number of Boolean values, thereby reducing memory usage;
2. Quickly judge whether an element exists: BitMap can quickly judge whether an element exists, just need to find the corresponding bit;
3. Deduplication: BitMap can be used for deduplication operations, using elements as indexes, setting the corresponding bit to 1, and repeated elements will only correspond to the same bit, so as to achieve deduplication;
4. Sorting: BitMap can be used for sorting, use the element as an index, set the corresponding bit to 1, and then traverse the bit array according to the index order to get an ordered sequence of elements;
5. In search engines such as ElasticSearch and Solr, when designing search pruning, it is necessary to save the historical information that has been searched, and you can use bitmaps to reduce the space occupied by historical information data;
4.2.3 Bloom filter
Bitmap (Bitmap) is a data storage structure. If the amount of data is large to a certain extent, such as 64bit data, simply calculate the storage space and you will know that the requirements for massive hardware resources are no longer realistic:
So another well-known industrial implementation - Bloom Filter (Bloom Filter) appeared. If BitMap maps every possible integer value through direct addressing, which is equivalent to using a hash function, then the Bloom filter introduces k ( k > 1 ) independent hashes The function ensures that the process of judging the weight of elements is completed under the given space and misjudgment rate. The figure below shows the Bloom filter when k = 3:
The inside of the Bloom filter relies on the hash algorithm. When detecting whether a piece of data has been seen, there is a certain probability of false positives (False Positive), but false negatives (False Negative) will never occur. That is to say, when the Bloom filter thinks that a piece of data has appeared, then the piece of data is likely to have appeared; but if the Bloom filter thinks that a piece of data has not appeared, then the piece of data must have never appeared. By introducing a certain error rate, the Bloom filter enables massive data weight judgment to be realized at an acceptable memory cost.
In the figure above, x, y, and z set their three positions in the Bitmap to 1 through the hash function mapping. When w appears, only when the three flag bits are all 1, it means that w is in the set. In the situation shown in the figure, the Bloom filter will determine that w is not in the set.
common implementation
• Implemented in the Guava toolkit in Java;
•Redis 4.0 began to provide the Bloom filter function in the form of plug-ins;
Applicable scene
• Web crawlers deduplicate URLs to avoid crawling to the same URL address. For example, Chrome browser uses a Bloom filter to identify malicious links;
• Spam filtering, judging whether a mailbox is an anti-spam mailbox from billions of spam lists;
• Solve database cache breakdown. When hackers attack the server, they will construct a large number of keys that do not exist in the cache to initiate requests to the server. When the amount of data is large enough, frequent database queries will cause hangups;
• Google Bigtable, Apache HBase, Apache Cassandra, and PostgreSQL use Bloom filters to reduce disk lookups for rows or columns that don't exist;
•Seckill the system to check whether users make repeated purchases;
4.3 Ring queue
A circular queue is a data structure used to represent a fixed-size, end-to-end connection, which is very suitable for caching data streams. It is used in various scenarios such as communication development (Socket, TCP/IP, RPC development), inter-process communication (IPC) in the kernel, video and audio playback, etc. Various middleware such as Dubbo, Netty, Akka, Quartz, ZooKeeper, and Kafka used in the daily development process also have the idea of circular queues. Two commonly used ring data structures are introduced below: Hash ring and time wheel.
4.3.1 Consistent Hash Ring
Let’s take a look first. The structure of a typical Hash algorithm is as follows:
The Hash strategy in the above figure is an example. When the number of nodes N changes, almost all the previous hash mappings become invalid. If the cluster is a stateless service, there is nothing wrong, but if it is a distributed cache scenario, it will lead to comparisons. serious problem. For example, Key1 was originally routed to Node1 and hits the cached Value1 data. But when the N node changes, Key1 may be routed to the Node2 node, which causes the problem that the cached data cannot be hit. Whether it is a machine failure or cache expansion, it will lead to changes in the number of nodes.
How to solve the problem of the above scenario? It is the consistent Hash algorithm introduced next.
Consistent hashing organizes the entire hash value space into a virtual ring, assuming that the value space of a certain hash function H is 0-2^32-1 (that is, the hash value is a 32-bit unsigned integer), All input values are mapped to 0-2^32-1, forming a circle. The entire hash space ring is as follows:
The process of routing data is as follows: Use the same function Hash to calculate the hash value of the data key, and determine the position of the data on the ring, and "walk" clockwise along the ring from this position, the first node encountered is the node that should be The located server. If the server of a certain node fails, its scope of influence is no longer the entire cluster, but limited to a part of the failed node and its upstream nodes.
When a node goes down, the requests that originally belonged to it will be re-hash-mapped to the downstream node, which will suddenly cause excessive pressure on the downstream node and may also cause the downstream node to go down, which is easy to form an avalanche. For this reason, virtual node to solve this problem.
According to the Node node, many virtual nodes are generated and distributed on the ring, and a real node map corresponds to multiple virtual nodes. In this way, when a node hangs up, the requests originally belonging to it will be evenly distributed to other nodes to reduce the situation of avalanche, and also solve the problem of uneven distribution of requests caused by the small number of physical nodes.
Hash ring with virtual nodes:
The consistent hash algorithm is widely used in the field of load balancing due to its balanced and persistent mapping characteristics. For example, nginx, dubbo, etc. have internal implementations of consistent hash.
4.3.2 Time Wheel Fragmentation
Timewheel (TimeWheel) is an exquisite algorithm to realize the delay function (timer), which can realize efficient delay queue. Take the time wheel implementation scheme in Kafka as an example. It is a circular queue for storing timed tasks. The bottom layer is implemented by an array. Each element in the array can store a timed task list (TimerTaskList). TimerTaskList is a circular doubly linked list. Each item in the linked list represents a timer task entry (TimerTaskEntry), which encapsulates the real timer task TimerTask.
It can be seen from the above figure that the time wheel algorithm no longer uses the task queue as a data structure, and the polling thread is no longer responsible for traversing all tasks, but only traversing the time scale. The time wheel algorithm is like the pointer constantly rotating and traversing on the clock. If one finds that there is a task (task queue) at a certain moment, then all the tasks on the task queue will be executed once.
Assuming that the interval between the expiration times of adjacent buckets is bucket=1s, the timing starts from 0s, and the scheduled tasks that expire after 1s are hung under bucket=1, and the scheduled tasks that expire after 2s are hung under bucket=2. When 1 second has passed, all nodes under bucket=1 execute the timeout action, and when the time reaches 2 seconds, all nodes under bucket=2 execute the timeout action. The time wheel uses a dial pointer (pointer) to indicate the number of times the current pointer of the time wheel jumps. You can use tickDuration * (pointer + 1) to indicate the next due task, and you need to process all the tasks in the TimeWheel corresponding to this bucket. Task.
Advantages of the time wheel
1. The addition and removal of tasks are O(1)-level complexity;
2. Only one thread is needed to advance the time wheel, which will not take up a lot of resources;
3. Compared with other task scheduling modes, CPU load and resource waste are reduced;
Applicable scene
The time wheel is a scheduling model for solving efficient scheduling tasks. It can be effective in scenarios such as periodic scheduled tasks, delayed tasks, and notification tasks.
Five Summary
This article explains the concepts of nouns related to data storage, focusing on the development history of database technology. In order to enrich the readability and practicability of the article, the extension and expansion of some technical practical capabilities are carried out from the data structure design level, and the application of related data structures such as zipper tables, bit operations, and ring queues in the field of software development is expounded. Hope this article Bring you the harvest.
Note: Individual pictures in this article are from the Internet