NoSQL (Not Only SQL) is the general term for non-relational databases. Databases other than mainstream relational databases are considered non-relational.

There is no need to pre-build databases and tables to define the data storage table structure. Each record can have different data types and number of fields (such as text, pictures, videos, music, etc. in WeChat group chats).

The mainstream NoSQL databases include Redis, MongBD, Hbase, Memcached, ElasticSearch, TSD, etc.

2. The difference between relational database and non-relational database

(1) Different data storage methods

The main difference between relational and non-relational databases is the way data is stored.

SQL databases are tabular in nature , so they are stored in rows and columns of data tables . Data tables can be associated with each other and stored collaboratively, and it is also easy to extract data.

NoSQL-type data is not suitable for storage in the rows and columns of the data table, but grouped together in large chunks. Non-relational data is usually stored in datasets , like documents, key-value pairs, or graph structures. Your data and its characteristics are the number one influencing factor in choosing how to store and extract your data.

(2) Different expansion methods

The biggest difference between relational and non-relational databases is in the way of expansion. To support the growing demand, of course, expansion is necessary.

SQL databases scale vertically , which means increasing processing power and using faster computers so that the same data set can be processed faster. Because data is stored in relational tables, performance bottlenecks for operations that may involve many tables need to be overcome by increasing computer performance. Although the sql database has a lot of room for expansion, it will definitely reach the upper limit of vertical expansion in the end.

NoSQL databases scale horizontally . Because non-relational data storage is naturally distributed, the expansion of NoSQL databases can be done by adding more common database servers (nodes) to the resource pool to share the load.

(3) Different support for transactional

If data operations require high transactionality or complex data queries need to control the execution plan, then the traditional SQL database is your best choice in terms of performance and stability.

SQL database supports fine-grained control over transaction atomicity, and it is easy to roll back transactions.

Although NoSQL databases can also use transaction operations, they cannot compare with relational databases in terms of stability. So their real shining value is in terms of operational scalability and large data volume processing.

3. Non-relational database usage scenarios

It can be used to deal with the three high problems (high concurrency, high performance, and high availability) of web2.0 purely dynamic website types.

High performance - high concurrent read and write requirements for the database;
Huge Storage——Requirements for efficient storage and access of massive data;
High Scalability and High Availability——Requirements for high scalability and high availability of the database.

        Relational databases and non-relational databases have their own characteristics and application scenarios. The close combination of the two will bring new ideas to the development of web2.0 databases:

        Relational databases focus on relationships and data consistency guarantees;

        Non-relational databases focus on storage and high efficiency.

        For example: in the Mysql database environment where read and write are separated, frequently accessed data can be stored in a non-relational database to improve access speed.

2. Overview of Redis

1 Introduction

Redis (Remote Dictionary Server) is an open source NoSQL database written in C language.

Redis runs based on memory and supports persistence. It adopts key-value (key-value pair) storage form, which is an indispensable part of the current distributed architecture.

The Redis server program is a single-process model, that is, multiple Redis processes can be started on one server at the same time, and the actual processing speed of Redis depends entirely on the execution efficiency of the main process.

2. Advantages

High data reading and writing speed : the data reading speed can reach up to 110,000 times/s, and the data writing speed can reach up to 81,000 times/s.

Support rich data types : support data type operations such as key-value, Strings, Lists, Hashes, Sets and Sorted Sets.

Support data persistence : the data in the memory can be saved in the disk, and can be loaded again for use when restarting.

Atomicity : All Redis operations are atomic.

Support data backup : it supports data backup in master-salve mode.

3. The reason why Redis reads and writes fast

Redis runs on memory , avoiding time-consuming operations such as disk I/O.

The core module of Redis command processing is single-threaded , which reduces the cost of lock competition, frequent creation and destruction of threads, and reduces the consumption of thread context switching.

Note: The newly added multi-threading in Redis 6.0 only uses multi-linearity for processing network requests, while the data read and write commands are still processed by a single thread.

The I/O multiplexing mechanism is adopted to reduce network I/O consumption and greatly improve concurrency efficiency.

4. Applicable scenarios

As a memory-based database, Redis is a high-performance cache that is generally used in session caches, queues
, leaderboards, counters, recent hottest articles, recent hottest comments, publish subscriptions, etc.

Redis is suitable for scenarios with high real-time data requirements, data storage with expiration and elimination characteristics, no need for persistence or only
weak consistency, and simple logic.

3. Redis installation and configuration

#将安装包放在/opt下
cd /opt
tar xf redis-5.0.7.tar.gz
cd redis-5.0.7/

#编译
make

#安装到指定目录
make install PREFIX=/usr/local/redis

You also need to go to utils/ in the installation package and execute install_server.sh

Then modify the listening address in the configuration file /etc/redis/6379.conf

Fourth, the use of Redis

tool	effect
redis-server	Tool for starting redis
redis-benchmark	Used to detect the operating efficiency of redis on the machine
redis-check-aof	Fix AOF persistent file
redis-check-rdb	Repair RDB persistent files
redis-cli	redis command-line tool

1. Command line tool redis-cli (login)

2. Test tool redis-benchmark (test)

redis-benchmark [option] [option value]
                                -h specifies the server host name
                                -p specifies the server port
                                -s specifies the server socket
                                -c specifies the number of concurrent connections
                                -n specifies the number of requests
                                -d specifies the SET/GET value in bytes Data size
                                -k 1 means keep alive to keep the connection; 0 means reconnect reconnection
                                -r SET, GET, INCR use random key; SADD use random value
                                -P transmit <numreg> request through pipeline
                                -q force quit redis only display query/sec value
                                --csv output in CSV format (, split field's text)
                                -l generate a loop, execute tests forever
                                -t only run a comma-separated list of test commands
                                -I Idle mode (only open N idle connections and wait)

#向IP地址为192.168.109.133、端口为6379的Redis服务器发送100个并发连接与100000个请求测试性能
redis-benchmark -h 192.168.116.10 -p 6379 -c 100 -n 100000

#测试存取大小为100字节的数据包的性能
redis-benchmark -h 192.168.116.10 -p 6379 -q -d 100

#测试本机上Redis服务在进行set与lpush操作时的性能
redis-benchmark -t set,lpush -n 100000 -q

3. Use of redis command

(1) Store key-value pairs

SET key value

(2) Get the value of the key

GET key

(3) Determine the data type of the key (the default data type of redis is string)

TYPE key

Five data types in Redis

name	type
String	string
List	the list
Hash	hash
Set	unordered collection
Sorted Set	ordered set

(4) View key

KEYS * View all keys

KEYS wildcards View specified keys matched by wildcards

(5) Determine whether the key exists

EXISTS key

(6) Delete key

DEL key

(7) Modify the key name

RENAME original key name new key name

The new key name to be changed already exists, the value of this key name will be overwritten (it is recommended to exist before changing the name) or use:

RENAMENX original key name new key name // judging whether the new key name exists before modification, if it exists, return 0, if it does not exist, return 1 and execute the modification

(8) Number of statistical keys

DBSIZE

(9) Set password

CONFIG SET REQUIREPASS password

Verify after login with AUTH password

(10) View current password

CONFIG GET REQUIREPASS

(11) Delete password

CONFIG SET REQUIREPASS ''

4. Redis multi-database common commands

Redis supports multiple databases. Redis contains 16 databases by default, and the database names are sequentially named with numbers 0-15 (the default login is database 0). Multiple databases are independent of each other and do not interfere with each other.

(1) Switch database

SELECT library number

(2) Move the data to the specified library

MOVE key bank number

Five, Redis performance management

1. View memory usage

info memory

mem fragmentation _ratio #memory fragmentation rate = used memory_rss / used memoryused

memory _rss # is the memory that Redis applies to the operating system.
used memory # is the memory occupied by the data in Redis.
used memory peak # The peak value of redis memory usage.

2. Clean up memory fragments

(1) How memory fragmentation occurs

Redis has its own internal memory manager to manage the application and release of memory in order to improve the efficiency of memory usage.

When the value in Redis is deleted, the memory is not directly released and returned to the operating system, but to the internal memory manager of Redis.

When applying for memory in Redis, first check whether there is enough memory available in your own memory manager.

This mechanism of Redis improves the memory utilization rate, but it will cause some memory in Redis that is not used by itself but not released, resulting in memory fragmentation.

(2) Memory fragmentation rate

Tracking memory fragmentation rate is very important to understand the resource performance of Redis instance

It is normal for the memory fragmentation rate to be between 1 and 1.5 . This value indicates that the memory fragmentation rate is relatively low, and it also indicates that Redis does not exchange memory.
If the memory fragmentation rate exceeds 1.5 , it means that Redis consumes 150% of the actual physical memory required , of which 50% is the memory fragmentation rate.
If the memory fragmentation rate is lower than 1 , it means that the Redis memory allocation exceeds the physical memory, and the operating system is exchanging memory . Need to increase available physical memory or reduce Redis memory usage.

(3) Clean up memory fragments

Redis version below 4.0

You need to enter the shutdown save command on the redis-cli tool to let the Redis database execute the save operation and close the Redis service, and then restart the server. After the Redis server restarts, Redis will return the useless memory to the operating system, and the fragmentation rate will drop.

Redis version 4.0 or above

Execute config set activedefrag yes to enable automatic defragmentation;

Perform memory purge , manual defragmentation.

3. Memory usage

If the memory usage of the redis instance exceeds the maximum available memory, the operating system will start exchanging memory and swap space, resulting in greatly reduced performance.

Ways to Avoid Memory Swapping

Choose to install a Redis instance for the size of the cached data
Use Hash data structure storage as much as possible
Set the TTL life cycle of the key (setex key name time (s) value)

4. Internal recovery key

The memory cleaning strategy ensures reasonable allocation of redis limited memory resources. By default, the recovery strategy is to prohibit deletion. When the set maximum threshold is reached, a key recovery strategy needs to be selected.

Modify the maxmemory-policy attribute value in the configuration file

volatile-lru	Use the LRU algorithm to eliminate data from the data set with the expiration time set (remove the least recently used key, for the key with TTL set)
volatile-ttl	Pick out the data that is about to expire from the data collection that has set the expiration time (remove the most recently expired key)
volatile-random	Randomly select data to be eliminated from the data collection with an expiration time set (randomly removed from the key with TTL set)
allkeys-lru	Use the LRU algorithm to eliminate data from all data sets (remove the least used key, for all keys)
allkeys-random	Randomly select data elimination from the data set (randomly remove key)
noenviction	It is forbidden to eliminate data (do not delete until it is full and report an error)

Redis overview and installation, use and management

1. NoSQL non-relational database

1. NoSQL overview