After working hard for so many years, looking back, it is almost all long setbacks and sufferings. For most people's life, smooth sailing is only occasional, and frustration, unbearable, anxiety and confusion are the main theme. We take the stage we did not choose, play the script we did not choose. Keep going!
Table of contents
2.3. Basic data types of Redis
2.4. Special data types of Redis
2.9, Redis cache hit rate problem
3.2, springBoot integrates MongoDB
4.2. Docker related instructions
1. Introduction to NoSQL
NoSQL (Not Only SQL), which means not just SQL, generally refers to non-relational databases. Nosql, a technical category, was proposed in the early days, and the trend became more and more high in 2009.
Compared with traditional relational databases, NoSQL has the following advantages:
(1) Easy to expand (there is no relationship between data, it is easy to expand)
(2) Large data volume and high performance (Redis writes 80,000 times per second, reads 110,000)
(3) Data types are diverse (no The database needs to be designed in advance, and it can be used at any time)
(4) Traditional RDBMS and NoSQL
Our commonly used non-relational databases are mainly redis and mongoDB, among which redis is a key-value pair type, and mongoDB is a document type.
2. Redis
Reference link: Redis learning articles
2.1. Introduction to Redis
Redis (Remote Dictionary Server), that is, remote dictionary service, is an open source, network-enabled, memory-based and persistent log-type, Key-Value database, and provides APIs in multiple languages.
Redis is an open source (BSD licensed), in-memory data structure server that can be used as a database, cache and message queue broker. It supports strings, hash tables, lists, sets, sorted sets, bitmaps, hyperloglogs and other data types. Built-in replication, Lua scripts, LRU reclamation, transactions, and different levels of disk persistence, while providing high availability through Redis Sentinel and automatic partitioning through Redis Cluster.
What is redis generally used for? Generally, it mainly has the following functions:
1. Memory storage and persistence. The data in the memory will be lost when power is off, so persistence is very important (RDB, AOF) 2.
High efficiency and can be used for cache
3. Publish and subscribe system
4. Map information analysis
5 , timer, counter (pageviews)
The characteristics of redis include the following points:
1. Various data types
2. Persistence
3. Cluster
4. Transaction
2.2. Basic knowledge of Redis
Redis has 16 databases by default, DB0~DB15, the 0th database is used by default, you can use select to switch databases. You can use Another Redis Desktop Manager visualization software to manage and manage redis.
Redis is fast, so why is Redis so fast?
Redis is based on memory operations and is single-threaded. CPU is not the performance bottleneck of Redis. The bottleneck of Redis is based on the memory and network bandwidth of the machine. Since a single thread can be used, use a single thread!
Redis puts all the data in memory, so using a single thread to operate is the most efficient. For the memory system, if there is no context switching, the efficiency is the highest, and multiple reads and writes are performed on one CPU.
2.3. Basic data types of Redis
Redis supports five data types: string (string), hash (hash), list (list), set (set) and zset (sorted set: ordered set).
Redis supports a variety of data types, and each data type has its specific application scenarios. The following are the data types supported by Redis and their application scenarios:
-
String (String): used to store simple values such as strings, integers, and floating-point numbers, and is often used in scenarios such as caches, counters, and distributed locks.
-
List (List): It is used to store a series of ordered string values, supports inserting and popping elements from both ends, and is often used in scenarios such as message queues and task queues.
-
Set (Set): used to store a set of unordered and unique string values, supports operations such as intersection, union, and difference, and is often used in scenarios such as counting the number of independent visitors and user tags.
-
Sorted Set (Sorted Set): Similar to a set, but each member is associated with a score, which can be sorted according to the score, supports range query and ranking operations, and is often used in scenarios such as leaderboards and scoring systems.
-
Hash: An unordered hash table used to store key-value pairs, supports single field read and write and batch read and write operations, and is often used in scenarios such as storing objects and user information.
According to the needs of specific scenarios, you can choose an appropriate data type to store and process data. In actual use, multiple data types can also be combined for complex data processing, such as using List and Hash to store multiple objects, and quickly searching and sorting through Set and Sorted Set.
2.4. Special data types of Redis
1. Geospatial (geographic location)
The Geospatial data type can store geographic coordinates and supports operations such as distance calculation, range query, and proximity query on locations. This data type is often used to implement scenarios such as map applications, nearby people functions, and location-based recommendation systems.
2.HyperLogLog (base)
The HyperLogLog data type is used to count the cardinality of a large amount of data (that is, the number of unique elements), and has the characteristics of high space efficiency and adjustable precision. This data type is often used to count the number of unique website visitors, user retention rate, advertising click rate and other scenarios.
3.Bitmaps (bitmap)
The Bitmaps data type can store binary bit arrays and supports logical operations on bit arrays, such as AND, OR, NOT and other operations. This data type is often used in scenarios such as counting user activity and analyzing user behavior.
for example:
-
Geospatial application scenario: A restaurant ordering platform needs to query nearby takeaway stores. You can store the latitude and longitude information of each store in the Geospatial data type of Redis, and use the ZRANGEBYSCORE command to query stores within a certain distance around the specified location.
-
HyperLogLog application scenario: A social network needs to count how many unique visitors it has every day. You can use the HyperLogLog data type of Redis to estimate the number of unique visitors to the website, so as to evaluate the traffic quality of the website.
-
Bitmaps application scenario: An e-commerce platform needs to count the page visits of each user over a period of time, and can store the bit array corresponding to each user in the Bitmaps data type of Redis, and use bit operation commands for data analysis, for example The BITCOUNT command calculates the number of active users, the BITOP command calculates the number of new users in two time periods, etc.
2.5. Affairs
The essence of Redis transactions: a collection of commands! All commands in a transaction will be serialized, and will be executed in order during transaction execution.
Redis supports the function of transactions, which can package a series of commands into an execution unit, and ensure that the execution unit is either executed successfully or not executed at all. Redis transactions are implemented using commands such as MULTI, EXEC, DISCARD, and WATCH.
1. Open the transaction
To open a transaction, you need to use the MULTI command, which will set the transaction status of the client to open, and then wait for the user to enter multiple commands. In a transaction, all commands are only recorded in a temporary storage area, but not actually executed.
2. Execute the transaction
In a transaction, the user can enter any number of Redis commands, and these commands will be recorded in the temporary storage area. After the user has entered all the commands, the EXEC command is required to execute the transaction, which will execute all the commands in the temporary storage area in order and return the execution result of each command.
3. Cancel transaction
During the transaction, if the user wants to cancel the transaction, he can use the DISCARD command, which will clear all the commands in the temporary storage area and set the transaction status of the client to not open.
4. Monitor key
During the transaction, if the user wants to cancel the transaction when a certain key changes, he can use the WATCH command, which will monitor the specified key and abort the execution of the transaction when any one of the keys is modified.
Redis's transaction function can guarantee the atomic execution of multiple commands, but does not support rollback operations. If an error occurs before the EXEC command is executed, the entire transaction will be canceled and an error message will be returned. Therefore, special attention should be paid to handling exceptions when using Redis transactions.
2.6, Redis persistence
Redis provides two persistence methods, RDB (Redis DataBase) and AOF (Append-Only File).
1. RDB persistence
RDB is a snapshot persistence method of Redis, which saves the data set of Redis at a certain point in time to a binary file on the hard disk. RDB persistence can be enabled or disabled through configuration files, and supports manual triggering. When the Redis server is restarted, the server state can be restored by loading the RDB file, thereby improving the startup speed and stability of Redis. However, there is a risk of data loss with RDB persistence, because if the Redis process crashes after saving the RDB file for the last time, the most recently modified data will be lost.
2. AOF persistence
AOF is an additional log persistence method of Redis, which will append each write command executed by Redis to a log file. AOF persistence can be turned on or off through the configuration file, and supports various strategies such as automatic saving every second or saving every command. When the Redis server restarts, all write commands can be re-executed by reading the AOF file, thereby ensuring data integrity and consistency. AOF persistence is safer than RDB persistence, but it also consumes more system resources, and may cause the AOF file to be too large and needs to be rewritten periodically.
3. RDB and AOF hybrid persistence
In some scenarios, RDB and AOF persistence methods can be used at the same time to make full use of their respective advantages. For example, you can enable RDB persistence, back up the data set every hour, and enable AOF persistence, and append all write commands to the AOF file. security and integrity. In short, when using the persistence function of Redis, it is necessary to select an appropriate persistence method according to the actual application scenario and system resource conditions, and perform reasonable configuration and management.
2.7, Redis message queue
Redis can be used as a lightweight message queue, usually implemented using the List data type. Specifically, messages generated by producers can be inserted at the end of a list, and then consumers can take messages from the head of the same list for processing. In order to ensure the order and mutual exclusivity of message processing among multiple consumers, you can use the blocking command BLPOP or BRPOP of Redis to pop up messages.
It should be noted that Redis's message queue usually does not support advanced features such as message re-entry and message expiration time, so it is suitable for processing simple and low-latency message scenarios, such as task distribution, notification push, logging, etc. If you need richer message queuing functions, you can consider using professional message queuing middleware, such as RabbitMQ, Kafka, etc.
2.8, Redis cluster
Redis Cluster is a distributed system designed to improve the scalability and redundancy of Redis. The principle is to disperse and store data on multiple nodes, use a hash function to map the data to a fixed slot, and each node is responsible for managing a part of the slot. The client can directly connect to any node for read and write operations.
The method of Redis cluster includes the following steps:
-
Initialize the cluster: start multiple Redis instances as nodes, and use the command line tool redis-trib.rb to create a cluster.
-
Adding a node: Add a new node to an existing node, make it join the cluster, and allocate some slots to the new node.
-
Failover: When a node fails, Redis will automatically transfer the slot of the node to other nodes, and elect a new master node to take over the work of the failed node.
-
Capacity expansion and contraction: When expansion or contraction is required, the allocation of slots can be changed by adding or reducing nodes.
-
Cluster maintenance: monitor node status, back up data, regularly clean up expired data and other operations to ensure the stable operation of the cluster.
Precautions:
- It is best to use an odd number of nodes in the Redis cluster to avoid split-brain problems.
- Redis cluster does not support operating on a key alone, because a key may be assigned to multiple nodes.
- Redis cluster needs to use the cluster mode of the Redis client library to connect.
2.9, Redis cache hit rate problem
① Cache penetration: A large number of requests for keys that do not exist at all, requests for resources that do not exist at all (DB itself does not exist, and Redis does not exist)
solution:
Use Bloom filter to cache null values
(Use BitMap as a Bloom filter, put all currently accessible resources into the Bloom filter through a simple mapping relationship (hash calculation), when a request comes When using the Bloom filter, the Bloom filter will be judged first, if there is, then it will be released, otherwise it will be directly intercepted)
② Cache avalanche: a large number of keys in redis collectively expire (details below)
③ Cache breakdown: A hotspot key in redis expires (a large number of users access the hotspot key, but the hotspot key expires)
Cache avalanche and cache penetration solutions:
Set the hot words in advance and adjust the key duration. Hot data preheating refers to loading some popular data into the cache in advance before the system goes online, so that these data can be obtained directly from the cache after the system is officially launched. , relieve the access pressure within a short period of time after the system goes online, and avoid problems such as cache breakdown and cache avalanche. Preheating hotspot data can effectively ensure system stability and high availability, and improve system performance and response speed.
The locking mechanism can be used to solve cache breakdown and avalanche problems. Specifically, you can try to acquire a mutex before acquiring cached data. If the acquisition is successful, it means that it is the first request. At this time, you can load data from the database into the cache, and then release the lock. If the acquisition fails, it means that other requests are loading cached data. At this time, you can wait for a period of time and try again or directly return to the default value or error prompt. (When the hot key expires, use the lock mechanism to prevent a large number of requests from directly hitting the DB)
3. MongoDB
3.1, mongoDB overview
MongoDB is a database system designed for rapid development of Internet web applications. MongoDB is designed to be minimal, flexible, and part of the web application stack. The data model of MongoDB is document-oriented. The so-called document is a structure similar to JSON. Simply understand that MongoDB stores various JSONs in the database. ( BSON )
There are three concepts in MongoDB:
Database ( database ): A database is a warehouse in which collections can be stored.
Collection ( collection ): A collection is similar to an array, and documents can be stored in the collection.
Document (document): The smallest unit in the document database, the content we store and manipulate is a document.
Advantages of mongoDB:
Easy to expand: There are many types of NoSQL databases, but a common feature is to remove the relational features of relational databases. There is no relationship between data, so it is very easy to expand
Large amount of data, high performance: NoSQL databases have very high read and write performance, especially in the case of large data volumes, they also perform well. Thanks to its non-relational nature, the structure of the database is simple
Flexible data model: NoSQL does not need to create fields for the data to be stored in advance, and can store custom data formats at any time. In a relational database, adding and deleting fields is a very troublesome thing.
3.2, springBoot integrates MongoDB
Just refer to this blog: "SpringBoot" 15. SpringBoot integrates MongoDB ultra-detailed tutorial (including installation tutorial)_springboot mongodb_Old boss Chen's blog-CSDN blog
Add dependencies, and use the mongoDBTemplate template to operate MongoDB data.
4. Docker
4.1, docker concept
Docker is an open source containerization platform that makes it easy to package, distribute, and run applications. The core components of Docker include Docker Engine, Docker Hub, and Docker Compose.
-
Docker Engine: is the core component of Docker, responsible for managing the life cycle of containers, including operations such as creating, starting, stopping, and deleting containers. Docker Engine also provides a series of command-line tools to manage Docker containers.
-
Docker Hub: It is the official image warehouse of Docker, where users can search and download various pre-built Docker images, or upload their own built images to Docker Hub for sharing.
-
Docker Compose: It is a multi-container orchestration tool for Docker. It can define the relationship and dependencies between multiple containers through a YAML file, and quickly start, stop, and delete multiple containers.
Using Docker brings many benefits such as:
-
Simplified application deployment: Encapsulating the application and its required dependencies in a Docker container allows the application to run in any environment, thus simplifying the application deployment process.
-
Improve development efficiency: Docker containers can be created and destroyed quickly, thus improving developer testing and debugging efficiency.
-
Improve system security: Docker provides a sandbox environment to isolate containers, thereby avoiding interference and attacks between applications.
In conclusion, Docker is a powerful tool that can help developers quickly build, release and manage applications, and improve the portability and security of applications.
4.2. Docker related instructions
You can refer to this blog: Spring Cloud Microservice Practical Article 3-docker_nuist__NJUPT's Blog-CSDN Blog
The following are commonly used Docker instructions, and they are classified by function:
mirror related
docker build: build image according to Dockerfiledocker images: List all mirrors on the local host.docker rmi: Delete one or more local mirrors.docker tag: tag the image
Container related operations
docker run: Run a new container. You can specify the image name, container name, and other options, such as mounting data volumes, port mapping, etc.docker ps: List running containers.docker stop: Stop one or more running containers.docker start: Start one or more stopped containers.docker restart: Restart one or more containers.docker rm: Delete one or more stopped containers.docker exec: Executes a command in a running container.docker logs: View a container's logs.docker inspect: View the details of the container.
Warehouse related operations
docker login: Log in to Docker Hub or a private repository.docker push: Push the local image to the remote warehouse.docker pull: Pull the image from the remote warehouse.
Network related operations
docker network create: Create a new Docker network.docker network ls: List Docker networks.docker network inspect: View the details of the Docker network.docker network connect: Connect a container to the specified Docker network.docker network disconnect: Disconnect a container from the Docker network.
Data Volume Related Operations
docker volume create: Create a new Docker volume.docker volume ls: List Docker volumes.docker volume inspect: View the details of the Docker data volume.docker volume rm: Delete one or more Docker volumes.docker run -v: Mount the Docker data volume into the container.
The above are commonly used Docker commands, and there are many other commands and options, which can be docker --helplearned through or official Docker documents.
Data volumes in Docker are a persistent storage mechanism for sharing data between containers and hosts. Data volumes can mount directories or files in the host file system to containers, and can also mount data volumes of one container to another container.
The benefits of using data volumes are:
- Separation from containers: Containers can be deleted and recreated, while data volumes can be retained, realizing data persistence.
- Shared data: Multiple containers can use the same data volume to achieve data sharing and collaboration.
- Data backup: Data volumes can be backed up to remote storage or cloud storage to achieve data reliability and disaster recovery capabilities.
There are two types of data volumes:
- Host mount: Directories or files in the host file system will be mounted to the container, and file modifications in the container will be directly reflected on the host.
- Container mounting: The data volume of one container can be mounted to another container, so as to realize data sharing between containers.
5. K8s
5.1. Docker and K8s
Docker is a containerization platform, and k8s is the coordinator of container platforms such as Docker.
Kubernetes (K8s) and Docker are two different concepts. Docker is a containerization technology used to package an application and its dependencies into a portable container and run it in different environments. Kubernetes is a container orchestration platform that automates the deployment, scaling, and management of containerized applications.
Although these two are different concepts, there is a close relationship between them. Kubernetes can use Docker or other containerization technologies to create and run containers. In Kubernetes, Docker is often used as the default container runtime.
Another relationship between Kubernetes and Docker is that it can take full advantage of Docker's functions and features, such as application packaging, image management, container networking, data volumes, and more. Kubernetes also provides its own functions and features, such as cluster management, automatic scaling, health check, load balancing, etc., to help users better manage containerized applications.
In conclusion, Kubernetes and Docker are complementary technologies, and through their combination, efficient, flexible and automated deployment and management of containerized applications can be achieved.
5.2 Overview of K8s
Kubernetes (commonly referred to as K8s) is an open source container orchestration platform for automating the deployment, scaling, and management of containerized applications. Kubernetes was originally developed by Google and donated to the Cloud Native Computing Foundation (CNCF) for maintenance and promotion.
Kubernetes provides a range of capabilities and features, including:
- Automated deployment: Kubernetes can automatically deploy containers to clusters and configure and manage their runtime environments.
- Automatic expansion: Kubernetes can automatically scale applications according to load conditions, thereby ensuring the reliability and performance of applications.
- Health check and self-healing mechanism: Kubernetes will periodically check the health status of the application and automatically restore the container in case of failure.
- Load balancing: Kubernetes supports multiple load balancing methods, including Round Robin, Session Affinity, etc.
- Network management: Kubernetes can automatically manage container networks, allowing communication and data sharing between multiple containers.
- Storage management: Kubernetes can manage persistent storage and data volumes to achieve data persistence and sharing.
- Configuration management: Kubernetes can manage application configuration information and sensitive information through ConfigMap and Secrets.
Kubernetes employs a layered architecture, which includes:
- Kubernetes API Server: Responsible for providing API services and receiving requests from clients.
- etcd: A distributed key-value store for storing state information of Kubernetes clusters.
- Kubernetes Controller Manager: Responsible for managing the controller and monitoring the status of the Kubernetes cluster.
- Kubernetes Scheduler: Responsible for scheduling containers to run on available nodes.
- Kubernetes Kubelet: Responsible for starting containers on nodes and communicating with Kubernetes API Server.
- Kubernetes Container Runtime: Responsible for running containers, usually using container runtimes such as Docker or rkt.
In short, Kubernetes is a very powerful and flexible container orchestration platform that can help users achieve high availability, elasticity, scalability, and automated application deployment and management.
5.3. Kubernetes instructions
The following lists some commonly used Kubernetes instructions, involving the basic operations of Kubernetes, application deployment, resource management, etc.:
basic operation
kubectl version: View the version information of Kubectl and Kubernetes Server.kubectl config get-contexts: List all available contexts (Context).kubectl config use-context <context-name>: Switch to the specified context.kubectl cluster-info: View the information of the current cluster.
application deployment
kubectl create deployment <deployment-name> --image=<image-name>: Create a Deployment for deploying containerized applications.kubectl expose deployment <deployment-name> --port=<port-number> --type=LoadBalancer: Expose the Deployment as a Service and bind it to the Load Balancer.kubectl scale deployment <deployment-name> --replicas=<num-replicas>: Expand or shrink the number of Pods in the Deployment.kubectl rollout status deployment/<deployment-name>: View the rolling upgrade status of the Deployment.kubectl rollout undo deployment/<deployment-name>: Undo the most recent rolling upgrade.
resource management
kubectl get pods: List all running Pods.kubectl describe pod <pod-name>: View the detailed information of the specified Pod.kubectl logs <pod-name>: View the logs of the specified Pod.kubectl exec -it <pod-name> -- /bin/bash: Run a bash shell in the specified Pod.kubectl get services: List all services (Service).kubectl describe service <service-name>: View detailed information of a specified service.kubectl delete <resource-type> <resource-name>: Delete a resource of the specified type and name.
The above are only some of the Kubernetes instructions, and there are many other instructions and options in Kubernetes. The specific usage methods can be kubectl --helplearned through or the official Kubernetes documentation.