Cache Category:
- Client cache: page cache, browser cache, APP client cache;
- Network cache: proxy cache, CDN cache;
- Server cache: database cache, platform cache level cache.
Ehcache: Ehcache is a tool used to manage cache. The cached data can be stored in memory or on the hard disk. Its core is CacheManager. All Ehcache applications start from CacheManager, which is used to manage Cache (cache). An application can have multiple CacheManagers, and one CacheManager can have multiple Cache. What is stored inside the Cache is each Element, and what is stored in an Element is a pairing of key and value, which is equivalent to an Entry in the Map.
Ehcache's late strategy:
- FIFO: First In First Out, first in, first out. Judging the time of storage, the data farthest from the present will be eliminated first.
- LRU: Least Recently Used, least recently used. Determine the most recently used time, and the furthest data will be eliminated first.
- LFU: Least Frequently Used, least frequently used. Within a period of time, data that has been used the least frequently will be eliminated first.
Starting from version 1.7, EhCache supports five cluster solutions, namely: Terracotta, RMI, JMS, JGroups, and EhCache Server.
Generally, ehcache is used as a secondary cache. When the redis server goes down, the ehcache cache can be queried, which can effectively handle the server request pressure.
Redis: An in-memory cache database based on the memory-basedsingle-process single-thread modelThe KV database is written in C language. The redis thread is safe. There is no need to consider various lock issues. There is no locking and releasing lock operations, and there is no performance consumption caused by possible deadlocks. It can reach 100000+ QPS (query rate per second).
Redis application scene：
- Token generation: Example: Generate a token when logging in, use the token as the key, use the user information as the value, store it in redis, and set the key expiration time.
- SMS verification code: Call Alibaba Cloud to send a text message, Redis saves the mobile phone number (Key) and value (Value), and sets the timeout.
- Distributed lock
- Counter: Applications such as counting clicks. Due to the single thread, concurrency problems can be avoided, error-free and 100% millisecond-level performance.
- Caching hotspot data
- Publish and subscribe: generally not recommended
Basic data types of Redis:
- String
- List
- Hash (dictionary)
- Set
- Sorted Set (ordered set)
Redis matter：
- Start transaction: MULTI
- Command enqueue:
- Execute transaction: EXEC
- Rollback: discard
Redis master-slave replication: A master database can have multiple slave databases, but a slave database can only have one master database. The replication function of redis can effectively separate the reading and writing of the database and improve the load capacity of the server. The master database mainly performs write operations, while the slave database is responsible for read operations. (Implementation method: modify the redis.conf file and configure the slave nodeslaveof)
Redis Sentinel Mechanism: used to manage multiple Redis servers. The system performs the following three tasks:
- Monitoring: Sentinel will constantly check whether your Master and Slave are operating normally.
- Notification: When there is a problem with a monitored Redis, the sentinel can send notifications to the administrator or other applications through the API.
- Automatic failover: When a Master fails to work properly, the sentinel will start an automatic failover operation, which will replace one of the failed Masters. The Slave is upgraded to the new Master, and other Slaves of the failed Master are changed to copy the new Master; when the client tries to connect to the failed Master, the cluster will also return the address of the new Master to the client, so that the cluster can use the Master to replace the failed Master. Master.
An example of a single sentinel is as follows:
Redis persistence: It is to save memory data to the hard disk. Redis persistent storage is divided into AOF and RDB two modes.
- RDB (used when data requirements are not strict): Write data to a temporary file at a certain point in time. After persistence, replace it with this temporary file. Secondary persistent files to achieve data recovery.
- AOF (used when strict data requirements are required, the file is relatively large, and recovery is slow): Append "operation + data" to the operation log file in the form of formatting instructions At the end, after the append operation returns (has been written to the file or is about to be written), the actual data changes are made. The "log file" saves all historical operation processes. There are three record synchronization methods: always, everysec, . no
Redis Cluster (supported after 3.0): Redis Cluster divides data into 16384 slots, and each slot is responsible for storing a part of the data. These slots are evenly distributed among the nodes in the cluster. Cluster information is exchanged between nodes through the Gossip protocol, including node status, owned slot information, etc. Nodes also use PING/PONG messages to detect whether each other is alive. Using the master-slave replication mechanism, the master node corresponding to each slot can have zero or more slave nodes.
It supports dynamic expansion. If an instance is added, the slots will be automatically reallocated, and the data migration process will be automatically started to migrate the slots from existing instances to new instances to achieve even distribution of data.
Cache avalanche: The cache is invalid (or the data is not loaded into the cache) and all requests that should have accessed the cache are querying the database.
- Solution: Distribute expiration time, cache warm-up, second-level cache, lock or queue, and primary and backup.
Cache penetration: Refers to the user's query data, which does not exist in the database and naturally does not exist in the cache. This will cause the user to not find it in the cache when querying, and have to go to the database to query again every time, and then return empty. In this way, the request bypasses the cache and directly checks the database. This is also a cache hit rate issue that is often raised.
- Solution: Cache empty values (set default values), limit query frequency (such as verification code).
Cache breakdown: It means that a hotspot key has expired or been deleted in the cache. At this time, a large number of concurrent requests access the hotspot key at the same time, causing these requests to penetrate the cache. Direct access to the underlying storage system.
- Solution: Mutex Lock or distributed lock to ensure that only one thread loads the cache. When the cache expires, use asynchronous mode to update the cache to avoid A large number of concurrent requests flush the cache at the same time.
Zookeeper: A distributed open source coordination service. Features are as follows:
- Distributed coordination: ZooKeeper provides a distributed coordination service for managing and coordinating operations and communications between multiple nodes.
- Naming Service: ZooKeeper provides distributed systems with a simple namespace in which services can be registered and found. This facilitates dynamic discovery and consumption of services in a distributed environment.
- Configuration management: ZooKeeper can be used to centrally manage configuration information to ensure that all nodes use the same configuration.
- Distributed lock: ZooKeeper provides a distributed lock mechanism that enables multiple nodes to collaboratively access shared resources without race conditions.
- load balancing
- Strong consistency: It can ensure the global uniqueness of node creation in distributed high concurrency situations, that is: multiple clients request the creation of /currentMaster node at the same time, and eventually it must Only one client request can be created successfully. Using this feature, you can easily select a cluster in a distributed environment.
Data structure: Hierarchical directory structure, naming conforms to conventional file system specifications (similar to file systems),
Znode with multiple types :
- Transient (ephemeral) (create -e /app1/test1 "test1" client disconnects zk deletes ephemeral type nodes)
- Persistent (create -s /app1/test2 "test2" client disconnects zk does not delete persistent type nodes)
Real-time, within a certain time range, the client can read the latest data.
CuratorFrameworkis a class in the Apache Curator library, a high-level client framework used to simplify interaction with Apache ZooKeeper. Apache Curator is a Java client library for Apache ZooKeeper that simplifies interaction with ZooKeeper and provides some useful abstractions and functions to simplify the development of distributed systems.
Distributed lock:
Zookeeper’s general idea of implementing distributed locks:Each participant creates a sequential temporary node, obtains the node list, and determines whether to acquire the lock (The process determines whether the node it created is the smallest node in the directory. If it is the smallest node, it means that the process successfully acquired the lock; otherwise, the process listens to the node smaller than itself, Wait for it to delete or release the lock.). After the process completes the task, it deletes the node it created, which means releasing the lock.
Compare the implementation of distributed locks with Redis:Use the SETNX command to try to acquire the lock (Set a specified key (representing the lock) in Redis, And set a unique value for the key (such as UUID)), set the expiration time of the lock
Comparison of database locks, cache locks, and zk locks:
- From the perspective of ease of understanding (low to high): Database > Cache > Zookeeper
- From an implementation complexity perspective (lowest to highest): Zookeeper >= Cache > Database
- From a performance perspective (from high to low): Cache > Zookeeper >= Database
- From a reliability perspective (from high to low): Zookeeper > Cache > Database
Zookeeper election: Based on the voting mechanism, nodes vote through mutual communication. When a node obtains more than half of the votes, it becomes the Leader, ensuring high availability and consistency.
- Leader-Follower mode: The nodes in the ZooKeeper cluster adopt the Leader-Follower mode, where one node is elected as the Leader and the other nodes become Followers.
- Zab protocol: ZooKeeper uses the Zab (ZooKeeper Atomic Broadcast) protocol to achieve consistency and election. In Zab, the Leader is responsible for processing write operations and then broadcasting the write operations to all Followers to ensure that they have the same status.
- Voting mechanism: When the Leader fails, ZooKeeper will elect a new Leader through the voting mechanism. To initiate a vote, a node needs the support of more than half of the nodes, including itself. The election process ensures that only one node becomes the new Leader.
Comparing the Redis election mechanism, the following are the characteristics of Redis:
- Master-slave replication mode: Redis adopts master-slave replication in cluster mode. One node is the Master and the other nodes are Slaves. The Master is responsible for handling write operations, and Slaves maintain consistency by replicating the Master's data.
- Sentinel Sentinel mechanism: Redis’s sentinel mechanism is used to monitor and manage nodes in the Redis cluster. When the Master node fails, Sentinel will assist in the election of a new Master.
- Quorum (majority): Redis’s Sentinel mechanism uses Quorum for voting to ensure the uniqueness of the election. Quorum is half the number of nodes plus one, ensuring support from more than half of the nodes.
- Automatic fault migration: Redis's sentinel mechanism is not only used to elect a new Master, but can also automatically perform fault migration and upgrade the Slave to the new Master.
Solution to website cross-domain:
- Use jsonp to solve website cross-domain issues
- Use HttpClient internal forwarding
- Allow cross-domain using setting response headers
- Build an enterprise-level API interface gateway based on Nginx
- Use Zuul to build a microservice API interface gateway
XXL-JOB: It is a distributed task scheduling platform based on the scheduling center and executor architecture, communicating through the HTTP protocol to achieve visual management, distributed scheduling and High availability, supporting mechanisms such as task sharding and distributed locks.
Apollo: It is a distributed configuration center developed by Ctrip’s framework department. It can centrally manage the configuration of different application environments and different clusters. After the configuration is modified, it can be pushed in real time. It reaches the application side and has standardized permissions, process management and other features, making it suitable for microservice configuration management scenarios.
ACID acid equilibrium:
- Atomicity: Refers to the fact that a transaction is an indivisible unit of work, and operations in a transaction either all occur or none occur.
- Consistency: The integrity of the data before and after the transaction must be consistent.
- Isolation: The isolation of transactions means that when multiple users access the database concurrently, the transactions opened by the database for each user cannot be affected by the operation data of other transactions. Interference, multiple concurrent transactions must be isolated from each other.
- Durability: Durability means that once a transaction is committed, its changes to the data in the database are permanent and will not change even if the database fails. should have no impact on it.
CAP hat principle :
- Consistency: All data backups in the distributed system have the same value at the same time, and the data read by all nodes at the same time is the latest copy of the data.
- Availability: Good responsiveness. Full availability means that under any failure model, the service will handle and respond within a limited time.
- Partition tolerance: The system continues to work despite some messages being lost on the network.
The CAP principle proves that any distributed system can only satisfy the above two points at the same time, and cannot take into account all three.
Base: It satisfies the CAP principle and obtains availability by sacrificing strong consistency. It is generally used in the application layer of service-oriented systems or big data processing systems, by achievingBase. a> to try to meet most of the business needs. Eventual consistency
- Basically Available: Refers to the fact that when a fault occurs in a distributed system, partial availability is allowed to be lost to ensure core availability. But that doesn’t mean it’s unavailable. (For example: the search engine returns query results in 0.5 seconds, but due to a failure, responds to query results in 2 seconds; when the web page access is too large, some users provide downgrade services, etc.
- Soft State: Refers to allowing the system to have an intermediate state, and the intermediate state will not affect the overall availability of the system, that is, when the system is allowed to synchronize replicas between different nodes There is a delay.
- Eventually Consistent: After a certain period of time, all data copies in the system can eventually reach a consistent state. There is no need to ensure strong consistency of system data in real time. Eventual consistency is a special case of weak consistency.
Rigid transaction: satisfies ACID theory
Flexible transactions: Satisfies BASE theory (basically available, ultimately consistent), divided into:
- two-stage type
- Compensation type
- Asynchronous guaranteed
- best effort notification
Distributed consistency related protocols:
-
XA interface: The reason why XA needs to introduce a transaction manager is because the two machines cannot theoretically reach a consistent state, and it needs to introduce a transaction manager Single point for coordination. The transaction manager controls global transactions, manages transaction lifecycle, and coordinates resources. The resource manager is responsible for controlling and managing the actual resources (such as databases or JMS queues).
-
JTA specification: Defines support for XA transactions. In fact, JTA is modeled based on the XA architecture. **Independent JTA implementation (**such as JOTM, Atomikos).
-
Two-stage submission agreement: The first stage is"voting stage", where all participants Feedback on whether the transaction can be successful is sent to the coordinator; the second stage is "execution phase". Based on the feedback from all participants, the coordinator Notify all participants and commit or rollback on all branches in unison. Disadvantages: If the coordinator goes down and the participants are not directed by the coordinator, they will remain blocked.
-
Three-stage commit protocol: It is an improved version of the two-stage commit protocol. It solves the blocking problem through the timeout mechanism. "Ask phase": The coordinator asks the participants whether they can complete the instructions. Participants only need to answer yes or no, without doing actual operations. This phase times out and causes abort. "Preparation phase": If all participants in the query phase return to perform operations, the coordinator sends a pre-execution request to the participants, and then the participants write redo and undo logs, execute the operation, but do not submit the operation; if in the query phase If any participant returns a result that the operation cannot be performed, the coordinator sends an abort request to the participant. The logic here is similar to the preparation phase of the two-phase commit protocol. This phase times out and leads to success. "Submission phase": If each participant returns preparation success in the preparation phase, that is, the resource reservation and execution operation are successful, the coordinator initiates a submission instruction to the participants, and the participants submit the resource change transaction and release the locked resources; if If any participant returns preparation failure, that is, reserving resources or executing operations fails, the coordinator initiates an abort instruction to the participants, and the participants cancel the changed transactions, execute the undo log, and release the locked resources. The logic here is the same as the two phases. The commit phases of the commit protocol are consistent.
The difference between 2PC and 3PC submission:
- An "inquiry phase" has been added. The inquiry phase can ensure that behaviors that cannot be performed and need to be aborted are discovered as early as possible. However, it cannot detect all such behaviors and will only reduce the occurrence of such situations. After the preparation phase, a timeout is added to the tasks performed by the coordinator and participants. Once the timeout occurs, the coordinator and participants continue to submit the transaction, and the default is success. This is also the correctness of the default success after timeout based on probability statistics. .
- Compared with the two-phase commit protocol, the three-phase commit protocol has the above advantages, but once a timeout occurs, the system will still be inconsistent, but this situation is rare. The advantage is that at least it will not block and lock resources forever.
TX-LCN transaction coordination framework: It consists of two major modules, TxClient and TxManager. As the dependency framework of the module, TxClient provides standard support for TX-LCN, and TxManager serves as the controller of distributed transactions. The transaction initiator or participant is controlled by TxClient.
LCN transaction mode, TCC transaction mode, TXC transaction mode****:****
- **LCN (chain transaction) transaction mode: **Through chained service calls, transaction context information is transferred on the call chain, and ultimate consistency is achieved through a reliable messaging mechanism.
- **TCC (Try-Confirm-Cancel) transaction mode: **Through the definition of business interfaces and the implementation of compensation logic, transaction consistency in a distributed environment is ensured.
- **TXC (Transaction eXtending Components) transaction mode: **Divide distributed transactions into multiple local transactions, and ensure the consistency of each local transaction through message communication.
Elasticsearch: An open source, distributed, RESTful interface full-text search engine built on Lucene. Its advantages are as follows:
- **Horizontal scalability:** You only need to add a server, do some configuration, and start Elasticsearch to merge it into the cluster.
- **The sharding mechanism provides better distribution: **The same index is divided into multiple shards (sharding), which is similar to the block mechanism of HDFS; the divide and conquer method can improve processing efficiency.
- **High availability:** Provides a replication (replica) mechanism. Multiple replications can be set up for one shard, so that if a server goes down, the cluster can still run as usual, and the data information lost when the server goes down will be Replication is restored to other available nodes.
Elasticsearch storage structure: It is a document-oriented database. A piece of data here is a document, and JSON is used as the document serialization format.
- Relational Database: Database ⇒ Table ⇒ Row ⇒ Column
- Elasticsearch: Index ⇒ Type ⇒ Document ⇒ Field
Elasticsearch forward and inverted index:
- Forward index: Mapping of documents to field values, stored in the order of documents
正排索引：
Document 1:
Title: Elasticsearch Basics
Content: Elasticsearch is a distributed search and analytics engine.
Document 2:
Title: Getting Started with Elasticsearch
Content: Learn the basics of Elasticsearch for search and analytics.
......
- Inverted index: mapping of field values to documents, used to quickly retrieve documents containing specific field values
倒排索引：
Elasticsearch:
Document IDs: 1, 2, 3
Basics:
Document IDs: 1, 2
Getting:
Document ID: 2
Started:
Document ID: 2
Advanced:
Document ID: 3
Queries:
Document ID: 3
..
Elasticsearch query method: One is a simple version of the query, and the other uses a complete JSON request body, called structured query (DSL). DSL query is to POST a JSON. Since the POST request is in JSON format, there is a lot of flexibility, so this method is mostly used.
GET /user_dao/user_table/_search
{
"from": 0,
"size": 3,
"query": {
"match": {
"name": "grand"
}
}
}
IK word segmenter: It is a commonly used Chinese word segmenter in Elasticsearch, mainly used to handle the word segmentation requirements of Chinese texts. IK Analyzer is an open source word segmentation project that provides fine-grained Chinese word segmentation capabilities. The es-ik word segmentation plug-in version must correspond to the version installed in es. Example of installation location:/usr/local/elasticsearch-6.4.3/plugins/ik
ES data type: supports basic types and complex types.
Basic type:
- String: string, the string type includes text and keyword.
- text: This type is used to index long texts. Before creating the index, these texts will be segmented into words, converted into word combinations, and indexed; es is allowed to retrieve these words. The text type cannot be used for sorting and aggregation.
- keyword: This type does not require word segmentation and can be used for retrieval, filtering, sorting and aggregation. If the keyword type is self-read, it can only be retrieved by itself (fuzzy retrieval after text segmentation is not available). Note: The keyword type cannot be segmented into words, and the Text type can be segmented into words for query.
Index type: long, integer, short, byte, double, float - Date type: date
- Boolean type: boolean
- Binary type: binary
- Array type: Array datatype
Additional type：
- Geographical location types (Geo datatypes): geographical coordinate types, geographical shape types
- Specialized datatypes: Pv4 type, Completion type, Token count type
ES cluster: It is a distributed system composed of multiple nodes working together, in which the master node manages the cluster status,Data nodes store and perform search operations,Sharding and replicas ensure data distribution and redundancy< /span>, to achieve high-availability, horizontally scalable search and analysis services.
- **Cluster:** represents a cluster. There are multiple nodes in the cluster, one of which is the master node. This master node can be elected;
- Shards: represents index sharding. es can divide a complete index into multiple shards. The advantage of this is that it can split a large index into multiple shards. Distributed to different nodes to form a distributed search. The number of shards can only be specified before the index is created, and cannot be changed after the index is created. The default index creation is to allocate 5 shards for storage. The backup shards corresponding to the primary shards cannot be stored on the same server.
- Replicas: Represents index replicas. es can set up multiple index replicas. The first role of replicas is to improve the fault tolerance of the system. When a certain fragment of a node is damaged or When lost, it can be restored from the copy. The second is to improve the query efficiency of es. es will automatically load balance search requests.
- **Recovery:** stands for data recovery or data redistribution. When a node joins or exits, ES will redistribute the index shards according to the load of the machine. Data recovery will also be performed when the dead node is restarted.
Calculation of the total number of shards in the ES cluster: The primary shard is 3 and the backup shard is 2. Each primary shard corresponds to 2 backup shards and 3 primary shards. The slice is placed on 3 nodes (servers). 总分片数 = 3（仓库数）*1（主分片）+ 2（备份分片）*3 = 9个
ES routing algorithm: When the client initiates the creation of a document, es needs to determine which shard the document is placed on in the index. This process is data routing. shard = hash(routing) % number_of_primary_shards
Remarks: The number of primary shards of the ES index cannot be changed, because ifnumber_of_primary_shards changes in the remainder when querying, it cannot be obtained. data.
ELK Nisshi Kasei Collection Course：
- Install the Logstash log collection system plug-in on each server cluster node.
- Each server node inputs logs into Logstash.
- Logstash formats the log into json format, creates different indexes based on each day, and outputs it to ElasticSearch.
- Use the browser to install Kibana to query log information.
Logstash example: format using json format to output myes.log on the console
input {
# 从文件读取日志信息 输送到控制台
file {
path => "/usr/local/elasticsearch-6.4.3/logs/myes.log"
codec => "json" ## 以JSON格式读取日志
type => "elasticsearch"
start_position => "beginning"
}
}
# filter {
#
# }
output {
# 标准输出
# stdout {}
# 输出进行格式化，采用Ruby库来解析日志
stdout {
codec => rubydebug }
}
Distributed global id generation strategy:
- UUID:The disadvantage is that the storage space is relatively large and the amount of data transmission is large.
- Database self-increment: Difficult to expand, different database syntax and implementation are different, need to be dealt with when database migration or when supporting multiple database versions.
- Generated based on Redis: It can be implemented using Redis's atomic operations INCR and INCRBY. Example
前缀+自增+并且设置有效期. - Snowflake algorithm: High bit random + millisecond number + machine code (data center + machine ID) + 10-digit serial number