Java preparation for server-side development 10

After working hard for so many years, looking back, it is almost all long setbacks and sufferings. For most people's life, smooth sailing is only occasional, and frustration, unbearable, anxiety and confusion are the main theme. We take the stage we did not choose, play the script we did not choose. Keep going!

Table of contents

1. The basic principle and underlying implementation of indexing?

2. What is the design principle of the index? Index failure?

3. What are the basic characteristics and isolation levels of transactions?

4. Explain MVCC in detail?

5. Clustered index and non-clustered index?

5. What is the difference between MyISAM and InnoDB?

6. What do the fields in the Explain statement result represent?

7. What is index covering?

8. How does InnoDB implement transactions?

9. What is the difference between B-tree and B+ tree? Why does MySQL use B+ trees?

10. What are the MySQL locks? How to understand?

11. What are RDB and AOF?

12. Redis expired key deletion strategy?

13. Briefly describe the transaction implementation of redis?

14. What is the core principle of redis master-slave replication?

15. What data structures does Redis have? Application scenario?

16. How is the bottom layer of Redis distributed lock implemented?

17. Redis cluster strategy?

18. Cache penetration, cache avalanche, cache breakdown?

19. How do Redis and MySQL ensure data consistency?

20. What is the persistence mechanism of Redis?

21. Why is redis single thread so fast?

22. Briefly describe the transaction implementation of redis?

23. CAP theory? BASE theory?

24. What is RPC?

25. What are the data consistency models?

26. What is distributed ID? What are the solutions?

27. What are the usage scenarios of distributed locks? What implementation options are there?

28. What is the ZAB protocol?

29. Why can Zookeeper be used as a registration center?

30. What is the leader election process in Zookeeper?

31. How is data synchronized between nodes in the Zookeeper cluster?

32. Which load balancing strategies does Dubbo support?

33. How does Dubbo complete the service export?

34. How does Dubbo complete the service introduction?

35. What is the architecture design of Dubbo?

36. What are the load balancing algorithms?

37. Under the distributed architecture, what is the solution for Session sharing?

38. How to achieve interface idempotency?

39. Briefly describe Zookeeper's naming service, configuration management, and cluster management?

40. Tell me about the watch mechanism of Zookeeper?

41. What is the difference between Zookeeper and eruka?

42. How to solve the unique primary key after storage split?

43. What is the snowflake algorithm? Snowflake algorithm principle?

44. What are the commonly used components in Spring Cloud and what are their functions?

45. How to avoid cache penetration, cache avalanche, and cache breakdown?

46. What are the commonly used caching schemes in distributed systems?

47. What are the strategies for cache expiration?

48. Common cache elimination algorithm?

49. Bloom filter principle? Advantages and disadvantages?

50. Distributed cache addressing algorithm?

51. What are the differences between SpringCloud and Dubbo?

52. What is Service Avalanche? What is service throttling?

53. What is a service circuit breaker? What is service degradation?

54. What is the difference between SOA, distributed, and microservices?

55. How to split microservices?

56. How to select the message queue?

57. How is the transaction message of RocketMQ implemented?

58. What is the underlying implementation principle of RocketMQ?

59. How does the message queue ensure the reliable transmission of messages?

60. What are the functions of the message queue?

61. What is a dead letter queue? What is a delay queue?

62. How does MQ ensure efficient reading and writing?

63. What is the difference between epoll and poll?

64. What is a cross-domain request? What is the problem? How to deal with it?

65. What is zero copy?

66. What is the difference between PostgreSQL and MySQL? Why do many project databases use PostgreSQL now?

67. What is distributed? What are microservices?

1. The basic principle and underlying implementation of indexing?

In the MySQL database, an index is a data structure used to improve query efficiency, and its basic principle is similar to other relational databases. Indexes speed up data access by storing ordered data on specific columns or groups of columns.

In the InnoDB storage engine, there are mainly the following index types:

B-tree index: B-tree index is the most common and important index type in InnoDB storage engine. It is realized by B-tree data structure, which is suitable for operations such as equivalence query, range query and sorting.
Full-text index: Full-text index can perform efficient fuzzy search on text-type columns, and supports functions such as keyword matching, Boolean search, and wildcard search.
Hash index: Hash index is suitable for equivalence query, by hashing the column value into an integer and building a hash table to achieve fast lookup.
Spatial index: Spatial index can perform efficient spatial query on data with spatial attributes, such as geographic location information.

In actual use, it is necessary to select the appropriate index type according to the specific scenario to achieve the best query efficiency and space utilization. At the same time, you need to pay attention to the maintenance cost of the index and the impact on insert and update operations.

Indexing is an important performance optimization technology in databases, and its underlying implementation can be divided into two aspects: storage structure and algorithm.

Storage structure: Usually, the index is stored on the disk in the form of B-tree or B+ tree. B-tree/B+ tree is a multi-fork balanced search tree, which has good balance and efficient search, insertion, deletion and other operations. Compared with B-tree, B+ tree is more suitable as the storage structure of database index, because it only stores keywords in leaf nodes, making it more efficient to traverse leaf nodes in range query, and because each non-leaf node only contains index column The value does not contain complete data row information, so the B+ tree can reduce memory usage and the number of IO operations, and improve query efficiency.
Algorithms: Operations such as creating indexes, querying indexes, and modifying indexes all need to use some algorithms to implement. Among the more common algorithms are:

Sorting algorithm: used to sort the values of index columns, such as quick sort, merge sort, etc.
Splitting algorithm: When the index node is full, it needs to be split into two nodes. At this time, a splitting algorithm is needed to ensure balance and correctness.
Search algorithm: used to locate the address of the data row on the index tree, such as binary search, B+ tree search, etc.
Compression algorithm: Compress the data in the index node to reduce disk storage space, such as prefix compression, Huffman coding, etc.

2. What is the design principle of the index? Index failure?

Index design principles:

Determine the index column: select the appropriate column as the index column according to factors such as query frequency, filtering degree, and sorting.
Index coverage: Try to make the index column contain all the data that needs to be returned to reduce query back to the table.
Consider composite indexes: When a single index cannot meet query requirements, you can consider using composite indexes composed of multiple columns.
Avoid too many indexes: Too many indexes increase the cost of modification operations and take up additional storage space.
Optimize index order: For composite indexes, you need to give priority to the most commonly used query conditions as the prefix of the index.

Index failure:

Functional operations on indexed columns: If a functional operation is performed on an indexed column in the WHERE clause, the index will not be used.
Non-leftmost prefix matching: For composite indexes, the composite index is only used when the leftmost index column is used in the query. If no leftmost prefix is used, the index will not be used.
Data type implicit conversion: If a data type different from the indexed column is used for comparison in the query, the index will not be used.
There is or must be an index.
like begins with %.
Full table scans are faster.

To sum up, the design and use of indexes need to be optimized and adjusted according to specific scenarios and requirements, so as to improve query efficiency and reduce index failures as much as possible.

3. What are the basic characteristics and isolation levels of transactions?

Basic properties of transactions include ACID:

Atomicity: All operations in a transaction are either completed or not completed, and there will be no partial completion.
Consistency: before and after transaction execution, the state of the database should remain consistent. In order to meet consistency, the database needs to perform some restrictions and checks during transaction execution.
Isolation: Multiple transactions executed concurrently should be isolated from each other, and the execution of each transaction should be independent of the execution of other transactions without interfering with each other.
Durability: Once the transaction is successfully committed, the changes made to the database should be permanently saved, and should not be lost even if the system crashes.

The isolation level refers to the degree of isolation between different transactions. There are four common isolation levels:

Read Uncommitted (Read Uncommitted): When a transaction has not been committed, the changes it makes can be seen by other transactions.
Read Committed (Read Committed): After a transaction is committed, the changes it makes will be seen by other transactions. This is the default isolation level for most database systems.
Repeatable Read: During the execution of a transaction, the data seen is always consistent, even if other transactions modify the data. The default isolation level of MySQL is Repeatable Read.
Serializable: All transactions are executed sequentially, so that the data seen by each transaction is the data modified by the previous transaction. This isolation level can avoid problems such as dirty reads, non-repeatable reads, and phantom reads, but it will reduce concurrency performance.

4. Explain MVCC in detail?

MVCC is the abbreviation of Multi-Version Concurrency Control (Multi-Version Concurrency Control), which is a technology used to achieve database isolation.

The basic idea of MVCC is to provide each transaction with an independent, visible snapshot of the database. When the transaction starts to run, the system will copy all the data in the current database to form a snapshot, and then the transaction can only see the data in the snapshot. Modifications to data by other transactions do not affect the data seen by this transaction. The modification made by the transaction will not be updated to the database immediately, but will be applied to the database when it is submitted, thus avoiding the use and waiting of locks.

Specifically, when a transaction reads a piece of data, the system first checks whether the data has been modified by other uncommitted transactions. If so, the system will return the historical version of the data to the transaction, otherwise it will directly return the latest version data. When a transaction modifies a piece of data, the system will create a new version for the modification and associate this version with the transaction, so as to ensure that modifications between different transactions will not interfere with each other.

By using MVCC, the database can achieve high concurrent read and write operations while ensuring data consistency and isolation. Common databases using MVCC include PostgreSQL and Oracle.

5. Clustered index and non-clustered index?

Both clustered and non-clustered indexes are commonly used index types in databases, and both can improve query efficiency.

A clustered index is an index structure that sorts the entire table according to the values of the indexed columns. When a clustered index is created, the physical order of data rows is the same as the logical order of the clustered index, so a table can only have one clustered index. When querying, if you use a clustered index, you can quickly locate the required data row directly through the index file, thereby speeding up the query. A clustered index is to construct a B+ tree according to the primary key of each table. At the same time, the leaf nodes store the row record data of the entire table, and the leaf nodes of the clustered index are also called data pages.

A non-clustered index stores the values of the index columns in a separate index file, and points to the corresponding data rows based on these values. Unlike clustered indexes, nonclustered indexes do not change the physical order of data rows, so a table can have multiple nonclustered indexes. When querying, if you use a non-clustered index, you need to find the corresponding data row address through the index file first, and then access the data row according to these addresses, so the query speed may be slightly slower than the clustered index. The leaf nodes of the Innodb auxiliary index do not contain all the data of the row records . In addition to the key values, the leaf nodes also contain the clustered index keys of the corresponding row data.

Back-to-table query means that when using a non-clustered index for query, if the data to be returned includes not only the values of the non-clustered index column but also the values of other columns, it is necessary to obtain these additional data through the back-to-table query. Specifically, when using a non-clustered index to locate a row of records, the database also needs to access the data page through the physical address (or row ID) of the row record to obtain the complete data of the row record.

Although table return query can obtain more comprehensive data and meet business needs, it may also cause performance problems in some cases, because each table return query requires additional IO operations and time for reading data. Therefore, when designing data table structures and indexes, it is necessary to weigh the selection and design of indexes according to actual query scenarios and data access patterns to achieve optimal query performance.

5. What is the difference between MyISAM and InnoDB?

Both MyISAM and InnoDB are commonly used storage engines in MySQL databases, and they have the following differences:

Transaction support: MyISAM does not support transactions, while InnoDB supports transactions.
Locking mechanism: MyISAM uses table-level locking. When a thread modifies a table, other threads must wait; while InnoDB uses row-level locking, which only locks the modified row and does not lock the entire table, thereby improving concurrency performance.
Foreign key constraints: MyISAM does not support foreign key constraints, while InnoDB supports foreign key constraints.
Performance: MyISAM is more efficient than InnoDB when performing a large number of simple queries, while InnoDB is better at processing a large number of complex transactional operations.

Based on the above differences, in general, if the application needs to support transactions, foreign key constraints or requires high concurrency performance, you can choose to use the InnoDB storage engine. And if the application is mainly query-based and needs fast performance when reading data, then you can choose the MyISAM storage engine. However, it should be noted that MySQL 5.5 and above have changed the storage engine to InnoDB by default, so if there is no special requirement, it is recommended to use the InnoDB storage engine.

6. What do the fields in the Explain statement result represent?

The EXPLAIN statement in MySQL is used to analyze the query statement and output the execution plan of the MySQL optimizer when the query statement is executed. The result of an EXPLAIN statement contains the following fields:

id: The unique identifier of the query block, including a sequence number and sub-sequence number.
select_type: query type, including simple query, joint query, subquery, etc.
table: Displays the name of the table involved in the query.
partitions: If the query involves a partitioned table, the partitions involved will be displayed.
type: The connection type used by the query, including constants, joins, ranges, indexes, etc.
possible_keys: Displays indexes that may be used for queries, but are not necessarily actually used.
key: The index actually used.
key_len: The length of the index field.
ref: Shows which field or constant the index is used by.
rows: Estimated number of rows to scan.
filtered: Percentage of remaining rows after filtering according to WHERE condition (0-100).
Extra: Any other notes and notes, including which temporary tables and file sorts are used, etc.

7. What is index covering?

Index Covering (Index Covering) means that during the query process, MySQL can directly use the index to meet the query needs without accessing table data. Queries with index coverage can reduce disk IO and CPU resource consumption, thus improving query performance.

When the columns involved in a query are covered by one or more indexes, it can be called an index covering query. For example, in a table with three columns (A, B, C), if you create a joint index that includes columns A and B, then the following query can use index coverage:

SELECT A, B FROM table WHERE A = 1;

In this case, MySQL can directly use the joint index to execute the query without further querying the table data. If there is no index coverage, MySQL needs to find the corresponding row in the table through the index, and then return the corresponding column value as needed, which will lead to more disk IO and CPU resource consumption.

8. How does InnoDB implement transactions?

InnoDB supports transactions by implementing ACID properties. ACID refers to Atomicity, Consistency, Isolation and Durability.

In InnoDB, a transaction consists of multiple operations, and the following conditions must be met for a successful commit:

Atomicity: All operations in a transaction are either executed successfully, or rolled back to the start state, ensuring the integrity of the transaction.
Consistency: The database must maintain a consistent state before and after transaction execution, that is, satisfy all constraints, triggers, and foreign key relationships.
Isolation: The operations between multiple concurrent transactions are isolated from each other and will not interfere with each other to ensure the correctness of data.
Persistence: Once a transaction is committed, its modifications will be permanently saved in the database, and data will not be lost even in the event of a system failure or downtime.

In order to achieve these features, InnoDB uses multi-version concurrency control (MVCC) technology, that is, each transaction operation will generate a version number to record the data status before modification, so as to avoid conflicts between concurrent transactions. At the same time, InnoDB also uses redo log and undo log to achieve transaction persistence and atomicity. The redo log is used to restore the modification to the database after the transaction is committed, and the undo log is used to undo the modification of the database by the uncommitted transaction.

9. What is the difference between B-tree and B+ tree? Why does MySQL use B+ trees?

B-tree and B+-tree are common index structures, both of which can be used to implement indexes in databases. Their differences are mainly reflected in the following aspects:

Structure: Each node of the B-tree stores both the key value and the pointer to the child node; while the internal nodes of the B+ tree only store the key value, not the data, and the leaf nodes store the key value and corresponding data, and use the linked list to store the leaf The nodes are connected.
Leaf nodes: The leaf nodes of the B-tree contain all the data, while the leaf nodes of the B+ tree only contain references to the data (usually a pointer to the location of the data on the disk), and the data itself is stored in the block below the leaf nodes.
Pointer: The nodes of the B tree contain pointers to child nodes, while the nodes of the B+ tree only contain key values and pointers to the next sibling node, which can reduce the size of each node so that the node can store more key values .

Why does MySQL use B+ trees? There are mainly the following reasons:

Disk access: B+ tree can reduce the number of disk I/O operations, because all its leaf nodes are stored together in order, and I/O efficiency can be improved by sequential reading.
Range query: Since the leaf nodes of the B+ tree are stored in order, it is very convenient to perform range query.
Memory usage: B+ trees are more suitable for use in memory than B trees, because each node needs to store less information, which means it can support larger indexes.
Update operation: The update operation of B+ tree is more efficient than that of B tree, because B+ tree only needs to modify leaf nodes, while B tree may need to modify multiple nodes.

10. What are the MySQL locks? How to understand?

MySQL's storage engine InnoDB is divided into global locks, table-level locks, and row-level locks according to the lock granularity.

Among them, global locks lock all tables in the database; table-level locks include table locks (shared read locks, exclusive write locks), metadata locks (different clients execute DML and DDL statements, locks will conflict, avoid reading and writing Inconsistency.), intent locks (you don’t need to check whether there is a row lock row by row, you only need to judge whether you can add a table lock based on the intent lock); row-level locks include row locks (shared locks/exclusive locks), Gap lock (to update a record that did not exist before, a gap lock will be added between two records), adjacent key lock (row lock + gap lock).

Row locks are divided into shared locks and exclusive locks.

Shared locks are also called: read locks. When a transaction acquires read locks on certain rows, other transactions are allowed to read these rows, but not to write them, and other transactions are not allowed to acquire exclusive locks on these rows, but read locks are allowed .

Exclusive lock is also called: write lock. When a transaction locks certain writes, other transactions are not allowed to write, but reads are allowed. No other transactions are allowed to lock any of these rows. Including write locks.

The key to understanding MySQL locks is to pay attention to the following points:

Lock conflicts: Lock conflicts occur when multiple transactions attempt to simultaneously acquire different types of locks on the same resource. For example, one transaction holds an exclusive lock during which no other transaction can acquire any locks on the resource.
Deadlock: A deadlock is a situation in which two or more transactions are waiting for each other to release the lock, resulting in the inability of all transactions to proceed. MySQL resolves deadlocks by detecting deadlocks and automatically rolling back one or more of the transactions.
Lock levels: MySQL supports different lock levels, including read uncommitted, read committed, repeatable read, and serializable. Different lock levels have different compromises in performance and data consistency, and you should choose an appropriate lock level based on specific business needs.
Concurrency: Locks are designed to increase data concurrency, but can cause performance issues if used incorrectly. Therefore, when using locks, it is necessary to balance concurrency and performance, and avoid problems such as excessive reliance on locks and long waiting times for locks.

11. What are RDB and AOF?

RDB (Redis DataBase) and AOF (Append Only File) are two different persistence methods in Redis.

RDB persistence refers to writing Redis data in memory to disk in the form of a snapshot. It will generate a complete data backup file (.rdb) at a point in time and save all the data of the Redis server at a point in time. When you need to restore data, you only need to load the backup file into memory. RDB persistence has little impact on Redis performance, and the backup file size is small, which is very suitable for scenarios such as backup and disaster recovery.

AOF persistence is to write every write operation performed by Redis into a log file, record all modification commands in an appended manner, and re-execute all write commands in the AOF file for data recovery when the service restarts. Compared with RDB, AOF persistence consumes more disk space, but it can guarantee the real-time performance of data, and is more suitable for business scenarios that require high data integrity.

At the same time, Redis also supports the combination of RDB and AOF persistence methods, which can take into account the efficiency of data backup and the guarantee of data security.

It should be noted that when using Redis persistence, it is necessary to weigh its performance and fault tolerance, and choose a suitable persistence solution according to the specific scenario.

12. Redis expired key deletion strategy?

There are two main deletion strategies for Redis expired keys: scheduled deletion and lazy deletion.

Timed deletion (Expired scanning): Redis will randomly select some keys with an expiration time set every second, check whether they are expired and delete the expired keys. This method is simple and efficient, but it may cause a large backlog of expired keys and increase system load.
Lazy deletion: Redis only checks whether the key is expired when accessing a key, and deletes it. This method can avoid excessive consumption of CPU resources, but may cause expired keys to exist in memory for a long time and occupy additional space.

In addition, Redis also adopts a hybrid expired key deletion strategy - regular deletion and lazy deletion. Specifically, Redis will add expired keys to a special dictionary, and scan this dictionary through scheduled tasks to clear expired keys. At the same time, Redis will also check whether the key is expired every time the data is accessed, and delete it immediately if it expires. This hybrid strategy can better balance the relationship between performance and system load, and improve the deletion efficiency of Redis expired keys.

It should be noted that when using Redis, the expiration time should be set reasonably according to the specific business scenario, and combined with an appropriate key deletion strategy to optimize performance and reduce memory overhead.

13. Briefly describe the transaction implementation of redis?

Redis implements simple transaction functions through instructions such as MULTI, EXEC, and WATCH, and supports packaging multiple commands into an atomic operation for execution.

Specifically, the implementation of Redis transactions is divided into the following steps:

Start a transaction with the MULTI command.
Execute the Redis commands that need to be executed in sequence. These commands will not be executed directly, but will be temporarily stored in a queue for subsequent batch execution.
Use the EXEC command to submit the transaction, and Redis executes all the commands in the queue. If any command fails, the entire transaction will be rolled back (that is, cancelled).
If during the execution of a transaction, a key is found to have been modified by another client, the transaction is automatically terminated and rolled back. To this end, you can use the WATCH command to monitor certain keys, detect whether these keys have been modified before the EXEC command is executed, and roll back the transaction if so.

It should be noted that Redis's transaction is an optimistic locking mechanism, which does not lock individual commands, so there may be concurrency conflicts. At the same time, since Redis transactions only package multiple commands, it cannot guarantee the full realization of ACID features. It is necessary to choose whether to use Redis transactions according to specific business scenarios.

14. What is the core principle of redis master-slave replication?

Redis master-slave replication is a common high-availability solution, which can synchronize data from one Redis server to multiple Redis slave servers, improving the fault tolerance and scalability of the system.

Its core principles are as follows:

First synchronization: When a Redis slave server starts, it sends a SYNC command to the master server. After the master server receives the SYNC command, it will start to execute the BGSAVE command, write the data snapshot to the disk, and transfer the snapshot file to the slave server. After that, the master server records all new write operations and sends these write operations to the slave server, and the slave server updates its own data according to the received write operations.
Incremental synchronization: After the first synchronization is completed, the master server will record each write operation and push these write operations to the slave server through the asynchronous replication mechanism. As soon as it receives a write operation from the server, it executes the operation and updates its own data.
Message confirmation: In order to ensure synchronization efficiency, the master server will not wait for the slave server to complete the execution of the write operation. Instead, it will implement message confirmation through the ACK mechanism. When the slave server receives a data packet, it will return an ACK response, telling the master server that the data packet has been successfully received.

The core principle of Redis master-slave replication is to record and asynchronously transmit the write operations of the master server to the slave server, and the slave server performs these write operations to update its own data. It should be noted that there may be delayed synchronization in master-slave replication, such as network delays, excessive load on the master server, and so on. Therefore, in actual use, it is necessary to select an appropriate master-slave replication scheme based on specific business needs and system performance, and to monitor and maintain it regularly.

15. What data structures does Redis have? Application scenario?

Redis supports five data types: string (string), hash (hash), list (list), set (set) and zset (sorted set: ordered set).

Redis supports a variety of data types, and each data type has its specific application scenarios. The following are the data types supported by Redis and their application scenarios:

String (String): used to store simple values such as strings, integers, and floating-point numbers, and is often used in scenarios such as caches, counters, and distributed locks.

List (List): It is used to store a series of ordered string values, supports inserting and popping elements from both ends, and is often used in scenarios such as message queues and task queues.

Set (Set): used to store a set of unordered and unique string values, supports operations such as intersection, union, and difference, and is often used in scenarios such as counting the number of independent visitors and user tags.

Sorted Set (Sorted Set): Similar to a set, but each member is associated with a score, which can be sorted according to the score, supports range query and ranking operations, and is often used in scenarios such as leaderboards and scoring systems.

Hash: An unordered hash table used to store key-value pairs, supports single field read and write and batch read and write operations, and is often used in scenarios such as storing objects and user information.

According to the needs of specific scenarios, you can choose an appropriate data type to store and process data. In actual use, multiple data types can also be combined for complex data processing, such as using List and Hash to store multiple objects, and quickly searching and sorting through Set and Sorted Set.

1. Geospatial (geographic location)

The Geospatial data type can store geographic coordinates and supports operations such as distance calculation, range query, and proximity query on locations. This data type is often used to implement scenarios such as map applications, nearby people functions, and location-based recommendation systems.

2.HyperLogLog (base)

The HyperLogLog data type is used to count the cardinality of a large amount of data (that is, the number of unique elements), and has the characteristics of high space efficiency and adjustable precision. This data type is often used to count the number of unique website visitors, user retention rate, advertising click rate and other scenarios.

3.Bitmaps (bitmap)

The Bitmaps data type can store binary bit arrays and supports logical operations on bit arrays, such as AND, OR, NOT and other operations. This data type is often used in scenarios such as counting user activity and analyzing user behavior.

for example:

Geospatial application scenario: A restaurant ordering platform needs to query nearby takeaway stores. You can store the latitude and longitude information of each store in the Geospatial data type of Redis, and use the ZRANGEBYSCORE command to query stores within a certain distance around the specified location.

HyperLogLog application scenario: A social network needs to count how many unique visitors it has every day. You can use the HyperLogLog data type of Redis to estimate the number of unique visitors to the website, so as to evaluate the traffic quality of the website.

Bitmaps application scenario: An e-commerce platform needs to count the page visits of each user over a period of time, and can store the bit array corresponding to each user in the Bitmaps data type of Redis, and use bit operation commands for data analysis, for example The BITCOUNT command calculates the number of active users, the BITOP command calculates the number of new users in two time periods, etc.

16. How is the bottom layer of Redis distributed lock implemented?

The implementation idea of Redis distributed lock is usually based on the atomic operation and expiration time of the SET command. Specifically, the underlying implementation of Redis distributed locks includes the following steps:

The client uses the SETNX command to try to acquire a specified key (that is, the lock). If it returns 1, it means the acquisition is successful, and the client has acquired the lock; if it returns 0, it means the acquisition fails, which means other clients already hold the lock.
If the client successfully acquires the lock, an expiration time can be set for the lock to prevent deadlocks caused by program exceptions and the like.
When the client completes the task, it needs to use the DEL command to delete the lock in time to release resources.

It should be noted that in a distributed environment, the implementation of Redis distributed locks also needs to solve the competition problem between multiple clients. For example, when multiple clients try to acquire the same lock at the same time, how to ensure that there is only one client Able to successfully acquire locks etc. To this end, technologies such as the Redlock algorithm can be used to increase the reliability and security of the lock.

In addition, in order to avoid the deadlock problem caused by the lock time being too long or the lock not being released, you should carefully consider when designing the lock expiration time, and set it reasonably according to the actual business scenario.

17. Redis cluster strategy?

Redis cluster is a common high-availability solution, which can distribute data to multiple nodes to improve the fault tolerance and scalability of the system. The main strategies of Redis cluster include the following aspects:

Data sharding: The Redis cluster uses hash slots for data sharding, and evenly distributes all keys to 16384 hash slots. Each node is responsible for maintaining some of the hash slots to realize distributed storage and query of data.
Inter-node communication: Redis cluster uses the Gossip protocol for inter-node communication to realize automatic discovery and update of node status. Through the Gossip protocol, each node periodically sends messages to other nodes to maintain the latest cluster status information.
High availability guarantee: For each hash slot, the Redis cluster will replicate it to multiple nodes to ensure that the data can still be accessed when some nodes go down. At the same time, the Redis cluster also supports the failover function, that is, when a master node goes down, a slave node will be selected as the new master node according to the preset rules, and the hash slot will be reassigned.
Client routing: When a client accesses a Redis cluster, it needs to route the request to the correct node first. The Redis cluster adopts the method of client-side sharding, that is, the hash slot is allocated to the client, and the client decides which node to send the request to.

It should be noted that when using a Redis cluster, an appropriate deployment strategy should be selected according to specific business needs, and the cluster should be monitored and maintained to ensure data reliability and high availability.

18. Cache penetration, cache avalanche, cache breakdown?

① Cache penetration: A large number of requests for keys that do not exist at all, requests for resources that do not exist at all (DB itself does not exist, and Redis does not exist)

solution:

Use Bloom filter to cache null values
(Use BitMap as a Bloom filter, put all currently accessible resources into the Bloom filter through a simple mapping relationship (hash calculation), when a request comes When using the Bloom filter, the Bloom filter will be judged first, if there is, then it will be released, otherwise it will be directly intercepted)
② Cache avalanche: a large number of keys in redis collectively expire (details below)

③ Cache breakdown: A hotspot key in redis expires (a large number of users access the hotspot key, but the hotspot key expires)

Cache avalanche and cache penetration solutions:

Set the hot words in advance, adjust the key duration,
adjust in real time, monitor which data is popular data, and adjust the key expiration time in real time.
Use the lock mechanism (when the hot key expires, use the lock mechanism to prevent a large number of requests from directly opening in DB)

19. How do Redis and MySQL ensure data consistency?

Redis and MySQL have the following common guarantee methods in terms of data consistency:

Implemented through the application: When performing read and write operations between Redis and MySQL, you can ensure data consistency by adding business logic to the application. For example, when writing data, first write the data to MySQL, and then synchronize the data to Redis through mechanisms such as message queues.
Automatic update through cache: When writing data, you can first write the data into the Redis cache, and then asynchronously write the data into the MySQL database. If the Redis cache fails, the latest data in MySQL can be synchronized back to Redis through the cache automatic update mechanism.
Achieved through read-write separation: separate read and write operations to different nodes, so that MySQL master-slave replication or PXC and other technologies can be used to achieve data consistency. The specific method is that when writing data, only write to the MySQL master node, and the MySQL master node is responsible for synchronizing the update operation to the slave nodes; while reading data, you can directly read data from the Redis cache, Thereby improving query efficiency.

It should be noted that when using Redis and MySQL, an appropriate strategy for ensuring data consistency should be selected according to the specific business scenario, and tested and verified in actual use to ensure the correctness and reliability of the data.

20. What is the persistence mechanism of Redis?

Redis has two persistence mechanisms:

RDB Persistence: Write Redis data in memory to the hard disk in the form of a snapshot, which can be triggered manually or automatically. RDB files are compact and compressed binary formats, suitable for situations with high data integrity requirements and large data volumes.
AOF persistence: Append each write command executed by Redis to the log file to achieve data persistence. The AOF file is a readable text format, and different synchronization strategies can be configured to balance data security and performance. AOF persistence is more robust than RDB persistence, but requires more disk space and IO operations.

21. Why is redis single thread so fast?

The reason why Redis single-thread can achieve high performance is mainly due to the following reasons:

Pure memory operation: Redis data is stored in memory, and memory access speed is faster than disk access speed, enabling Redis to achieve efficient read and write operations.
Non-blocking I/O model: Redis uses a non-blocking I/O model, which can process requests from other clients before processing a client request. This model enables Redis to efficiently handle a large number of concurrent connections.
Simple data structure: Redis has a variety of built-in data structures, such as strings, hashes, lists, sets, and ordered sets. These data structures are very simple and efficient to operate.
Single thread avoids lock competition: In a multi-thread environment, multiple threads need to lock and unlock when accessing shared resources at the same time, and lock competition will affect the performance of the program and even lead to deadlock. Redis's single thread avoids lock competition, reduces thread context switching overhead, and improves program execution efficiency.

Combining the advantages of the above aspects, Redis single-thread can achieve a high performance level.

22. Briefly describe the transaction implementation of redis?

The transaction implementation of Redis mainly includes the following three steps:

Transaction start: set the state of the client to the transaction state through the MULTI command, and Redis will cache all subsequent commands sent by the client.
Command enqueuing: All commands sent in the transaction state will not be executed immediately, but will be added to a queue, and all commands will not be executed at one time until the EXEC command is executed.
Transaction submission: Trigger the transaction submission through the EXEC command, and Redis will traverse the previously cached command queue, execute them sequentially, and return the result of each command execution.

If an error occurs during transaction execution, such as a command execution failure, Redis will automatically roll back the transaction and undo all executed commands. After the transaction is committed, Redis cannot roll back. It should be noted that Redis transactions can be nested, but cannot span multiple databases.

Redis's transaction implementation is not a strict ACID (atomicity, consistency, isolation, and durability) model, because the transaction does not make any changes to the data on the Redis server during the command enqueuing phase, so it cannot guarantee the transaction isolation and persistence

23. CAP theory? BASE theory?

CAP theory and BASE theory are two important concepts in the field of distributed systems.

CAP theory: CAP theory points out that in a distributed system, the three characteristics of consistency (Consistency), availability (Availability) and partition tolerance (Partition Tolerance) can only satisfy at most two of them at the same time. That is, we need to make a trade-off between consistency and availability in the face of network partitions.
BASE theory: BASE theory is an extension and supplement to the CAP theory. It believes that in a distributed system, due to network partitions, node failures, etc., strong consistency cannot be guaranteed, so an availability-based software design strategy should be adopted. BASE means:
- Basically Available: the system must ensure basic availability;
- Soft state: The system allows for an intermediate state that does not affect the overall availability of the system;
- Eventually consistent (Eventually consistent): The system eventually achieves consistency, but does not require real-time guarantee of strong consistency.

BASE theory requires us to prioritize availability and partition tolerance in distributed systems, and to accept a certain degree of data inconsistency in exchange for higher availability and performance.

24. What is RPC?

RPC (Remote Procedure Call) is a technology for communication between distributed systems, which allows a computer program to call a subroutine or service in another computer program, just like a local call.

The basic principle of RPC is to encapsulate remote method calls into local method calls. When the client calls a remote service, the client interface will package the parameters into a message and send it to the server. After receiving the message, the server parses the parameters and performs the corresponding operation, and then returns the result to the client.

The advantage of RPC is that it can shield complex network communication details, making it as easy for clients to call remote services as calling local services, improving development efficiency and maintainability. Common RPC frameworks include Dubbo, gRPC, Thrift, etc.

25. What are the data consistency models?

In a distributed system, in order to ensure data consistency, the commonly used data consistency models are as follows:

Strong Consistency: Under the strong consistency model, all nodes can see the same version of data and must maintain data consistency at all times. For example, in the ACID transaction model, data update operations must be atomic. For other nodes, these operations are either all completed or all fail.
Sequential consistency (Sequential Consistency): Under the sequential consistency model, the order of operations of all nodes must be consistent, that is, if a node executes an operation first, other nodes must also execute the corresponding operation first .
Causal consistency (Causal Consistency): Under the causal consistency model, if node A notifies node B after updating a certain data, node B must update according to the update order of node A.
Eventual Consistency: Eventual Consistency means that at a given point in time, the data of all nodes may be in an inconsistent state, but after a period of synchronization, they will eventually reach a consistent state. The eventual consistency model is suitable for scenarios where consistency requirements are not high and read operations are more frequent than write operations, such as caches in Internet applications.

26. What is distributed ID? What are the solutions?

In a distributed system, it is usually necessary to generate a unique ID for each node to distinguish different nodes and avoid ID conflicts. This process is called distributed ID generation.

Common distributed ID generation solutions are as follows:

Based on UUID: Use the UUID (Universally Unique Identifier) algorithm to generate a globally unique ID, which can ensure that the generated ID will not be repeated among different nodes. However, the ID generated by UUID is relatively long, which is not suitable for scenarios such as database indexes or URL parameters.
Based on the Snowflake algorithm: The Snowflake algorithm is an open source algorithm for generating distributed IDs from Twitter. It divides a 64-bit binary number into different parts to achieve monotonically increasing and global uniqueness. The algorithm can be easily deployed and has good performance.
Redis-based auto-increment counter: Generate a globally unique ID by storing an auto-increment counter in Redis. Due to the high performance and reliability of Redis, this solution is very suitable for scenarios with high concurrency and low latency.
Zookeeper-based auto-increment ID: Use the atomic operations provided by Zookeeper to realize ID auto-increment in a distributed environment. This scheme can guarantee the global uniqueness and incrementality of the ID, but due to the high latency and low throughput of Zookeeper, it is not suitable for high concurrency scenarios.

27. What are the usage scenarios of distributed locks? What implementation options are there?

Distributed locks are a technology used to protect shared resources. There are many usage scenarios in distributed systems, such as:

Database operation: prevent multiple nodes from modifying the same record in the database at the same time, causing data inconsistency.
Distributed task scheduling: Ensure that only one node executes scheduled tasks at the same time to avoid repeated execution.
Distributed current limiting: limit the number of concurrent requests to avoid service overload and crash.
Cache penetration protection: During cache invalidation, prevent a large number of requests from falling on the back-end data source.

Common implementations of distributed locks are as follows:

Redis-based distributed locks: Distributed locks are implemented through Redis' atomic operations (such as SETNX). This solution has the advantages of high reliability, high performance, and easy deployment, and is one of the more commonly used distributed lock implementation solutions.
Zookeeper-based distributed lock: use the temporary node and watch mechanism provided by Zookeeper to realize the function of distributed lock. This scheme has good reliability and scalability, but it is cumbersome to implement.
Database-based distributed lock: Use the row-level lock or pessimistic lock of the database to realize the function of distributed lock. This solution is simple to implement, but has poor concurrency performance and is not suitable for high-concurrency scenarios.

28. What is the ZAB protocol?

The ZAB (ZooKeeper Atomic Broadcast) protocol is the core of Zookeeper's high availability. Zookeeper guarantees the data consistency between multiple replicas in the cluster by implementing the ZAB protocol, and can automatically switch to the standby node when the master node is down.

Based on the idea of Atomic Broadcast, the ZAB protocol serializes all write operations into a globally unique sequence, and then broadcasts according to this sequence. When any server in the Zookeeper cluster receives a write request, it will initiate a broadcast request to other servers and wait for responses from most nodes. Only when a majority of nodes respond successfully will the write operation be committed.

The ZAB protocol is divided into two phases:

Leader election: When the Leader node in the Zookeeper cluster goes down or the network is interrupted, a new Leader node needs to be re-elected. The ZAB protocol uses a voting mechanism similar to the Paxos algorithm to elect a new Leader.
Data synchronization: When a new leader is elected, the data that has been submitted but not yet synchronized to all nodes needs to be synchronized. The ZAB protocol uses a method similar to log replication to record the write operation sequence of the Leader node and send a synchronization request to the Follower node.

The ZAB protocol can guarantee that at any time, all nodes in the Zookeeper cluster have the same data status, thus achieving high availability and data consistency.

29. Why can Zookeeper be used as a registration center?

Zookeeper can be used as a registration center mainly because it has the following characteristics:

Data consistency: Zookeeper uses the ZAB protocol to ensure data consistency. All clients can read the same service list information to avoid inconsistencies in service lists between different clients.
High availability: Zookeeper supports multi-node cluster deployment. When a node goes down, other nodes in the cluster can automatically take over, ensuring high availability of services.
Monitoring mechanism: Zookeeper provides a Watcher mechanism. When the services in the registry change, the client can receive notifications in time, so that service registration and discovery can be processed more efficiently.
Node hierarchy: Zookeeper stores data in a hierarchical structure similar to a file system, which can facilitate service classification and management.

The above characteristics make Zookeeper very suitable as a service registry in a distributed system, which can meet the needs of service governance and has good performance and scalability.

30. What is the leader election process in Zookeeper?

In the Zookeeper cluster, only the Leader node can process the client's write request, while the Follower node can only process the client's read request and forward the write request to the Leader node. When the Leader node goes down or loses contact with the Follower node, a new Leader node needs to be re-elected. The specific process is as follows:

The node enters the LOOKING state: When a node has just joined the cluster or the current Leader node is down, the node will enter the LOOKING state.
Initiate a vote: Nodes in the LOOKING state will send a voting request (called proposals) to other nodes in the cluster and wait for responses from other nodes.
Election algorithm: When a node receives the approval votes of the majority of nodes (that is, more than half of the nodes), the node is selected as the new Leader node. In the election algorithm, each node has a zxid (transaction ID) to represent the number of the proposal proposed by itself. During the election process, the value of zxid is used as the priority. If two nodes have the same zxid, compare their myid (node id).
Synchronize data: After a new Leader node is selected, it is necessary to synchronize the data that has been submitted but not yet synchronized to all Follower nodes. This process is implemented using the ZAB protocol.
Client connection: After data synchronization is completed, the new Leader node starts to process the client's write request and synchronizes the request to other nodes.

In short, Zookeeper's leader election process ensures the uniqueness of the Leader node through the election algorithm, zxid and myid, and uses the ZAB protocol to ensure data consistency.

31. How is data synchronized between nodes in the Zookeeper cluster?

The nodes in the Zookeeper cluster synchronize data through the ZAB protocol. The specific process is as follows:

The Leader node records the write request to the local disk and assigns a globally unique transaction ID (zxid).
The Leader node serializes the write request and zxid into a Proposal object and sends it to all Follower nodes.
After the Follower node receives the Proposal object, it will store it on its own disk and reply an ACK message to the Leader node.
After receiving ACK messages from most Follower nodes, the Leader node considers that the Proposal has been submitted. Then the Leader node will broadcast the submitted Proposal to all Follower nodes, and wait for the confirmation message from the Follower nodes.
After the Follower node receives the submitted Proposal, it will perform a data update operation and reply a COMMIT message to the Leader node.
After receiving the COMMIT messages from most Follower nodes, the Leader node considers that the Proposal has been submitted, and sends ACK confirmation messages to all Follower nodes.
After the Follower node receives the ACK confirmation message, it deletes the Proposal from the memory, indicating that the processing has been completed.
The Follower node periodically synchronizes the latest data from the Leader node. This process is implemented using SNAP and DIFF modes. When the Follower node starts, it will connect to the Leader node and request the latest SNAP snapshot, and then use the DIFF mode to synchronize the changed part.

In general, the ZAB protocol is used between the nodes in the Zookeeper cluster to ensure data consistency. The Leader node is responsible for broadcasting Proposal and ACK/COMMIT confirmation messages to all Follower nodes, and considers that the Proposal has been submitted only after most Follower nodes have confirmed it. Follower nodes are responsible for receiving Proposal and ACK/COMMIT messages, and performing data update operations based on their contents.

32. Which load balancing strategies does Dubbo support?

Dubbo supports the following seven load balancing strategies:

Random (Random): Randomly select an available service provider.
Round Robin: Select the available service providers in order and cycle through them.
Least Active: Select the service provider with the least number of active requests to call, which can make the distribution of requests more even and reduce the pressure on the service provider.
Consistent Hash: Calculate the requested parameters through a hash function to obtain a hash value, and then select the server provider closest to the hash value in the server list to call. This method can effectively avoid the problem of data skew caused by the service provider going offline.
Weight (Weighted Random/Balanced): Randomly select an available service provider according to the configured weight. The higher the weight, the greater the probability of being selected.
Weighted Round Robin: Select available service providers in turn according to the configured weight, and the number of selections is calculated according to the weight ratio.
Smooth Weighted Round Robin (Smooth Weighted Round Robin): The weighted round robin algorithm is improved, and the average load of the service provider is more balanced by dynamically adjusting the weight.

The above are the load balancing strategies supported by Dubbo. It is necessary to select an appropriate load balancing strategy according to actual needs.

33. How does Dubbo complete the service export?

Dubbo completes the process of exporting services as follows:

When the service provider starts, it will load the ServiceConfig Bean provided by Dubbo through the Spring container, and the ServiceConfig Bean contains the relevant configuration information of the service.
ServiceConfig will encapsulate the service as an Invoker object and bind the Invoker object to the URL address. This URL address includes information such as protocol, IP address, port number, service name, method name, etc., and is used to uniquely identify a service.
ServiceConfig will register the Invoker object to the Registry center, which is Dubbo's registration center and is responsible for maintaining the relationship between service providers and consumers.
When a consumer needs to call the service, the consumer will obtain the address list of the service provider from the Registry center, and select a suitable service provider to call.
The consumer uses the Proxy proxy mechanism to initiate a request to the selected service provider.
After receiving the request, the service provider parses the request into an Invocation object and hands it to the corresponding Invoker for processing.
Invoker invokes the specific service implementation logic based on the information contained in the Invocation, and returns the execution result to the service provider.
The service provider serializes the execution result into a byte stream and returns it to the consumer.

In general, Dubbo's service export process involves the interaction between multiple components, including ServiceConfig, Invoker, Registry, Proxy, etc. Dubbo completes the export and registration of the service by encapsulating the service as an Invoker object, binding the Invoker object to the URL address, and registering the Invoker object to the Registry center. The consumer can obtain the address list of the service provider from the Registry center, and initiate a request to call the service through the proxy mechanism.

34. How does Dubbo complete the service introduction?

Dubbo completes the service introduction process as follows:

When the consumer starts, it will load the ReferenceConfig Bean provided by Dubbo through the Spring container, and the ReferenceConfig Bean contains the relevant configuration information of the service.
ReferenceConfig will obtain a list of available service provider addresses from the Registry Center and select a suitable service provider. If you are using direct connection, skip this step.
ReferenceConfig will encapsulate the service as an Invoker object and bind the Invoker object to the URL address. This URL address includes information such as protocol, IP address, port number, service name, method name, etc., and is used to uniquely identify a service.
ReferenceConfig generates the corresponding Service interface implementation class through the proxy mechanism, and calling the implementation class can initiate a call to the remote service.
In the specific call process, the Service interface implementation class will convert the method call into an Invocation object, and pass the Invocation object to the Invoker for processing.
Invoker initiates a request to the service provider according to the information contained in the Invocation, and returns the execution result to the Service interface implementation class.
The Service interface implementation class deserializes the execution result into the corresponding type and returns it to the caller.

In general, Dubbo's service introduction process also involves the interaction between multiple components, including ReferenceConfig, Invoker, Registry, Proxy, etc. Dubbo encapsulates the service as an Invoker object, binds the Invoker object to the URL address, and uses the proxy mechanism to generate the Service interface implementation class to complete the introduction and invocation of the service. The service consumer can initiate a request through the Service interface implementation class to complete the call to the remote service.

35. What is the architecture design of Dubbo?

Dubbo's architecture design is based on the distributed service framework, including three core components: Provider, Consumer and Registry Center. Their functions in the Dubbo system are as follows:

Provider: Responsible for packaging the specific service implementation logic into a jar package and publishing it to the Dubbo service. The provider encapsulates the service as an Invoker object, and binds the Invoker object to the URL address to export the service.
Consumer: Responsible for importing services from Dubbo services and calling remote service interfaces. The consumer generates the proxy class of the service interface through the proxy mechanism, converts the method call into an Invocation object, and passes the Invocation object to the Invoker for processing, and finally deserializes the execution result into the corresponding type.
Registration center: Dubbo's registration center is responsible for maintaining the relationship between service providers and consumers. The provider registers the Invoker object to the Registry center, and the consumer obtains a list of available service provider addresses from the Registry center, and selects a suitable service provider to call. Dubbo's registration center supports multiple protocols, including Zookeeper, Redis, Multicast, etc.

In addition to the above three core components, Dubbo also has some auxiliary components, including:

Monitoring Center: Dubbo's monitoring center is responsible for collecting runtime data of service providers and consumers, such as request volume, response time, error rate, etc. Dubbo's monitoring center supports multiple protocols, including Zookeeper, Redis, Simple, etc.
Configuration Center: Dubbo's configuration center is responsible for managing service configuration information, such as timeout, retry times, load balancing strategy, etc. Dubbo's configuration center supports multiple protocols, including Zookeeper, Properties, File, etc.
RPC communication framework: Dubbo uses its own RPC communication framework for remote calls and data transmission, which has the characteristics of high performance and low latency.

In general, Dubbo's architecture design is based on the distributed service framework, and through the cooperation of core components such as providers, consumers, and registration centers, functions such as service export, import, and relationship maintenance are realized.

36. What are the load balancing algorithms?

Common load balancing algorithms include:

Round Robin: Each request is allocated to different servers in sequence, and the cycle repeats.
Least Connections Scheduling Algorithm (Least Connections): Allocate new requests to the server with the fewest current connections.
IP Hash Scheduling Algorithm (IP Hash): Calculate the hash value according to the client IP address, then match the hash value with the server list, and select the matching server to process the request.
Weighted Round Robin: Set different weights for different servers, and allocate requests according to the weight ratio.
Weighted Least Connections Scheduling Algorithm (Weighted Least Connections): Set different weights for different servers, and assign the request to the server with the smallest product of the current number of connections and the weight.
Random Scheduling Algorithm (Random): Randomly select a server to process the request.
Dynamic Response Time Scheduling Algorithm (Dynamic Response Time): Adjust the weight of the server according to the average response time of the server to achieve dynamic load balancing.

37. Under the distributed architecture, what is the solution for Session sharing?

Under the distributed architecture, the Session sharing scheme includes:

Use sticky session (Sticky Session): Bind the user's request to a certain server, so that all the user's requests are processed by the same server. This solution can be implemented with a load balancer.
Use a database or cache to store Session data: Store Session data in a database or cache shared by all servers, and different servers read and update Session data by accessing the database or cache.
Use unified authentication service (Single Sign-On, referred to as SSO): store the user's authentication and authorization information in a central authentication service, and realize cross-domain authentication and authentication through standardized protocols (such as OAuth, OpenID Connect, etc.). authorized. This solution can realize Session sharing of cross-domain multi-application systems.
Use Token-based authentication mechanism: save the user's identity authentication information in a Token, which can be passed to the back-end server for verification through HTTP request headers, URL parameters, etc. shared.

38. How to achieve interface idempotency?

To achieve the idempotency of the interface, the following measures can be taken:

Use Unique Identifiers: Assign a unique identifier (such as a UUID) to each request and include it with each request. After the server receives the request, it checks to see if the identifier already exists, and if so returns the previously saved response instead of performing the same operation.
Limit repeated requests: Set the repeated request limit on the client or server, that is, only allow the same request to be sent a certain number of times, and the request will be rejected if the threshold is exceeded.
Use optimistic lock: use the optimistic lock mechanism in database operations, that is, compare the version numbers before and after the modification when updating records. If the version numbers are the same, it means that the database records have not been modified by other users and can be updated, otherwise an error will be reported.
Return an idempotent response: Include information such as an idempotent token or version number in the response so that the client can identify and handle duplicate responses.
Avoid operations with side effects: For requests that cause state changes, such as insert, update, delete, etc., confirm before execution to avoid data corruption caused by misoperation.

39. Briefly describe Zookeeper's naming service, configuration management, and cluster management?

Zookeeper is a distributed coordination service that provides functions such as naming services, configuration management, and cluster management.

Naming service: Zookeeper implements naming service by maintaining a tree-structured namespace. Clients can create, delete, and move nodes on Zookeeper, and perform read and write operations on nodes. These nodes can be used to store various information, such as server address, system configuration items, etc. By using the Watcher mechanism, the client can monitor changes in nodes and obtain the latest information in a timely manner.
Configuration management: Zookeeper can be used as a manager for some shared configuration in the cluster. For example, a Hadoop cluster can use Zookeeper to store configuration information for all nodes, as well as failover status information. When a node needs to change the configuration, it can send the changes to Zookeeper and notify other nodes to update the configuration. This way, the entire cluster can maintain a consistent configuration state.
Cluster management: Zookeeper also provides a variety of cluster management capabilities. For example, it can select a leader through an election algorithm, and the leader manages the state of the entire cluster. In addition, Zookeeper also provides the function of mutual exclusion locks, which can help realize synchronization operations in a distributed environment. For example, in a distributed lock, only the node that acquires the lock can perform an operation, and other nodes need to wait for the lock to be released before attempting to acquire the lock.

40. Tell me about the watch mechanism of Zookeeper?

The watch mechanism of Zookeeper is an event monitoring mechanism, which can help the client monitor the state changes of nodes on Zookeeper.

When a client connects to Zookeeper and registers a watcher, the watcher will be registered in the watcher list of the corresponding node on the server side. When the state of the node changes (for example, node data or child nodes change, node is deleted, etc.), the server will send notification information to all clients registered with the node observer.

When the client receives the notification, it needs to get the node status from Zookeeper again, and check the reason for triggering the watcher, such as the node is deleted, the node data changes, etc. Handle according to different situations, such as re-registering watcher, updating local cache, etc.

It should be noted that Zookeeper's watch mechanism is one-off. That is, when the watcher is triggered, it will be removed from the observer list of the node. If you need to continue monitoring, you need to re-register the watcher.

At the same time, due to the delay and uncertainty of the watch mechanism, it cannot be relied on to ensure the reliability and consistency of the distributed system, and it needs to be supported by other mechanisms.

41. What is the difference between Zookeeper and eruka?

Both Zookeeper and Eureka are distributed coordination services, but their design goals and functional focus are different:

Design goal: Zookeeper is mainly used to build a reliable, high-performance distributed system, providing naming services, configuration management, cluster management and other functions, aiming to solve the coordination problem in a distributed environment; while Eureka is a network developed by Netflix. A service registration and discovery framework designed to provide a service governance solution for applications based on microservice architecture.
Function focus: The focus of Zookeeper is to provide high-throughput, low-latency, high-reliability coordination services, support node elections, distributed locks, data publishing and subscription functions, and provide Java client APIs and command-line tools. Eureka focuses on service registration and discovery, checks the status of service instances through the heartbeat mechanism, and provides REST-style HTTP API and Java client libraries.
Application scenarios: Zookeeper is usually used in distributed systems such as Hadoop and Kafka for coordination, synchronization, configuration, etc.; Eureka is usually used in smaller microservice architecture applications for service registration, Discovery, load balancing, and more.

In general, both Zookeeper and Eureka are excellent distributed coordination services, with differentiated advantages in different scenarios.

42. How to solve the unique primary key after storage split?

After the storage is split, how to solve the unique primary key problem is a key issue that needs to be considered. Here are some common solutions:

Horizontal split + UUID: Use UUID instead of the self-increasing ID in the database as the unique primary key to ensure that the ID generated on each node after horizontal split is unique.
Vertical splitting + distributed transactions: Split the master table and slave table to different nodes through vertical splitting, and then use distributed transactions to ensure data consistency among multiple nodes.
Database sharding + distributed ID generator: shard the database to different nodes, then use the distributed ID generator to generate unique IDs, and ensure that the IDs generated on all nodes are unique.
Distributed lock: When operating the database, first obtain a globally unique lock to ensure that only one node can modify the database at the same time to avoid duplicate primary keys.
Application layer solution: implement the uniqueness check at the application layer instead of at the database level. For example, use caches such as Redis to save IDs that have already been used. Every time a new ID is generated, first check whether it exists in the cache, and regenerate it if it exists.

Each of the above solutions has its advantages and disadvantages. Which one to choose needs to comprehensively consider business requirements, performance, reliability and other factors.

43. What is the snowflake algorithm? Snowflake algorithm principle?

The Snowflake algorithm (Snowflake) is an algorithm for generating unique IDs, which can generate globally unique and ordered IDs in a distributed system.

The principle of the snowflake algorithm is as follows:

A 64-bit binary number, in which the first bit is the sign bit, which is always 0; the next 41 bits are millisecond-level timestamps, which are used to record the time when the ID is generated; and then 10 bits are used to record the machine ID, 1024 nodes can be deployed; the last is 12 bits, which are used to record different ID serial numbers generated within the same millisecond.
A self-incrementing sequence number counter is maintained on each node to ensure that the ID sequence numbers generated within the same millisecond are incremented.
The current timestamp, node ID and serial number are spliced into a 64-bit binary number according to certain rules, and converted into a decimal number as the final ID value.

The advantage of the snowflake algorithm is that it generates IDs quickly, takes up less space, has high reliability, and is well ordered. Since the ID structure contains timestamp information, it can be sorted by time, which is convenient for storage and query. At the same time, due to the uniqueness and order of the ID, it is also convenient for data fragmentation and load balancing in distributed systems.

It should be noted that the snowflake algorithm has relatively high requirements on the clock. If the system clock is dialed back, duplicate IDs will be generated. Therefore, it is necessary to ensure the synchronization and accuracy of the system clock when using the snowflake algorithm. In addition, if the number of nodes increases, node IDs may be duplicated, so you need to be careful when deploying nodes.

44. What are the commonly used components in Spring Cloud and what are their functions?

Spring Cloud is a series of tools built on Spring Boot for developing and deploying applications for distributed systems. The following are commonly used components in Spring Cloud and their functions:

Eureka: A REST-based service registration and discovery component, used to implement registration and discovery in the microservice architecture.
Ribbon: Client load balancer, which implements load balancing inside the client, and can choose different algorithms to achieve load balancing.
Feign: An annotation-based declarative REST client that simplifies the code for clients to call REST APIs.
Hystrix: A fault-tolerant management tool for dealing with latency and fault-tolerance issues in distributed systems. Provides functions such as circuit breaker mode, thread pool isolation, and request caching.
Zuul: gateway service, which provides functions such as dynamic routing and access filtering, and can realize a unified API gateway.
Config: Distributed configuration center, which realizes centralized management of application configuration files and supports multiple back-end storage.
Bus: A message bus that connects nodes in a distributed system to implement change notification and dissemination of configuration and status information.
Stream: A message-driven framework, based on message processing implemented by RabbitMQ or Kafka, provides a simplified way to process event stream data.
Security: The security framework provides an OAuth2-based authentication and authorization mechanism to protect data security in the microservice architecture.

The above are commonly used components in Spring Cloud, and they can play different roles in different scenarios. By integrating these components, functions such as service registration and discovery, load balancing, fault-tolerant management, and API gateway can be realized in the distributed system, which improves development efficiency and system reliability.

45. How to avoid cache penetration, cache avalanche, and cache breakdown?

① Cache penetration: A large number of requests for keys that do not exist at all, requests for resources that do not exist at all (DB itself does not exist, and Redis does not exist)

solution:

③ Cache breakdown: A hotspot key in redis expires (a large number of users access the hotspot key, but the hotspot key expires)

Cache avalanche and cache penetration solutions:

46. What are the commonly used caching schemes in distributed systems?

In a distributed system, the commonly used caching schemes are as follows:

Memcached: A memory-based distributed key-value pair storage system that supports multi-threaded concurrent access and provides client libraries in various languages. Memcached is suitable for storing simple data types, such as strings and numbers, and provides fast read and write performance.
Redis: A high-performance memory-based key-value pair storage database that supports multiple complex data types, such as hash tables, lists, sets, etc., and supports data persistence. Redis has high-speed read and write performance and a reliable data protection mechanism, which is suitable for fast reading and frequent data writing scenarios.
Ehcache: A caching framework developed by Java that supports two modes of local caching and distributed caching. Ehcache achieves very high read and write performance by caching inside the JVM, and also supports multiple caching strategies and expiration mechanisms.
Hazelcast: An open source distributed data cache and computing platform that provides distributed cache, distributed Map, distributed Queue and other data structure support, and supports dynamic scaling and high availability features.
Caffeine: A local cache based on Java 8, the goal is to provide the best cache performance and complete cache features. Caffeine adopts a new caching algorithm, supports high-speed reading and writing, asynchronous loading and expiration mechanism, and is suitable for high-concurrency and low-latency scenarios.

It is necessary to select an appropriate caching solution based on actual business needs and system architecture, and configure, optimize and monitor it to improve system performance and reliability.

47. What are the strategies for cache expiration?

Cache expiration means that the data in the cache is automatically deleted after a period of time, so that the data can be updated in real time. The commonly used cache expiration strategies are as follows:

Timed expiration: That is, set a fixed expiration time for the data when writing to the cache, and it will automatically expire after the specified time. This method is suitable for scenarios with few data changes, and the expiration time needs to be updated manually.
Expiration strategy based on the LRU (Least Recently Used) algorithm: When the cache space is insufficient, determine which data needs to be cleared according to the latest usage time of the data. This method is suitable for scenarios where hot data is obvious, and can ensure that the latest hot data always exists in the cache.
Expiration strategy based on the LFU (Least Frequently Used) algorithm: that is, the data expiration time is determined according to the frequency of data access. The more frequently accessed data, the longer the expiration time. This method is suitable for scenarios where the access frequency is relatively stable, the amount of data is large, and hot data changes infrequently.
Lazy expiration: that is, when the data is accessed, it is checked whether it has expired, and if it has expired, the data is reloaded and the cache is updated. This method is suitable for scenarios where data changes rapidly and hotspot data may change at any time.

It is necessary to select an appropriate cache expiration strategy based on actual business needs and system architecture, and configure, optimize, and monitor it to ensure data consistency and system stability.

48. Common cache elimination algorithm?

The cache elimination algorithm is used to decide which cache data needs to be cleared to make room for new cache data. Common cache elimination algorithms are as follows:

FIFO (First In, First Out): First-in-first-out algorithm, which cleans up data in the order in which they first entered the cache. This algorithm is simple and easy to implement, but it does not consider the access frequency and importance of data, which may cause hot data in the cache to be cleared.
LRU (Least Recently Used): That is, the least recently used algorithm, which determines which data needs to be cleaned up according to the latest usage time of the data. This algorithm keeps the most recently accessed data and avoids the problem of hot data being cleaned up.
LFU (Least Frequently Used): It is the least frequently used algorithm, which determines which data needs to be cleaned according to the frequency of data access. This algorithm retains data with high access frequency and can ensure that hot data always exists in the cache.
Random Replacement: It is a random replacement algorithm, which randomly selects some data for cleaning. This algorithm is simple and easy to implement, but it cannot guarantee the storage of hot data in the cache.
Clock: Also known as the second chance algorithm, it combines the FIFO algorithm and the LRU algorithm. On the basis of the FIFO algorithm, a clock pointer is used to mark the last access time of each data. If the data has been accessed, it is marked as "have a chance", otherwise it is marked as "no chance". When it comes time to clean up data, only clean up data that has "no chance" and hasn't been accessed for the longest time.

It is necessary to select an appropriate cache elimination algorithm based on actual business needs and system architecture, and configure, optimize, and monitor it to ensure data consistency and system stability.

49. Bloom filter principle? Advantages and disadvantages?

Bloom Filter (Bloom Filter) is a data structure used to quickly determine whether an element exists in a collection. It uses bitmaps and hash functions to achieve fast lookup.

The principle of Bloom filter is as follows:

Initialization: Create a bitmap of size m and initialize all bits in it to 0.
Adding elements: For the element x to be added, k hash values h1(x), h2(x), ..., hk(x) are calculated using k different hash functions, and then the corresponding bitmap position is set to 1.
Query element: For the element y to be queried, k hash functions are also used to calculate k hash values h1(y), h2(y), ..., hk(y), and then check the corresponding Whether the positions are all 1. If any of the positions is 0, it means that y must not exist in the set; if all of them are 1, then y may exist in the set.

The advantages of Bloom filters include:

Efficiency: Bloom filters can add and query elements in constant time, and have very high efficiency.
High space efficiency: Compared with other data structures, Bloom filters take up very little space, requiring only a small amount of memory to store a large number of elements.

Disadvantages of Bloom filters include:

Prone to misjudgment: Since the hash function of the Bloom filter is deterministic, different elements may map to the same bitmap position, resulting in misjudgment.
Unable to delete elements: Elements cannot be deleted because each element's corresponding bit is set to 1.

Bloom filters are suitable for scenarios that need to quickly determine whether an element exists in a collection, such as URL deduplication, spam filtering, etc. But it should be noted that it cannot guarantee 100% correctness, it can only provide a certain degree of accuracy and efficiency.

50. Distributed cache addressing algorithm?

The distributed cache addressing algorithm is used to map key-value pairs to specific nodes in the distributed cache cluster. The common distributed cache addressing algorithms are as follows:

Consistent Hashing: Consistent hashing is a hashing algorithm widely used in distributed systems. It maps both nodes and cached data to a ring, and solves data imbalance through virtual nodes. question. When a key-value pair needs to be looked up or added, hash calculation is performed on it first, and then the key-value pair is looked up on the next node on the ring.
Modulo-based hashing algorithm (Modulo Hashing): Modulo the hash value of the key-value pair to the number of cache nodes to obtain a remainder, and then store the key-value pair corresponding to the remainder on the corresponding node. This algorithm is simple and easy to understand, but there may be data skew problems, causing some nodes to be overloaded.
Random Hashing: Randomly assign key-value pairs to cache nodes. This algorithm can avoid the problem of data skew, but it may also lead to a low cache hit rate.
Weighted Round-Robin algorithm (Weighted Round-Robin): Polling is performed according to the weight value of the cache node. The higher the weight of the node, the greater the probability of being selected. This algorithm is suitable for scenarios where the load between nodes is unbalanced, and can ensure the balanced distribution of data.

It is necessary to select an appropriate distributed cache addressing algorithm according to actual business needs and system architecture, and configure, optimize and monitor it to ensure data consistency and system stability.

51. What are the differences between SpringCloud and Dubbo?

Both Spring Cloud and Dubbo are commonly used microservice frameworks, and they have the following differences:

Language support: Dubbo is mainly oriented to the Java language, while Spring Cloud can support multiple languages.
Protocol support: Dubbo supports multiple RPC protocols, such as Dubbo, gRPC, etc., while Spring Cloud supports protocols such as RESTful API and WebSocket.
Service registration and discovery: Dubbo uses external registration centers such as ZooKeeper or Consul for service registration and discovery, while Spring Cloud provides various registration center implementations such as Eureka, Consul, and ZooKeeper, and can also use its own service registration and discovery components.
Configuration management: Dubbo uses an external configuration center (such as Zookeeper) to manage configuration information, while Spring Cloud provides Config Server to centrally manage the configuration information of all microservices.
Fuse mechanism: Dubbo provides a fuse mechanism, but it needs to be implemented through Hystrix, while Spring Cloud has its own circuit breaker (Circuit Breaker), which can be directly implemented at the application layer level.
Project ecology: The Dubbo ecology is relatively simple, only providing core functions and some extension plug-ins, while Spring Cloud has a richer ecology, including sub-projects such as Netflix OSS, Spring Cloud Alibaba, and various third-party library support.

In general, Dubbo focuses on distributed service governance, emphasizing high performance and stability, and is suitable for more traditional enterprise applications; while Spring Cloud focuses more on development efficiency and development experience, providing more components and tools to Support microservice architecture. Choosing which framework to use also requires trade-offs and choices based on specific business needs and technology stacks.

52. What is Service Avalanche? What is service throttling?

Both service avalanche and service throttling are common problems in distributed systems.

Service avalanche (Service Avalanche) refers to the situation that when a service fails or is abnormal, multiple services it depends on cannot operate normally, resulting in a chain reaction, which eventually leads to the unavailability of the entire system. The reason for the service avalanche may come from many reasons such as too close dependencies between services, unreasonable service design, and wrong environment configuration. In order to avoid service avalanche, some measures can be taken, such as setting timeout, fuse, current limit, etc.

Service Limit refers to a mechanism that rejects or delays requests through certain algorithms in high-concurrency scenarios to ensure the stable operation of the system. Service throttling can effectively control system load and avoid service avalanches. Common service flow limiting algorithms include token bucket algorithm, leaky bucket algorithm, fixed window algorithm, etc. In practical applications, an appropriate current limiting algorithm can be selected according to specific business requirements and system characteristics, and reasonable thresholds and parameters can be set to achieve optimal system performance and stability.

53. What is a service circuit breaker? What is service degradation?

Service fusing and service degradation are some important mechanisms in the microservice architecture to ensure the stability and reliability of system operation.

Circuit Breaker refers to the monitoring of various indicators of the system (such as response time, error rate, etc.), when a service is found to be abnormal or unavailable, the service request is quickly failed, so as to avoid other service impact. By setting the fuse, the request can be cut off in time when the service fails, preventing the failure from spreading and improving the stability and availability of the entire system.

Service Degradation refers to giving up or reducing some non-core functions or services to ensure the normal operation of the core business in the face of resource constraints or high system pressure. For example, turn off some functions or delete some page elements during peak hours to reduce system load and relieve system pressure. Service degradation usually requires setting policies and rules in advance, and adjusting and optimizing according to the actual situation to achieve the best system performance and user experience.

Both service breaking and service degradation play an important role in distributed systems, which can effectively improve the reliability and robustness of the system. It is necessary to select the appropriate mechanism according to the specific business needs and system characteristics, and conduct detailed tests and evaluations to ensure the stability and reliability of the system.

54. What is the difference between SOA, distributed, and microservices?

SOA (Service-Oriented Architecture), distributed and microservices are all important concepts in modern software architecture. There are several differences between them:

Scale: SOA is usually a large-scale, enterprise-level architecture, dedicated to integrating various types of systems into a whole; while distributed and microservices focus more on specific application scenarios and business needs, and are usually smaller.
Technology stack: The SOA technology stack is relatively complex and needs to be implemented using a variety of technologies, such as Web Services, XML, SOAP, etc.; while the technology stack used by distributed and microservices is relatively simple, usually using HTTP/REST protocol, JSON wait.
Service granularity: The service granularity of SOA is relatively coarse, and is usually organized in units of business functions, such as order service, inventory service, etc.; while the service granularity of distributed and microservices is relatively fine, usually in units of specific business needs Division, and the coupling between services is low.
System complexity: Since SOA involves large-scale system integration, its system complexity and management difficulty are relatively high; while distributed and microservices are more flexible and lightweight, and easy to expand and upgrade.

Generally speaking, SOA, distributed, and microservices are all important concepts in modern software architecture. They have their own advantages and disadvantages, and they can play their unique value in different application scenarios and business requirements. It is necessary to select the appropriate architecture according to the specific situation, and carry out detailed design and practice to meet business needs and improve system performance and reliability.

55. How to split microservices?

The core idea of the microservice architecture is to split a large system into multiple small services, each of which is responsible for a specific function and can be deployed, upgraded and expanded independently. When splitting microservices, the following aspects need to be considered:

Business function: First, the system needs to be divided according to business modules or domain models according to actual business requirements. Usually each microservice should have a clear business function and be relatively independent from other microservices.
Database: For each business function, it is necessary to determine the required data storage method, including database type, table structure, etc. Usually each microservice should have its own data store, but it is also possible for multiple microservices to share the same data source.
Interface design: Define a clear interface for each microservice, including request and response parameters, interface address, protocol, etc. Usually, RESTful API or gRPC is used for interface design.
Code coupling: When splitting microservices, you need to pay attention to the code coupling between different modules. Avoid too tight dependencies to ensure the independence and maintainability of each microservice.
Deployment method: Each microservice should have an independent deployment method, which can be deployed, upgraded and expanded independently. It is usually necessary to use containerization technology (such as Docker) to encapsulate microservices and deploy them to cloud platforms or virtualized environments.
System monitoring: Each microservice should have independent system monitoring and logging mechanisms, as well as failure recovery and fault tolerance mechanisms. These functions can be implemented using some open source tools and frameworks, such as Prometheus, Grafana, etc.

Through the above steps, a large-scale system can be effectively split into microservices to improve the maintainability and scalability of the system, and at the same time better adapt to rapidly changing business needs.

56. How to select the message queue?

The selection of message queue needs to consider many aspects, including the following factors:

Application scenarios: Different message queues are suitable for different application scenarios. For example, Kafka is suitable for scenarios with high throughput and high concurrent reading and writing, and RabbitMQ is suitable for scenarios with low real-time requirements but high stability. When selecting a model, you need to choose a suitable message queue according to your own business needs and technical architecture.
Performance requirements: The performance indicators of message queues include throughput, delay, stability, etc., and message queues need to be selected according to specific performance requirements.
Scalability: The scalability of the message queue refers to whether it has good processing capabilities for large-scale and high-peak message traffic. It is necessary to select a message queue that supports horizontal expansion according to the expected scale and growth trend of the system.
Data security: Message queues involve important business data, and attention needs to be paid to issues such as message transmission security and data privacy protection, such as supporting mechanisms such as SSL/TLS encrypted communication, data backup, and authority control.
Community support: The degree of community support for message queues is also an important factor in model selection, including development documents, API interface support, open source activity, etc. These factors will have an important impact on subsequent maintenance and expansion.
Cost: Message queuing products and their usage costs are also one of the factors that need to be considered when selecting a model. Different message queuing products have different prices and expenses in terms of performance, scalability, and data security. Excellent solution.

When selecting a message queue, it is necessary to comprehensively consider the above factors and conduct various tests and evaluations to ensure that the selected message queue can meet business needs and have good performance and reliability.

57. How is the transaction message of RocketMQ implemented?

RocketMQ's transaction message is a mechanism to ensure the consistency of distributed transactions, which is mainly realized through the half-message and message review mechanism:

Half message: The sender sends the message to MQ, but the session is not committed. At this time, the MQ server only records the transaction branch and does not persist it. If the transaction execution fails at this time, RocketMQ will roll back the transaction branch. If successful, continue to the second step.
Formal message: The sender executes a local transaction. If an exception occurs or the transaction times out, the sender needs to send a rollback request to the MQ server, and the MQ server will delete the half-message sent before; if it is executed normally, send a commit Request, at this time the MQ server will persist the message.
Checkback mechanism: When the RocketMQ server detects that a transaction message has not received a commit or rollback instruction, it will start the transaction message checkback program. The checkback program will query the status of the transaction branch from the sender. If it gets "submitted", it will not process it. Otherwise, the transaction branch will be considered as failed and the consumer will be notified that the message is invalid.

RocketMQ's transaction messages can guarantee the consistency of messages and business logic in distributed transaction scenarios. At the same time, RocketMQ also provides a visual console for real-time monitoring and management of the running status of the message queue cluster.

58. What is the underlying implementation principle of RocketMQ?

RocketMQ is an open source distributed message queuing system of Alibaba. Its underlying implementation principles mainly include the following aspects:

Storage model: RocketMQ uses a file-based storage method to store messages to improve read and write performance and reliability. Each topic corresponds to a file directory. Each file has a fixed size and quantity limit. When the file reaches a certain size, it will be divided into multiple files. Each file has an index file and a log file for recording messages. and location information.
Message transmission protocol: RocketMQ uses a custom protocol for message transmission, including the protocol for producers to send messages to Brokers, the protocol between Brokers, and the protocol for consumers to request messages from Brokers. Among them, the producer sends the message to the Master node through the TCP connection, and the Master node is responsible for persisting the message to the disk and passing the message to all Slave nodes; the consumer pulls the message from the Broker through the TCP connection and consumes it .
Message routing and load balancing: RocketMQ uses a Broker cluster to achieve high availability and load balancing. Each Broker node corresponds to one or more Topic shards, and each shard contains part of the message data under the Topic. The sending and receiving of messages are managed by NameServer. NameServer maintains the routing information of Broker when it is started, and is responsible for routing messages to the correct Broker node.
High availability and fault tolerance: RocketMQ adopts a master-slave replication method to ensure high availability and fault tolerance of data. Each Master node has one or more Slave nodes. When the Master node goes down, all fragments on the node will automatically switch to the corresponding Slave nodes to continue to provide services.
Message order guarantee: RocketMQ provides a strict message order guarantee, that is, messages sent by the same Producer can only be consumed by the same Consumer in the same partition in the order in which they were sent.

Generally speaking, RocketMQ provides high-performance, high-availability and reliable message transmission services through excellent design and implementation, which is suitable for large-scale distributed systems and business needs of Internet enterprises.

59. How does the message queue ensure the reliable transmission of messages?

The reliable transmission of message queues usually includes the following aspects:

Producer confirmation mechanism: After sending a message to the message queue, the producer needs to wait for the confirmation response of the message queue. Only when the confirmation response is successfully returned, the producer will consider the message sent successfully.
Message persistence mechanism: Message queues usually use a message persistence mechanism to ensure that messages can be recovered in abnormal situations such as downtime. Message persistence can be achieved by writing messages to disk, databases, or distributed storage systems.
Message retry mechanism: In the process of message transmission, there may be network failures, service failures, etc., resulting in messages not being consumed normally. In response to this situation, the message queue usually provides a message retry mechanism, that is, the retry operation is performed when the message is not consumed until the message is normally consumed.
Message deduplication mechanism: Due to various reasons, messages may be sent repeatedly. In order to avoid repeated processing of messages, the message queue usually adopts a message deduplication mechanism, such as using message IDs, unique identifiers, etc. to deduplicate.
Message monitoring and alarm mechanism: In order to discover and solve problems in the process of message transmission in a timely manner, message queues are usually equipped with message monitoring and alarm mechanisms, including error log records, performance monitoring, etc., to ensure system stability and reliability.

It is necessary to select the appropriate message queuing product according to the specific business requirements and technology stack, and configure, optimize and monitor it to ensure the stability and reliability of message delivery. At the same time, you also need to pay attention to the capacity limit of the message queue, message security and other issues.

60. What are the functions of the message queue?

Message Queue (Message Queue) is an asynchronous communication mechanism based on message passing, which is mainly used for decoupling, asynchrony and scalability. Message queues have the following roles in modern distributed systems:

Asynchronous communication and application decoupling: message queues can decouple the direct dependencies between producers and consumers. Producers only need to send messages to message queues, and no longer need to directly call consumer methods, thus reducing the Coupling between systems.
Traffic peak shaving and peak shaving and valley filling: Under high concurrency, there will be traffic peaks and valleys. At this time, message queues can play a role in buffering and smoothing traffic, avoiding system crashes or slow response times.
Data distribution and processing: The message queue can distribute data to multiple consumers for processing, which improves the processing capacity and efficiency of the system.
Transaction management: The message queue can support the transaction mode to ensure the atomicity of the message, so as to ensure that the entire system can maintain consistency.
Message persistence: Message queues can store messages persistently to avoid message loss due to sudden downtime.
System decoupling and scalability: By introducing message queues, different components and services can be decoupled, making the system more modular and scalable.

In short, message queues play an important role in modern distributed systems, which can effectively improve system performance, stability and reliability, while also reducing system coupling and improving system scalability.

61. What is a dead letter queue? What is a delay queue?

Dead letter queues and delay queues are commonly used concepts in message queues.

Dead Letter Queue (Dead Letter Queue, DLQ) is a queue for processing messages that cannot be consumed normally. When a message cannot be processed normally due to some reasons (such as message format error, message expiration, etc.), these messages will be sent to the dead letter queue for subsequent processing. Dead letter queues are usually used to ensure the reliability and consistency of message delivery, and to troubleshoot problems that arise during system operation.

Delayed Queue (Delayed Queue) is a queue that supports delayed sending of messages. It allows the producer to set a delay time when the message is sent. When the delay time is reached, the message will be sent to the specified queue for consumption. Delay queues are usually used to deal with scenarios that take a certain amount of time to process, such as order payment timeout, verification code expiration, etc.

Delay queues and dead letter queues have some similarities, both of which involve issues such as message expiration and reprocessing. But their main difference is: the delay queue temporarily stores messages and delays delivery; while the dead letter queue transfers and reprocesses messages that cannot be processed normally. Both can improve the stability and reliability of message delivery, and the appropriate solution needs to be selected according to specific business needs and technology stacks.

62. How does MQ ensure efficient reading and writing?

MQ (Message Queue) is an asynchronous communication mechanism based on message passing, mainly used for decoupling, asynchrony and scalability. In MQ, efficient reading and writing of data involves the following aspects:

Asynchronous processing: In MQ, the producer and consumer communicate asynchronously through the message queue. The producer does not have to wait for the response of the consumer, but returns immediately after sending the message to the queue, which improves the execution efficiency.
Batch processing: In order to reduce the storage and transmission overhead of a single message, MQ usually supports batch message processing, which can combine multiple messages into one transmission or processing, thereby improving performance.
Clustered deployment: MQ usually adopts a clustered deployment method, and multiple nodes can jointly be responsible for the storage and transmission of messages, which improves the fault tolerance and scalability of the system.
Efficient storage and transmission mechanism: In order to improve the efficiency of reading and writing data, MQ usually uses an efficient message storage and transmission mechanism, such as using file mapping technology to achieve zero copy (Zero Copy), using memory mapping technology to improve data reading speed wait.
Message compression: For a large number of text messages, MQ usually uses message compression technology to reduce message size and reduce network transmission and storage overhead.

It is necessary to select the appropriate MQ product according to the specific business requirements and technology stack, and configure, optimize and monitor it to ensure the stability and reliability of message delivery

63. What is the difference between epoll and poll?

Both epoll and poll are I/O multiplexing mechanisms provided by the Linux system. They are used to monitor events on multiple file descriptors at the same time. When an event occurs on a certain file descriptor, the application is notified to process it.

Their main differences are:

Trigger method: poll uses polling to check whether there are events in the file descriptor. However, epoll uses a callback function. When an event occurs on a certain file descriptor, the event is passed to the application through the callback function, which can reduce unnecessary polling overhead.
The required kernel space data structure: poll needs to pass a pollfd array to the kernel, which includes each file descriptor to be monitored and its corresponding event type. And epoll registers the list of file descriptors to be monitored through the epoll_ctl() function, and passes the returned epoll_fd as a parameter into the epoll_wait() function.
Quantity limitation: Since poll adopts the polling method, as the number of file descriptors to be monitored increases, the number of polling times and CPU overhead also increase accordingly, which may lead to system performance degradation. The way epoll uses callback functions can effectively avoid this situation and support a larger number of concurrent connections.

Generally speaking, epoll is more efficient and flexible than poll, and supports a larger number of concurrent connections, especially for high-concurrency server application scenarios. However, it should be noted that in practical applications, factors such as actual application scenarios, system performance, and stability need to be considered when choosing which I/O multiplexing mechanism to use.

64. What is a cross-domain request? What is the problem? How to deal with it?

Cross-domain request refers to the behavior that a browser sends a request from one source (protocol + domain name + port) to another different source. Common problems with cross-domain requests are:

Security issues: Cross-domain requests may lead to security vulnerabilities. For example, website A refers to the page of website B through an iframe, and then injects malicious scripts into the page of website B to attack users.
Data isolation: Due to browser restrictions on cross-domain requests, data between different sources cannot be directly shared and accessed, which affects data transmission and interoperability.
Access control: When making cross-domain requests, APIs restricted by CORS only allow access to requests under specific domain names, and requests under other domain names will be rejected.

There are several ways to solve the cross-domain request problem:

JSONP (JSON with Padding): JSONP uses the feature that script tags can make cross-domain requests, carries the name of the callback function when making the request, and wraps the returned result into a function call and returns it to the browser.
CORS (Cross-Origin Resource Sharing): CORS is a mechanism based on HTTP header information, which allows the server to tell the browser whether to allow cross-domain requests, and can also specify the domain names and request headers that are allowed to be accessed.
Reverse proxy: By setting a reverse proxy on the server side, the cross-domain request is forwarded to the local service for processing, so as to realize the cross-domain request.
WebSocket: The WebSocket protocol allows full-duplex communication between the browser and the server, and can easily realize cross-domain data transmission.

It is necessary to select an appropriate solution based on specific business needs and technology stack, and configure, optimize and monitor it to ensure data security and stability.

65. What is zero copy?

Zero Copy (Zero Copy) is a computer system optimization technology, mainly used to reduce CPU and memory overhead during data transmission. The basic principle of zero copy is to avoid copying data between the application program and the kernel space as much as possible, so that the data can be directly transmitted between the kernel and the hardware device, thereby reducing CPU usage and memory bandwidth.

The traditional data transmission method usually needs to copy data multiple times between the application program and the kernel space. For example, when an application program sends data to the network, it needs to copy the data from the user space to the kernel space, and then copy it from the kernel space into the buffer of the network device. These copy operations consume a lot of CPU resources and memory bandwidth, seriously affecting system performance.

The use of zero-copy technology can avoid these unnecessary data copy operations. The specific implementation methods include:

DMA (Direct Memory Access): The DMA controller is used to directly copy data from the memory to the buffer of the network device, avoiding the CPU from participating in this process.
sendfile(): The sendfile() system call is provided in the Linux system, which can directly transfer the file content to the buffer of the network device without copying data between user space and kernel space.
mmap(): Use the mmap() system call to map the file into the memory, and directly transfer the data from the memory to the buffer of the network device through the DMA controller, while avoiding the data copy operation between the user space and the kernel space.

Zero-copy technology can effectively improve data transmission efficiency, reduce CPU overhead and memory bandwidth, and is especially suitable for large-scale data transmission and high-performance computing scenarios.

66. What is the difference between PostgreSQL and MySQL? Why do many project databases use PostgreSQL now?

Both PostgreSQL and MySQL are open source relational database management systems, and they have the following differences:

Data types: PostgreSQL supports more data types, such as arrays, JSON, etc. At the same time, PostgreSQL also supports custom types, which is convenient for users to expand types according to business needs.
ACID compatibility: PostgreSQL strictly follows the ACID (Atomicity, Consistency, Isolation, Persistence) specification, while MySQL does not fully support the ACID specification by default, and needs to manually set the InnoDB storage engine to ensure ACID compatibility.
Performance: MySQL has good performance when processing large amounts of data, while PostgreSQL performs better when processing complex queries.
Stored procedure: PostgreSQL supports more programming languages, such as Python, Perl, etc., and supports recursive calls of stored procedures. MySQL only supports the SQL language and does not support other programming languages, and stored procedures do not support recursive calls.
Scalability: PostgreSQL is more flexible and scalable, and various functions can be realized through extension modules. However, the scalability of MySQL is weak, and some advanced functions need to be implemented manually.

At present, more and more projects choose to use PostgreSQL as the database management system. The main reasons are as follows:

Rich functions: PostgreSQL provides a lot of advanced functions, such as complex queries, stored procedures, triggers, partitions, etc., which can meet the needs of most application scenarios.
Good scalability: PostgreSQL supports various plug-ins and extension modules, which is convenient for users to customize development according to their own business needs.
High security: PostgreSQL has good security and supports multiple security functions such as SSL connection, data encryption, and access control.
Active community: PostgreSQL has an active community, a large number of users have contributed excellent extensions and tools, and a strong development team continues to promote the development of PostgreSQL.

It should be noted that although PostgreSQL performs better than MySQL in some aspects, which database system to choose should be evaluated and selected based on factors such as actual needs, technology stack, and personnel skills.

67. What is distributed? What are microservices?

A distributed system refers to multiple computers working together to complete a set of common tasks through message passing. These computers, called nodes, are connected together by a network and coordinate the functions of a larger system. Distributed systems usually have advantages such as high availability, scalability, and fault tolerance.

Microservices is an architectural style that divides an application into a set of small, independent, loosely coupled services, each of which can be deployed independently and communicate through APIs. Each service focuses on a specific business domain and is developed and deployed using an independent technology stack. The advantages of microservice architecture include flexibility, scalability, maintainability, etc.

Compared with monolithic applications, distributed systems and microservice architectures are more suitable for large-scale and complex application scenarios. In a distributed system, each node can run on a different physical or virtual machine and work together through a network connection. The microservice architecture splits a large single application into multiple small independent services, so that each service can be independently expanded as needed, improving the scalability and maintainability of the system.

It should be noted that both distributed systems and microservice architectures involve cross-process and cross-service communication and coordination, so they also bring some challenges, such as network delay, data consistency, and service governance.