Problems encountered by the traditional MySQL+ Memcached architecture
In fact , MySQL is suitable for mass data storage. Memcached is used to load hot data into cache to speed up access. Many companies have used this architecture, but with the continuous increase of business data volume and the continuous growth of access volume, we Encountered many problems:
1. MySQL needs to continuously dismantle the database and table, and Memcached also needs to continue to expand, and the expansion and maintenance work take up a lot of development time.
2. The data consistency between Memcached and MySQL database.
3. The Memcached data hit rate is low or the machine is down, and a large number of accesses directly penetrate the DB, which MySQL cannot support.
4. Cross-machine room cache synchronization problem.
Many NoSQL flowers bloom, how to choose
In recent years, many and various NoSQL products have been emerging in the industry , so how to use these products correctly and maximize their strengths is a problem that we need to study and think about. In the final analysis, the most important thing is Understand the positioning of these products, and understand the tradeoffs of each product, so as to avoid weaknesses in practical applications. In general, these NoSQL are mainly used to solve the following problems
1. A small amount of data storage, high-speed read and write access. This type of product ensures high-speed access by means of all in-momery data, and provides the function of data landing. In fact, this is the main application scenario of Redis.
2. Mass data storage, distributed system support, data consistency guarantee, convenient cluster node addition /deletion.
3. The most representative in this regard are the ideas described in the two papers of dynamo and bigtable. The former is a completely non-centralized design. The nodes transmit cluster information through gossip, and the data guarantees eventual consistency. The latter is a centralized solution design, which ensures strong consistency through a distributed lock service similar to that of data writing. Write to memory and redo log first, and then periodically compat to disk, optimize random writes to sequential writes, and improve write performance.
4. Schema free, auto-sharding, etc. For example, some common document databases currently support schema-free, directly store data in json format, and support functions such as auto-sharding, such as mongodb.
Faced with these different types of NoSQL products, we need to choose the most suitable product according to our business scenario.
Redis applicable scenarios, how to use it correctly
As has been analyzed before, Redis is most suitable for all data in-momory scenarios. Although Redis also provides persistence functions, it is actually more of a disk-backed function, which is quite different from persistence in the traditional sense. Then you may have questions. It seems that Redis is more like an enhanced version of Memcached, so when to use Memcached and when to use Redis?
Comparison of Redis and Memcached
1. Network IO model
Memcached is a multi-threaded, non-blocking IO multiplexing network model. It is divided into monitoring main thread and worker sub-thread. The monitoring thread listens for network connections. After accepting the request, it passes the connection description word pipe to the worker thread for reading and writing IO, network The layer uses the event library encapsulated by libevent. The multi-threading model can play a multi-core role, but it introduces the problem of cache coherency and lock. For example, the most commonly used stats command of Memcached, in fact, all operations of Memcached must lock this global variable and count it. And so on, it brings performance loss.
2.
3.
( Memcached network IO model)
4.
Redis uses a single-threaded IO multiplexing model and encapsulates a simple AeEvent event processing framework, which mainly implements epoll, kqueue and select. For only IO operations, single-threaded can maximize the speed advantage, but Redis It also provides some simple computing functions, such as sorting, aggregation, etc. For these operations, the single-threaded model will actually seriously affect the overall throughput. During the CPU computing process, the entire IO scheduling is blocked.
5.
6. Memory management
Memcached uses a pre-allocated memory pool, uses slabs and chunks of different sizes to manage memory, Item selects the appropriate chunk storage according to the size, and the memory pool method can save the overhead of applying/releasing memory, and can reduce memory fragmentation However, this method will also lead to a certain degree of space waste, and when there is still a lot of space in the memory, new data may also be eliminated. The reason can refer to Timyang's article: http://timyang.net /data/Memcached-lru-evictions/
7.
Redis uses the method of on-site application for memory to store data, and rarely uses free-list and other methods to optimize memory allocation, and there will be memory fragmentation to a certain extent. Redis will store data with expiration time separately according to the storage command parameters. Together, and call them temporary data, non-temporary data will never be culled, even if the physical memory is not enough, resulting in swap will not cull any non-temporary data (but will try to cull some temporary data), this point Redis is more suitable as a storage rather than a cache.
8.
9. Data Consistency Issues
Memcached provides the cas command, which can ensure the consistency of multiple concurrent access operations on the same data. Redis does not provide the cas command, and cannot guarantee this, but Redis provides the function of transaction, which can ensure the atomicity of a series of commands, and will not be interrupted by any operation in the middle.
10.
11. Storage method and other aspects
Memcached basically only supports simple key-value storage, does not support enumeration, does not support functions such as persistence and replication
12.
In addition to key/value, Redis also supports list, set, sorted set, hash and many other data structures, and provides KEYS
13.
Perform enumeration operations, but cannot be used online. If you need to enumerate online data, Redis provides tools to scan its dump files directly and enumerate all data. Redis also provides functions such as persistence and replication.
14.
15. About client support in different languages
In terms of clients in different languages, Memcached and Redis have rich third-party clients to choose from. However, because Memcached has been developed for a longer time, many clients of Memcached are more mature and stable in terms of client support. Redis, because its protocol itself is more complicated than Memcached, and the author keeps adding new functions, etc., the follow-up speed of the corresponding third-party client may not keep up, and sometimes you may need to make some modifications on the basis of the third-party client to be better. usage of.
16.
According to the above comparison, it is not difficult to see that when we do not want the data to be kicked out, or when we need more data types than key/value, or when we need the landing function, it is more suitable to use Redis than to use Memcached.
About some peripheral functions of Redis
In addition to being used as storage, Redis also provides some other functions, such as aggregate computing, pubsub, scripting, etc. For such functions, it is necessary to understand the implementation principle and clearly understand its limitations before it can be used correctly, such as pubsub function, this actually does not have any persistence support, the messages from the consumer's connection flash or reconnect will be lost, and functions such as aggregate computing and scripting are limited by the Redis single-threaded model. It is possible to achieve very high throughput and needs to be used with caution.
In general , the author of Redis is a very diligent developer. You can often see that the author is trying various new ideas and ideas. The functions of these aspects require us to have a deep understanding before using them.
Summarize:
1. The best way to use Redis is to use all data in-memory.
2. More scenarios of Redis are used as a replacement for Memcached.
3. When more data type support than key/value is required, Redis is more suitable.
4. When the stored data cannot be eliminated, it is more appropriate to use Redis.
Follow-up plans for Redis articles:
1. Redis data types and capacity planning.
2. How to build a stable, reliable and scalable Redis cluster according to business scenarios.
3. Redis parameters, code optimization and basic practice of secondary development.