redis related notes (three .redis Design and Implementation (notes))

1. Data Structure

1.1 simple dynamic strings:

And their attributes are int len: length, int free: free length, char [] bur: character array (content)
string length obtain simple;
prevent buffer overflow;
the number of memory allocation required for reducing the weight modifying the string length;
binary security; compatible portion C string functions;

1.2 list: the list is implemented using listNode

listnode and their attributes are struct listNode * prev: previous node, struct listNode * next: After a node, void * value: The current node value
list and their attributes are listNode * head: the head node, listNode * tail: tail node, long len: number of nodes (length)

1.3 dictionary:

The k hash values acquired, then acquisition of the index table dictht in accordance hash & sizemask, if there is a conflict dictEntry point to a next list,
when the hash table to expand or contract, the program requires the hash table contains all existing rehash the new key-value pair into the hash table, and a gradual accomplished
1.4 jump tables: ordered underlying collection implemented by zskiplist and zskiplistNode composition, zskiplist jump table information stored as header node, tail node table, a length ;
. 1.5 integers: one set of keys underlying implementation, the underlying array implemented in an orderly manner without duplication of storage elements
1.6 Packing list: list of keys and hash key underlying implementation one, comprising a plurality of nodes, each. nodes may be stored in a byte array or an integer value

2. Object: each object is represented by a structure represented redisObject

And their attributes are unsigned type: type, encoding: encoding, void * ptr: implementation pointer points to the underlying data structure
type (type refers to the value stored value, i.e., types) values are REDIS_STRING: string, REDIS_LIST: list, REDIS_HASH : hash, REDIS_SET: collection, REDIS_ZSET: ordered set
encoding (decision pointer to that data structure) of the underlying data structure of encoded values are corresponding
REDIS_ENCODING_INT long integer type
REDIS_ENCODING_EMBSTR embstr simple string encoded
REDIS_ENCODING_RAW simple dynamic character string
REDIS_ENCODING_HT dictionary
REDIS_ENCODING_LINKEDLIST double-ended list
REDIS_ENCODING_ZIPLIST compression listing
REDIS_ENCODING_INTSET integers
REDIS_ENCODING_SKIPLIST jumping and dictionaries
and their attributes are the refcount: counting for recovery and object sharing memory
garbage collection:
reference count, the reference count program may be tracked through the information object, appropriate time and automatically releases the object memory recovery
when creating a new object is 1; when a new program is used, a self-energizing; no longer used, since minus 1; and when is 0, the memory is released;
object sharing:
database It refers to the value of the pointer A target current value; to refer to the same two keys string object; reference count is incremented;
redis when initializing the server, create ten thousand string objects, including all integer values from 0-9999, the server will use these shared objects, but not new
and their attributes are lru: record the object was last accessed command program time, the recovery of the memory using
the server if the configuration maxmemory option, the server memory algorithm for recovering volatile-lru or allkeys-lru, and more than the set limit, long high during idling is recovered freed memory

3. Database:

Server struct redisServer
its properties redisDb * db: an array of all database servers
its properties int dbnum: Database number of servers,
clients struct redisClient
its properties db: one point to the database server in the array

database key space
typedef struct redisDb {
dict * dict; / * the keyspace for this DB * / key space
...
} redisDb;
the actual realization of the dict field operation for the operation of the database
key to survival and delete
survival time or expire by pexpire command specified key
three kinds delete a policy:
Timing deleted, expired at the same time create a set timer expired is deleted. For cpu unfriendly
inert deleted, regardless of the key expired, obtain the key to check if you delete expired, unexpired is returned. Memory unfriendly
periodically delete, regular inspection database, which deletes expired check. When you delete a need to set reasonable frequency and CEO

4. Persistence: to save the data in the database to disk

1.RDB persistence

Create and load 1.RDB file: by saving data to a database record database state

Generation, and BGSAVE SAVE command
SAVE block the server process until the RDB file created
BGSAVE spawns a child process, RDB file created by this process, the server process continues to process other commands
loaded, RDB file is automatically loaded when the server starts (not open AOF persistence time)

2. In the configuration redis.conf interval can be set automatically saved: as follows

save 900 1 900 seconds once modified is performed
with a modification 10 is performed within seconds save 300 10 300
10,000 This modification is performed within save 60 10000 60 seconds
serverCron every 100 milliseconds save configuration check conditions are satisfied, as long as there is a meet is performed BGSAVE

3.RDB file structure

2.AOF persistence: write commands executed by saving redis server to record the state of the database

1. instruction adding

There aof_buf redisServer attribute associated execution command storage server

2. file write and synchronization

Server before the complete implementation of a client request event, will consider whether to write to aof_buf buffer to a file
server has configuration items appendfsync, its value can be
always: the aof_buf all writes to the file and synchronize
everysec: All content written to the file, the last synchronization event than a second, there is a special thread is responsible for synchronizing
no: write to a file, but not synchronized, decide when to synchronize operating system

3.AOF rewrite:

File is growing more and more, to rewrite a new command will merge,
as two different times on the same set of operations of the AOF file has two commands, this command can be combined into a
background rewriting: the parent process to continue process commands from the process of rewriting, rewriting period, the parent command processing AOF into the buffer, then re-written to AOF and synchronize files.

5. Event: redis server is an event-driven program

1. File event: server socket abstraction operation

I / O multiplexing a plurality of sockets listening program, ordered into a synchronization queue, the queue each time the file is transmitted to a socket event dispatcher, the dispatcher calls the appropriate file generated according to the time event handler
most common event handler:
connect an answering processor, client connections listening socket server listening socket generate AE_READABLE event, executed by the processor
command requesting processor, the client sends a command request, the server listening socket sockets produce AE_READABLE event, executed by the processor
command processor reply, the client socket generate AE_WRITABLE event, executed by the processor

2. Time Event: timer time (execution time) and periodic events (how often executed once).

Three attributes, id: identity, when: the time you need to perform, timeProc: things need to deal with
all the time event placed in the random list, perform through all the lists, the execution time of the event is
currently redis server time event application examples serverCron function, periodic events run every so often will be executed once, until the server is shut down
there hz in redis.conf configuration options, the number of times per second to perform this function, you can configure the default 10
file events and time events for the partnership, both servers take turns processing time, processing will not seize all the time event might be time to complete the set.

6. Client: struct client

Appear as follows by connecting redis client list command input
id = 2 addr = 192.168.37.134: 43836 fd = 6 name = age = 181355 idle = 1 flags = S db = 0 sub = 0 psub = 0 multi = -1 qbuf = 0 qbuf 0 = 0 = OBL -free OLL OMEM = 0 = 0 = R & lt cmd = replconf Events
ID = 192.168.37.134 addr = 14: FD = 50888 = 258. 7 Age name = IDLE = the flags = 0 N = 0 DB = 0 Sub Psub Multi qbuf -1 = 0 = 0 = 32768 = OBL Free qbuf-OLL = 0 = 0 = 0 OMEM Events cmd = = R & lt Client
ID = addr = 192.168.37.1 15: 55242 FD = name = Age =. 8. 4. 4 = IDLE flags = N db = 15 sub = 0 psub = 0 multi = -1 qbuf = 0 qbuf-free = 0 obl = 0 oll = 0 omem = 0 events = r cmd = select
the server has three client connection
id: unique identifier, addr: connection address, fd: flag, -1 pseudo client (e.g. aof or lua script), age: length of connecting to the server, flags: the role flag (e.g., S for the server), db: database number, qbuf : input buffer, cmd: command last executed

7. server: establish a connection with multiple clients, save the data, maintain server operation through Explorer

1. command execution

1. The server reads the resp agreement, saved to client input buffer, command buffer for analysis, and the number of parameters of the parameter
2 to save the client argv properties and properties in the argc
3. Call command executor , execute the command: 1. Find command, performs a pre-operation 2, 3 implement a function call instruction, the implementation of the follow-up 4..
4. The client socket writable reply command to the client

2.serverCron function

Update server status information, resource management client and the database state, checking the implementation of persistence

3. Start the server

1. The initial state of the server
2. The server configuration loaded
3. Initial server data structures
4. Restore the state of the database
5. Perform event loop
8. Copy
sync all
1. SYNC command transmitted from the server
2. The server receives a command from the master server RDB generate a file, and a record of all ordered write buffer now begin
3. the main server sends the file to the RDB, RDB loaded from the server from the server
4. the server main buffer write commands sent from the server to the
synchronization section: efficient replication of the processing from the main heavy broken
by copying offset \ replication backlog buffer \ id server running is achieved three parts

9. Sentinel: sentinel examples monitor the primary server and will be relevant to offline from a server to a primary server from the primary server by one or more

1. Initialize instance master record all information of sentinel surveillance, and linked to create two primary server, and the subscription link command links
2. Links 10 seconds by default command info transmitting information acquired by the master server command
3. acquired created from the server information from the command links and subscribe to the linked server, available through links from command to save the information to a server instance
4. configure the server configuration is the primary objective offline, if it is determined off the assembly line, each sentinel surveillance consultations election to lead off the assembly line sentinel primary server failover (raft algorithm)
5. lead sentinel selected from all servers in a conversion from the primary server-based servers, so that other new master copy from the server, the server becomes the master has been off the assembly line as the new master server from the server

Cluster 10: distributed database programs, share data through slicing, and providing failover and copy

Saving information on cluster nodes clusterNode structure to
the cluster by the stored key-value pair fragment, the entire cluster database is divided into 16,384 slots, different slots assigned to different nodes
MOVED error: When a node receives the command request, where the key check groove are responsible for their own, if not, then returned to the client MOVED error, MOVED wrong turn in the right node carries information
to re-slice: touch keys that belong to all slots to be transferred from one node to another node
ASK error: redistribution films, the node is not found in its own database command specifies the key database, it will return an error ASK to guide the client to continue to look for another node

11. subscriptions and publishing

Clients subscribe to one or more channels to subscribers, when a client sends a message to other channels, channel all subscribers will receive the message
subscription: subscribe "channel" ... to send: publish "channel" "Message "unsubscribe: unsubscribe" channel "...
save relationship subscribe to all channels in the dictionary pubsub_channels of redisServer, pointing to a channel subscribe to the client list for all channels, subscribe to add, delete unsubscribe
subscribe to all modes of save pubsub_patterns redisServer relation to the dictionary, a pattern matching character includes client and related information

12. things

Multi, exec, watch and other things in order to achieve through
receipt of process execution 1. Server multi command, open something, 2. Enter the other as set \ get command, 3. Enter the exec submit
a client state of affairs marked, server receives the command after, it is determined whether the client is in a state of affairs, the object status commands into the client (server records) of the command queue (FIFO), the queue until the command execution EXEC
watch optimistic locking, before executing the exec monitoring database, the keys were modified refused to carry things, returns an empty
error exec submit the command will not execute all things, canceled things, if some of the things an error, you can submit exec, an error command is not executed, the other continues

13.Lua script

Redis lua script to execute commands contained in a pseudo-client, use a script to save all loaded dictionaries eval command is executed or the script load script
eval command script defines a client input function, call the function to execute scripts
such as : eval "return redis.call ( 'del ', unpack (redis.call ( 'keys', ARGV [1])))" 0 'USER_ID_ *'
to remove all keys to the beginning of the USER_ID_

14. Sort: SORT command key list, set the value of a set of keys or key study tour sort

The key elements contained loaded into the array, and then sort the array, there are various options to return after ordering different results, BY \ LIMIT \ GET \ ALPHA \ DESC \ ASC

15. binary array

SDS is used to save, save reverse the median group, BITCOUNT algorithm using look-up table and variable-precision SWAR algorithms to optimize

16. slow query log: a recording execution time from exceeding a given duration of the command request to monitor and optimize query speed log function generated by this

Configuring slowlog-log-slower-than microseconds: command execution time exceeds the number of microseconds to record log
slowlog-max-len number of: how many slow query log store up to the specified server

Related Information command execution client Monitor, will become their own monitor, the print server and receive real-time processing of the current request: 17. Monitor

192.168.37.134:8340> monitor
OK
1569290062.401279 [0 192.168.37.1:60115] "AUTH" "admin.1231"
1569290062.402377 [0 192.168.37.1:60115] "PING"
1569290062.403760 [0 192.168.37.1:60115] "INFO" "ALL"
1569290063.409147 [0 192.168.37.1:60115] "set" "t" "hh"
1569290075.303194 [0 192.168.37.1:60115] "set" "tt" "hh"