1. Introduction to quicklist
The Redis list is a simple list of strings, sorted in order of insertion. You can add an element to the head (left) or tail (right) of the list.
A list can contain up to 2 32 - 1 element (4,294,967,295, each element of a list of more than 4 billion).
The internal data structure that its underlying implementation depends on is quicklist, whose main features are:
1. list is a doubly linked list.
2. It is extremely convenient to add and delete data at both ends of the list, and the time complexity is O (1).
3. The list also supports access operations at any intermediate position, with a time complexity of O (N).
Before looking at the source code (version 3.2.2), let's take a look at a few main data structures in quicklist:
A quicklist consists of multiple quicklistNodes, each quicklistNode points to a ziplist, and a ziplist contains multiple entry elements. Each entry element is an element of a list.
Figure 1: quicklist
Second, the source code of quicklist data structure
Let's take a look at the source code of quicklist and quicklistNode respectively (the code file is Quicklist.h, and the following articles in ziplist will be analyzed):
quicklist:
/* The quicklist structure occupies 32 bytes (64-bit system), where the fields: head: points to the first quicklistNode. tail: points to the last quicklistNode. count: The total number of entries in all ziplists. len: the number of quicklistNode. fill: The size of ziplist is limited, given by server.list_max_ziplist_size. compress: Node compression depth setting, given by server.list-compress-depth, 0 means turn off compression. */ typedef struct quicklist { quicklistNode *head; quicklistNode *tail; unsigned long count; /* total count of all entries in all ziplists */ unsigned int len; /* number of quicklistNodes */ int fill : 16; /* fill factor for individual nodes */ unsigned int compress : 16; /* depth of end nodes not to compress;0=off */ } quicklist;
quicklistNode:
/* prev: points to the previous quicklistNode. next: Point to the next quicklistNode. zl: point to the ziplist of the current node. sz: The number of bytes occupied by ziplist. count: The number of elements in the ziplist. encoding: encoding type, RAW == 1 or LZF == 2. container: container type, NONE == 1 or ZIPLIST == 2 recompress: bool type, true means that the data of the node is temporarily decompressed. attempted_compress: bool type, used in the test phase. extra: Fill the dictionary, which may be used in the future. */ typedef struct quicklistNode { struct quicklistNode *prev; struct quicklistNode *next; unsigned char *zl; unsigned int sz; /* ziplist size in bytes */ unsigned int count : 16; /* count of items in ziplist */ unsigned int encoding : 2; /* RAW==1 or LZF==2 */ unsigned int container : 2; /* NONE==1 or ZIPLIST==2 */ unsigned int recompress : 1; /* was this node previous compressed? */ unsigned int attempted_compress : 1; /* node can't compress; too small */ unsigned int extra : 10; /* more bits to steal for future usage */ } quicklistNode;
3. Addition, deletion, and modification of quicklist
1. Create quicklist
When the push command is executed (for example, lpush), if there is no such key, a quicklist is created and initialized, as follows:
void pushGenericCommand(client *c, int where) { int j, waiting = 0, pushed = 0; robj *lobj = lookupKeyWrite(c->db,c->argv[1]); if (lobj && lobj->type != OBJ_LIST) { addReply(c,shared.wrongtypeerr); return; } for (j = 2; j < c->argc; j++) { c- > argv [j] = tryObjectEncoding (c-> argv [j]); if (! lobj) { // key does not exist, first create the key object and add it to the db lobj = createQuicklistObject (); // initialize quicklist Object quicklistSetOptions (lobj-> ptr, server.list_max_ziplist_size, server.list_compress_depth); // Use redis server configuration items to initialize dbAdd (c-> db, c-> argv [ 1 ], lobj); // Add quicklist to redis db } // Add new elements to list in listTypePush (Lobj, the C-> the argv [J], WHERE ); pushed++; } addReplyLongLong(c, waiting + (lobj ? listTypeLength(lobj) : 0)); if (pushed) { char *event = (where == LIST_HEAD) ? "lpush" : "rpush"; signalModifiedKey(c->db,c->argv[1]); notifyKeyspaceEvent(NOTIFY_LIST,event,c->argv[1],c->db->id); } server.dirty += pushed; }
Look at createQuicklistObject again:
/* Create a new quicklist. * Free with quicklistRelease(). */ quicklist *quicklistCreate(void) { struct quicklist *quicklist; quicklist = zmalloc(sizeof(*quicklist)); quicklist->head = quicklist->tail = NULL; quicklist->len = 0; quicklist->count = 0; quicklist->compress = 0; quicklist->fill = -2; return quicklist; }
2. Add elements
Continue to look at the listTypePush method in the source code above:
void listTypePush(robj *subject, robj *value, int where) { if (subject->encoding == OBJ_ENCODING_QUICKLIST) { int pos = (where == LIST_HEAD) ? QUICKLIST_HEAD : QUICKLIST_TAIL; value = getDecodedObject(value); size_t len = sdslen (value-> ptr); // Calculate the length of the new element quicklistPush (subject-> ptr, value-> ptr, len, pos); // Add to quicklist decrRefCount (value); } else { serverPanic("Unknown list encoding"); } }
Continue to watch quicklistPush:
/* Wrapper to allow argument-based switching between HEAD/TAIL pop */ void quicklistPush(quicklist *quicklist, void *value, const size_t sz, int where) { if (where == QUICKLIST_HEAD) { // 添加到list头部 quicklistPushHead(quicklist, value, sz); } else if ( where == QUICKLIST_TAIL) { // Add to the end of the list quicklistPushTail (quicklist, value, sz); } } /* Add new entry to head node of quicklist. * * Returns 0 if used existing head. * Returns 1 if new head created. Add a new element to the head node of quicklist: If the new element is added to the head, it returns 0, otherwise it returns 1. */ int quicklistPushHead(quicklist *quicklist, void *value, size_t sz) { quicklistNode * orig_head = quicklist-> head; // If head is not empty and the size of the space meets the storage requirements of the new element, the new element is added to the head, otherwise a quicklistNode if (likely ( _quicklistNodeAllowInsert(quicklist->head, quicklist->fill, sz))) { quicklist->head->zl = ziplistPush(quicklist->head->zl, value, sz, ZIPLIST_HEAD); quicklistNodeUpdateSz(quicklist->head); } else { // Create a new quicklistNode quicklistNode * node = quicklistCreateNode (); // Add new elements to the newly created ziplist node-> zl = ziplistPush (ziplistNew (), value, sz, ZIPLIST_HEAD); // Update ziplist The length to the sz field of quicklistNode, and then add the new node to quicklist, which is added to the quicklistNodeUpdateSz (node) in front of the original head ; _quicklistInsertNodeBefore(quicklist, quicklist->head, node); } quicklist->count++; quicklist->head->count++; return (orig_head != quicklist->head); }
ziplistpush will add new elements to the ziplist, which will be analyzed later in this article.
3. Get elements
The method of obtaining elements is the quicklistPop method (quicklist.c), as follows:
/* Default pop function * * Returns malloc'd value from quicklist */ int quicklistPop(quicklist *quicklist, int where, unsigned char **data, unsigned int *sz, long long *slong) { unsigned char * vstr; unsigned int vlen; long long vlong; if (quicklist->count == 0) return 0; // pop一个元素 int ret = quicklistPopCustom(quicklist, where, &vstr, &vlen, &vlong, _quicklistSaver); if (data) *data = vstr; if (slong) *slong = vlong; if (sz) *sz = vlen; return ret; }
The specific implementation is in quicklistPopCustom:
/* pop from quicklist and return result in 'data' ptr. Value of 'data' * is the return value of 'saver' function pointer if the data is NOT a number. * * If the quicklist element is a long long, then the return value is returned in * 'cool'. * * Return value of 0 means no elements available. * Return value of 1 means check 'data' and 'sval' for values. * If 'data' is set, use 'data' and 'sz'. Otherwise, use 'sval'. If there are no elements in quicklist, return 0, otherwise return 1. When returning 1, you need to check the two fields of data and sval: 1. If it is of type string, store the result address in the data pointer and the length in sz. 2. If it is of type long long, store the value in the sval field. */ int quicklistPopCustom(quicklist *quicklist, int where, unsigned char **data, unsigned int *sz, long long *sval, void *(*saver)(unsigned char *data, unsigned int sz)) { unsigned char *p; unsigned char * vstr; unsigned int vlen; long long vlong; int pos = (where == QUICKLIST_HEAD) ? 0 : -1; if (quicklist->count == 0) return 0; if (data) *data = NULL; if (sz) *sz = 0; if (sval) *sval = -123456789; quicklistNode *node; if (where == QUICKLIST_HEAD && quicklist->head) { node = quicklist->head; } else if (where == QUICKLIST_TAIL && quicklist->tail) { node = quicklist->tail; } else { return 0; } // p: 0 takes the first element of ziplist; -1 takes the last element of ziplist ; p = ziplistIndex (node-> zl, pos); if (ziplistGet (p, & vstr, & vlen, & vlong)) { if (vstr) { if (data) * data = saver (vstr, vlen); if (sz) * sz = vlen; } else { if (data) *data = NULL; if (sval) *sval = vlong; } // Remove the element from quicklist quicklistDelIndex (quicklist, node, & p); return 1 ; } return 0; }
Look at quicklistDelIndex again:
/* Delete one entry from list given the node for the entry and a pointer * to the entry in the node. * * Note: quicklistDelIndex() *requires* uncompressed nodes because you * already had to get *p from an uncompressed node somewhere. * * Returns 1 if the entire node was deleted, 0 if node still exists. * Also updates in/out param 'p' with the next offset in the ziplist. Remove an entry from quicklistNode: 1. Delete entry from ziplist. 2. The number of entries in quicklistNode is reduced by 1: If the number of entries in quicklistNode is 0, delete the current quicklistNode from quicklist. Otherwise, update the sz field in quicklistNode. */ REDIS_STATIC int quicklistDelIndex(quicklist *quicklist, quicklistNode *node, unsigned char **p) { int gone = 0; node -> zl = ziplistDelete (node-> zl, p); node->count--; if (node->count == 0) { gone = 1; __quicklistDelNode(quicklist, node); } else { quicklistNodeUpdateSz(node); } quicklist->count--; /* If we deleted the node, the original node is no longer valid */ return gone ? 1 : 0; }
At this point, the main structure code of quicklist, and the code of the main two methods push and pop are over. The next article analyzes the source code of the quicklist bottom storage ziplist.