redis supports multiple data types, sds (simple dynamic string) is the most basic kind, type a string of mostly use redis sds save, which supports dynamic expansion and compression, and offers many utility functions. This article will analyze sds in redis it is how to achieve.
1. sds type
sds in redis in fact, is a char * type alias, the following statement:
typedef char *sds;
However, having sds point storage format string constant rule, i.e., storing the corresponding header information before the data string, the header information includes the:. 1 alloc- memory space allocated length. 2. len- effective string length. 3. flags- head type.
There redis sdshdr5, sdshdr8, sdshdr16, sdshdr32, sdshdr64 this category 5 type head, which is declared as follows:
struct __attribute__ ((__packed__)) sdshdr8 {
uint8_t len; /* used */
uint8_t alloc; /* excluding the header and null terminator */
unsigned char flags; /* 3 lsb of type, 5 unused bits */
char buf[];
};
Several other similar types, but with alloc len field type to use according to different head types, sdshdr16 use uint16_t, sdshdr32 use uint32_t, sdshdr64 use uint64_t. In addition, different types of head sdshdr5 other, with no alloc len field, and the actual length of the string stored in the high 5bits flags field.
Because of uint8_t sdshdr8 alloc type, thus it can be represented by the string up to 255 bytes; alloc due to uint16_t sdshdr16 string type, can be represented by up to 65536 bytes; similarly, Redis select the most suitable head unit to store the string, when little space saving (I believe that when a shorter string, using sdshdr8 save space, but when the string is longer than the range represented uint8_t using sdshdr16 and sdshdr32, the proportion of the length of the head space used not much differences). What the head selection function used to achieve the following:
static inline char sdsReqType(size_t string_size) { if (string_size < 1<<5) return SDS_TYPE_5; if (string_size < 1<<8) return SDS_TYPE_8; if (string_size < 1<<16) return SDS_TYPE_16; #if (LONG_MAX == LLONG_MAX) if (string_size < 1ll<<32) return SDS_TYPE_32; return SDS_TYPE_64; #else return SDS_TYPE_32; #endif }
2. sds operation
2.1 sds conversion between an actual memory allocation type
sds type points to a valid character string initial position, the effective storage space of the header information is a string of uniform distribution, continuous, the length of the forward movement of the head sds their memory space, the actual distribution can be obtained memory starting address. Sds header length is determined by the type head, which is implemented as follows:
static inline int sdsHdrSize(char type) { switch(type&SDS_TYPE_MASK) { case SDS_TYPE_5: return sizeof(struct sdshdr5); case SDS_TYPE_8: return sizeof(struct sdshdr8); case SDS_TYPE_16: return sizeof(struct sdshdr16); case SDS_TYPE_32: return sizeof(struct sdshdr32); case SDS_TYPE_64: return sizeof(struct sdshdr64); } return 0; }
sds head type 3bits flags stored in the lower head, the head type can be obtained by the following manner
S = oldtype [- . 1 ] & SDS_TYPE_MASK; // S type is sds
The actual distribution obtained from the memory address in the following manner
sh = (char*)s-sdsHdrSize(oldtype);
This is, sdsfree operation to achieve the following:
void sdsfree(sds s) { if (s == NULL) return; s_free((char*)s-sdsHdrSize(s[-1])); }
2.2 distribution sds type c_str
Sds new type of function is declared as:
sds sdsnewlen(const void *init, size_t initlen)
It may be generally described as follows steps:
- The appropriate length selection c_str sds head type, this step () function is realized by sdsReqType
- Allocated enough space to store the valid header string (sds end byte string memory requires a '\ 0'), this step is implemented by function s_malloc ().
- Set header information, content copy to the c_str sds in.
Specific achieve the following functions:
sds sdsnewlen(const void *init, size_t initlen) { void *sh; sds s; char type = sdsReqType(initlen); /* Empty strings are usually created in order to append. Use type 8 * since type 5 is not good at this. */ if (type == SDS_TYPE_5 && initlen == 0) type = SDS_TYPE_8; int hdrlen = sdsHdrSize(type); unsigned char *fp; /* flags pointer. */ sh = s_malloc(hdrlen+initlen+1); if (init==SDS_NOINIT) init = NULL; else if (!init) memset(sh, 0, hdrlen+initlen+1); if (sh == NULL) return NULL; s = (char*)sh+hdrlen; fp = ((unsigned char*)s)-1; switch(type) { case SDS_TYPE_5: { *fp = type | (initlen << SDS_TYPE_BITS); break; } case SDS_TYPE_8: { SDS_HDR_VAR(8,s); sh->len = initlen; sh->alloc = initlen; *fp = type; break; } case SDS_TYPE_16: { SDS_HDR_VAR(16,s); sh->len = initlen; sh->alloc = initlen; *fp = type; break; } case SDS_TYPE_32: { SDS_HDR_VAR(32,s); sh->len = initlen; sh->alloc = initlen; *fp = type; break; } case SDS_TYPE_64: { SDS_HDR_VAR(64,s); sh->len = initlen; sh->alloc = initlen; *fp = type; break; } } if (initlen && init) memcpy(s, init, initlen); s[initlen] = '\0'; return s; }
2.3 sds to increase the storage space
Increase the storage space for the type already exists sds operation function is declared as:
sds sdsMakeRoomFor(sds s, size_t addlen)
It steps as follows:
- Is there enough free space to hold the string length addlen, there is returned, no other operation is continued in view sds.
- Calculating a desired length reallocate storage space, including the original length sds addlen, additional preliminary portion of the remaining space.
- The new length, obtain a new type sds head, if the head of the same type as the new type of original is used to allocate more space s_realloc; if the new head type is not the same type as the original, using s_alloc reallocated memory, and the original copy sds content to the newly allocated space.
Specific achieve the following functions:
sds sdsMakeRoomFor(sds s, size_t addlen) { void *sh, *newsh; size_t avail = sdsavail(s); size_t len, newlen; char type, oldtype = s[-1] & SDS_TYPE_MASK; int hdrlen; /* Return ASAP if there is enough space left. */ if (avail >= addlen) return s; len = sdslen(s); sh = (char*)s-sdsHdrSize(oldtype); newlen = (len+addlen); if (newlen < SDS_MAX_PREALLOC) newlen *= 2; else newlen += SDS_MAX_PREALLOC; type = sdsReqType(newlen); /* Don't use type 5: the user is appending to the string and type 5 is * not able to remember empty space, so sdsMakeRoomFor() must be called * at every appending operation. */ if (type == SDS_TYPE_5) type = SDS_TYPE_8; hdrlen = sdsHdrSize(type); if (oldtype==type) { newsh = s_realloc(sh, hdrlen+newlen+1); if (newsh == NULL) return NULL; s = (char*)newsh+hdrlen; } else { /* Since the header size changes, need to move the string forward, * and can't use realloc */ newsh = s_malloc(hdrlen+newlen+1); if (newsh == NULL) return NULL; memcpy((char*)newsh+hdrlen, s, len+1); s_free(sh); s = (char*)newsh+hdrlen; s[-1] = type; sdssetlen(s, len); } sdssetalloc(s, newlen); return s; }
2.4 release unused storage space sds
sds recording head alloc allocated memory space, the memory space len recorded actual use, when required to release unused memory space according len recording may be used or the use of a compression space s_realloc s_alloc redistribute smaller space, release old memory. The corresponding function is declared as:
sds sdsRemoveFreeSpace(sds s)
Substantially following steps:
- Calculating a new desired sds type memory according len.
- If the new type sds same as the original type, the use of compression s_realloc original memory space.
- If the new type sds original type, does not use s_alloc reallocate space to copy the original content sds newly allocated, and the release of the original memory.
sdsRemoveFreeSpace sds sdsMakeRoomFor achieved with substantially the same, not listed here.
Further sds tool provides many functions, such as cat operation, Copy operation, ll2str (Integer String), vprintf, operation (formatting operation) and the like. See the specific implementation source code file sds.c.