Redis from entry to abandonment series (2) Hash

Redis from entry to abandonment series (2) Hash

The example in this article is based on: 5.0.4 Hash is a relatively common data structure in Redis. It is implemented as hashtable/ziplist, and it is ziplist when it is created by default. When it reaches a certain order of magnitude, redis will convert ziplist into hashtable

Redis from entry to abandonment series (1) String

First let's take a look at how to use the Hash type in redis

//将hash表中key的域field的值设为value
//如果key不存在，一个新的哈希表被创建并进行HSET操作
//如果域field已经存在于哈希表中，旧值将被覆盖
hset key field value

Code example:

//创建不存在的field
>hset user:1 id 1
(integer) 1
//覆盖原先的field
>hset user:1 id 2
(integer) 0
>hget user:1 id
"2"
//获取不存在的field
>hget user:1 not_exist
(nil)
----------------------------------
// hsetnx key field value
//当不存在该field 设置成功返回1 ，否则返回0
> hsetnx user:1 id 1
(integer) 1
> hsetnx user:1 id 1
(integer) 0
> hget user:1 id
"1"
----------------------------------
// hmset key field value [field value ....]
//批量设置多个键值对
>HMSET user:1 id 1 name "黑搜丶D" wechat "black-search"
OK
----------------------------------
//hget key field
//获取hash表key中给定的field的值
>hget user:1 id 
"1"
----------------------------------
// hmget key field[field...]
//按照我们输入的field的顺序返回
>hmget user:1 name wechat id  not_exist
1) "黑搜丶D"
2) "black-search"
3) "1"
4) (nil)
----------------------------------
// hdel key field 删除返回被成功移除的域的数量 
> hgetall user:1
1) "id"
2) "1"
3) "name"
4) "black-search"
> HDEL user:1 name
(integer) 1
> HDEL user:1 name
(integer) 0
----------------------------------
// HINCRBY key field increment
// 为hash表某个整数类型的field增加increment ,返回增加increment之后的大小
> hset user:1 wechat "black-search"
(integer) 1
> HINCRBY user:1 wechat 2
(error) ERR hash value is not an integer
> HINCRBY user:1 id 21
(integer) 22
> hget user:1 id
"22"

So far, the usage of redis hash has come to an end.

---

debug object key

At the beginning of this article, it is said that it is created as a ziplist by default. When it reaches a certain level of magnitude, it is converted into a hashtable, so when will it be converted into a hashtable?

# Hashes are encoded using a memory efficient data structure when they have a
# small number of entries, and the biggest entry does not exceed a given
# threshold. These thresholds can be configured using the following directives.
hash-max-ziplist-entries 512
hash-max-ziplist-value 64

From the above we can know that only when we meet the following two conditions will the ziplist be converted into a hashtable structure

1. The number of all key-value pairs saved is less than 512 (this limit is controlled by the hash-max-ziplist-entries parameter, the default is 512)
2. The length of all saved key-value pairs is less than 64 bytes (this limit is controlled by the hash-max-ziplist-value parameter, the default is 64)

// 这里测试当键值对小于等于512时，hash的类型
@RequestMapping("/")
public void test(){
	List<Long> list = redisTemplate.executePipelined(new RedisCallback<Long>() {
		@Override
		public Long doInRedis(RedisConnection redisConnection) throws DataAccessException {
			redisConnection.openPipeline();
			for (int i=0;i<512;i++){
				redisConnection.hSet("key".getBytes(),("field"+i).getBytes(),"value".getBytes());
			}
			return null;
		}
	});
	System.out.println("结束");
}
//我们发现这里hash的类型就是ziplist
> debug object key
Value at:0xbc6f80 refcount:1 encoding:ziplist serializedlength:2603 lru:14344435 lru_seconds_idle:17
//让我们调大一下循环的次数，改为513，我们发现
> debug object key
Value at:0xbc6f80 refcount:1 encoding:hashtable serializedlength:7587 lru:14344656 lru_seconds_idle:4

Source code analysis

//首先我们来看一下dict的结构
typedef struct dict {
    dictType *type;
    void *privdata;
    dictht ht[2];
    long rehashidx; /* rehashing not in progress if rehashidx == -1 */
    unsigned long iterators; /* number of iterators currently running */
} dict;
typedef struct dictType {
    uint64_t (*hashFunction)(const void *key);
    void *(*keyDup)(void *privdata, const void *key);
    void *(*valDup)(void *privdata, const void *obj);
    int (*keyCompare)(void *privdata, const void *key1, const void *key2);
    void (*keyDestructor)(void *privdata, void *key);
    void (*valDestructor)(void *privdata, void *obj);
} dictType;
/* This is our hash table structure. Every dictionary has two of this as we
 * implement incremental rehashing, for the old to the new table. */
typedef struct dictht {
    dictEntry **table;
    unsigned long size;
    unsigned long sizemask;
    unsigned long used;
} dictht;
typedef struct dictEntry {
    void *key;
    union {
        void *val;
        uint64_t u64;
        int64_t s64;
        double d;
    } v;
    struct dictEntry *next;
} dictEntry;

From the above, we can know that the dict contains two dictthts (ps:hashtable), and usually only one dicttht has a value. But when the dict expands/shrinks, it needs to allocate a new dicttht, and then gradually relocate, when After the migration, the old dicttht is deleted and only the new dicttht dict is kept. How to solve the hash conflict? In fact, the principle is the same as that of Java's HashMap, which is solved by using an array + linked list.

progressive rehash

We know that redis is a single process. If it is time-consuming to expand a large dictionary, other requests may be suspended. So redis uses progressive rehash to complete this difficult task~

dictEntry *dictAddRaw(dict *d, void *key, dictEntry **existing)
{
    long index;
    dictEntry *entry;
    dictht *ht;
    //这里每次都会进行搬迁~
    if (dictIsRehashing(d)) _dictRehashStep(d);

    /* Get the index of the new element, or -1 if
     * the element already exists. */
    if ((index = _dictKeyIndex(d, key, dictHashKey(d,key), existing)) == -1)
        return NULL;

    /* Allocate the memory and store the new entry.
     * Insert the element in top, with the assumption that in a database
     * system it is more likely that recently added entries are accessed
     * more frequently. */
    //当字典处于搬迁中，将新添加的元素挂到新的数组下面
    ht = dictIsRehashing(d) ? &d->ht[1] : &d->ht[0];
    entry = zmalloc(sizeof(*entry));
    entry->next = ht->table[index];
    ht->table[index] = entry;
    ht->used++;

    /* Set the hash entry fields. */
    dictSetKey(d, entry, key);
    return entry;
}

In this way, every time the client requests (hset/hdel, etc.), it will judge whether it needs to be relocated, so when the client does not request us, it is possible that there is no complete relocation? no no no redis will scan the dict in the rehash in the scheduled task, and then complete the rest of the relocation~ The code is as follows

/* This function handles 'background' operations we are required to do
 * incrementally in Redis databases, such as active key expiring, resizing,
 * rehashing. */
void databasesCron(void) {
    /* Expire keys by random sampling. Not required for slaves
     * as master will synthesize DELs for us. */
    if (server.active_expire_enabled) {
        if (server.masterhost == NULL) {
            activeExpireCycle(ACTIVE_EXPIRE_CYCLE_SLOW);
        } else {
            expireSlaveKeys();
        }
    }

    /* Defrag keys gradually. */
    if (server.active_defrag_enabled)
        activeDefragCycle();

    /* Perform hash tables rehashing if needed, but only if there are no
     * other processes saving the DB on disk. Otherwise rehashing is bad
     * as will cause a lot of copy-on-write of memory pages. */
    if (server.rdb_child_pid == -1 && server.aof_child_pid == -1) {
        /* We use global counters so if we stop the computation at a given
         * DB we'll be able to start from the successive in the next
         * cron loop iteration. */
        static unsigned int resize_db = 0;
        static unsigned int rehash_db = 0;
        int dbs_per_call = CRON_DBS_PER_CALL;
        int j;

        /* Don't test more DBs than we have. */
        if (dbs_per_call > server.dbnum) dbs_per_call = server.dbnum;

        /* Resize */
        for (j = 0; j < dbs_per_call; j++) {
            tryResizeHashTables(resize_db % server.dbnum);
            resize_db++;
        }

        /* Rehash */
        //重点在这里rehash
        if (server.activerehashing) {
            for (j = 0; j < dbs_per_call; j++) {
                int work_done = incrementallyRehash(rehash_db);
                if (work_done) {
                    /* If the function did some work, stop here, we'll do
                     * more at the next cron loop. */
                    break;
                } else {
                    /* If this db didn't need rehash, we'll try the next one. */
                    rehash_db++;
                    rehash_db %= server.dbnum;
                }
            }
        }
    }
}

Application scenarios

To store business data, we found that the usage of hset is actually very simple. Review the last application scenario of the previous lecture.

//上一讲使用string 
>set user:1 '{"id":1,"name":"黑搜丶D","wechat":"black-search"}'
//让我们使用hash来实现相似的做法
> HMSET user:1 id 1 name "黑搜丶D" wechat "black-search"
OK
//获取key的某个field的值
>hget user:1 wechat
"black-search"
//获取到key的所有 field:value组合
> HGETALL user:1
1) "id"
2) "1"
3) "name"
4) "\xe9\xbb\x91\xe6\x90\x9c\xe4\xb8\xb6D"
5) "wechat"
6) "black-search"

Compared with the usage of string, we can save a lot of bandwidth by using hash to get a field or set a field~

{{o.name}}

{{m.name}}