Teach you how to implement a high-performance delayed message queue based on Redis!

Recently, I am building my own blog back-end system. I need to use the function of delayed tasks, but I only have a set of MySQL and Redis at hand. If it is a bit expensive to build a set of MQ, I want to use redis to implement delayed message queues. In some scenarios, the function of delaying messages can be easily realized by using the regular scan table of the database, but for my actual scenario (such as the counting system), the data is actually stored in redis. If the database is used to implement delayed messages, it will affect the database. There is a lot of pressure.

system design

Here is a reference to the praiseworthy delay queue design

data structure design

event message body

type EventEntity struct {
	EventId    int64
	Topic      string
	Body       string
	EffectTime time.Time
}
复制代码

• EVENT_POOL: Use the hash of redis, which stores the complete information of task events, key=prefix+namespace+topic, field=EventId, val=EventEntity;
• EVENT_BUCKET: Use redis zset, which stores an ordered collection of task events, key=prefix+namespace+topic, score=EffectTime, member=EventId;
• EVENT_QUEUE: Use the redis list, which stores the EventId of the expected consumption task.

The execution flow of the delay queue

1. When a new delayed task comes, a record will be added to the topic corresponding to EVENT_POOL, and the task will also be added to EVENT_BUCKET, sorted by effective time;

2. The handling thread will regularly scan the expired tasks in EVENT_BUCKET, push these tasks to the queue corresponding to the topic of EVENT_QUEUE, and then delete these tasks from EVENT_BUCKET;

3. Each topic of EVENT_QUEUE will have a listening thread. When a task to be consumed in the current topic queue is found, the task will be popped out, and the task details will be queried from EVENT_POOL and handed over to the consumer for consumption.

Code

core code

Publish delayed tasks

func (q *DelayQueue) PublishEvent(ctx context.Context, event *EventEntity) error {
	pipeline := q.redisClient.WithContext(ctx).Pipeline()
	defer pipeline.Close()

    // 向EVENT_POOL中添加任务
	pipeline.HSet(q.genPoolKey(event.Topic), strconv.FormatInt(event.EventId, 10), util.ToJsonString(event))
	// 将任务id添加到EVENT_BUCKET中，按生效时间排序
	pipeline.ZAdd(q.genBucketKey(event.Topic), redis.Z{
		Member: strconv.FormatInt(event.EventId, 10),
		Score:  float64(event.EffectTime.Unix()),
	})
	_, err := pipeline.Exec()
	if err != nil {
		logs.CtxWarn(ctx, "pipeline.Exec", logs.String("err", err.Error()))
		return err
	}
	return nil
}
复制代码

Handling thread scan due tasks

func (q *DelayQueue) carryEventToQueue(topic string) error {
	ctx := context.Background()
	// 扫描zset中到期的任务
	members, err := q.redisClient.WithContext(ctx).ZRangeByScoreWithScores(q.genBucketKey(topic), redis.ZRangeBy{Min: "0", Max: util.ToString(time.Now().Unix())}).Result()
	if err != nil && err != redis.Nil {
		logs.CtxWarn(ctx, "[carryEventToQueue] ZRangeByScoreWithScores", logs.String("err", err.Error()))
		return err
	}
	if len(members) == 0 {
		return nil
	}

	errMap := make(map[string]error)
	// 将任务添加到对应topic的待消费队列里
	for _, m := range members {
		eventId := m.Member.(string)
		err = q.redisClient.WithContext(ctx).LPush(q.genQueueKey(topic), eventId).Err()
		if err != nil {
			logs.CtxWarn(ctx, "[carryEventToQueue] LPush", logs.String("err", err.Error()))
			errMap[eventId] = err
		}
	}

	// 从Bucket中删除已进入待消费队列的事件
	var doneMembers []interface{}
	for _, m := range members {
		eventId := m.Member.(string)
		if _, ok := errMap[eventId]; !ok {
			doneMembers = append(doneMembers, eventId)
		}
	}
	if len(doneMembers) == 0 {
		return nil
	}

	err = q.redisClient.WithContext(ctx).ZRem(q.genBucketKey(topic), doneMembers...).Err()
	if err != nil {
		logs.CtxWarn(ctx, "[carryEventToQueue] ZRem", logs.String("err", err.Error()))
	}
	return nil
}

复制代码

Monitor thread consumption tasks

Here, the BLPop command of List is used. When there is data, it will return immediately, and if there is no data, it will be blocked until there is data; this can avoid wasting resources by regularly scanning the list.

func (q *DelayQueue) runConsumer(topic string, subscriberList []IEventSubscriber) error {
	for {
		ctx := context.Background()
		kvPair, err := q.redisClient.WithContext(ctx).BLPop(60*time.Second, q.genQueueKey(topic)).Result()
		if err != nil {
			logs.CtxWarn(ctx, "[InitOnce] BLPop", logs.String("err", err.Error()))
			continue
		}
		if len(kvPair) < 2 {
			continue
		}

		eventId := kvPair[1]
		data, err := q.redisClient.WithContext(ctx).HGet(q.genPoolKey(topic), eventId).Result()
		if err != nil && err != redis.Nil {
			logs.CtxWarn(ctx, "[InitOnce] HGet", logs.String("err", err.Error()))
			if q.persistFn != nil {
				_ = q.persistFn(&EventEntity{
					EventId: util.String2Int64(eventId),
					Topic:   topic,
				})
			}
			continue
		}
		event := &EventEntity{}
		_ = jsoniter.UnmarshalFromString(data, event)

		for _, subscriber := range subscriberList {
			util.Retry(3, 0, func() (success bool) {
				err = subscriber.Handle(ctx, event)
				if err != nil {
					logs.CtxWarn(ctx, "[InitOnce] subscriber.Handle", logs.String("err", err.Error()))
					return false
				}
				return true
			})
		}

		err = q.redisClient.WithContext(ctx).HDel(q.genPoolKey(topic), eventId).Err()
		if err != nil {
			logs.CtxWarn(ctx, "[InitOnce] HDel", logs.String("err", err.Error()))
		}
	}
}
复制代码

other

1. Graceful shutdown

The DelayQueue object uses three variables: wg, isRunning, and stop to achieve graceful shutdown. For details, please refer to the source code.

type DelayQueue struct {
	namespace   string
	redisClient *redis.Client
	once        sync.Once
	wg          sync.WaitGroup
	isRunning   int32
	stop        chan struct{}
	persistFn   PersistFn
}
复制代码

// gracefully shudown
func (q *DelayQueue) ShutDown() {
	if !atomic.CompareAndSwapInt32(&q.isRunning, 1, 0) {
		return
	}
	close(q.stop)
	q.wg.Wait()
}
复制代码

2. Persistent task after consumption failure

The persistence method persistFn can be set for the DelayQueue object, which is used to persist the task id for manual processing when the monitoring thread consumption task fails.

...

q.redisClient.WithContext(ctx).HGet(q.genPoolKey(topic), eventId).Result()
if err != nil && err != redis.Nil {
	logs.CtxWarn(ctx, "[InitOnce] HGet", logs.String("err", err.Error()))
	if q.persistFn != nil {
		_ = q.persistFn(&EventEntity{
			EventId: util.String2Int64(eventId),
			Topic:   topic,
		})
	}
	continue
}

...
复制代码

source address

redis_delay_queue: github.com/hudingyu/re…