1. Background
Recently I saw the number generator algorithm on Sina Weibo, and I thought I would also study the principles, and then check various information. To this end, I consolidated bit operations, computer original code, complement code, inverse code, etc. related information.
It feels good because there is one thing or goal that drives you to learn. I feel good. If you are aimless, it will be difficult to persist, so I recommend it to everyone.
2. Why should we implement the number sender?
In many places we need a globally unique number, which is a uuid. To give a common scenario, when the e-commerce system generates an order, it needs a corresponding order number. On composer we can also see that there are many excellent components that can generate uuid. So, why do we have to implement the number sender ourselves to generate uuid? After thinking about it, there are two main reasons:
1. I hope that the uuid can be desolved. By deconstructing the uuid, I can get data related to my business. The components related to uuid in composer that I saw generated a string of strings in a specified format, and it was difficult for me to associate it with a specific business.
2. I hope that uuid can be adjusted along with the increase in volume. For example, the original support can generate 1,000 uuid in one second, but as the business scale grows, I hope to support the generation of 10,000 uuid in one second. Moreover, it is best to just change the configuration.
For the above two reasons, we need our own numberer to generate uuid. So, the next question is, how should we implement the number sender, and what is the principle of implementing the number sender?
3. snowFlake algorithm
Regarding the implementation principle of the number generator, you may have heard of the famous snowflake algorithm --Snowflake algorithm, Twitter's distributed self-increasing ID algorithm,
Twitter's distributed self-increasing ID algorithm uses long (8 × 8 = 64 byte) to save uuid. Among them, 1 bit is reserved for fixed sign bit 0, 41 bits are reserved for millisecond timestamp, 10 bits are reserved for MachineID, which means that the machine must be pre-configured, and the remaining 12 bits are reserved for Sequence (which can support 4096 requests within 1 millisecond).
Some people may ask what to do if 4096 requests exceed 1 millisecond? The general approach is to let it wait for 1 millisecond to cause the 41-bit timestamp to change.
Here we split the MachineId, 5byte is reserved for the machine (can support up to 32 machines), and 5byte is reserved for the business number (can support up to 32 businesses)
The timestamp here saves the difference between the current time and the fixed past time, not the current time. The advantage of this is that it can be used for a longer period of time and is not limited by the year. It only depends on when it is used, 2^41 / 1000360024*365=69 years.
If the current timestamp is saved, it can only be used up to 2039. 2^41=2199023255552=2039/9/7 23:47:35
Theoretical stand-alone speed: 2^12*1000 = 4 096 000/s
4. How to ensure continuous increase within unit time
Through a preliminary understanding of snowflake, I found that the number issuer is actually based on timestamps, because time is the only natural element. However, how to ensure that the Sequence continues to increase within a unit of time, such as one second or one millisecond, is the key to the implementation of the serializer.
The way we implement it here is relatively simple, directly using the incr of redis for counting, and the corresponding key is the millisecond timestamp. For the purpose of redis memory recycling, we need to set the expiration time for each key. If the key is a timestamp at the second level, the expiration time is 1 second; if the key is a timestamp at the millisecond level, the expiration time is 1 millisecond.
At the same time, in order to ensure that the execution of incr and expire (pexpire) is atomic, we use lua to implement it.
5. demo
<?php
class SignGenerator
{
CONST BITS_FULL = 64;
CONST BITS_PRE = 1;//固定
CONST BITS_TIME = 41;//毫秒时间戳 可以最多支持69年
CONST BITS_SERVER = 5; //服务器最多支持32台
CONST BITS_WORKER = 5; //最多支持32种业务
CONST BITS_SEQUENCE = 12; //一毫秒内支持4096个请求
CONST OFFSET_TIME = "2021-11-12 00:00:00";//时间戳起点时间
/**
* 服务器id
*/
protected $serverId;
/**
* 业务id
*/
protected $workerId;
/**
* 实例
*/
protected static $instance;
/**
* redis 服务
*/
protected static $redis;
/**
* 获取单个实例
*/
public static function getInstance($redis)
{
if(isset(self::$instance)) {
return self::$instance;
} else {
return self::$instance = new self($redis);
}
}
/**
* 构造初始化实例
*/
protected function __construct($redis)
{
if($redis instanceof \Redis || $redis instanceof \Predis\Client) {
self::$redis = $redis;
} else {
throw new \Exception("redis service is lost");
}
}
/**
* 获取唯一值
*/
public function getNumber()
{
if(!isset($this->serverId)) {
throw new \Exception("serverId is lost");
}
if(!isset($this->workerId)) {
throw new \Exception("workerId is lost");
}
do{
$id = pow(2,self::BITS_FULL - self::BITS_PRE) << self::BITS_PRE;
echo "id=",$id,"\n";
//时间戳 41位
$nowTime = (int)(microtime(true) * 1000);
$startTime = (int)(strtotime(self::OFFSET_TIME) * 1000);
$diffTime = $nowTime - $startTime;
$shift = self::BITS_FULL - self::BITS_PRE - self::BITS_TIME;
$id |= $diffTime << $shift;
echo "diffTime=",$diffTime,"\n";
echo "id=",$id,"\n";
//服务器
$shift = $shift - self::BITS_SERVER;
$id |= $this->serverId << $shift;
echo "serverId=",$this->serverId,"\n";
echo "id=",$id,"\n";
//业务
$shift = $shift - self::BITS_WORKER;
$id |= $this->workerId << $shift;
echo "workerId=",$this->workerId,"\n";
echo "id=",$id,"\n";
//自增值
$sequenceNumber = $this->getSequence($id);
echo "sequenceNumber=",$sequenceNumber,"\n";
if($sequenceNumber > pow(2, self::BITS_SEQUENCE)) {
usleep(1000);
} else {
$id |= $sequenceNumber;
return $id;
}
} while(true);
}
/**
* 反解获取业务数据
*/
public function reverseNumber($number)
{
$uuidItem = [];
$shift = self::BITS_FULL - self::BITS_PRE - self::BITS_TIME;
$uuidItem['diffTime'] = ($number >> $shift) & (pow(2, self::BITS_TIME) - 1);
$shift -= self::BITS_SERVER;
$uuidItem['serverId'] = ($number >> $shift) & (pow(2, self::BITS_SERVER) - 1);
$shift -= self::BITS_WORKER;
$uuidItem['workerId'] = ($number >> $shift) & (pow(2, self::BITS_WORKER) - 1);
$shift -= self::BITS_SEQUENCE;
$uuidItem['sequenceNumber'] = ($number >> $shift) & (pow(2, self::BITS_SEQUENCE) - 1);
$time = (int)($uuidItem['diffTime']/1000) + strtotime(self::OFFSET_TIME);
$uuidItem['generateTime'] = date("Y-m-d H:i:s", $time);
return $uuidItem;
}
/**
* 获取自增序列
*/
protected function getSequence($id)
{
$lua = <<<LUA
local sequenceKey = KEYS[1]
local sequenceNumber = redis.call("incr", sequenceKey);
redis.call("pexpire", sequenceKey, 1);
return sequenceNumber
LUA;
$sequence = self::$redis->eval($lua, [$id], 1);
$luaError = self::$redis->getLastError();
if(isset($luaError)) {
throw new \ErrorException($luaError);
} else {
return $sequence;
}
}
/**
* @return mixed
*/
public function getServerId()
{
return $this->serverId;
}
/**
* @param mixed $serverId
*/
public function setServerId($serverId)
{
$this->serverId = $serverId;
return $this;
}
/**
* @return mixed
*/
public function getWorkerId()
{
return $this->workerId;
}
/**
* @param mixed $workerId
*/
public function setWorkerId($workerId)
{
$this->workerId = $workerId;
return $this;
}
}
$redis = new Redis;
$redis->connect("127.0.0.1", 6379);
$instance = SignGenerator::getInstance($redis);
$instance->setWorkerId(2)->setServerId(1);
$number = $instance->getNumber();
echo "number".$number."\n";
$item = $instance->reverseNumber($number);
var_dump($item);
?>Run it and see that the generated parameters are the same as those obtained by reverse analysis.
id=0
diffTime=54767209
id=229710323777536
serverId=1
id=229710323908608
workerId=2
id=229710323916800
sequenceNumber=1
number229710323916801
array(5) {
["diffTime"]=>
int(54767209)
["serverId"]=>
int(1)
["workerId"]=>
int(2)
["sequenceNumber"]=>
int(1)
["generateTime"]=>
string(19) "2021-11-12 15:12:47"
}