Copyright statement: All articles in this blog are owned by Wuhan Flow Network Technology Co., Ltd., welcome to reprint, please indicate the source.
background
Primary Key (Primary Key), used to uniquely identify each piece of data in the table. Therefore, the most basic requirement for a qualified primary key should be uniqueness.
How to ensure uniqueness? I believe that most developers choose the self-increasing id of the database when they first enter the industry, because this is a very simple way, just configure it in the database. But the advantages and disadvantages of auto-incrementing primary keys are obvious.
The advantages are as follows:
- No coding is required, the database is automatically generated, fast, and stored in order.
- Digital format, takes up little space.
The disadvantages are as follows:
- There is a quantity limit. There is a risk of running out.
- When importing old data, there may be id duplication or id being reset.
- It is too cumbersome to handle the sub-database and sub-table scenarios.
GUID
GUID, globally unique identifier, is a 128-bit digital identifier generated by an algorithm. In an ideal situation, no computer or computer cluster will generate two identical GUIDs, so uniqueness can be guaranteed. But there are also advantages and disadvantages. They are as follows:
The advantages are as follows:
- The only distributed scenario.
- It is convenient to merge data across merge servers.
The disadvantages are as follows:
- Storage space takes up a lot.
- Disordered, poor performance in scenarios involving sorting.
The biggest disadvantage of GUID is disorder, because the primary key of the database is a clustered index by default, and disordered data will cause performance degradation when sorting scenarios are involved. Although an ordered GUID can be generated according to the algorithm, the corresponding storage space is still relatively large.
Concept introduction
So, here comes the focus of this article. If you can optimize the shortcomings of self-increment and GUID, is it a better choice? A good primary key needs to have the following characteristics:
- Uniqueness.
- Increasing orderliness.
- The storage space is as small as possible.
- Distributed support.
The optimized snowflake algorithm can perfectly support the above features.
The following figure is the composition diagram of the snowflake algorithm:
Snowflake id is composed of 1 bit sign bit + 41 bit timestamp + 10 bit working machine id + 12 bit self-incrementing serial number, a long type composed of a total of 64 bits.
1 sign bit : Because the highest bit of long is the sign bit, positive numbers are 0 and negative numbers are 1. We require that the generated id are all positive numbers, so the sign bit value is set to 0.
41-bit time stamp : The largest time stamp that can be represented by 41 bits is 2199023255552 (1L<<41), and the usable time is 2199023255552/(1000 60 60 24 365) ≈69 years. There may be people who can’t think about it. The time stamp 2199023255552 should be 2039-09-07 23:47:35, which is less than 20 years from now. Why did the author calculate 69 years?
In fact, the algorithm of timestamp is the number of milliseconds or seconds elapsed from January 1, 1970 to the designated time. Then we can extend the maximum time that can be expressed by 41-bit timestamp by starting the start time in 2020.
10-digit working machine id : This represents the id corresponding to each machine in the cluster in a distributed scenario, so we need to number each machine. The 10-bit binary system supports up to 1024 machine nodes.
12-bit serial number : self-incremental, supports up to 4096 IDs in milliseconds, that is, up to 4096000 IDs can be generated per second . To put it aside, if you use Snowflake id as the order number, will Taobao's double eleven have this amount of orders per second?
At this point, the structure of the snowflake id algorithm has been introduced, so how to package it into usable components based on this algorithm?
Development program
As a programmer, writing code according to the algorithm logic is a basic operation, but before writing, you need to think clearly about the possible pits in the algorithm. Let's go through the structure of snowflake id again.
First of all, the 41-bit timestamp part does not require special attention. You can use 1970 as the starting time. Anyway, it will be enough for more than ten or twenty years (it's my shit after twenty years). Or, if you think your system can run for more than half a century, then use the time closest to you as the starting time.
Secondly, for the 10-digit work machine id, you can make a number for each machine, 0-1023 you can choose, but it seems a bit silly to do this manually, if there are only two or three machines, it doesn't matter if you match them manually. However, how to configure it in docker or k8s environment? Therefore, we need a function to automatically assign machine IDs. When the program is started, an unused value of 0-1023 is assigned to the current node. At the same time, there may be a situation where a node is restarted, or a version is issued frequently, so that every time a new unused id is generated, the 1024 numbers will be quickly used up. Therefore, we also need to realize the function of automatic recovery of machine id.
To sum up, the algorithm for automatically assigning machine IDs needs to limit the maximum number of generations. Since there is a maximum number limit, the redistribution caused by node restart may quickly use up all the numbers. Then, our algorithm must support number recovery. Features. There are many ways to implement this function, but they all need the help of a database or middleware. The java platform may use zookeeper more, and it can also be implemented with a database (the distributed id algorithm of Baidu and Meituan is based on the snowflake algorithm, with the help of the database Realized), because the author is based on the .net platform platform development, here we use redis to achieve this program.
First, when the program is started, call the redis incr command to obtain the value of an auto-incremented key, and determine whether the key value is less than or equal to the maximum machine id number allowed by the snowflake id. If the conditions are met, the current number is not used yet. The value of the key is the workid of the current node. At the same time,
with the help of the ordered collection command of redis, the key value is added to the ordered collection, and the corresponding timestamp of the current time is used as the score. Then use the background service to refresh the score of the key every specified time.
The reason why the score needs to be refreshed regularly is that we can determine whether the machine node corresponding to the specified key still exists based on the score. For example, if the score is refreshed in 5 minutes set by the program, if the timestamp corresponding to the key's score is 5 minutes ago, it means that the node corresponding to this key is offline. Then this key can be allocated to other nodes again.
Therefore, when the value returned by the incr command of redis is greater than 1024, it means that all the numbers between 0-1023 have been used up, then we can call redisu to get the command of the specified score interval to get the id with score greater than five minutes. The id obtained can be used again. This perfectly solves the problem of machine id recovery and reuse.
Finally, there is also a pit that cannot be ignored, the clock is set back. When explaining this concept formally, let's first look at a story. To be precise, it should be an accident.
During the first Gulf War in February 1991, the US Patriot missile system deployed in Dhahran, Saudi Arabia, failed to track and intercept the incoming Iraqi Scud missile. As a result, the Scud missile hit the US military camp.
Loss: 28 soldiers died, more than 100 injured
Cause of failure: Inaccurate time calculation and computer arithmetic errors caused system failure. From a technical point of view, this is a small truncation error. At that time, the Patriot anti-missile system responsible for the defense of the base had been in continuous operation for 100 hours. For every hour of operation, the clock in the system would have a slight millisecond delay. This was the source of this tragedy of failure. The clock register of the Patriot anti-missile system is designed to be 24 bits, so the time accuracy is limited to 24 bits. After working for a long time, this tiny precision error is gradually magnified. After 100 hours of work, the system time delay is one third of a second.
0.33 seconds is trivial to ordinary people. But for a radar system that needs to track and destroy an aerial missile, this is catastrophic. The Scud missile has an airspeed of Mach 4.2 (1.5 kilometers per second). This "trivial" 0.33 seconds is equivalent to an error of approximately 600 meters. In the Dhahran missile incident, the radar found the missile in the air, but the anti-missile missile was not launched and intercepted because of the clock error.
Because of the millisecond time delay, such a large loss is caused. Imagine if the code we wrote caused the loss of the company's property, would it be caught and sacrificed to heaven? Therefore, we need to pay attention to the issue of clock callback. After talking about such a long period of nonsense, what is a clock back?
To put it simply, the internal timer of the computer cannot be guaranteed to be 100% accurate when it is running for a long time. There is a problem of too fast or too slow. Therefore, a time synchronization mechanism is required. During the time synchronization process, the current The computer time, adjust back, this is the clock back (personal understanding, if there is an error, you can move to the comment area), reference: https://zhuanlan.zhihu.com/p/150340199.
So how to solve the problem of machine callback? Jun and patiently look down.
Before I wrote the code to implement the snowflake algorithm, I read a lot of open source code that implements the snowflake algorithm, and most of the solutions given are waiting. For example, if the obtained timestamp is less than the timestamp corresponding to the previous time, an infinite loop is written for judgment until the currently obtained timestamp is greater than the timestamp corresponding to the previous time. Generally speaking, this approach is okay, because theoretically the time callback caused by the machine will not be too bad, basically in milliseconds, for the program, it will not have much impact. However, this is still not a robust solution.
why would you said this? I don’t know if you have heard of winter time and summer time. I believe that most people don't understand this, because we all use Beijing time in the celestial dynasty. But if you have lived or worked abroad, you may know about winter time or summer time. I won't talk about specific concepts. If you are interested, please Baidu yourself. Here I only explain one phenomenon, that is, countries that use daylight saving time will have the clock back one hour. If you write an infinite loop to solve the callback when generating the id, then I really can't imagine whether you will be sacrificed to heaven, anyway, I will.
I personally think that to solve this problem fundamentally, the best way is to switch to a new workid. However, if you directly obtain the workid recovered 5 minutes ago as I described above, there will still be problems. It may exist before the clock is called back, and the workid has just been offline, then if the workid is reassigned to a clock back at this time If you dial the node for 1 hour, it is very likely that there will be a duplicate id. Therefore, when we obtain the recycled workid from the ordered list, we can obtain it sequentially, that is, obtain the workid with the longest offline time.
The coding idea is over, so how can we take a look at the specific code implementation.
The SnowflakeIdMaker class is the main code to implement this scheme, as shown below:
public class SnowflakeIdMaker : ISnowflakeIdMaker
{
private readonly SnowflakeOption _option;
static object locker = new object();
//最后的时间戳
private long lastTimestamp = -1L;
//最后的序号
private uint lastIndex = 0;
/// <summary>
/// 工作机器长度，最大支持1024个节点，可根据实际情况调整，比如调整为9，则最大支持512个节点，可把多出来的一位分配至序号，提高单位毫秒内支持的最大序号
/// </summary>
private readonly int _workIdLength;
/// <summary>
/// 支持的最大工作节点
/// </summary>
private readonly int _maxWorkId;
/// <summary>
/// 序号长度，最大支持4096个序号
/// </summary>
private readonly int _indexLength;
/// <summary>
/// 支持的最大序号
/// </summary>
private readonly int _maxIndex;
/// <summary>
/// 当前工作节点
/// </summary>
private int? _workId;
private readonly IServiceProvider _provider;
public SnowflakeIdMaker(IOptions<SnowflakeOption> options, IServiceProvider provider)
{
_provider = provider;
_option = options.Value;
_workIdLength = _option.WorkIdLength;
_maxWorkId = 1 << _workIdLength;
//工作机器id和序列号的总长度是22位，为了使组件更灵活，根据机器id的长度计算序列号的长度。
_indexLength = 22 - _workIdLength;
_maxIndex = 1 << _indexLength;
}
private async Task Init()
{
var distributed = _provider.GetService<IDistributedSupport>();
if (distributed != null)
{
_workId = await distributed.GetNextWorkId();
}
else
{
_workId = _option.WorkId;
}
}
public long NextId(int? workId = null)
{
if (workId != null)
{
_workId = workId.Value;
}
if (_workId > _maxWorkId)
{
throw new ArgumentException($"机器码取值范围为0-{_maxWorkId}");
}
lock (locker)
{
if (_workId == null)
{
Init().Wait();
}
var currentTimeStamp = TimeStamp();
if (lastIndex >= _maxIndex)
{
//如果当前序列号大于允许的最大序号，则表示，当前单位毫秒内，序号已用完，则获取时间戳。
currentTimeStamp = TimeStamp(lastTimestamp);
}
if (currentTimeStamp > lastTimestamp)
{
lastIndex = 0;
lastTimestamp = currentTimeStamp;
}
else if (currentTimeStamp < lastTimestamp)
{
//throw new Exception("时间戳生成出现错误");
//发生时钟回拨，切换workId，可解决。
Init().Wait();
return NextId();
}
var time = currentTimeStamp << (_indexLength + _workIdLength);
var work = _workId.Value << _workIdLength;
var id = time | work | lastIndex;
lastIndex++;
return id;
}
}
private long TimeStamp(long lastTimestamp = 0L)
{
var current = (DateTime.Now.Ticks - _option.StartTimeStamp.Ticks) / 10000;
if (lastTimestamp == current)
{
return TimeStamp(lastTimestamp);
}
return current;
}
}
The important logic in the above code is commented, so I won’t explain it in detail here. Just talk about the next few more important points.
First, in the constructor, obtain the configuration information from IOptions, and then calculate the length of the serial number according to the value of WorkIdLength in the configuration. Some people may not understand the reason for this design, so I need to expand it a bit here. When the author developed the first version, the length of the working machine and the length of the serial number were completely specified according to the snowflake algorithm, that is, the length of the working machine id was 10 and the length of the serial number was 12, so the design would have a problem. As I mentioned above, a 10-bit machine ID supports a maximum of 1024 nodes, and a 12-bit serial number supports a maximum of 4096 IDs per millisecond. But if the length of the machine id is changed to 9 and the length of the serial number is changed to 13, then the machine supports a maximum of 512 nodes, which is sufficient in theory. The 13-digit serial number can theoretically generate 8192 per millisecond. Therefore, through such a design, the efficiency and performance of ID generation by a single node can be greatly improved, as well as the number generated per unit time.
In addition, in the Init method, try to obtain an instance of the IDistributedSupport interface, which has two methods. code show as below:
public interface IDistributedSupport
{
/// <summary>
/// 获取下一个可用的机器id
/// </summary>
/// <returns></returns>
Task<int> GetNextWorkId();
/// <summary>
/// 刷新机器id的存活状态
/// </summary>
/// <returns></returns>
Task RefreshAlive();
}
The purpose of this design is also to allow interested readers to more easily expand according to their actual situation. As mentioned above, I rely on redis to implement the dynamic allocation of machine IDs. Maybe some people want to use the database method, so you only need to implement the IDistributedSupport interface method. The following is the code of the implementation class of this interface:
public class DistributedSupportWithRedis : IDistributedSupport
{
private IRedisClient _redisClient;
/// <summary>
/// 当前生成的work节点
/// </summary>
private readonly string _currentWorkIndex;
/// <summary>
/// 使用过的work节点
/// </summary>
private readonly string _inUse;
private readonly RedisOption _redisOption;
private int _workId;
public DistributedSupportWithRedis(IRedisClient redisClient, IOptions<RedisOption> redisOption)
{
_redisClient = redisClient;
_redisOption = redisOption.Value;
_currentWorkIndex = "current.work.index";
_inUse = "in.use";
}
public async Task<int> GetNextWorkId()
{
_workId = (int)(await _redisClient.IncrementAsync(_currentWorkIndex)) - 1;
if (_workId > 1 << _redisOption.WorkIdLength)
{
//表示所有节点已全部被使用过，则从历史列表中，获取当前已回收的节点id
var newWorkdId = await _redisClient.SortedRangeByScoreWithScoresAsync(_inUse, 0,
GetTimestamp(DateTime.Now.AddMinutes(5)), 0, 1, Order.Ascending);
if (!newWorkdId.Any())
{
throw new Exception("没有可用的节点");
}
_workId = int.Parse(newWorkdId.First().Key);
}
//将正在使用的workId写入到有序列表中
await _redisClient.SortedAddAsync(_inUse, _workId.ToString(), GetTimestamp());
return _workId;
}
private long GetTimestamp(DateTime? time = null)
{
if (time == null)
{
time = DateTime.Now;
}
var dt1970 = new DateTime(1970, 1, 1);
return (time.Value.Ticks - dt1970.Ticks) / 10000;
}
public async Task RefreshAlive()
{
await _redisClient.SortedAddAsync(_inUse, _workId.ToString(), GetTimestamp());
}
}
The above is the core code of my snowflake id algorithm. The call is also very simple. First, add the following code in Startup:
services.AddSnowflakeWithRedis(opt =>
{
opt.InstanceName = "aaa:";
opt.ConnectionString = "10.0.0.146";
opt.WorkIdLength = 9;
opt.RefreshAliveInterval = TimeSpan.FromHours(1);
});
When you need to call, you only need to get the ISnowflakeIdMaker instance, and then call the NextId method.
idMaker.NextId()
end
At this point, the composition of the snowflake id and the pits that may be encountered during the encoding process have been shared.