Distributed ID generation scheme
- UUID
- database self-increment
- number segment mode
- Redis implementation
- Snowflake algorithm (SnowFlake)
- Baidu Uidgenerator
- Meituan Leaf
- Didi TinyID
This article focuses on Leaf and TinyID that can auto-increment ID
number segment mode
This mode is also a method for generating distributed IDs. The idea is to obtain a range of numbers from the database, such as [1,1000], generate auto-increment IDs from 1 to 1000, and load them into memory. The table structure is as follows :
CREATE TABLE id_generator (
id int(10) NOT NULL,
max_id bigint(20) NOT NULL COMMENT '当前最大id',
step int(20) NOT NULL COMMENT '号段的布长',
biz_type int(20) NOT NULL COMMENT '业务类型',
version int(20) NOT NULL COMMENT '版本号',
PRIMARY KEY (`id`)
)
biz_type: different business types
max_id: the current largest id
step: the step size of the representative number segment
version: the version number, just like MVCC, can be understood as an optimistic lock
, wait for the ID to be used, then go to the database to get it, and then change the maximum value
update id_generator set max_id = #{max_id+step}, version = version + 1 where version = # {version} and biz_type = XXX
Advantages: There are relatively mature solutions, such as Baidu Uidgenerator, Meituan Leaf
Disadvantages: Depends on database implementation
Meituan Leaf
Leaf provides two ways to generate IDs: number segment mode (Leaf-segment) and snowflake mode (Leaf-snowflake). You can enable both methods at the same time, or you can specify a method to enable. The default is to disable the two methods.
git location
GitHub - Meituan-Dianping/Leaf: Distributed ID Generate Service
Introduction document
https://github.com/Meituan-Dianping/Leaf/blob/master/README_CN.md
Create data table
CREATE DATABASE leaf
CREATE TABLE `leaf_alloc` (
`biz_tag` varchar(128) NOT NULL DEFAULT '',
`max_id` bigint(20) NOT NULL DEFAULT '1',
`step` int(11) NOT NULL,
`description` varchar(256) DEFAULT NULL,
`update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`biz_tag`)
) ENGINE=InnoDB;
insert into leaf_alloc(biz_tag, max_id, step, description) values('leaf-segment-test', 1, 2000, 'Test leaf Segment Mode Get Id')get item
git clone [email protected]:Meituan-Dianping/Leaf.gitUpgrade mysql driver
If the connection is above mysql1.8, you need to upgrade the mysql driver, respectively
Leaf/poe.xml and leaf_core/poe.xml
# Leaf/poe.xml
<mybatis-spring.version>1.2.5</mybatis-spring.version>
#leaf_core/poe.xml
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.25</version>
</dependency>
Change setting
The following is the configuration of the number segment mode
leaf_server/resouces/leaf.properties
leaf.name=com.sankuai.leaf.opensource.test
leaf.segment.enable=true
leaf.jdbc.url=jdbc:mysql://127.0.0.1:3306/leaf?useSSL=false
leaf.jdbc.username=root
leaf.jdbc.password=123456
leaf.snowflake.enable=false
#leaf.snowflake.zk.address=
#leaf.snowflake.port=Pack
cd leaf
mvn clean install -DskipTests
run
cd leaf-server
#mvn method
mvn spring-boot:run
#script method
sh deploy/run.sh
test
#segment
curl http://localhost:8080/api/segment/get/leaf-segment-test
#snowflake
curl http://localhost:8080/api/snowflake/get/test
Monitoring page
number segment mode: http:// localhost:8080/cache#Note that after adding a biz_type, it will take effect after 10s .
Didi TinyID
github location
GitHub - request/request: ID Generator identification
Introduction document
get item
git clone https://github.com/didi/tinyid.gitCreate data table
tinyid-server/db.sql
CREATE TABLE `tiny_id_info` (
`id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '自增主键',
`biz_type` varchar(63) NOT NULL DEFAULT '' COMMENT '业务类型，唯一',
`begin_id` bigint(20) NOT NULL DEFAULT '0' COMMENT '开始id，仅记录初始值，无其他含义。初始化时begin_id和max_id应相同',
`max_id` bigint(20) NOT NULL DEFAULT '0' COMMENT '当前最大id',
`step` int(11) DEFAULT '0' COMMENT '步长',
`delta` int(11) NOT NULL DEFAULT '1' COMMENT '每次id增量',
`remainder` int(11) NOT NULL DEFAULT '0' COMMENT '余数',
`create_time` timestamp NOT NULL DEFAULT '2010-01-01 00:00:00' COMMENT '创建时间',
`update_time` timestamp NOT NULL DEFAULT '2010-01-01 00:00:00' COMMENT '更新时间',
`version` bigint(20) NOT NULL DEFAULT '0' COMMENT '版本号',
PRIMARY KEY (`id`),
UNIQUE KEY `uniq_biz_type` (`biz_type`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COMMENT 'id信息表';
CREATE TABLE `tiny_id_token` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT COMMENT '自增id',
`token` varchar(255) NOT NULL DEFAULT '' COMMENT 'token',
`biz_type` varchar(63) NOT NULL DEFAULT '' COMMENT '此token可访问的业务类型标识',
`remark` varchar(255) NOT NULL DEFAULT '' COMMENT '备注',
`create_time` timestamp NOT NULL DEFAULT '2010-01-01 00:00:00' COMMENT '创建时间',
`update_time` timestamp NOT NULL DEFAULT '2010-01-01 00:00:00' COMMENT '更新时间',
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COMMENT 'token信息表';
INSERT INTO `tiny_id_info` (`id`, `biz_type`, `begin_id`, `max_id`, `step`, `delta`, `remainder`, `create_time`, `update_time`, `version`)
VALUES
(1, 'test', 1, 1, 100000, 1, 0, '2018-07-21 23:52:58', '2018-07-22 23:19:27', 1);
INSERT INTO `tiny_id_info` (`id`, `biz_type`, `begin_id`, `max_id`, `step`, `delta`, `remainder`, `create_time`, `update_time`, `version`)
VALUES
(2, 'test_odd', 1, 1, 100000, 2, 1, '2018-07-21 23:52:58', '2018-07-23 00:39:24', 3);
INSERT INTO `tiny_id_token` (`id`, `token`, `biz_type`, `remark`, `create_time`, `update_time`)
VALUES
(1, '0f673adf80504e2eaa552f5d791b644c', 'test', '1', '2017-12-14 16:36:46', '2017-12-14 16:36:48');
INSERT INTO `tiny_id_token` (`id`, `token`, `biz_type`, `remark`, `create_time`, `update_time`)
VALUES
(2, '0f673adf80504e2eaa552f5d791b644c', 'test_odd', '1', '2017-12-14 16:36:46', '2017-12-14 16:36:48');
Change setting
cd tinyid-server/src/main/resources/offline
vi application.properties
server.port=9999
server.context-path=/tinyid
batch.size.max=100000
#datasource.tinyid.names=primary
#如果希望数据库能够高可用，可以设置多个不同节点，两个节点上的数据保持一致。
#注意添加配置的时候，多个节点都要添加
datasource.tinyid.names=primary,secondary
datasource.tinyid.type=org.apache.tomcat.jdbc.pool.DataSource
datasource.tinyid.primary.driver-class-name=com.mysql.jdbc.Driver
datasource.tinyid.primary.url=jdbc:mysql://localhost:3306/db1?autoReconnect=true&useUnicode=true&characterEncoding=UTF-8
datasource.tinyid.primary.username=root
datasource.tinyid.primary.password=123456
#datasource.tinyid.primary.testOnBorrow=false
#datasource.tinyid.primary.maxActive=10
datasource.tinyid.secondary.driver-class-name=com.mysql.jdbc.Driver
datasource.tinyid.secondary.url=jdbc:mysql://localhost:3306/db2?autoReconnect=true&useUnicode=true&characterEncoding=UTF-8
datasource.tinyid.secondary.username=root
datasource.tinyid.secondary.password=123456
datasource.tinyid.secondary.testOnBorrow=false
datasource.tinyid.secondary.maxActive=10
Pack
cd tinyid
mvn clean install -DskipTests
#or
cd tinyid-server/
sh build.sh offline
run
cd tinyid-server/
#sh build.sh offline
java -jar output/tinyid-server-xxx.jar
test
nextId:
curl 'http://localhost:9999/tinyid/id/nextId?bizType=test&token=0f673adf80504e2eaa552f5d791b644c'
response:{"data":[2],"code":200,"message":""}
nextId Simple:
curl 'http://localhost:9999/tinyid/id/nextIdSimple?bizType=test&token=0f673adf80504e2eaa552f5d791b644c'
response: 3
with batchSize:
curl 'http://localhost:9999/tinyid/id/nextIdSimple?bizType=test&token=0f673adf80504e2eaa552f5d791b644c&batchSize=10'
response: 4,5,6,7,8,9,10,11,12,13
Get nextId like 1,3,5,7,9...
bizType=test_odd : delta is 2 and remainder is 1
curl 'http://localhost:9999/tinyid/id/nextIdSimple?bizType=test_odd&batchSize=10&token=0f673adf80504e2eaa552f5d791b644c'
response: 3,5,7,9,11,13,15,17,19,21
client use
client packaging
cd tinyid
mvn clean install -DskipTests
#tinyid-client\target\ tinyid-client-0.1.0-SNAPSHOT.jar is the client that can be used
client profile
tinyid_client.properties
Put tinyid_client.properties under resources
tinyid.server=localhost:9999
tinyid.token=0f673adf80504e2eaa552f5d791b644c
use
public class ClientTest {
@Test
public void testNextId() {
for (int i = 0; i < 100000; i++) {
Long id = TinyId.nextId("test_odd");
System.out.println("current id is: " + id);
}
}
}
Note: Every time you restart the program, the id will add a step forward, no matter whether you have used it up or not, if the program restarts frequently, the step should not be set too large. But the step should not be set too small, otherwise the interaction with the database will be too frequent.
source: