There are no two identical leaves in the world.
There are no two identical leaves in the world.
— Leibniz
After slackening for a period of time, you have to choose between career and love. For me, I can only single-mindedly. Maybe my brain is not enough?
There are no two identical leaves in the world, and there is no perfect ending? Want to know what I chose in the end? Leave a suspense...
I will sort out the materials that I want to study recently. I hope it can bring me some help. I also hope that only a few words can help everyone, and I have a clear conscience.
Introduction：
The earliest requirement of Leaf is the order ID generation requirement of each business line. In the early days of Meituan, some businesses generated IDs directly through DB self-increment, some businesses generated IDs through redis caching, and some businesses directly generated IDs using UUID. The above methods have their own problems, so we decided to implement a set of distributed ID generation services to meet the needs. For specific Leaf design documents, see: leaf Meituan Distributed ID Generation Service
At present, Leaf covers many business lines of Meituan Dianping's internal finance, catering, food delivery, hotel travel, Maoyan movies, etc. On the basis of 4C8G VM, through the company RPC method, the QPS pressure test result is nearly 5w/s, TP999 1ms.
Use leaf-starter annotation to start leaf
git clone [email protected]:Meituan-Dianping/Leaf.git
git checkout feature/spring-boot-starter
cd leaf
mvn clean install -Dmaven.test.skip=true Introduce dependencies
<dependency>
<artifactId>leaf-boot-starter</artifactId>
<groupId>com.sankuai.inf.leaf</groupId>
<version>1.0.1-RELEASE</version>
</dependency>Configure leaf.properties to your classpath
springboot配置：
sagement1：properties
leaf.name=com.sankuai.leaf.opensource.test
leaf.segment.enable=false
#leaf.segment.url=
#leaf.segment.username=
#leaf.segment.password=
sagement2：yaml
leaf:
name: leaf-name
segment:
url: jdbc:mysql://**********.mysql.rds.aliyuncs.com:3306/test?useUnicode=true&characterEncoding=utf-8&serverTimezone=Asia/Shanghai&useSSL=false&zeroDateTimeBehavior=convertToNull&allowMultiQueries=true
username: ******
password: ******
enable: true
snowflake1:
leaf:
snowflake:
enable: false
address: ******
port: *******
snowflake2:
leaf.snowflake.enable=false
#leaf.snowflake.address=
#leaf.snowflake.port=Start leaf with annotations and use api
//EnableLeafServer 开启leafserver
@SpringBootApplication
@EnableLeafServer
public class LeafdemoApplication {
public static void main(String[] args) {
SpringApplication.run(LeafdemoApplication.class, args);
}
}
//直接使用 spring注入
public class T {
@Autowired
private SegmentService segmentService;
@Autowired
private SnowflakeService snowflakeService;
}
The configuration of Leaf Server is in leaf-server/src/main/resources/leaf.properties
| Configuration item | meaning | Defaults |
|---|---|---|
| leaf.name | leaf service name | |
| leaf.segment.enable | Whether to open the number segment mode | false |
| leaf.jdbc.url | mysql library address | |
| leaf.jdbc.username | mysql username | |
| leaf.jdbc.password | mysql password | |
| leaf.snowflake.enable | Whether to enable snowflake mode | false |
| leaf.snowflake.zk.address | zk address in snowflake mode | |
| leaf.snowflake.port | Service registration port in snowflake mode |
Number mode
If you use the number segment mode, you need to create a DB table and configure leaf.jdbc.url, leaf.jdbc.username, leaf.jdbc.password
If you don't want to use this mode, configure leaf.segment.enable=false
Create data table
CREATE DATABASE leaf
CREATE TABLE `leaf_alloc` (
`biz_tag` varchar(128) NOT NULL DEFAULT '',
`max_id` bigint(20) NOT NULL DEFAULT '1',
`step` int(11) NOT NULL,
`description` varchar(256) DEFAULT NULL,
`update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`biz_tag`)
) ENGINE=InnoDB;
insert into leaf_alloc(biz_tag, max_id, step, description) values('leaf-segment-test', 1, 2000, 'Test leaf Segment Mode Get Id')Configure related data items
Configure leaf.jdbc.url, leaf.jdbc.username, leaf.jdbc.password parameters in leaf.properties
Snowflake mode
Leaf-snowflakeBasically it follows the design of snowflake, ID composition structure: 正数位(occupies 1 bit) + 时间戳(occupies 41 bits) + 机器ID(occupies 5 bits) + 机房ID(occupies 5 bits) + 自增值(occupies 12 bits), a Long composed of 64 bits in total Types of
Leaf-snowflakeDifferent from the original snowflake algorithm, it mainly Leaf-snowflakedepends on the Zookeepergeneration of workId workId, which is the above 机器ID(5 bits) + 机房ID(5 bits). LeafThe workId is 顺序Idgenerated based on ZooKeeper . When each application uses Leaf-snowflake, it will generate a sequential Id in Zookeeper when it starts, which is equivalent to a machine corresponding to a sequential node, that is, a workId.
The algorithm is taken from twitter's open source snowflake algorithm.
If you do not want to use this mode, configure leaf.snowflake.enable=false.
Configure zookeeper address
Configure leaf.snowflake.zk.address in leaf.properties and configure the port leaf.snowflake.port that the leaf service listens to.
Run Leaf Server
Package service
git clone [email protected]:Meituan-Dianping/Leaf.git
//按照上面的号段模式在工程里面配置好
cd leaf
mvn clean install -DskipTests
cd leaf-serverRun the service
Tips: First, you must configure the database table or zk address first
ForExample：
Usually when using the number segment mode, the timing of the number segment is taken when the previous number segment is exhausted, but an ID has just been taken, but the database has been updated max_id, which means that one more number segment leafhas been obtained. Number section, what is this operation?
Why is Leaf designed this way?
In order to explore this question, with curiosity, test the results of the research...
Then observe the database changes, max_id becomes 1001 according to step(1000)
answer:
It is for non-blocking in the process of taking number segments in DB!
When the number segment is exhausted, go to the DB to take the next number segment. If the network jitters or the DB has a slow query at this time, the business system cannot get the number segment, which will cause the response time of the entire system to slow down, and the traffic Huge business, this is intolerable.
So Leafwhen the current number segment reaches a certain point, the next number segment is asynchronously loaded into the memory. There is no need to wait until the number segment is exhausted to update the number segment. This greatly reduces the risk of the system.
LeafIn 双bufferthe method adopted , there are two number segment buffers within its service segment. When the current number segment has consumed 10%, and the next number segment is not available, another update thread will be started to update the next number segment.
Segment advantages:
- The Leaf service can be easily expanded linearly, and its performance can fully support most business scenarios.
- High disaster tolerance: The Leaf service has internal number segment cache, even if the DB is down, Leaf can still provide services normally in a short time.
Disadvantages:
- The ID number is not random enough and can reveal information about the number of issued numbers, which is not very secure.
- The DB downtime will cause the entire system to be unavailable (it is possible to use the database).
Advantages of SnowFlake:
- The ID number is a 64-bit number with an increasing trend of 8 bytes, which meets the primary key requirements of the database storage.
Disadvantages:
- Relying on ZooKeeper, there is a risk of service unavailability (I really don’t know the shortcomings)