BitMap function demonstration
Our check-in function can be completely completed through mysql, for example, the following table
A user's check-in is a record. If there are 10 million users and the average number of check-ins per person per year is 10, then the amount of data in this table per year will be 100 million.
Each check-in requires a total of 22 bytes of memory (8 + 8 + 1 + 1 + 3 + 1), which requires at least 600 bytes per month. This solution consumes too much memory.
We can use a solution like this to realize our check-in needs.
We count user sign-in information on a monthly basis.The sign-in record is 1, and the non-sign-in record is 0.
Each bit corresponds to each day of the month, forming a mapping relationship. Using 0 and 1 to mark the business status is called a BitMap. In this way, we can use a very small space to represent a large amount of data.
Redis uses string type data structure to implement BitMap, so the maximum upper limit is 512M, which is 2^32 bits when converted into bits.
The operating commands of BitMap are:
SETBIT: Store a 0 or 1 in the specified position (offset)
GETBIT: Get the bit value of the specified position (offset)
BITCOUNT: Counts the number of bits with a value of 1 in the BitMap
BITFIELD: operate (query, modify, increment) the value of the specified position (offset) in the bit array in BitMap
BITFIELD_RO: Get the bit array in BitMap and return it in decimal form
BITOP: Perform bit operations (AND, OR, XOR) on the results of multiple BitMap
BITPOS: Find the position where the first 0 or 1 appears in the specified range in the bit array
Use setbit to assign a value, which is used to set the check-in status. If it is not assigned a value, it will be initialized to 0.
Use getbit to get check-in status
Implement the check-in function
Requirements: Implement the check-in interface and save the current user's check-in information for the day to Redis.
Idea: We can use the year and month as the key of the bitMap, and then save it in a bitMap. Every time you check in, go to the corresponding position and change the number from 0 to 1,As long as the corresponding value is 1, it means that you have signed in on this day, otherwise, you have not signed in.
UserController
@PostMapping("/sign")
public Result sign(){
return userService.sign();
}UserServiceImpl
Since the front end does not pass the corresponding time parameters, we only need to get them ourselves on the back end.
@Override
public Result sign() {
// 1.获取当前登录用户
Long userId = UserHolder.getUser().getId();
// 2.获取日期
LocalDateTime now = LocalDateTime.now();
// 3.拼接key
String keySuffix = now.format(DateTimeFormatter.ofPattern(":yyyyMM"));
String key = USER_SIGN_KEY + userId + keySuffix;
// 4.获取今天是本月的第几天
int dayOfMonth = now.getDayOfMonth();
// 5.写入Redis SETBIT key offset 1
stringRedisTemplate.opsForValue().setBit(key, dayOfMonth - 1, true);
return Result.ok();
}Since today is the 5th, the fifth digit from left to right is 1
Sign-in statistics
Starting from the last check-in and counting forward until the first failure to check-in is encountered, the total number of check-ins is calculated, which is the number of consecutive check-in days.
Java logic code: Get the last check-in data of the current month, define a counter, and then keep counting forward until the first non-0 number is obtained. Every time a non-0 number is obtained, the counter + 1, until After traversing all the data, you can get the total number of check-in days in the current month.
Suppose today is the 10th, then we can start from the first day of the current month and get the number of digits of the current day. If it is the 10th, then it is 10 digits. If we get the data for this period, we can get all The data is there, so how many times have you checked in in these 10 days? Just count how many 1's there are. We only need to execute the following redis command
BITFIELD key GET u[dayOfMonth] 0
We also need to solve the problem of how to traverse these bits from back to front
Note: bitMap returns data in decimal format. If it returns a number 8, how do I know which ones are 0? , which ones are 1? We only need to do the AND operation between the obtained decimal number and 1, because 1 is 1 only when it meets 1, and other numbers are 0. We perform the AND operation between the sign-in result and 1, each time, and push it in sequence, and we can complete the effect of traversing one by one. just move the check-in result one position to the right
Requirement: Implement the following interface to count the number of consecutive check-in days of the current user as of the current time in this month.
If a user has time, we can organize the corresponding key. At this time, we can find all the check-in records of this user as of this day. Then based on this set of algorithms, we can count the number of consecutive check-ins he has had.
UserController
@GetMapping("/sign/count")
public Result signCount(){
return userService.signCount();
}UserServiceImpl
@Override
public Result signCount() {
// 1.获取当前登录用户
Long userId = UserHolder.getUser().getId();
// 2.获取日期
LocalDateTime now = LocalDateTime.now();
// 3.拼接key
String keySuffix = now.format(DateTimeFormatter.ofPattern(":yyyyMM"));
String key = USER_SIGN_KEY + userId + keySuffix;
// 4.获取今天是本月的第几天
int dayOfMonth = now.getDayOfMonth();
// 5.获取本月截止今天为止的所有的签到记录，返回的是一个十进制的数字 BITFIELD sign:5:202203 GET u14 0
List<Long> result = stringRedisTemplate.opsForValue().bitField(
key,
BitFieldSubCommands.create()
.get(BitFieldSubCommands.BitFieldType.unsigned(dayOfMonth)).valueAt(0)
);
if (result == null || result.isEmpty()) {
// 没有任何签到结果
return Result.ok(0);
}
Long num = result.get(0);
if (num == null || num == 0) {
return Result.ok(0);
}
// 6.循环遍历
int count = 0;
while (true) {
// 6.1.让这个数字与1做与运算，得到数字的最后一个bit位 // 判断这个bit位是否为0
if ((num & 1) == 0) {
// 如果为0，说明未签到，结束
break;
}else {
// 如果不为0，说明已签到，计数器+1
count++;
}
// 把数字右移一位，抛弃最后一个bit位，继续下一个bit位
num >>>= 1;
}
return Result.ok(count);
}UV statistics
Related concepts
UV: The full name is Unique Visitor, also called unique visitors, which refers to natural people who access and browse this webpage through the Internet. If the same user visits the website multiple times within a day, it will only be recorded once.
PV: The full name is Page View, also called page views or clicks. Every time a user visits a page on the website, one PV is recorded. If the user opens the page multiple times, multiple PVs are recorded. Often used to measure website traffic.
Generally speaking, UV is much larger than PV, so when measuring the number of visits to the same website, we need to consider many factors, so we simply use these two values as a reference value.
It will be more troublesome to do UV statistics on the server side, because in order to determine whether the user has been counted, the counted user information needs to be saved. But if every visiting user is saved to Redis, the amount of data will be very scary, so how to deal with it?
Hyperloglog (HLL) is a probabilistic algorithm derived from the Loglog algorithm and is used to determine the cardinality of very large sets without the need to store all of its values. For related algorithm principles, you can refer to: Explanation of the principles of the HyperLogLog algorithm and how Redis applies it - Nuggets HLL in Redis is implemented based on the string structure , the memory of a single HLL is always less than 16kb, memory usage is low People are outraged! As a trade-off, the measurement results are probabilistic, with an error of less than 0.81%. But for UV statistics, this is completely negligible.
Millions of data tests
test code
String[] users = new String[1000];
int index = 0;
for (int i = 1; i < 1000000; i++) {
index = i % 1000;
users[index++] = "user_" + i;
if (i % 1000 == 0) {
index = 0;
stringRedisTemplate.opsForHyperLogLog().add("hills", users);
}
}
Long size = stringRedisTemplate.opsForHyperLogLog().size("hills");
System.out.println("size=" + size);The memory space of redis before testing
After testing, it was found that the memory consumption is indeed not large.
