Bitmap, also known as bitmap, is a data structure in redis. What is the meaning of its appearance for redis, what are its functions, and how is it used? What is its realization mechanism? Now let us walk into the world of bitmap.
Introduction to Bitmap
What are bitmaps?
Before the text description, let’s take a look at a picture.
This represents the number 7 in binary (00000111) 2 = (7) 10.
So what does this represent in bitmap?
It indicates that bits 6, 7, and 8 are all 1 in the bitmap, which means that bits 6, 7, and 8 have data. [Note: The low bit of the bitmap is on the left, and the high bit is on the right]
Summary: Each bit in the bitmap represents the number 1, and the index of the array is the storage bit of the bit data. For example, 0010 means that there is data on bit 2, 0100 means that there is data on bit 3, and 1000 means that there is data on bit 4. Then you can imagine, in the scene where a large amount of data needs to be stored, is it more convenient and space-saving to use bitmap than string?
The implementation mechanism of Bitmap?
The bottom layer of the Bitmap is a bytes array, and the array supports automatic expansion. If a certain offset position is set beyond the existing content range, the bit array will be automatically expanded. The memory model is as follows
How to use Bitmap?
Redis has officially configured some operation functions for bitmap
- get bitmap / set bitmap value Find/set all bitmap data
For example
set bitmap1 hello
ok
get bitmap1
"hello"
- bitcount bitmap [startIndex endIndex] counts the number of 1s in the bitmap
For example:
# hello二进制（低位在左）:
# 01101000 01100101 01101100 01101100 01101111
bitcount bitmap1
(Integer) 21
bitcount bitmap 0 1 # 取0-1字符上1的个数（注意不是0、1位）
(Integer)7
- bitpos bitmap bit [startIndex endIndex] Count the position of the first bit (0 or 1) in the bitmap
For example:
bitpos bitmap1 0
(Integer)0 #第一个0在最左边的0号位
bitpos bitmap 0 1
(Integer)8 #从第一个字符开始算起，第一个0在8号位（从左往右数）
- getbit bitmap index /setbit bitmap index value Find the data at a certain position in the bitmap/set the data at a certain position in the bitmap
For example:
getbit bitmap1 0
(Integer) 0
# 下面两步将hello 改成 heolo (“l” -----01101100, “0” ------01101111)
setbit bitmap1 22 1
(Integer) 0
setbit bitmap1 23 1
(Integer) 0
get bitmap1
"heolo"
- Redis 3.2+ new features: bitfield, mainly to solve the disadvantage that setbit/getbit can only operate a single bit.
- get
example:
# 从bitmap1二进制数的0号位开始取4位无符号数【注意：高位在左边】
bitfield bitmap1 get u4 0 #（0110）2
(integer) 6
# 从bitmap1二进制数的1号位开始取4位无符号数【注意：高位在左边】
bitfield bitmap1 get u4 1
(integer) 13 #（1101）2
# 从bitmap1二进制数的1号位开始取4位有符号数【注意：高位在左边】
bitfield bitmap1 get i4 1
(integer) -3 #（1101）2
特殊用法【语法糖】：
bitfield bitmap1 get u4 0 get u4 1 get i4 1
(integer) 6
(integer) 13
(integer) -3
- set
example:
# 将第八位开始的一个字符替换成‘a’
bitfield bitmap1 set u8 8 97
(integer) 101
Get bitmap1
”haolo”
Bitmap application and function
Because bitmap is convenient for storing a large amount of data, it can be applied to scenarios such as storing current online users, calculating the number of current online users, storing users' daily check-in, and calculating the number of consecutive days of user's check-in. In addition, it can also be used for filtering URL black and white lists, and the famous application is Bloom filter.
Scene analysis:
- Calculate the number of current online users
Each customer has a user_id (the default is a number and self-incrementing), and since each user_id is different, the converted binary is also different, then we can use each bit of the bitmap to store the current user_id user Status, 0-indicates that the customer is offline, 1-indicates that the customer is online, and then realize the statistics of online users through the function of bitcount in the bitmap. - Calculate the number of consecutive check-in days of a user
for 365 days a year, then we can use a 365-bit bit array to store the number of consecutive check-in days of the current user. If we have one million users, then only about 365MB of space is needed to store data , Compared with the string string data structure, it has storage advantages.
Java implementation of Bitmap
package bitmap;
import java.util.Arrays;
/**
* 尝试实现一下redis的bitmap
*/
public class MyBitmap {
/**
* init_factor 初始化因子,模仿hashmap
*/
private static final int INIT_FACTOR = 16;
/**
* bitmap扩容临界点1M
* 为了防止浪费内存，1M前每次扩容两倍扩容，否则就1M扩容
*/
private static final int RESIZE_THRESHOLD = 1 * 1024 * 1024 * 8;
/**
* myBits - 位数组
*/
private byte[] myBits;
public MyBitmap() {
myBits = new byte[INIT_FACTOR];
}
/**
* 存放位数组
* @param numb 位的索引
* @param bit 当前的位
* @return this
*/
public MyBitmap setBit(int numb, byte bit) {
if (numb > myBits.length) {
resize(numb);
}
myBits[numb] = bit;
return this;
}
/**
* 获取位数组长度
* @return 位数组长度
*/
public int getLength() {
return myBits.length;
}
/**
* 存放位数组
* @param numb 位的索引
* @param bit 当前的位
* @return this
*/
public byte getBit(int numb) {
if (numb > myBits.length) {
return -1;
}
return myBits[numb];
}
/**
* 扩容
* @param newSize 新的内存空间
*/
private void resize(int newSize) {
byte[] temp;
if (newSize < RESIZE_THRESHOLD) {
// 两倍扩容
newSize = bitTableSizeFor(newSize);
} else if (newSize < Integer.MAX_VALUE) {
// 每次增加1M
newSize = bitTableSizeForOverThreshold(newSize);
} else {
// integer的最大值
newSize = Integer.MAX_VALUE;
}
temp = new byte[newSize];
//复制数据
for (int i = 0; i < myBits.length; i++) {
temp[i] |= myBits[i];
}
myBits = temp;
}
/**
* 获取离目标容量最近的1M的倍数
* @param newSize
* @return 离目标容量最近的1M的倍数
*/
private int bitTableSizeForOverThreshold(
int newSize
) {
int temp = (newSize - 1) >> 23; // 获取高位
return (temp + 1) << 23;
}
/**
* 获取离目标容量最近的2的幂次方,模仿hashmap jdk7.0
* @param newSize
* @return 离目标容量最近的2的幂次方
*/
private int bitTableSizeFor(int newSize) {
int temp = newSize - 1;
temp |= temp >> 1;
temp |= temp >> 2;
temp |= temp >> 4;
temp |= temp >> 8;
temp |= temp >> 16;
return (temp < 0) ? 1 : (temp + 1);
}
public static void main(String[] args) {
MyBitmap map = new MyBitmap();
Stopwatch sw = new Stopwatch();
System.out.println(map.setBit(1000000200,(byte) 1).getBit(1000000200));
System.out.println(map.getLength());
System.out.println("持续时间: " + sw.countTime());
}
}
output result
1
1006632960
持续时间: 0.391