7.1 bitmap review
1 what is
- bit array representing binary bits by 0 and 1 states
2 what can you do
- for status statistics
- Y, N similar to AutomicBoolean
- need
- Whether the user has logged in Y, N, such as Jingdong daily sign-in to send Jingdong
- Whether movies and advertisements have been clicked to play
- Dingding punch card to work, sign-in statistics
3. Jingdong sign in and receive Jingdong
-
Small factory method, traditional mysql method
-
CREATE TABLE user_sign ( keyid BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT, user_key VARCHAR(200),#京东用户ID sign_date DATETIME,#签到日期(20210618) sign_count INT #连续签到天数 ) INSERT INTO user_sign(user_key,sign_date,sign_count) VALUES ('20210618-xxxx-xxxx-xxxx-xxxxxxxxxxxx','2020-06-18 15:11:12',1); SELECT sign_count FROM user_sign WHERE user_key = '20210618-xxxx-xxxx-xxxx-xxxxxxxxxxxx' AND sign_date BETWEEN '2020-06-17 00:00:00' AND '2020-06-18 23:59:59' ORDER BY sign_date DESC LIMIT 1; -
The number of sign-in users is small, this is ok, how to solve this problem
-
One check-in record corresponds to one record, which will occupy more and more space.
-
There are up to 31 days in a month. It happens that our int type is 32 bits. Then such an int type can handle a month. 32 bits are greater than 31 days. If a bit comes on the day, it is 1 and if it does not come, it is 0.
-
A piece of data directly stores a month's check-in record, instead of storing a day's check-in record.
-
-
-
Big factory method
-
Realize check-in calendar based on redis Bitmap
-
In the sign-in statistics, each user's day's sign-in can be represented by 1 bit,
One month (assuming 31 days) sign-in situation can use 31 bits, and one-year sign-in only needs to use 365 bits, so there is no need for too complicated collection type
-
-
4. Basic commands
SETBIT key offset value // 将第offset的值设为value value只能是0或1 offset 从0开始
GETBIT key offset // 获得第offset位的值
STRLEN key // 得出占多少字节 超过8位后自己按照8位一组一byte再扩容
BITCOUNT key // 得出该key里面含有几个1
BITOP and destKey key1 key2 // 对一个或多个 key 求逻辑并，并将结果保存到 destkey
BITOP or destKey key1 key2 // 对一个或多个 key 求逻辑或，并将结果保存到 destkey
BITOP XOR destKey key1 key2 // 对一个或多个 key 求逻辑异或，并将结果保存到 destkey
BITOP NOT destKey key1 key2 // 对一个或多个 key 求逻辑非，并将结果保存到 destkey
7.2 Bloom filters
1. what is
It consists of a bit array whose initial value is zero and multiple hash functions, which are used to quickly determine whether an element exists in the set
- Purpose
- Reduce memory usage
- Way
- Do not save data information, just make a flag in the memory whether it exists or not
- Nature
- Determine whether the specific data is in a large collection
- Bloom filter is a data structure similar to set, but the statistical results are a little flawed and not perfect under the huge amount of data
- It is actually a very long binary array (00000000) + a series of random hash algorithm mapping functions, mainly used to determine whether an element is in the set.
- Usually we will encounter many business scenarios where we need to judge whether an element is in a certain collection. The general idea is to save all the elements in the collection and then determine through comparison.
- Data structures such as linked lists, trees, hash tables, etc. are all of this kind of thinking. But as the number of elements in the collection increases, the storage space we need will also increase linearly, eventually reaching the bottleneck. At the same time, the retrieval speed is getting slower and slower. The retrieval time complexities of the above three structures are O(n), O(logn), and O(1) respectively. At this time, the Bloom Filter (Bloom Filter) came into being
2. What can I do?
- Insert and query efficiently, occupy less space, and return results with uncertainty + imperfection
- If an element is judged: if it exists, the element does not necessarily exist; if it does not exist, the element must not exist
- Bloom filters can add elements, but cannot delete elements
- When it comes to the hashcode judgment basis, deleting elements will increase the misjudgment rate
- Why can't it be deleted?
- Because it has multiple hash functions, it performs multiple hash operations on a value, and stores 1 in the corresponding position for each value obtained. It is easy to cause this 1 to represent other values. Once deleted, another value cannot pass
3. Implementation principle and data structure
Bloom Filter is an advanced data structure specially designed to solve the deduplication problem.
The essence is a large bit array and several different unbiased hash functions (unbiased means uniform distribution). It consists of a bit array whose initial value is zero and multiple hash functions, which are used to quickly determine whether a certain data exists. But like HyperLogLog, it is also a little imprecise, and there is a certain probability of misjudgment
-
When adding a key
- Use multiple hash functions to perform a hash operation on the key to obtain an integer index value, and perform a modulo operation on the length of the bit array to obtain a position
- Each hash function will get a different position, set these positions to 1 to complete the add operation
-
When querying key
- As long as one of them is zero, it means that the key does not exist, but if they are all 1, the corresponding key does not necessarily exist
-
in conclusion
- yes, maybe
- no sure no
-
Hash conflicts lead to inaccurate data 1
- When querying, check whether these points are 1 or not, and there is a high probability until there is no it in the collection. If any of these points is zero, the variable being queried must not exist. If they are all 1, the variable being queried likely to exist
- It is based on the rapid detection feature of the Bloom filter, we can use the Bloom filter to make a mark when writing data to the database, and when the cache is missing, when the application queries the database, we can quickly judge by querying the Bloom filter Whether the data exists, if not, there is no need to go to the database to query. In this way, even if cache penetration occurs, a large number of requests will only query Redis and Bloom filters, and will not backlog to the database, and will not affect the normal operation of the database. Bloom filters can be implemented using redis, which can bear a lot of concurrent access pressure
-
Hash conflicts lead to inaccurate data 2
-
The concept of a hash function is: a function that converts input data of any size into output data of a specific size. The converted data is called a hash value or hash code, also called a hash value
-
If the two hash values are not identical (according to the same function) then the original input of the two hash values is also not identical.
-
This property is the deterministic result of the hash function, and the hash function with this property is called a one-way hash function.
-
The input and output of the hash function are not uniquely corresponding. If two hash values are the same, the two input values are likely to be the same, but they may also be different. This situation is called "hash collision (collision)"
-
When using a hash table to store a large amount of data, the space efficiency is still very low, and when there is only one hash function, hash collisions are easy to occur.
-
4. Use three steps
- Essentially consists of a bit vector or bit list (a list containing only 0 or 1 bit values) of length m, initially all values are set to 0
-
To add data, in order to avoid address conflicts as much as possible, use multiple hash functions to calculate a subscript index value, and then perform a modulo operation on the data length to obtain a position. Each hash function will calculate a different position. Then set these positions of the bit array to 1 to complete the add operation.
Perform multiple hash(key) on the string → modulo operation → get the pit position
-
When querying the Bloom filter for the existence of a certain key, first operate the key through the same multiple hash functions to check whether the corresponding positions are all 1, as long as one bit is zero, it means that the Bloom filter This key does not exist; if these positions are all 1, then it is very likely to exist;
- Summarize
- does it exist
- yes, maybe
- no sure no
- It is best not to let the actual number of elements be much larger than the initialization number, one time is enough to avoid expansion
- When the actual number of elements exceeds the initialization number, the Bloom filter should be rebuilt, a filter with a larger size should be reassigned, and all historical elements should be added in batches
- does it exist
5. Try to write a simple Bloom filter by hand, combined with bitmap
1. Overall structure
2. Step design
- The construction process of setBit
- @PostConstruct initializes whitelist data
- Calculate element hash value
- Get the hash value to calculate the pit position of the corresponding binary array
- Change the value of the corresponding pit position to the number 1, indicating the existence
- getBit query exists
- Calculate element hash value
- Get the hash value to calculate the pit position of the corresponding binary array
- Return the value of the corresponding pit position, 0 means no, 1 means exists
3 springboot + redis + mybatis + Bloom filter integration
-
Automatically generated using Mapper4
-
Create the mybatis-generator project first
-
Create table sql
CREATE TABLE `t_customer` ( `id` int(20) NOT NULL AUTO_INCREMENT, `cname` varchar(50) NOT NULL, `age` int(10) NOT NULL, `phone` varchar(20) NOT NULL, `sex` tinyint(4) NOT NULL, `birth` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`id`), KEY `idx_cname` (`cname`) ) ENGINE=InnoDB AUTO_INCREMENT=10 DEFAULT CHARSET=utf8mb4 -
pom
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.xfcy</groupId> <artifactId>mybatis_generator</artifactId> <version>1.0-SNAPSHOT</version> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.6.10</version> <relativePath/> </parent> <properties> <!-- 依赖版本号 --> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <maven.compiler.source>1.8</maven.compiler.source> <maven.compiler.target>1.8</maven.compiler.target> <java.version>1.8</java.version> <hutool.version>5.5.8</hutool.version> <druid.version>1.1.18</druid.version> <mapper.version>4.1.5</mapper.version> <pagehelper.version>5.1.4</pagehelper.version> <mysql.version>5.1.39</mysql.version> <swagger2.version>2.9.2</swagger2.version> <swagger-ui.version>2.9.2</swagger-ui.version> <mybatis.spring.version>2.1.3</mybatis.spring.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <!--Mybatis 通用mapper tk单独使用，自己带着版本号--> <dependency> <groupId>org.mybatis</groupId> <artifactId>mybatis</artifactId> <version>3.4.6</version> </dependency> <!--mybatis-spring--> <dependency> <groupId>org.mybatis.spring.boot</groupId> <artifactId>mybatis-spring-boot-starter</artifactId> <version>${ mybatis.spring.version}</version> </dependency> <!-- Mybatis Generator --> <dependency> <groupId>org.mybatis.generator</groupId> <artifactId>mybatis-generator-core</artifactId> <version>1.4.0</version> <scope>compile</scope> <optional>true</optional> </dependency> <!--通用Mapper--> <dependency> <groupId>tk.mybatis</groupId> <artifactId>mapper</artifactId> <version>${ mapper.version}</version> </dependency> <!--persistence--> <dependency> <groupId>javax.persistence</groupId> <artifactId>persistence-api</artifactId> <version>1.0.2</version> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <optional>true</optional> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> <exclusions> <exclusion> <groupId>org.junit.vintage</groupId> <artifactId>junit-vintage-engine</artifactId> </exclusion> </exclusions> </dependency> </dependencies> <build> <resources> <resource> <directory>${ basedir}/src/main/java</directory> <includes> <include>**/*.xml</include> </includes> </resource> <resource> <directory>${basedir}/src/main/resources</directory> </resource> </resources> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> <configuration> <excludes> <exclude> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> </exclude> </excludes> </configuration> </plugin> <plugin> <groupId>org.mybatis.generator</groupId> <artifactId>mybatis-generator-maven-plugin</artifactId> <version>1.3.6</version> <configuration> <configurationFile>${basedir}/src/main/resources/generatorConfig.xml</configurationFile> <overwrite>true</overwrite> <verbose>true</verbose> </configuration> <dependencies> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>${mysql.version}</version> </dependency> <dependency> <groupId>tk.mybatis</groupId> <artifactId>mapper</artifactId> <version>${mapper.version}</version> </dependency> </dependencies> </plugin> </plugins> </build> </project> -
src/main/resources
-
New config.properties
#t_customer表包名 #com.xfcy 输自己用的包名 要不然不能直接cv到自己用的工程里 package.name=com.xfcy jdbc.driverClass = com.mysql.jdbc.Driver jdbc.url = jdbc:mysql://localhost:3306/db_boot jdbc.user = root jdbc.password =123456 -
New generatorConfig.xml
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE generatorConfiguration PUBLIC "-//mybatis.org//DTD MyBatis Generator Configuration 1.0//EN" "http://mybatis.org/dtd/mybatis-generator-config_1_0.dtd"> <generatorConfiguration> <properties resource="config.properties"/> <context id="Mysql" targetRuntime="MyBatis3Simple" defaultModelType="flat"> <property name="beginningDelimiter" value="`"/> <property name="endingDelimiter" value="`"/> <plugin type="tk.mybatis.mapper.generator.MapperPlugin"> <property name="mappers" value="tk.mybatis.mapper.common.Mapper"/> <property name="caseSensitive" value="true"/> </plugin> <jdbcConnection driverClass="${jdbc.driverClass}" connectionURL="${jdbc.url}" userId="${jdbc.user}" password="${jdbc.password}"> </jdbcConnection> <javaModelGenerator targetPackage="${package.name}.entities" targetProject="src/main/java"/> <sqlMapGenerator targetPackage="${package.name}.mapper" targetProject="src/main/java"/> <javaClientGenerator targetPackage="${package.name}.mapper" targetProject="src/main/java" type="XMLMAPPER"/> <!-- 输入表名 t_customer --> <table tableName="t_customer" domainObjectName="Customer"> <generatedKey column="id" sqlStatement="JDBC"/> </table> </context> </generatorConfiguration>
-
-
One-click generation
ps: Just generate these two
-
-
Integrate redis
-
New Construction
-
POM
<parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.6.10</version> <relativePath/> </parent> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <maven.compiler.source>1.8</maven.compiler.source> <maven.compiler.target>1.8</maven.compiler.target> <junit.version>4.12</junit.version> <log4j.version>1.2.17</log4j.version> <lombok.version>1.16.18</lombok.version> </properties> <dependencies> <!--SpringBoot通用依赖模块--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <!--jedis--> <dependency> <groupId>redis.clients</groupId> <artifactId>jedis</artifactId> <version>4.3.1</version> </dependency> <!--lettuce--> <!--<dependency> <groupId>io.lettuce</groupId> <artifactId>lettuce-core</artifactId> <version>6.2.1.RELEASE</version> </dependency>--> <!--SpringBoot与Redis整合依赖--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-redis</artifactId> </dependency> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-pool2</artifactId> </dependency> <!--swagger2--> <dependency> <groupId>io.springfox</groupId> <artifactId>springfox-swagger2</artifactId> <version>2.9.2</version> </dependency> <dependency> <groupId>io.springfox</groupId> <artifactId>springfox-swagger-ui</artifactId> <version>2.9.2</version> </dependency> <!--Mysql数据库驱动--> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>5.1.47</version> </dependency> <!--SpringBoot集成druid连接池--> <dependency> <groupId>com.alibaba</groupId> <artifactId>druid-spring-boot-starter</artifactId> <version>1.1.10</version> </dependency> <dependency> <groupId>com.alibaba</groupId> <artifactId>druid</artifactId> <version>1.1.16</version> </dependency> <!--mybatis和springboot整合--> <dependency> <groupId>org.mybatis.spring.boot</groupId> <artifactId>mybatis-spring-boot-starter</artifactId> <version>1.3.0</version> </dependency> <!--hutool--> <dependency> <groupId>cn.hutool</groupId> <artifactId>hutool-all</artifactId> <version>5.2.3</version> </dependency> <!--persistence--> <dependency> <groupId>javax.persistence</groupId> <artifactId>persistence-api</artifactId> <version>1.0.2</version> </dependency> <!--通用Mapper--> <dependency> <groupId>tk.mybatis</groupId> <artifactId>mapper</artifactId> <version>4.1.5</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-autoconfigure</artifactId> </dependency> <!--通用基础配置junit/devtools/test/log4j/lombok/--> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>${ junit.version}</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>log4j</groupId> <artifactId>log4j</artifactId> <version>${ log4j.version}</version> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>${ lombok.version}</version> <optional>true</optional> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> -
YML
server.port=7070 # ========================swagger===================== spring.swagger2.enabled=true #在springboot2.6.X结合swagger2.9.X会提示documentationPluginsBootstrapper空指针异常， #原因是在springboot2.6.X中将SpringMVC默认路径匹配策略从AntPathMatcher更改为PathPatternParser， # 导致出错，解决办法是matching-strategy切换回之前ant_path_matcher spring.mvc.pathmatch.matching-strategy=ant_path_matcher # ========================redis单机===================== spring.redis.database=0 ## 修改为自己真实IP spring.redis.host=192.168.238.111 spring.redis.port=6379 spring.redis.password=123456 spring.redis.lettuce.pool.max-active=8 spring.redis.lettuce.pool.max-wait=-1ms spring.redis.lettuce.pool.max-idle=8 spring.redis.lettuce.pool.min-idle=0 # ========================logging===================== logging.level.root=info logging.level.com.atguigu.redis7=info logging.pattern.console=%d{ yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger- %msg%n logging.file.name=D:/mylogs2023/redis7_study.log logging.pattern.file=%d{ yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger- %msg%n # ========================alibaba.druid===================== spring.datasource.type=com.alibaba.druid.pool.DruidDataSource spring.datasource.driver-class-name=com.mysql.jdbc.Driver spring.datasource.url=jdbc:mysql://localhost:3306/db_boot?useUnicode=true&characterEncoding=utf-8&useSSL=false spring.datasource.username=root spring.datasource.password=123456 spring.datasource.druid.test-while-idle=false # ========================mybatis=================== mybatis.mapper-locations=classpath:mapper/*.xml mybatis.type-aliases-package=com.xfcy.entities -
startup class
import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import tk.mybatis.spring.annotation.MapperScan; /** * customer 是有关简单手写布隆过滤器 */ @MapperScan("com.xfcy.mapper") // import tk.mybatis.spring.annotation.MapperScan; @SpringBootApplication public class Redis7Study2Application { public static void main(String[] args) { SpringApplication.run(Redis7Study2Application.class, args); } } -
bloom filter
-
filter package BloomFilterInit
import lombok.extern.slf4j.Slf4j; import org.springframework.data.redis.core.RedisTemplate; import org.springframework.stereotype.Component; import javax.annotation.PostConstruct; import javax.annotation.Resource; /** * @author 晓风残月Lx * @date 2023/3/29 17:05 * 布隆过滤器白名单初始化工具类，一开始就设置一部分数据为白名单所有 * 白名单业务默认规定：布隆过滤器有，redis 有可能有 布隆过滤器无 redis一定无 * 白名单业务默认规定：whitelistCustomer */ @Component @Slf4j public class BloomFilterInit { @Resource private RedisTemplate redisTemplate; @PostConstruct // 初始化白名单数据 public void init() { // 1.白名单客户加载到布隆过滤器 String key = "customer:11"; // 2.计算hashValue，由于存在计算出来负数的可能，取绝对值 int hashValue = Math.abs(key.hashCode()); // 3.通过hashValue 和 2的32次方后取余，获得对应的下标坑位 long index = (long) (hashValue % Math.pow(2, 32)); log.info(key + "对应的坑位 index： {}", index); // 4.设置redis里面的bitmap对应类型的坑位，将该值设置为1 redisTemplate.opsForValue().setBit("whitelistCustomer", index, true); } } -
utils包 CheckUtils
import javax.annotation.Resource; /** * @author 晓风残月Lx * @date 2023/3/29 17:21 */ @Component @Slf4j public class CheckUtils { @Resource private RedisTemplate redisTemplate; public boolean checkWithBloomFilter(String checkItem, String key){ int hashValue = Math.abs(key.hashCode()); long index = (long) (hashValue % Math.pow(2, 32)); boolean existOK = redisTemplate.opsForValue().getBit(checkItem, index); log.info("---->:" + key +"对应坑位下标index："+ index + "是否存在：" + existOK); return existOK; } }
-
-
business class
-
entity Copy the generated Customer class
-
mapper copy generated CustomerMapper class
-
src/main/resources/mapper Copy the generated CustomerMapper.xml
-
Service class (personally modified a part and added double check and lock)
import com.xfcy.entities.Customer; import com.xfcy.mapper.CustomerMapper; import com.xfcy.utils.CheckUtils; import lombok.extern.slf4j.Slf4j; import org.springframework.data.redis.core.RedisTemplate; import org.springframework.stereotype.Service; import javax.annotation.Resource; import java.util.concurrent.TimeUnit; /** * @author 晓风残月Lx * @date 2023/3/29 15:48 */ @Service @Slf4j public class CustomerService { public static final String CACHE_KEY_CUSTOMER = "customer:"; @Resource private CustomerMapper customerMapper; @Resource private RedisTemplate redisTemplate; @Resource private CheckUtils checkUtils; /** * 写操作 * @param customer */ public void addCustomer(Customer customer){ int i = customerMapper.insertSelective(customer); if (i > 0){ // mysql 插入成功，需要重新查询一次将数据写进redis Customer result = customerMapper.selectByPrimaryKey(customer.getId()); // redis缓存key String key = CACHE_KEY_CUSTOMER + customer.getId(); // 写入redis redisTemplate.opsForValue() .set(key, result); } } /** * 读操作 + 双检加锁 */ public Customer findCustomerById(Integer customerId) { Customer customer = null; // 缓存redis的key名称 String key = CACHE_KEY_CUSTOMER + customerId; // 1.去redis上查询 customer = (Customer) redisTemplate.opsForValue().get(key); // 2. 如果redis有，直接返回 如果redis没有，在mysql上查询 if (customer == null) { // 3.对于高QPS的优化，进来就先加锁，保证一个请求操作，让外面的redis等待一下，避免击穿mysql（大公司的操作 ） synchronized (CustomerService.class) { // 3.1 第二次查询redis，加锁后 customer = (Customer) redisTemplate.opsForValue().get(key); // 4.再去查询我们的mysql customer = customerMapper.selectByPrimaryKey(customerId); // 5.mysql有，redis无 if (customer != null) { // 6.把mysql查询到的数据会写到到redis， 保持双写一致性 7天过期 redisTemplate.opsForValue().set(key, customer, 7L, TimeUnit.DAYS); } } } return customer; } /** * 业务逻辑没有写错，对于QPS <= 1000 可以使用 * @param customerId * @return */ // public Customer findeCustomerById(Integer customerId) { // Customer customer = null; // // 缓存redis的key名称 // String key = CACHE_KEY_CUSTOMER + customerId; // // 1.去redis上查询 // customer = (Customer) redisTemplate.opsForValue().get(key); // // 2. 如果redis有，直接返回 如果redis没有，在mysql上查询 // if(customer == null) { // customer = customerMapper.selectByPrimaryKey(customerId); // if (customer == null) { // redisTemplate.opsForValue().set(key, customer); // } // } // return customer; // } /** *布隆过滤器 + 双检加锁 注意单个服务，在分布式系统锁将锁不住 * @param customerId * @return */ public Customer findCustomerByIdWithBloomFilter(Integer customerId) { Customer customer = null; // 1.缓存redis的key名称 String key = CACHE_KEY_CUSTOMER + customerId; if (!checkUtils.checkWithBloomFilter("whitelistCustomer", key)) { log.info("白名单没有此信息，不可以访问" + key); return null; } // 2. 如果redis有，直接返回 如果redis没有，在mysql上查询 if (customer == null) { // 3.对于高QPS的优化，进来就先加锁，保证一个请求操作，让外面的redis等待一下，避免击穿mysql（大公司的操作 ） synchronized (CustomerService.class) { // 3.1 第二次查询redis，加锁后 customer = (Customer) redisTemplate.opsForValue().get(key); // 4.再去查询我们的mysql customer = customerMapper.selectByPrimaryKey(customerId); // 5.mysql有，redis无 if (customer != null) { // 6.把mysql查询到的数据会写到到redis， 保持双写一致性 7天过期 redisTemplate.opsForValue().setIfAbsent(key, customer, 7L, TimeUnit.DAYS); } } } return customer; } } -
controller
import com.xfcy.entities.Customer; import com.xfcy.service.CustomerService; import io.swagger.annotations.Api; import io.swagger.annotations.ApiOperation; import lombok.extern.slf4j.Slf4j; import org.springframework.web.bind.annotation.PathVariable; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestMethod; import org.springframework.web.bind.annotation.RestController; import javax.annotation.Resource; import java.sql.Date; import java.time.LocalDateTime; import java.time.ZoneId; import java.util.Random; /** * @author 晓风残月Lx * @date 2023/3/29 15:48 */ @Api("客户Customer接口+布隆过滤器讲解") @RestController @Slf4j public class CustomerController { @Resource private CustomerService customerService; @ApiOperation("数据库初始化2条customer记录插入") @RequestMapping(value = "/customer/add", method = RequestMethod.POST) public void addCustomer() { for (int i = 0; i < 2; i++) { Customer customer = new Customer(); customer.setCname("customer" +i); customer.setAge(new Random().nextInt(30)+1); customer.setPhone("12345678910"); customer.setSex((byte) new Random().nextInt(2)); customer.setBirth(Date.from(LocalDateTime.now().atZone(ZoneId.systemDefault()).toInstant())); customerService.addCustomer(customer); } } @ApiOperation("单个customer查询操作，按照customerId查询") @RequestMapping(value = "/customer/{customerId}", method = RequestMethod.GET) public Customer findCustomerById(@PathVariable("customerId") Integer customerId){ return customerService.findCustomerById(customerId); } @ApiOperation("布隆过滤器单个customer查询操作，按照customerId查询") @RequestMapping(value = "/bloomfilter/{customerId}", method = RequestMethod.GET) public Customer findCustomerByIdWithBloomFilter(@PathVariable("customerId") Integer customerId){ return customerService.findCustomerByIdWithBloomFilter(customerId); } }
-
-
-
Start the swagger test, pay attention to look at the console
6. Advantages and disadvantages of Bloom filter
advantage
- Insert and query efficiently, occupying less memory bit space
shortcoming
- cannot delete element
- Because deleting an element will lead to an increase in the false positive rate, because hash conflicts may store things in the same location that are shared by multiple people, when you delete an element, you may also delete the other
- There are misjudgments and cannot be accurately filtered
- yes, maybe
- None, Absolutely None
In order to solve the problem that the Bloom filter cannot delete elements, the cuckoo filter was born.