Jedis
reference:
Redis not only uses commands to operate, but now basically mainstream languages have client support, such as java, C, C#, C++, php, Node.js, Go, etc. Some Java clients are listed on the official website, including Jedis, Redisson, Jredis, JDBC-Redis, etc. Jedis and Redisson are officially recommended.
The most used in the enterprise is Jedis. Jedis basically implements all Redis commands, and also supports advanced usages such as connection pools and clusters, and is easy to use, making it very simple to use Redis services in Java.
Dependency, common API
rely
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.9.0</version> <!--若与springboot集成，推荐不写版本，由springboot控制-->
</dependency>
jedis common APIs
// 创建jedis对象，参数host是redis服务器地址，参数port是redis服务端口
new Jedis(host, port)
// 释放资源
public void close()
// 设置字符串类型的数据
public String set(String key, String value)
// 获得字符串类型的数据
public String get(String key)
// 删除指定的key
public Long del(String... keys)
// 设置哈希类型的数据
public Long hset(String key, String field, String value)
// 获得哈希类型的数据
public String hget(String key, String field)
// 设置列表类型的数据
public Long rpush(String key, String... strings)
public Long lpush(String key, String... strings)
// 列表左面弹栈
public String lpop(String key)
// 列表右面弹栈
public String rpop(String key)
jedis connection pool
The creation and destruction of jedis connection resources consumes a lot of program performance, so jedis provides jedis pooling technology. jedisPool initializes some connection resources and stores them in the connection pool when it is created. When using jedis connection resources, it does not need to be created, but obtains a resource from the connection pool for redis operations.
Encapsulation of JedisUtils tool class:
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.JedisPoolConfig;
public class JedisUtils {
private static JedisPool jedisPool =null;
static {
// 创建jedis连接池的配置对象
JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
// 连接池初始化的最大连接数
jedisPoolConfig.setMaxTotal(40);
// 最大空闲连接数
jedisPoolConfig.setMaxIdle(10);
// 当池内没有可用连接时，最大等待时间
jedisPoolConfig.setMaxWaitMillis(10000);
// 创建jedis的连接池
jedisPool = new JedisPool(jedisPoolConfig, "localhost", 6379);
}
//获取jedis
public static Jedis getJedis() {
// 从连接池中获取jedis
Jedis jedis = jedisPool.getResource();
return jedis;
}
}
Spring Data Redis
overview, dependencies
Spring Data Redis is part of the Spring Data family. The Jedis client is encapsulated and integrated with spring. It is very convenient to realize the configuration and operation of redis.
-
When Redis is operated as a database or message queue, it is generally operated using the RedisTemplate tool class
redisTemplate is a special tool class for Spring to integrate Redis and operate redis
-
When Redis is used as a cache, it can be used as the implementation of Spring Cache and used directly through annotations
Spring Cache is a complete set of caching solutions provided by Spring. It provides a complete set of interfaces and code specifications, configurations, annotations, etc. It is not a specific cache implementation, and the specific implementation is implemented by the respective third parties themselves. Such as Guava, EhCache, Redis, local cache, etc.
Spring Boot integrates Redis automatic configuration principle
rely
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
Configuration files and configuration classes
Stand-alone version yml configuration
spring:
redis:
# 连接模式（自定义的配置项）
model: standalone
# 配置redis地址（单机模式）
host: 192.168.85.135
# 配置redis端口（单机模式）
port: 6379
# 库。不配置默认为0
database: 0
# 密码。不配置默认无密码
password: passwd@123
pool:
# 连接池初始化的最大连接数
max-active: 8
# 最大空闲连接数
max-idle: 8
# 最小空闲连接数
min-idle: 0
# 当池内没有可用连接时，最大等待时间。-1 为一直等待
max-wait: -1
# 集群模式
cluster:
nodes: ip1:6379,ip2:6379,ip3:6379
# 最大重试次数
max-redirects: 6
# 哨兵模式
sentinel:
nodes: ip1:6379,ip2:6379,ip3:6379
master: mymaster
redis connection configuration class
import com.fasterxml.jackson.annotation.JsonAutoDetect;
import com.fasterxml.jackson.annotation.PropertyAccessor;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.data.redis.RedisProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisClusterConfiguration;
import org.springframework.data.redis.connection.RedisNode;
import org.springframework.data.redis.connection.RedisSentinelConfiguration;
import org.springframework.data.redis.connection.RedisStandaloneConfiguration;
import org.springframework.data.redis.connection.jedis.JedisClientConfiguration;
import org.springframework.data.redis.connection.jedis.JedisConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.Jackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;
import redis.clients.jedis.JedisPoolConfig;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
@Configuration
public class RedisConfig {
/**
* redis 连接模式
*/
@Value("${spring.redis.model:standalone}")
private String model;
@Bean
public JedisPoolConfig jedisPoolConfig(RedisProperties properties){
// 创建jedis连接池的配置对象
JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
RedisProperties.Pool pool = properties.getJedis().getPool();
if (pool == null) pool = new RedisProperties.Pool();
// 连接池初始化的最大连接数
jedisPoolConfig.setMaxTotal(pool.getMaxActive());
// 最大空闲连接数
jedisPoolConfig.setMaxIdle(pool.getMaxIdle());
// 最小空闲连接数
jedisPoolConfig.setMaxIdle(pool.getMinIdle());
// 当池内没有可用连接时，最大等待时间
jedisPoolConfig.setMaxWaitMillis(pool.getMaxWait().toMillis());
return jedisPoolConfig;
}
/**
* redis 连接
*/
@Bean
public JedisConnectionFactory jedisConnectionFactory(RedisProperties properties, JedisPoolConfig jedisPoolConfig) {
// 设置jedis连接池
JedisClientConfiguration.JedisClientConfigurationBuilder jedisConfBuilder = JedisClientConfiguration.builder();
JedisClientConfiguration jedisClientConfiguration = jedisConfBuilder.usePooling().poolConfig(jedisPoolConfig).build();
if ("sentinel".equalsIgnoreCase(model)){
// 哨兵模式连接
List<String> serverList = properties.getSentinel().getNodes();
Set<RedisNode> nodes = serverList.stream().map(ipPortStr -> {
String[] ipPortArr = ipPortStr.split(":");
return new RedisNode(ipPortArr[0].trim(), Integer.parseInt(ipPortArr[1]));
}).collect(Collectors.toSet());
RedisSentinelConfiguration redisConfig = new RedisSentinelConfiguration();
redisConfig.setSentinels(nodes);
redisConfig.setDatabase(properties.getDatabase());
redisConfig.setMaster(properties.getSentinel().getMaster());
redisConfig.setPassword(properties.getPassword());
return new JedisConnectionFactory(redisConfig, jedisClientConfiguration);
} else if ("cluster".equalsIgnoreCase(model)) {
// 集群模式连接
List<String> serverList = properties.getCluster().getNodes();
Set<RedisNode> nodes = serverList.stream().map(ipPortStr -> {
String[] ipPortArr = ipPortStr.split(":");
return new RedisNode(ipPortArr[0].trim(), Integer.parseInt(ipPortArr[1]));
}).collect(Collectors.toSet());
RedisClusterConfiguration redisConfig = new RedisClusterConfiguration();
redisConfig.setClusterNodes(nodes);
redisConfig.setMaxRedirects(properties.getCluster().getMaxRedirects());
redisConfig.setPassword(properties.getPassword());
return new JedisConnectionFactory(redisConfig, jedisClientConfiguration);
} else {
// 单节点模式连接
RedisStandaloneConfiguration redisConfig = new RedisStandaloneConfiguration();
redisConfig.setHostName(properties.getHost());
redisConfig.setPort(properties.getPort());
redisConfig.setDatabase(properties.getDatabase());
redisConfig.setPassword(properties.getPassword());
return new JedisConnectionFactory(redisConfig, jedisClientConfiguration);
}
}
/**
* RedisTemplate 配置
*/
@Bean
public RedisTemplate<String, Object> redisTemplate(JedisConnectionFactory jedisConnectionFactory) {
// 设置序列化。redisTemplate序列化默认使用的jdkSerializeable，存储二进制字节码，导致key会出现乱码，所以自定义
Jackson2JsonRedisSerializer jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer(Object.class);
ObjectMapper om = new ObjectMapper();
om.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);
// 指定序列化输入的类型，类必须是非final修饰的，final修饰的类，比如String, Integer等会抛出异常
om.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL);
jackson2JsonRedisSerializer.setObjectMapper(om);
// 配置redisTemplate
RedisTemplate<String, Object> redisTemplate = new RedisTemplate<>();
redisTemplate.setConnectionFactory(jedisConnectionFactory);
StringRedisSerializer stringSerializer = new StringRedisSerializer();
// key序列化
redisTemplate.setKeySerializer(stringSerializer);
// value序列化
redisTemplate.setValueSerializer(jackson2JsonRedisSerializer);
// Hash key序列化
redisTemplate.setHashKeySerializer(stringSerializer);
// Hash value序列化
redisTemplate.setHashValueSerializer(jackson2JsonRedisSerializer);
redisTemplate.afterPropertiesSet();
return redisTemplate;
}
RedisTemplate
Spring Data provides a tool class for Redis: RedisTemplate. It encapsulates various operations on the five data structures of Redis, including:
- redisTemplate.opsForValue() : operation string
- redisTemplate.opsForHash() : operation hash
- redisTemplate.opsForList(): operation list
- redisTemplate.opsForSet(): operation set
- redisTemplate.opsForZSet(): operate zset
Some common commands, such as del, can be called directly through redisTemplate.xx()
StringRedisTemplate
When RedisTemplate is created, its generic type can be specified:
- K : represents the data type of the key
- V : represents the data type of value
Note: The type here is not the data type stored in Redis, but the data type in Java. RedisTemplate will automatically convert the Java type to the data type supported by Redis: string, byte, binary, etc.
However, RedisTemplate will use JDK's own serialization (Serialize) to convert objects by default. The generated data is very large, so the key and value are generally specified as String type, so that the developer can serialize the object into a json string for storage.
In most cases, RedisTemplate whose key and value are both Strings is used, so Spring provides such an implementation by default:
public class StringRedisTemplate extends RedisTemplate<String, String>
redisService tool class
You can directly call the relevant methods of the tool class in the code
import cn.hutool.core.util.ObjectUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.*;
import org.springframework.data.redis.support.atomic.RedisAtomicLong;
import org.springframework.stereotype.Service;
import java.io.Serializable;
import java.util.*;
import java.util.concurrent.TimeUnit;
@Service
public class RedisService {
private final Logger logger = LoggerFactory.getLogger(this.getClass());
@Autowired
private RedisTemplate redisTemplate;
/**
* 模糊查询key
*/
public Set<String> listKeys(final String key) {
Set<String> keys = redisTemplate.keys(key);
return keys;
}
/**
* 重命名
*/
public void rename(final String oldKey, final String newKey) {
redisTemplate.rename(oldKey, newKey);
}
/**
* 模糊获取
*/
public List<Object> listPattern(final String pattern) {
List<Object> result = new ArrayList<>();
Set<Serializable> keys = redisTemplate.keys(pattern);
for (Serializable str : keys) {
ValueOperations<Serializable, Object> operations = redisTemplate.opsForValue();
Object obj = operations.get(str.toString());
if (!ObjectUtil.isEmpty(obj)) {
result.add(obj);
}
}
return result;
}
/**
* 写入缓存
*/
public boolean set(final String key, Object value) {
boolean result = false;
try {
ValueOperations<Serializable, Object> operations = redisTemplate.opsForValue();
operations.set(key, value);
result = true;
} catch (Exception e) {
logger.error("set fail ,key is:" + key, e);
}
return result;
}
/**
* 批量写入缓存
*/
public boolean multiSet(Map<String, Object> map) {
boolean result = false;
try {
ValueOperations<Serializable, Object> operations = redisTemplate.opsForValue();
operations.multiSet(map);
result = true;
} catch (Exception e) {
logger.error("multiSet fail ", e);
}
return result;
}
/**
* 集合出栈
*/
public Object leftPop(String key) {
ListOperations list = redisTemplate.opsForList();
return list.leftPop(key);
}
public Object llen(final String key) {
final ListOperations list = this.redisTemplate.opsForList();
return list.size((Object) key);
}
/**
* 写入缓存设置时效时间
*/
public boolean set(final String key, Object value, Long expireTime) {
boolean result = false;
try {
ValueOperations<Serializable, Object> operations = redisTemplate.opsForValue();
operations.set(key, value);
redisTemplate.expire(key, expireTime, TimeUnit.SECONDS);
result = true;
} catch (Exception e) {
logger.error("set fail ", e);
}
return result;
}
/**
* 写入缓存设置时效时间
*/
public boolean setnx(final String key, Object value, Long expireTime) {
boolean res = false;
try {
ValueOperations<Serializable, Object> operations = redisTemplate.opsForValue();
res = operations.setIfAbsent(key, value);
if (res) {
redisTemplate.expire(key, expireTime, TimeUnit.SECONDS);
}
} catch (Exception e) {
logger.error("setnx fail ", e);
}
return res;
}
/**
* 缓存设置时效时间
*/
public void expire(final String key, Long expireTime) {
redisTemplate.expire(key, expireTime, TimeUnit.SECONDS);
}
/**
* 自增操作
*/
public long incr(final String key) {
RedisAtomicLong entityIdCounter = new RedisAtomicLong(key, redisTemplate.getConnectionFactory());
return entityIdCounter.getAndIncrement();
}
/**
* 批量删除
*/
public void removeKeys(final List<String> keys) {
if (keys.size() > 0) {
redisTemplate.delete(keys);
}
}
/**
* 批量删除key
*/
public void removePattern(final String pattern) {
Set<Serializable> keys = redisTemplate.keys(pattern);
if (keys.size() > 0) {
redisTemplate.delete(keys);
}
}
/**
* 删除对应的value
*/
public void remove(final String key) {
if (exists(key)) {
redisTemplate.delete(key);
}
}
/**
* 判断缓存中是否有对应的value
*/
public boolean exists(final String key) {
return redisTemplate.hasKey(key);
}
/**
* 判断缓存中是否有对应的value(模糊匹配)
*/
public boolean existsPattern(final String pattern) {
if (redisTemplate.keys(pattern).size() > 0) {
return true;
} else {
return false;
}
}
/**
* 读取缓存
*/
public Object get(final String key) {
ValueOperations<Serializable, Object> operations = redisTemplate.opsForValue();
return operations.get(key);
}
/**
* 哈希 添加
*/
public void hmSet(String key, Object hashKey, Object value) {
HashOperations<String, Object, Object> hash = redisTemplate.opsForHash();
hash.put(key, hashKey, value);
}
/**
* 哈希 添加
*/
public Boolean hmSet(String key, Object hashKey, Object value, Long expireTime, TimeUnit timeUnit) {
HashOperations<String, Object, Object> hash = redisTemplate.opsForHash();
hash.put(key, hashKey, value);
return redisTemplate.expire(key, expireTime, timeUnit);
}
/**
* 哈希获取数据
*/
public Object hmGet(String key, Object hashKey) {
HashOperations<String, Object, Object> hash = redisTemplate.opsForHash();
return hash.get(key, hashKey);
}
/**
* 哈希获取所有数据
*/
public Object hmGetValues(String key) {
HashOperations<String, Object, Object> hash = redisTemplate.opsForHash();
return hash.values(key);
}
/**
* 哈希获取所有键值
*/
public Object hmGetKeys(String key) {
HashOperations<String, Object, Object> hash = redisTemplate.opsForHash();
return hash.keys(key);
}
/**
* 哈希获取所有键值对
*/
public Object hmGetMap(String key) {
HashOperations<String, Object, Object> hash = redisTemplate.opsForHash();
return hash.entries(key);
}
/**
* 哈希 删除域
*/
public Long hdel(String key, Object hashKey) {
HashOperations<String, Object, Object> hash = redisTemplate.opsForHash();
return hash.delete(key, hashKey);
}
/**
* 列表添加
*/
public void rPush(String k, Object v) {
ListOperations<String, Object> list = redisTemplate.opsForList();
list.rightPush(k, v);
}
/**
* 列表删除
*/
public void listRemove(String k, Object v) {
ListOperations<String, Object> list = redisTemplate.opsForList();
list.remove(k, 1, v);
}
public void rPushAll(String k, Collection var2) {
ListOperations<String, Object> list = redisTemplate.opsForList();
list.rightPushAll(k, var2);
}
/**
* 列表获取
*/
public Object lRange(String k, long begin, long end) {
ListOperations<String, Object> list = redisTemplate.opsForList();
return list.range(k, begin, end);
}
/**
* 集合添加
*/
public void add(String key, Object value) {
SetOperations<String, Object> set = redisTemplate.opsForSet();
set.add(key, value);
}
/**
* 判断元素是否在集合中
*/
public Boolean isMember(String key, Object value) {
SetOperations<String, Object> set = redisTemplate.opsForSet();
return set.isMember(key, value);
}
/**
* 集合获取
*/
public Set<Object> setMembers(String key) {
SetOperations<String, Object> set = redisTemplate.opsForSet();
return set.members(key);
}
/**
* 有序集合添加
*/
public void zAdd(String key, Object value, double scoure) {
ZSetOperations<String, Object> zset = redisTemplate.opsForZSet();
zset.add(key, value, scoure);
}
/**
* 有序集合获取
*/
public Set<Object> rangeByScore(String key, double scoure, double scoure1) {
ZSetOperations<String, Object> zset = redisTemplate.opsForZSet();
return zset.rangeByScore(key, scoure, scoure1);
}
/**
* 有序集合根据区间删除
*/
public void removeRangeByScore(String key, double scoure, double scoure1) {
ZSetOperations<String, Object> zset = redisTemplate.opsForZSet();
zset.removeRangeByScore(key, scoure, scoure1);
}
/**
* 列表添加
*/
public void lPush(String k, Object v) {
ListOperations<String, Object> list = redisTemplate.opsForList();
list.rightPush(k, v);
}
/**
* 获取当前key的超时时间
*/
public Long getExpireTime(final String key) {
return redisTemplate.opsForValue().getOperations().getExpire(key, TimeUnit.SECONDS);
}
public Long extendExpireTime(final String key, Long extendTime) {
Long curTime = redisTemplate.opsForValue().getOperations().getExpire(key, TimeUnit.SECONDS);
long total = curTime.longValue() + extendTime;
redisTemplate.expire(key, total, TimeUnit.SECONDS);
return total;
}
public Set getKeys(String k) {
return redisTemplate.keys(k);
}
}
Spring Cache
overview
Spring has defined org.springframework.cache.Cache and org.springframework.cache.CacheManager interfaces since 3.1 to unify different caching technologies; and supports the use of JCache (JSR-107) annotations to simplify development.
-
The Cache (caching) interface is defined by the component specification of the cache, including a collection of various operations of the cache
Under the Cache interface, Spring provides various xxxCache implementations; such as RedisCache, EhCacheCache, ConcurrentMapCache, etc.
-
CacheManager (cache manager) manages various cache (Cache) components, responsible for adding, deleting, modifying and querying the cache
The cache medium of CacheManager can be configured, such as: ConcurrentMap/EhCache/Redis, etc.
When no EhCache or Redis dependency is added, the cache implemented by concurrentMap is used by default, which is stored in memory, and the cache is cleared when the server is restarted
The principle of Spring Cache
AOP idea (aspect-oriented programming) based on Proxy / AspectJ dynamic proxy technology.
Every time a method that requires caching is called, Spring will check whether the specified target method of the specified parameter has been called; if so, it will directly obtain the result of the method call from the cache, if not, call the method and cache the result and return it to the user. The next call is fetched directly from the cache.
Configuration files and configuration classes
yaml configuration file properties
spring:
cache:
#cache-names: # 可以自动配置
#type: redis # 可以自动配置
redis:
time-to-live: 3600000 # 指定存活时间。单位毫秒，缺省默认为 -1 （永不过时）
key-prefix: CACHE_ # key前缀，缺省默认使用缓存的名称（@Cacheable注解的value参数值）作为前缀
use-key-prefix: true # 是否使用前缀，默认为true，指定为false时不使用任何key前缀
cache-null-values: true # 是否缓存空值。默认为true。Spring Cache 对缓存穿透问题的解决方案
Spring cache configuration class
Generally, only CacheManager and KeyGenerator can be customized, and other customizations belong to advanced use
import com.fasterxml.jackson.annotation.JsonAutoDetect;
import com.fasterxml.jackson.annotation.PropertyAccessor;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.autoconfigure.cache.CacheProperties;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.cache.Cache;
import org.springframework.cache.CacheManager;
import org.springframework.cache.annotation.CachingConfigurerSupport;
import org.springframework.cache.annotation.EnableCaching;
import org.springframework.cache.interceptor.CacheErrorHandler;
import org.springframework.cache.interceptor.CacheResolver;
import org.springframework.cache.interceptor.KeyGenerator;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.cache.RedisCacheConfiguration;
import org.springframework.data.redis.cache.RedisCacheManager;
import org.springframework.data.redis.connection.jedis.JedisConnectionFactory;
import org.springframework.data.redis.serializer.Jackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.RedisSerializationContext;
import org.springframework.data.redis.serializer.StringRedisSerializer;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.List;
@Configuration
@EnableCaching
@EnableConfigurationProperties(CacheProperties.class)
@Slf4j
public class RedisCachingConfig extends CachingConfigurerSupport {
/**
* 自定义缓存管理器
*/
@Bean
public RedisCacheManager redisCacheManager(JedisConnectionFactory jedisConnectionFactory, CacheProperties cacheProperties) {
Jackson2JsonRedisSerializer jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer(Object.class);
ObjectMapper om = new ObjectMapper();
om.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);
om.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL);
jackson2JsonRedisSerializer.setObjectMapper(om);
// 配置序列化（解决乱码的问题，因为默认使用JDK的序列化机制，转换为二进制数据）
RedisCacheConfiguration config = RedisCacheConfiguration.defaultCacheConfig()
// 设置Key的序列化方式
.serializeKeysWith(
RedisSerializationContext.SerializationPair.fromSerializer(new StringRedisSerializer()))
// 设置值的序列化方式
.serializeValuesWith(
RedisSerializationContext.SerializationPair.fromSerializer(jackson2JsonRedisSerializer))
.disableCachingNullValues();
// 注：若使用自定义的 RedisCacheConfiguration，则不会自动从配置文件中取出来配置，需要手动注册配置文件中所有的配置项
CacheProperties.Redis redisProperties = cacheProperties.getRedis();
if (redisProperties.getTimeToLive() != null) config = config.entryTtl(redisProperties.getTimeToLive());
if (redisProperties.getKeyPrefix() != null) config = config.prefixKeysWith(redisProperties.getKeyPrefix());
if (!redisProperties.isCacheNullValues()) config = config.disableCachingNullValues();
if (!redisProperties.isUseKeyPrefix()) config = config.disableKeyPrefix();
return RedisCacheManager
.builder(jedisConnectionFactory)
.cacheDefaults(config)
.build();
}
/**
* 自定义 key 生成器
*/
@Bean
@Override
public KeyGenerator keyGenerator() {
return new KeyGenerator() {
@Override
public Object generate(Object target, Method method, Object... params) {
StringBuilder sb = new StringBuilder();
sb.append(target.getClass().getName());
sb.append("$").append(method.getName());
for (int i = 0; i < params.length; i++) {
if (i == 0) {
sb.append("[").append(params[i].toString());
} else {
sb.append(",").append(params[i].toString());
}
}
sb.append("]");
return sb.toString();
}
};
}
/**
* 自定义错误处理器
*/
@Override
@Bean
public CacheErrorHandler errorHandler() {
// 当缓存读写异常时，忽略异常
return new CacheErrorHandler(){
@Override
public void handleCacheGetError(RuntimeException e, Cache cache, Object o) {
log.error(e.getMessage(), e);
}
@Override
public void handleCachePutError(RuntimeException e, Cache cache, Object o, Object o1) {
log.error(e.getMessage(), e);
}
@Override
public void handleCacheEvictError(RuntimeException e, Cache cache, Object o) {
log.error(e.getMessage(), e);
}
@Override
public void handleCacheClearError(RuntimeException e, Cache cache) {
log.error(e.getMessage(), e);
}
};
}
/**
* 自定义缓存解析器
*/
@Override
@Bean
public CacheResolver cacheResolver() {
// 通过Guava实现的自定义堆内存缓存管理器
// CacheManager guavaCacheManager = new GuavaCacheManager();
CacheManager redisCacheManager = this.cacheManager();
List<CacheManager> list = new ArrayList<>();
// 优先读取堆内存缓存
// list.add(concurrentMapCacheManager);
// 堆内存缓存读取不到该key时再读取redis缓存
list.add(redisCacheManager);
return new CustomCacheResolver(list);
}
}
Custom cache resolver class
public class CustomCacheResolver implements CacheResolver, InitializingBean {
@Nullable
private List<CacheManager> cacheManagerList;
public CustomCacheResolver(){
}
public CustomCacheResolver(List<CacheManager> cacheManagerList){
this.cacheManagerList = cacheManagerList;
}
public void setCacheManagerList(@Nullable List<CacheManager> cacheManagerList) {
this.cacheManagerList = cacheManagerList;
}
public List<CacheManager> getCacheManagerList() {
return cacheManagerList;
}
@Override
public void afterPropertiesSet() {
Assert.notNull(this.cacheManagerList, "CacheManager is required");
}
@Override
public Collection<? extends Cache> resolveCaches(CacheOperationInvocationContext<?> context) {
Collection<String> cacheNames = context.getOperation().getCacheNames();
if (cacheNames == null) {
return Collections.emptyList();
}
Collection<Cache> result = new ArrayList<>();
for(CacheManager cacheManager : getCacheManagerList()){
for (String cacheName : cacheNames) {
Cache cache = cacheManager.getCache(cacheName);
if (cache == null) {
throw new IllegalArgumentException("Cannot find cache named '" +
cacheName + "' for " + context.getOperation());
}
result.add(cache);
}
}
return result;
}
}
CachingConfigurerSupport Description
Read and write mechanisms that support custom caching:
-
cacheManager (cache manager)
By default, SpringBoot will use the SimpleCacheConfiguration cache configuration class. Then create a ConcurrentMapCacheManager cache manager that can get ConcurrentMap to use as a cache component.
After the redis starter is introduced, the RedisCacheConfiguration cache configuration class will take effect and a RedisCacheManager will be created
- The RedisCacheManager created by default is RedisTemplate<Object, Object> when operating redis
- RedisTemplate<Object, Object> is the serialization mechanism that uses JDK by default
- You can customize CacheManager if you want to save it in JSON format
- Note: From the perspective of execution time, JdkSerializationRedisSerializer is the most efficient (after all, it is native to JDK), but the serialized result string is the longest. Due to the compactness of its data format, the serialization length of JSON is the smallest, and the time is longer than the former. And OxmSerialiabler is the longest in terms of time (it was related to the use of specific Marshaller at the time). Therefore, it is recommended to use JacksonJsonRedisSerializer as the POJO serializer.
-
keyGenerator (key generator)
When the cache-related annotation is not specified, the SimpleKeyGenerator (combining all parameter values of the method) is automatically used to generate the key by default. If different methods specify the same cache partition and the parameter values are the same, the key automatically generated by SimpleKeyGenerator is the same. You can customize the keyGenerator to avoid this situation
-
errorHandler (error handler)
When an exception occurs in the redis connection, calling the method marked with cache-related comments will throw an exception and affect the normal business process. You can customize the errorHandler to handle the exception of cache read and write
If a cache read exception occurs:
- The cache error handler can ignore exceptions and continue to read data from the database without affecting the business
- However, if the amount of requests is large, there will be a cache avalanche problem. A large number of query requests are sent to the database, causing the database to be overloaded and blocked or even downtime.
- It is recommended to use a multi-layer cache
If an exception occurs in the cache write , it may cause inconsistency between the data in the database and the data in the cache:
- In order to solve this problem, it is necessary to continue to extend the handleCachePutError and handleCacheEvictError methods of CacheErrorHandler
- The idea is to save the keys whose redis write operations failed, and delete the caches corresponding to these keys by retrying tasks to solve the problem of inconsistency between database data and cached data
-
cacheResolver (cache resolver)
Dynamic selection of CacheManager can be realized by customizing CacheResolver
Multiple caching mechanisms can be used: first read the cache from the heap memory, read the cache from redis when the heap memory cache does not exist, and finally read data from the database when the redis cache does not exist, and write the read data to redis and heap memory in turn.
By customizing the CacheResolver, developers can implement more custom functions, such as the scene of automatic upgrading and upgrading of hotspot caches:
-
In most cases, the project only uses redis as a cache. When individual data becomes hot data in some scenarios, the project caches the hot data in the heap memory to relieve the load pressure on the network and redis after the hot data is counted in real time, such as through storm.
-
This kind of scenario can be realized by customizing CacheResolver. Storm collects hot data in real time. Before calling resolveCaches to select CacheManager, the custom CacheResolver first judges whether the cache key for reading and writing is hot data. If it is hot data, use the CacheManager of heap memory, otherwise use the CacheManager of redis.
-
Main annotations in Spring Cache
- @EnableCaching : Enable the annotation-based caching function, and mark the @EnableCaching annotation on the configuration class (no need to reconfigure Redis)
- @Cacheable : Cache data or get cached data, generally used in query methods
- @CachePut : Modify cached data. The method is guaranteed to be called, and the result is expected to be cached. Generally used in the new method
- @CacheEvict : Clear the cache. Generally used in update or delete methods
- @CacheConfig : Unified configuration of the value in @Cacheable, mainly marked on the class, can also be marked on the method
- @Caching : Combining multiple Cache annotations
@Cacheable: cache data | get cache
Cache data or obtain cache data, generally used in query methods .
When the marked method is called for the first time, the returned result is cached according to the method (note: the saved data is the data returned by return). When the next request is made, if the cache exists , the cached data is directly read and returned; if the cache does not exist , the method is executed and the returned result is stored in the cache.
The cached data uses the JDK serialization mechanism by default (converting data to binary), and the default expiration time is TTL -1 (never passes).
The main parameters:
-
value / cacheNames attribute: Specify the name of the cache (cached prefix/partition/cache space, divided by business type)
Required, specify at least one; @CacheConfig can also be used instead
Example:
@Cacheable(value="testcache") @Cacheable(value={ "testcache1","testcache2"} -
**key** attribute: the key suffix of the cache
When it is empty, it is used by default
SimpleKeyGenerator(combining all parameter values of the method) to generate a key. If specified, it should be written according to the SpEL expressionNote: The format of the complete cache key-value pair is: value attribute value::key attribute value=method return value
value属性值::It is the cache prefix, which can be specified through the configuration file spring.cache.redis.key-prefix property
Example:
@Cacheable(value="testcache", key="#id")SimpleKeyGenerator source code (understand):
public SimpleKey(Object... elements) { Assert.notNull(elements, "Elements must not be null"); this.params = new Object[elements.length]; System.arraycopy(elements, 0, this.params, 0, elements.length); this.hashCode = Arrays.deepHashCode(this.params); } -
condition attribute: the condition of the cache
Can be empty, written in SpEL, returns true or false
Cache/clear the cache only if it is true, and it can be judged before and after calling the method
Example:
@Cacheable(value="testcache", condition="#id.length()>2") -
unless attribute: veto cache conditions
If the condition is true, it will not be cached, and if it is false, it will be cached
Only judge after the method is executed, at this time you can get the return value result for judgment
Example:
@Cacheable(value="testcache", condition="#result == null") -
sync attribute: whether to use asynchronous mode
That is, whether to lock when executing the method. Default is false
Sping Cache's solution to the problem of cache breakdown (a large number of concurrent queries for a data that just expires at the same time)
-
keyGenerator attribute: key generator
You can specify the component id of the key, which can only be used with the key attribute
-
cacheManager attribute: specify the cache manager
-
cacheResolver attribute: specify to get the resolver
@CacheEvict: clear cache data
The method using this annotation flag will clear the specified cache . It is generally used in the update or delete method (that is, the cache is cleared immediately after updating the database data, and the cache is also cleared immediately after deleting the database data, so that the next query can retrieve the cache).
Delete the cache according to the corresponding value and key, and the value and key must be the same to delete (Note: value + key is combined to form a redis key); if no key value is specified and allEntries=false, the key value will default to the parameter value to delete the cache, and if there is no parameter, the cache will not be cleared.
Main parameters: value, key, condition, allEntries, beforeInvocation
-
allEntries attribute: Whether to clear all cached content
The default is false, if specified as true, all caches will be cleared immediately after the method call
Example:
@CachEvict(value="testcache", allEntries=true) -
beforeInvocation attribute: Whether to clear the cache data before the method is executed
The default is false. By default, if the method execution throws an exception, the cache will not be cleared
If specified as true, the cache will be cleared before the method is executed
Example:
@CachEvict(value="testcache", beforeInvocation=true)Note: There is only one function, which is to clear the cache first and then execute the method
@CachePut: Add or update cache
The method using this annotation mark will be executed every time, and the result will be stored in the specified cache , which is generally used in new methods
First query the cache according to the value and key, if it exists, modify it; if it does not exist, add it.
Main parameters: value, key, condition
Note: The saved data is the data returned by return
@CacheConfig: Unified configuration value value
Uniformly configure the value value in the @Cacheable annotation, which is mainly marked on the class, and can also be marked on the method
If there is no value in the @Cacheable annotation, use the value in @CacheConfig; if there is a value in the @Cacheable annotation, use the value in @Cacheable (proximity principle).
@Caching: combining multiple annotations
Combining annotations, multiple annotations can be combined
@Caching(put = {
@CachePut(value = "user", key = "#user.id"),
@CachePut(value = "user", key = "#user.username"),
@CachePut(value = "user", key = "#user.email")
})
public User save(User user) {
}
Cache SpEL expression
Cache SpEL expression syntax
| name | Location | describe | example |
|---|---|---|---|
| methodName | root object | The name of the currently called method | #root.methodName |
| method | root object | the currently called method | #root.method.name |
| target | root object | the currently called target object | #root.target |
| targetClass | root object | The currently called target object class | #root.targetClass |
| args | root object | The parameter list of the currently called method | #root.args[0] |
| caches | root object | The cache list used by the current method call (such as @Cacheable(value={"cache1", "cache2"}), there are two caches) | #root.caches[0].name |
| argument name | evaluation context | The name of the method parameter, you can directly #parameter name, or use the form of #p0 or #a0, 0 represents the index of the parameter | #iban、#a0、#p0 |
| result | evaluation context | The return value after the method is executed (only when the judgment after the method is executed is valid, such as the expression of 'unless', 'cache put', the expression of 'cache evict' beforeInvocation=false) | #result |
SpEL operator
| type | operator |
|---|---|
| relational operator | < ，> ， <= ，>=，==，!=,lt，gt，le，ge，eq，ne |
| arithmetic operator | +，-，*，/ ，%，^ |
| Logical Operators | &&， |
| conditional operator | ? : (ternary),? : (elvis) |
| regular expression | matches |
| other types | ?. ，?[…] ，![…] ，1，$[…] |
Redisson
overview
In some scenarios, it may be necessary to implement different types of distributed locks, such as: fair locks, mutex locks, reentrant locks, read-write locks, red locks (redLock), etc. It is more troublesome to realize. The open source framework Redisson implements the above-mentioned lock functions, and has many other powerful functions.
Redisson is a Java in-memory data grid (In-Memory Data Grid) implemented on the basis of Redis. It not only provides a series of distributed common Java objects, but also provides many distributed services. These include ( BitSet, Set, Multimap, SortedSet, Map, , List, Queue, BlockingQueue, Deque, , , , , , , , , , , BlockingDeque) Semaphore, Locketc. AtomicLongRedisson CountDownLatchprovides Publish / Subscribethe Bloom filtereasiest Remote serviceand Spring cachemost convenient way to use Redis. The purpose of Redisson is to promote users' separation of concerns about Redis (Separation of Concern), so that users can focus more on processing business logic.Executor serviceLive Object serviceScheduler service
Dependency and client configuration
rely
<dependency>
<groupId>org.redisson</groupId>
<artifactId>redisson</artifactId>
<version>3.10.6</version>
</dependency>
Configure the Redisson client:
import org.redisson.Redisson;
import org.redisson.api.RedissonClient;
import org.redisson.config.Config;
import org.springframework.boot.autoconfigure.data.redis.RedisProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class RedisConfig {
@Bean
public RedissonClient redissonClient(RedisProperties prop) {
Config config = new Config();
config.useSingleServer().setAddress("redis://" + prop.getHost() + ":" + prop.getPort());
return Redisson.create(config);
}
}
Note: A property named RedisProperties is read here, because SpringDataRedis is introduced, Spring has automatically loaded RedisProperties, and read the Redis information in the configuration file.
Common APIs
RedissonClient interface
// 创建锁对象，并指定锁的名称
RLock getLock(String name)
RLock interface
acquire lock method
// 获取锁，`waitTime`默认0s，即获取锁失败不重试，`leaseTime`默认30s
boolean tryLock()
// 获取锁，设置锁等待时间`waitTime`、时间单位`unit`。释放时间`leaseTime`默认的30s
boolean tryLock(long waitTime, TimeUnit unit)
// 获取锁，设置锁等待时间`waitTime`、释放时间`leaseTime`，时间单位`unit`。
boolean tryLock(long waitTime, long leaseTime, TimeUnit unit)
// 如果获取锁失败后，会在`waitTime`减去获取锁用时的剩余时间段内继续尝试获取锁，如果依然获取失败，则认为获取锁失败；
// 获取锁后，如果超过`leaseTime`未释放，为避免死锁会自动释放。
// 释放锁
void unlock()
Redis distributed lock
Redis Distributed Lock Principle
The key to distributed locks is the memory tag shared by multiple processes , so you only need to place such a tag in Redis.
When implementing distributed locks, pay attention to the following goals:
-
Multi-process visible: multi-process visible, otherwise the distributed effect cannot be achieved
- Redis itself is shared by multiple services, so you don’t need to pay too much attention
-
Avoid deadlock: There are many situations of deadlock. It is necessary to consider the situation of various abnormalities leading to deadlock, and ensure that the lock can be released
-
Lock release problem after service downtime: It is best to set the validity period of the lock when setting the lock. If the service is down, the lock will be automatically deleted when the validity period expires
> set lock 001 nx ex 20 OK > get lock 001 > ttl lock 10 > set lock 001 nx ex 20 null
-
-
Exclusive: at a time, only one process can acquire the lock
-
It can be realized by using the setnx command (set when not exits) of Redis. When the setnx command is executed multiple times, only the first execution will succeed and return 1, otherwise it will return 0
> setnx lock 001 1 > get lock 001 > setnx lock 001 0Define a fixed key, multiple processes execute setnx, set the value of this key, the service that returns 1 acquires the lock, and the service that returns 0 does not acquire
-
-
High availability: remedial measures to avoid lock service downtime or handle downtime
- Use Redis master-slave, sentinel, cluster to ensure high availability
Distributed non-reentrant lock
process
According to the theory mentioned above, the process of distributed lock is roughly as follows:
Basic process:
- 1. Set the lock through the set command
- 2. Determine whether the returned result is OK
- 1) Nil, acquisition failed, end or retry (spin lock)
- 2) OK, the lock is acquired successfully
- perform business
- release lock
- 3. Abnormal circumstances, service downtime. When the timeout period EX ends, the lock will be released automatically
Note: When releasing the lock, you need to judge that the release of the value of the lock is consistent with the value stored in it
Otherwise, there will be a problem of releasing the lock in the following scenarios:
-
Three processes: A, B, and C are executing tasks and competing for locks. At this time, A acquires the lock and sets the automatic expiration time to 10s
-
A starts to execute the business. For some reason, the business is blocked and it takes more than 10 seconds. At this time, the lock is automatically released.
-
B starts to try to acquire the lock at this time, because the lock has been automatically released, and the lock is successfully acquired
-
At this time, A completes the business execution, and executes the release lock logic (delete key), so B's lock is released, and B is actually still executing the business
-
At this time, process C tries to acquire the lock, and it succeeds, because A deletes B's lock.
B and C acquire the lock at the same time, violating exclusivity!
Code
Define a lock interface:
public interface RedisLock {
boolean lock(long releaseTime);
void unlock();
}
Define a lock utility:
import org.springframework.data.redis.core.StringRedisTemplate;
import java.util.UUID;
import java.util.concurrent.TimeUnit;
public class SimpleRedisLock implements RedisLock{
private StringRedisTemplate redisTemplate;
/**
* 设定好锁对应的 key
*/
private String key;
/**
* 存入的线程信息的前缀，防止与其它JVM中线程信息冲突
*/
private final String ID_PREFIX = UUID.randomUUID().toString();
public SimpleRedisLock(StringRedisTemplate redisTemplate, String key) {
this.redisTemplate = redisTemplate;
this.key = key;
}
public boolean lock(long releaseTime) {
// 获取线程信息作为值，方便判断是否是自己的锁
String value = ID_PREFIX + Thread.currentThread().getId();
// 尝试获取锁
Boolean boo = redisTemplate.opsForValue().setIfAbsent(key, value, releaseTime, TimeUnit.SECONDS);
// 判断结果
return boo != null && boo;
}
public void unlock(){
// 获取线程信息作为值，方便判断是否是自己的锁
String value = ID_PREFIX + Thread.currentThread().getId();
// 获取现在的锁的值
String val = redisTemplate.opsForValue().get(key);
// 判断是否是自己
if(value.equals(val)) {
// 删除key即可释放锁
redisTemplate.delete(key);
}
}
}
Use locks in scheduled tasks:
import com.test.task.utils.SimpleRedisLock;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
@Slf4j
@Component
public class HelloJob {
@Autowired
private StringRedisTemplate redisTemplate;
@Scheduled(cron = "0/10 * * * * ?")
public void hello() {
// 创建锁对象
RedisLock lock = new SimpleRedisLock(redisTemplate, "lock");
// 获取锁,设置自动失效时间为50s
boolean isLock = lock.lock(50);
// 判断是否获取锁
if (!isLock) {
// 获取失败
log.info("获取锁失败，停止定时任务");
return;
}
try {
// 执行业务
log.info("获取锁成功，执行定时任务。");
// 模拟任务耗时
Thread.sleep(500);
} catch (InterruptedException e) {
log.error("任务执行异常", e);
} finally {
// 释放锁
lock.unlock();
}
}
}
distributed reentrant lock
Overview of reentrant locks
A reentrant lock, also called a recursive lock, means that in the same thread, after the outer function acquires the lock, the inner recursive function can still acquire the lock. In other words: when the same thread enters the synchronization code again, it can use the lock it has acquired.
Reentrant locks can avoid deadlocks caused by multiple lock acquisitions in the same thread.
How to implement a reentrant lock: When the lock has been used, determine whether the lock is your own, and if so, acquire it again .
When setting the value of the lock, you can store the information of the thread that acquired the lock , so that when you come back next time, you can know whether the person currently holding the lock is yourself, and if so, you are allowed to acquire the lock again.
It should be noted that because the acquisition of the lock is reentrant , the number of reentries must be recorded, so that the lock will not be released at once when the lock is released, but released layer by layer.
Therefore, the simple key-value structure can no longer be used, and the hash structure is recommended here:
- key：lock
- hashKey: thread information
- hashValue: number of reentries, default 1
When releasing the lock, the number of reentries is reduced by one each time to 0, indicating that the logic of multiple lock acquisitions has been executed before the key can be deleted and the lock released
flow chart
The focus here is on the process of acquiring locks:
The following assumes that the key of the lock is "lock", the hashKey is the id of the current thread: "threadId", and the automatic release time of the lock is assumed to be 20 s
Steps to acquire a lock:
-
Determine whether the lock exists
EXISTS lockIt exists, indicating that the lock has been acquired, and then judge whether it is your own lock
Determine whether the current thread id exists as a hashKey:
HEXISTS lock threadId- If it does not exist, it means that the lock has been acquired, and it is not acquired by itself. The lock acquisition fails, end
- It exists, indicating that it is a lock acquired by itself, and the number of reentries +1:
HINCRBY lock threadId 1, go to step 3
-
Does not exist, indicating that the lock can be acquired,
HSET key threadId 1 -
Set the lock auto-release time,
EXPIRE lock 20
Steps to release the lock:
- Determine whether the current thread id exists as a hashKey:
HEXISTS lock threadId- If it does not exist, it means that the lock has expired, so don’t worry about it
- It exists, indicating that the lock is still there, and the number of reentries is reduced by 1:
HINCRBY lock threadId -1, to obtain a new number of reentries
- Determine whether the number of reentries is 0:
- If it is 0, it means that all locks are released and the key is deleted:
DEL lock - If it is greater than 0, it means that the lock is still in use, and the valid time is reset:
EXPIRE lock 20
- If it is 0, it means that all locks are released and the key is deleted:
The biggest problem with the above process is that there are a lot of judgments, so when multi-threaded running, there will be thread safety issues, unless the atomicity of executing commands can be guaranteed .
Common distributed reentrant lock implementations:
- Redisson distributed lock
- Execute lua scripts. Multiple statements can be defined in a lua script, and statement execution is atomic.
Redisson distributed lock
code example
import lombok.extern.slf4j.Slf4j;
import org.redisson.api.RLock;
import org.redisson.api.RedissonClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.Scheduled;
@Slf4j
@Component
public class RedsssionJob {
@Autowired
private RedissonClient redissonClient;
@Scheduled(cron = "0/10 * * * * ?")
public void hello() {
// 创建锁对象，并制定锁的名称
RLock lock = redissonClient.getLock("taskLock");
// 获取锁，自动失效时间默认为50s
boolean isLock = lock.tryLock();
// 判断是否获取锁
if (!isLock) {
// 获取失败
log.info("获取锁失败，停止定时任务");
return;
}
try {
// 执行业务
log.info("获取锁成功，执行定时任务。");
// 模拟任务耗时
Thread.sleep(500);
} catch (InterruptedException e) {
log.error("任务执行异常", e);
} finally {
// 释放锁
lock.unlock();
log.info("任务执行完毕，释放锁");
}
}
}
Lua script distributed lock (understand)
For a detailed introduction to Lua scripts, see Extended Redis Lua Scripts
Distributed lock Lua scripting
Suppose there are 3 parameters:
- KEYS[1]: is the key of the lock
- ARGV[1]: is the thread id information
- ARGV[2]: lock expiration time
Acquire the lock:
if (redis.call('EXISTS', KEYS[1]) == 0) then
redis.call('HSET', KEYS[1], ARGV[1], 1);
redis.call('EXPIRE', KEYS[1], ARGV[2]);
return 1;
end;
if (redis.call('HEXISTS', KEYS[1], ARGV[1]) == 1) then
redis.call('HINCRBY', KEYS[1], ARGV[1], 1);
redis.call('EXPIRE', KEYS[1], ARGV[2]);
return 1;
end;
return 0;
Release the lock:
if (redis.call('HEXISTS', KEYS[1], ARGV[1]) == 0) then
return nil;
end;
local count = redis.call('HINCRBY', KEYS[1], ARGV[1], -1);
if (count > 0) then
redis.call('EXPIRE', KEYS[1], ARGV[2]);
return nil;
else
redis.call('DEL', KEYS[1]);
return nil;
end;
Java executes Lua script
RedisTemplate provides a method to execute Lua scripts:
public <T> T execute(RedisScript<T> script, List<K> keys, Object... args)
parameter:
- RedisScript script: an object that encapsulates a Lua script
- List keys : the value of the key in the script
- Object ... args : the values of the arguments in the script
To encapsulate the script into a RedisScript object, there are two ways to build a RedisScript object:
-
Method 1: Customize the object of RedisScript's implementation class DefaultRedisScript (commonly used)
// 场景脚本对象 DefaultRedisScript<Long> script = new DefaultRedisScript<Long>(); // 设置脚本数据源，从 classpath 读取 script.setScriptSource(new ResourceScriptSource(new ClassPathResource("lock.lua"))); // 设置返回值类型 script.setResultType(Long.class);You can write the script file to a certain location under the classpath, then get the script content by loading this file, and set it to the DefaultRedisScript instance
-
Method 2: Through the static method in RedisScript (you need to write the script content into the code and pass it as a parameter, which is not elegant enough)
static <T> RedisScript<T> of(String script) static <T> RedisScript<T> of(String script, Class<T> resultType)parameter:
- String script: Lua script
- Class resultType : return value type
Implementation of reentrant distributed lock
-
Write two Lua script files in the classpath
-
Define a class (ReentrantRedisLock) to implement the RedisLock interface
Basic logic: use static code blocks to load and initialize scripts, and implement the lock and unlock methods of the RedisLock interface
public class ReentrantRedisLock implements RedisLock { private StringRedisTemplate redisTemplate; /** * 设定好锁对应的 key */ private String key; /** * 存入的线程信息的前缀，防止与其它JVM中线程信息冲突 */ private final String ID_PREFIX = UUID.randomUUID().toString(); public ReentrantRedisLock(StringRedisTemplate redisTemplate, String key) { this.redisTemplate = redisTemplate; this.key = key; } private static final DefaultRedisScript<Long> LOCK_SCRIPT; private static final DefaultRedisScript<Object> UNLOCK_SCRIPT; static { // 加载释放锁的脚本 LOCK_SCRIPT = new DefaultRedisScript<>(); LOCK_SCRIPT.setScriptSource(new ResourceScriptSource(new ClassPathResource("lock.lua"))); LOCK_SCRIPT.setResultType(Long.class); // 加载释放锁的脚本 UNLOCK_SCRIPT = new DefaultRedisScript<>(); UNLOCK_SCRIPT.setScriptSource(new ResourceScriptSource(new ClassPathResource("unlock.lua"))); } // 锁释放时间 private String releaseTime; @Override public boolean lock(long releaseTime) { // 记录释放时间 this.releaseTime = String.valueOf(releaseTime); // 执行脚本 Long result = redisTemplate.execute( LOCK_SCRIPT, Collections.singletonList(key), ID_PREFIX + Thread.currentThread().getId(), this.releaseTime); // 判断结果 return result != null && result.intValue() == 1; } @Override public void unlock() { // 执行脚本 redisTemplate.execute( UNLOCK_SCRIPT, Collections.singletonList(key), ID_PREFIX + Thread.currentThread().getId(), this.releaseTime); } } -
Create a new scheduled task to test the reentrant lock:
import com.leyou.task.utils.RedisLock; import com.leyou.task.utils.ReentrantRedisLock; import lombok.extern.slf4j.Slf4j; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.data.redis.core.StringRedisTemplate; import org.springframework.scheduling.annotation.Scheduled; import org.springframework.stereotype.Component; @Slf4j @Component public class ReentrantJob { @Autowired private StringRedisTemplate redisTemplate; private int max = 2; @Scheduled(cron = "0/10 * * * * ?") public void hello() { // 创建锁对象 RedisLock lock = new ReentrantRedisLock(redisTemplate, "lock"); // 执行任务 runTaskWithLock(lock, 1); } private void runTaskWithLock(RedisLock lock, int count) { // 获取锁,设置自动失效时间为50s boolean isLock = lock.lock(50); // 判断是否获取锁 if (!isLock) { // 获取失败 log.info("{}层 获取锁失败，停止定时任务", count); return; } try { // 执行业务 log.info("{}层 获取锁成功，执行定时任务。", count); Thread.sleep(500); if(count < max){ runTaskWithLock(lock, count + 1); } } catch (InterruptedException e) { log.error("{}层 任务执行失败", count, e); } finally { // 释放锁 lock.unlock(); log.info("{}层 任务执行完毕，释放锁", count); } } }
expand
Cache penetration, avalanche, breakdown
-
cache penetration
It refers to querying a data that must not exist (the data does not exist in the cache or the database), and the database will be queried every time. When the concurrency is high, the database may hang or io blocking may occur.
The reason is: Generally, the database is queried for caching when the first request misses, and for fault tolerance considerations, if the data cannot be found from the database, it will not be written into the cache, which will cause non-existent data to be queried to the database every time it is requested, losing the meaning of caching.
If someone uses a non-existent key to frequently attack the application, this is a vulnerability.
solution:
- Solution 1: Use the Bloom filter to hash all possible data into a sufficiently large bitMap, and a data that must not exist will be intercepted by this bitMap, thereby avoiding the query pressure on the underlying storage system.
- Solution 2: If the data returned by a query is empty (whether the data does not exist or the system fails), store a null character in the cache and set a short expiration time, no more than five minutes.
-
cache avalanche
It means that the same expiration time is used when setting the cache, causing the cache to fail at a certain moment at the same time, all requests are forwarded to the DB, and the instantaneous pressure on the DB is too heavy to cause an avalanche.
The avalanche effect of cache invalidation has a terrible impact on the underlying system.
solution:
-
Solution 1: Use locks or queues to ensure single-threaded (process) writing of the cache, so as to avoid a large number of concurrent requests from falling on the underlying storage system when it fails
-
Solution 2: Disperse the cache expiration time. For example, you can add a random value based on the original expiration time, such as 1-5 minutes random, so that the repetition rate of each cache expiration time will be reduced, and it will be difficult to trigger collective invalidation events.
-
-
cache breakdown
A key with an expiration time is set to be accessed at high concurrency when it fails (very "hot" data), the cache expires, and the data has not been reloaded into the cache, and the concurrency pressure is instantly transferred to the database. At this time, large concurrent requests may instantly cause the database to hang or io to block.
The difference between cache breakdown and cache avalanche is that cache breakdown is invalid for a certain key cache, while cache avalanche is that many keys are invalid at the same time.
solution:
-
Use a mutex (mutex key)
Commonly used mutex. To put it simply, when the cache fails (judging that the retrieved value is empty), instead of loading db immediately, first use some operations of the cache tool with a successful operation return value (such as Redis's SETNX or Memcache's ADD) to set a mutex key. When the operation returns successfully, then perform the load db operation and reset the cache; otherwise, retry the entire get cache method.
-
Lua script for Redis
introduce
There are many ways to implement Redis atomic operations, such as Redis transactions, but in comparison, using Redis Lua scripts is better and has irreplaceable benefits:
- Atomicity: Redis will execute the entire script as a whole and will not be inserted by other commands.
- Reuse: The script sent by the client will be permanently stored in redis, and can be reused in the future, and can be shared by each Redis client.
- Efficient: After the Lua script is parsed, it will form a cache, so it does not need to be parsed every time it is executed.
- Reduce network overhead: After the Lua script is cached, the SHA value can be formed as the cache key. Later calls can directly call the script according to the SHA value, without sending the complete script every time, reducing network usage and delay
Redis script commands
Common commands:
-
Execute a script directly : EVAL script numkeys key [key ...] arg [arg ...]
parameter:
- script: script content, or script address
- numkeys: the number of keys used in the script, the next numkeys parameters will be used as key parameters, and the rest will be used as arg parameters
- key: As a parameter of key, it will be stored in the KEYS array in the script environment, and the subscript starts from 1
- arg: other parameters, which will be stored in the ARGV array in the script environment, and the subscript starts from 1
Example:
> eval "return 'hello world!'" 0 hello world!in:
- "return 'hello world!'": It is the content of the script, directly returns the string, no other commands
- 0: It means that the key parameter is not used, and it returns directly
-
Compile and cache a script, generate and return a SHA1 value as the key of the script dictionary : SCRIPT LOAD script
parameter:
- script: script content, or script address
Example:
> script load "return 'hello world!'" ada0bc9efe2392bdcc0083f7f8deaca2da7f32ecin:
-
What is returned
ada0bc9efe2392bdcc0083f7f8deaca2da7f32ecis the sha1 value obtained after the script cacheIn the script dictionary, each such sha1 value corresponds to a parsed script
-
Execute a script by its sha1 value : EVALSHA sha1 numkeys key [key ...] arg [arg ...]
Similar to EVAL, the difference is that through the sha1 value of the script, go to the script cache to look it up, and then execute
parameter:
- sha1: is the sha1 value corresponding to the script
Example:
> evalsha ada0bc9efe2392bdcc0083f7f8deaca2da7f32ec 0 hello world!
Basic syntax of Lua
Lua scripts follow the basic syntax of Lua, several commonly used ones:
-
**Two functions that call the redis command: **redis.call() and redis.pcall()
The difference is that if an error occurs during call execution, the error will be returned directly; pcall will continue to execute downward after encountering an error. The basic syntax is similar to:
redis.call("命令名称", 参数1， 参数2 ...)Example:
eval "return redis.call('set', KEYS[1], ARGV[1])" 1 name Jackin:
- 'set': is to execute the set command
- KEYS[1]: Take the first key parameter from the KEYS array in the script environment
- ARGV[1]: Take the first arg parameter from the ARGV array in the script environment
- 1: There is only one declared key, and the next first parameter is used as the key parameter
- name: key parameter, which will be stored in the KEYS array
- Jack: the arg parameter, which will be stored in the ARGV array
-
Conditional judgment syntax : if (conditional statement) then ...; else ...; end;
Variable receiving syntax : local variable name = variable value;
Example:
local val = redis.call('get', KEYS[1]); if (val > ARGV[1]) then return 1; else return 0; end;Basic logic: Get the value of the specified key, judge whether it is greater than the specified parameter, return 1 if greater, otherwise return 0
Example:
> set num 321 OK > script load "local val = redis.call('get', KEYS[1]); if (val > ARGV[1]) then return 1; else return 0; end;" ad4bc448c3c264aeaa475a0407683c35bf1bc7af > evalsha ad4bc448c3c264aeaa475a0407683c35bf1bc7af 1 num 400 0in:
- num starts with 321
- save script
- Then execute and pass num, 400. Determine whether the value of num is greater than 400
- result returns 0
… ↩︎