Jedisconnection pool
RedisThe connection pool, like the database connection pool, also creates and manages a set of connections in advance, so that when it needs to Redisinteract with the server, the connection can be reused directly. All Redisclients have implemented the function of connection pool. In this article, we first take as an example , and introduce the specific implementation functions in detail from the aspects of connection acquisition, return, closure, and creation.JedisLettuceJedis
use of multithreading
In the database, if multiple threads reuse a connection, there will be database transaction problems, so Redislet's first look at what problems will arise when using a connection in a multi-threaded environment?
Firstly, start two threads to jointly operate the same Jedisinstance, each thread loops 500 times, and reads the values of key a and b respectively
java
copy code
Jedis jedis = new Jedis("127.0.0.1", 6379); new Thread(() -> { for (int i = 0; i < 500; i++) { String result = jedis.get("a"); System.out.println(result); } }).start(); new Thread(() -> { for (int i = 0; i < 500; i++) { String result = jedis.get("b"); System.out.println(result); } }).start();
Execute the program many times, and you can see that there are various strange exception information in the log, some unknown reply errors, and some connection close exceptions, etc.
错误1：redis.clients.jedis.exceptions.JedisConnectionException: Unknown reply: 1
错误2：java.io.IOException: Socket Closed
Then let's take a look at Jedisthe source code of the commonly used (3.x) version
java
copy code
public class Jedis extends BinaryJedis implements JedisCommands, MultiKeyCommands, AdvancedJedisCommands, ScriptingCommands, BasicCommands, ClusterCommands, SentinelCommands, ModuleCommands{} public class BinaryJedis implements BasicCommands, BinaryJedisCommands, MultiKeyBinaryCommands, AdvancedBinaryJedisCommands, BinaryScriptingCommands, Closeable { protected final Client client; } public class Client extends BinaryClient implements Commands {} public class BinaryClient extends Connection {} public class Connection implements Closeable { private Socket socket; private RedisOutputStream outputStream; private RedisInputStream inputStream; }
First of all, Jedisit inherits BinaryJedis, BinaryJedisand saves a single Clientinstance of , and Clientfinally inherits Connection, Connectionand saves a single Socketinstance of , and the corresponding two read and write streams, one RedisOutputStreamis one RedisInputStream.
BinaryClientVarious commands are encapsulated Redis, and the method will be called eventually sendCommand, and it is found that when sending a command, it directly operates RedisOutputStreamand writes bytes.
java
copy code
private static void sendCommand(final RedisOutputStream os, final byte[] command, final byte[]... args) { try { os.write(ASTERISK_BYTE); os.writeIntCrLf(args.length + 1); os.write(DOLLAR_BYTE); os.writeIntCrLf(command.length); os.write(command); os.writeCrLf(); for (final byte[] arg : args) { os.write(DOLLAR_BYTE); os.writeIntCrLf(arg.length); os.write(arg); os.writeCrLf(); } } catch (IOException e) { throw new JedisConnectionException(e); } }
So using it in a multi-threaded environment Jedisis actually multiplexing RedisOutputStream. If multiple threads are executing the operation, there is no guarantee that the entire command will be written atomically Socket. For example, write operations interfere with each other, and if multiple commands are interspersed with each other, they must not be legal Rediscommands, which will lead to various problems.
This also shows that Jedisit is not thread safe. However, instances can JedisPoolbe managed through the connection pool, and each thread has its own independent Jedisinstance in the case of multi-threading, which can become thread-safe.
java
copy code
//使用redis连接池,不会有线程安全问题 private static JedisPool jedisPool = new JedisPool("127.0.0.1", 6379); new Thread(() -> { try (Jedis jedis = jedisPool.getResource()) { for (int i = 0; i < 1000; i++) { String result = jedis.get("a"); System.out.println(result); } } }).start(); new Thread(() -> { try (Jedis jedis = jedisPool.getResource()) { for (int i = 0; i < 1000; i++) { String result = jedis.get("b"); System.out.println(result); } } }).start();
Connection pool management
JedisPoolThe connection pool is based on the realization Apache Commons Poolof the GenericObjectPool. Let's first understand Apache Commons Poolthe implementation
Apache Commons Pool
Apache Commons PoolIt is an open source general object pool implementation, which provides the basic functions of the object pool, such as object creation, destruction, borrowing and returning, etc. Apache Commons PoolThere are three core components as follows, which are mainly responsible for the general configuration of objects, the creation of objects, and the management of object pools.
GenericObjectPoolConfig
GenericObjectPoolConfigThe class is responsible for general object pool configuration information, such as the maximum number of objects, the minimum number of free objects, and so on.
JedisPoolConfigThrough inheritance GenericObjectPoolConfig, many personalized configurations about idle connection detection are set.
java
copy code
public class GenericObjectPoolConfig<T> extends BaseObjectPoolConfig<T> { /** * 对象池中最大对象数 * @see GenericObjectPool#getMaxTotal() */ public static final int DEFAULT_MAX_TOTAL = 8; /** * 对象池中最大空闲对象数 * @see GenericObjectPool#getMaxIdle() */ public static final int DEFAULT_MAX_IDLE = 8; /** * 对象池中最小空闲对象数 * @see GenericObjectPool#getMinIdle() */ public static final int DEFAULT_MIN_IDLE = 0; } public class JedisPoolConfig extends GenericObjectPoolConfig<Jedis> { public JedisPoolConfig() { //空闲时是否进行对象有效性检查 setTestWhileIdle(true); //连接空闲的最小时间 setMinEvictableIdleTimeMillis(60000); //“空闲链接”检测线程，检测的周期，毫秒数 setTimeBetweenEvictionRunsMillis(30000); //对所有连接做空闲监测 setNumTestsPerEvictionRun(-1); } }
PooledObjectFactory
PooledObjectFactoryThis object factory is mainly responsible for the creation and destruction of objects. It is an interface that JedisFactoryimplements the corresponding interface functions.
java
copy code
public interface PooledObjectFactory<T> { //"激活"对象 void activateObject(PooledObject<T> var1) throws Exception; //销毁对象 void destroyObject(PooledObject<T> var1) throws Exception; default void destroyObject(PooledObject<T> p, DestroyMode destroyMode) throws Exception { this.destroyObject(p); } //创建一个新对象 PooledObject<T> makeObject() throws Exception; // "钝化"对象, void passivateObject(PooledObject<T> var1) throws Exception; //检测对象是否"有效" boolean validateObject(PooledObject<T> var1); }
GenericObjectPool
GenericObjectPoolIt is mainly responsible for operating objects in the object pool, obtaining objects from the object pool, returning objects, and other operations. And GenericObjectPoolby holding the above PooledObjectFactoryobject factory, and then operate the corresponding object.
java
copy code
public interface ObjectPool<T> extends Closeable { //从池中获取对象 T borrowObject() throws Exception, NoSuchElementException, IllegalStateException; //清除池，池可用 void clear() throws Exception, UnsupportedOperationException; //关闭池，池不可用 @Override void close(); //将对象放回池中 void returnObject(T obj) throws Exception; }
summary
The above are Apache Commons Poolthe specific core components and functions. Next, let's see JedisPoolhow the connection pool implements specific functions based on it.
get connection
When we use it JedisPool, we use getResource()the method to get it Jedis, the following code
Jedis jedis = jedisPool.getResource()
Let's look at the source code first. In the end, the method in the object pool getResource()is actually called .GenericObjectPoolborrowObject
java
copy code
//Pool#getResource public T getResource() { try { return internalPool.borrowObject(); } catch (NoSuchElementException nse) { if (null == nse.getCause()) { //异常是连接池耗尽导致的 throw new JedisExhaustedPoolException( "Could not get a resource since the pool is exhausted", nse); } //异常是 activateObject() or ValidateObject()导致的 throw new JedisException("Could not get a resource from the pool", nse); } catch (Exception e) { throw new JedisConnectionException("Could not get a resource from the pool", e); } }
Let's look at the overall process first. The actual thing to do is to obtain the object from the idle queue. If there is no object, create the object information, then activate the object instance, verify the validity of the object, and finally return a corresponding object instance.
Next, let's look at the specific source code, the method implementation GenericObjectPoolin the object poolborrowObject
java
copy code
//GenericObjectPool#borrowObject private final LinkedBlockingDeque<PooledObject<T>> idleObjects; public T borrowObject(final Duration borrowMaxWaitDuration) throws Exception { //检查对象池状态，看看是否已经被关闭了 assertOpen(); //清除废弃的对象 final AbandonedConfig ac = this.abandonedConfig; if (ac != null && ac.getRemoveAbandonedOnBorrow() && (getNumIdle() < 2) && (getNumActive() > getMaxTotal() - 3)) { removeAbandoned(ac); } PooledObject<T> p = null; final boolean blockWhenExhausted = getBlockWhenExhausted(); boolean create; final long waitTimeMillis = System.currentTimeMillis(); while (p == null) { create = false; //从LinkedBlockingDeque队列中拿出第一个元素 p = idleObjects.pollFirst(); if (p == null) { //创建对象 p = create(); if (p != null) { //创建成功，创建标识置为true create = true; } } if (blockWhenExhausted) { //上面没有创建成功 if (p == null) { //如果maxWaitDuration设置的为负数 if (borrowMaxWaitDuration.isNegative()) { // 从空闲队列获取，但是该方法会阻塞，一直等到有可用空闲对象。 p = idleObjects.takeFirst(); } else { // 如果设置了一个有效的等待时间,最多等待borrowMaxWaitMillis毫秒。还取不到就返回空 p = idleObjects.pollFirst(borrowMaxWaitDuration); } } if (p == null) { throw new NoSuchElementException(appendStats( "Timeout waiting for idle object, borrowMaxWaitDuration=" + borrowMaxWaitDuration)); } } else if (p == null) { throw new NoSuchElementException(appendStats("Pool exhausted")); } // 如果分配失败（可认为被别人抢走了），p置为空（可以进行下一次循环遍历） if (!p.allocate()) { p = null; } if (p != null) { try { //通过对象池工厂，激活这个对象 //jedis连接池的实现是JedisFactory，做了一个redis的select连库请求 factory.activateObject(p); } catch (final Exception e) { try { // 如果激活对象时，发生了异常,销毁对象 destroy(p, DestroyMode.NORMAL); } catch (final Exception e1) {} p = null; if (create) { final NoSuchElementException nsee = new NoSuchElementException( appendStats("Unable to activate object")); nsee.initCause(e); throw nsee; } } if (p != null && getTestOnBorrow()) { boolean validate = false; Throwable validationThrowable = null; try { //激活成功，开始校验对象。jedis的实现是，发一条redis的ping命令来校验连接的有效性 validate = factory.validateObject(p); } catch (final Throwable t) { PoolUtils.checkRethrow(t); validationThrowable = t; } if (!validate) { try { //校验对象失败，开始销毁对象 destroy(p, DestroyMode.NORMAL); destroyedByBorrowValidationCount.incrementAndGet(); } catch (final Exception e) { // Ignore - validation failure is more important } p = null; if (create) { final NoSuchElementException nsee = new NoSuchElementException( appendStats("Unable to validate object")); nsee.initCause(validationThrowable); throw nsee; } } } } } //更新对象池统计信息 updateStatsBorrow(p, Duration.ofMillis(System.currentTimeMillis() - waitTimeMillis)); //返回对象实例 return p.getObject(); }
return link
Then let's see how to return the connection. The main process is actually to add the connection to the idle queue.
The connection return is triggered by the method Jedisinside , and the actual call is still in the class . We mainly look at this methodclose()GenericObjectPoolreturnObject()
java
copy code
//Jedis#close public void close() { if (dataSource != null) { JedisPoolAbstract pool = this.dataSource; this.dataSource = null; if (isBroken()) { pool.returnBrokenResource(this); } else { pool.returnResource(this); } } else { super.close(); } }
java
copy code
//GenericObjectPool#returnObject public void returnObject(final T obj) { //从ConcurrentHashMap中获取原始对象的PooledObject对象 final PooledObject<T> p = getPooledObject(obj); //如果p为空，说明这个要还的对象，已经不在池子中了 if (p == null) { if (!isAbandonedConfig()) { throw new IllegalStateException( "Returned object not currently part of this pool"); } return; } //使用同步锁，标记返回对象的状态 markReturningState(p); //获取对象使用时间 final Duration activeTime = p.getActiveDuration(); //如果testOnReturn配置为true，需要校验有效性 if (getTestOnReturn() && !factory.validateObject(p)) { try { //如果校验不通过，则销毁该对象 destroy(p, DestroyMode.NORMAL); } catch (final Exception e) { swallowException(e); } try { ensureIdle(1, false); } catch (final Exception e) { swallowException(e); } updateStatsReturn(activeTime); return; } //钝化对象，也就是反初始化，也就是释放核心资源，JedisFactory里面是什么都没有实现的 try { factory.passivateObject(p); } catch (final Exception e1) { swallowException(e1); try { destroy(p, DestroyMode.NORMAL); } catch (final Exception e) { swallowException(e); } try { ensureIdle(1, false); } catch (final Exception e) { swallowException(e); } updateStatsReturn(activeTime); return; } //变更状态为 IDLE if (!p.deallocate()) { throw new IllegalStateException( "Object has already been returned to this pool or is invalid"); } //获取对象池配置的最大空闲对象数量 final int maxIdleSave = getMaxIdle(); //目前空闲对象数量已经达到规定的最大值，直接销毁对象 if (isClosed() || maxIdleSave > -1 && maxIdleSave <= idleObjects.size()) { try { destroy(p, DestroyMode.NORMAL); } catch (final Exception e) { swallowException(e); } try { ensureIdle(1, false); } catch (final Exception e) { swallowException(e); } } else { // 如果对象可以被正常归还，那么把对象添加到空闲队列 if (getLifo()) { // 如果是后进先出，那么把空闲对象添加到队列开头 idleObjects.addFirst(p); } else { // 如果是先进先出，那么把空闲对象添加到队列末尾 idleObjects.addLast(p); } // 判断一下对象池状态，如果是关闭状态，那么调用clear方法，清空对象池 if (isClosed()) { clear(); } } //更新统计信息 updateStatsReturn(activeTime); }
close connection
Closing the connection is actually destroying the object, calling the method GenericObjectPoolin the class , and finally calling the implemented method of closing the physical connection to close the server connection.destroy()JedisFactorysocket
java
copy code
//GenericObjectPool#destroy private void destroy(final PooledObject<T> toDestroy, final DestroyMode destroyMode) throws Exception { //设置状态为无效状态 toDestroy.invalidate(); //空闲列表移除当前对象 idleObjects.remove(toDestroy); //所有对象列表移除当前对象 allObjects.remove(new IdentityWrapper<>(toDestroy.getObject())); try { //调用 JedisFactory的destroyObject方法 factory.destroyObject(toDestroy, destroyMode); } finally { //增加销毁数量 destroyedCount.incrementAndGet(); //减少创建的数量 createCount.decrementAndGet(); } } //JedisFactory#destroyObject public void destroyObject(PooledObject<Jedis> pooledJedis) throws Exception { final BinaryJedis jedis = pooledJedis.getObject(); if (jedis.isConnected()) { try { jedis.close(); } catch (RuntimeException e) { logger.debug("Error while close", e); } } } //Connection#disconnect //关闭服务器socket连接 public void disconnect() { if (isConnected()) { try { outputStream.flush(); socket.close(); } catch (IOException ex) { broken = true; throw new JedisConnectionException(ex); } finally { IOUtils.closeQuietly(socket); } } }
create connection
To create a connection, in fact, when the connection is obtained above, if it is found that there is no free connection in the free list, create()the method will be called to create the object.
java
copy code
private PooledObject<T> create() throws Exception { // 获取对象池的最大数量配置 int localMaxTotal = getMaxTotal(); if (localMaxTotal < 0) { //配置为负数，则为无限 localMaxTotal = Integer.MAX_VALUE; } final long localStartTimeMillis = System.currentTimeMillis(); final long localMaxWaitTimeMillis = Math.max(getMaxWaitDuration().toMillis(), 0); Boolean create = null; while (create == null) { synchronized (makeObjectCountLock) { // createCount先自增 final long newCreateCount = createCount.incrementAndGet(); //当前数量大于配置的最大数量，池子满了 if (newCreateCount > localMaxTotal) { createCount.decrementAndGet(); //当前没有创建的对象数量，则无需创建对象 if (makeObjectCount == 0) { create = Boolean.FALSE; } else { //否则等待对象的返回 makeObjectCountLock.wait(localMaxWaitTimeMillis); } } else { // 对象池未达到容量。创建新对象 makeObjectCount++; create = Boolean.TRUE; } } //超过了最大等待时间 if (create == null && (localMaxWaitTimeMillis > 0 && System.currentTimeMillis() - localStartTimeMillis >= localMaxWaitTimeMillis)) { create = Boolean.FALSE; } } if (!create.booleanValue()) { return null; } final PooledObject<T> p; try { //创建对象调用的是JedisFactory中实现的 p = factory.makeObject(); //如果testOnReturn配置为true，需要校验有效性 if (getTestOnCreate() && !factory.validateObject(p)) { //不合法减少创建数量，返回空 createCount.decrementAndGet(); return null; } } catch (final Throwable e) { //创建失败，减少创建数量，抛出异常 createCount.decrementAndGet(); throw e; } finally { //释放锁，通知其他的等待线程 synchronized (makeObjectCountLock) { makeObjectCount--; makeObjectCountLock.notifyAll(); } } //清除废弃的对象配置 final AbandonedConfig ac = this.abandonedConfig; if (ac != null && ac.getLogAbandoned()) { p.setLogAbandoned(true); p.setRequireFullStackTrace(ac.getRequireFullStackTrace()); } //增加createdCount数量 createdCount.incrementAndGet(); //新的对象创建好了，需要把他添加到池子里,allObjects用的一个ConcurrentHashMap allObjects.put(new IdentityWrapper<>(p.getObject()), p); return p; }
JedisFactoryThe object creation method implemented in is actually creating an Jedisinstance
java
copy code
//JedisFactory#makeObject public PooledObject<Jedis> makeObject() throws Exception { Jedis jedis = null; try { //创建redis连接 jedis = new Jedis(jedisSocketFactory, clientConfig); jedis.connect(); return new DefaultPooledObject<>(jedis); } catch (JedisException je) { if (jedis != null) { try { jedis.close(); } catch (RuntimeException e) { logger.debug("Error while close", e); } } throw je; } }
Connection pool configuration
Next, let's look at the commonly used configuration parameters and suggestions.
| parameter | illustrate | Defaults | suggestion |
|---|---|---|---|
| maxTotal | The maximum number of connections in the resource pool | 8 | |
| maxIdle | The maximum number of idle connections allowed by the resource pool | 8 | |
| minIdle | The minimum number of idle connections guaranteed by the resource pool | 0 | |
| blockWhenExhausted | Whether the caller should wait when the resource pool is exhausted. Only when the value is true, the following maxWaitMillis will take effect. | true | The default is suggested. |
| maxWaitMillis | When the resource pool connection is exhausted, the caller's maximum waiting time (in milliseconds). | -1 (never timeout) | The default is not recommended. |
| testOnBorrow | Whether to perform connection validity detection (ping) when borrowing a connection from the resource pool. Invalid connections detected will be removed. | false | It is recommended to set it to false to reduce the overhead of a ping if the business volume is large. |
| testOnReturn | Whether to check the validity of the connection (ping) when returning the connection to the resource pool. Invalid connections detected will be removed. | false | It is recommended to set it to false to reduce the overhead of a ping if the business volume is large. |
| jmxEnabled | Whether to enable JMX monitoring | true | It is recommended to enable, please note that the application itself also needs to be enabled |
| testWhileIdle | Whether to enable idle resource detection. | false | true |
| timeBetweenEvictionRunsMillis | The detection cycle of idle resources (in milliseconds) | -1 (do not detect) | Suggested settings, choose the cycle yourself |
| minEvictableIdleTimeMills | The minimum idle time of resources in the resource pool (in milliseconds), after reaching this value, idle resources will be removed. | 30 minutes | You can decide according to your own business, generally the default value is enough |
| numTestsPerEvictionRun | When doing idle resource detection, the number of resources is detected each time. | 3 | It can be fine-tuned according to the number of connections in its own application. If it is set to -1, it will perform idle monitoring on all connections. |
The most important of these is the maximum number of connections ( maxTotal) . It can be estimated according to the following formula first, but the actual situation is QPSto Redisevaluate the size of the connection pool used by each node as a whole based on the size of the client called by the total business.
Maximum number of connections = average command execution time (S) ∗ business QPS maximum number of connections = average command execution time (S) * business QPS maximum number of connections = average command execution time (S) ∗ business QPS
If redisthe average command time is about 1ms, and the business expects QPS10000, then the theoretically required number of connections is 0.001/10000=100.001 / 10000 = 100.001/10000=10
Summarize
JedisPoolThe connection pool is based on Apache Commons Pool the GenericObjectPoolimplementation, which is HikariCPsimpler to implement than the database connection pool. You can also use it Apache Commons Poolto implement other connection pool technologies, such as FTPconnection pools, etc.