When storm writes a large number of cedis, the problem of tack deadlock and inability to write

Error code example:

private void batchExec(RedisBatchExector rbe, ThreadData td) throws Exception {
		List<Object> readData = null;
		Jedis jedis = null;
		try {
			//The following is the correct code, temporarily blocked
			//jedis = ((RoundRobinJedisPool)jedisPool).getResource(1, TimeUnit.SECONDS);
			jedis = jedisPool.getResource();
			//logger.info("--BatchHllTableWriter-batchExec-td.index={}----",td.index );

			Pipeline pipeline = jedis.pipelined();
			for (Cell<String, String, HLL> cell : td.cells) {
				byte[] hashKey = cell.getRowKey().getBytes(Charsets.UTF_8);
				byte[] fieldName = cell.getColumnKey().getBytes(Charsets.UTF_8);
				pipeline.hget(hashKey, fieldName);
			}
			readData = pipeline.syncAndReturnAll();
			jedis.close();
		}catch(Exception e){
			if(jedis!=null){
				jedis.close();
			}
			throw e;
		}
		for (int i = 0; i < batchSize; i++) {
			Cell<String, String, HLL> cell = td.cells[i];
			byte[] hashKey = cell.getRowKey().getBytes(Charsets.UTF_8);
			byte[] fieldName = cell.getColumnKey().getBytes(Charsets.UTF_8);
			HLL smallHll = cell.getValue();

			byte[] bigBytes = (byte[]) readData.get(i);
			bigBytes = union(bigBytes, smallHll);
			rbe.hset(hashKey, fieldName, bigBytes);
		}
	}

 

@Override
	public Jedis getResource() {
		return internalPool.take();
	}

	public Jedis getResource(long timeout, TimeUnit timeUnit){
	    return internalPool.tryTake(timeout, timeUnit);
    }

 Added internalPool.tryTake(timeout, timeUnit); method

When a deadlock occurs in the tack, use tryTack to obtain a new redis connection

jedis.minIdle: 20
jedis.maxIdle: 200
jedis.maxTotal: 200
jedis.testOnBorrow: false
jedis.addresses:
    - 10.221.193.2:6402
    - 10.221.193.2:6403
    - 10.221.193.2:6404

 

public synchronized RoundRobinJedisPool getJedisPool() {
		if (jedisPool != null) {
			return jedisPool;
		}
		Map<String, Object> props = mapPrefix(JEDIS_PREFIX);
		Number minIdle = (Number) props.get("minIdle");
		Number maxTotal = (Number) props.get("maxTotal");
		List<String> addresses = (List<String>) props.get("addresses");
		String password = null;
		if (props.get("password") != null
				&& StringUtils.isNotBlank(String.valueOf(props.get("password")))
				&& !StringUtils.equals("null",String.valueOf(props.get("password")))) {
			password = String.valueOf(props.get("password"));
		}
		jedisPool = new RoundRobinJedisPool(addresses, null, minIdle.intValue(), maxTotal.intValue());
		return jedisPool;
	}

 

public RoundRobinJedisPool(List<String> addresses, String password,
			int initialSize, int maxSize) {

		super();
		// The starting index of each process is random to avoid links being concentrated in a certain instance.
		Random random = new Random(System.currentTimeMillis());
		int offset = random.nextInt(addresses.size() + 1);
		jedisIndex.addAndGet(offset);

		this.addresses = addresses;
		this.password = password;
		initialSize = (initialSize / addresses.size() + 1) * addresses.size();
		maxSize = (maxSize / addresses.size() + 1) * addresses.size();
		internalPool = new ConcurrentLinkedPool<Jedis>(this, initialSize,
				maxSize, false);

	}

 It can be seen from the above code that maxTotal cannot be too small, otherwise, when a large amount of data is written, it is easy to cause a deadlock of the stack

 

 

Summarize:

In my actual work environment, I encountered this problem because of the following reasons:

1. jedis.maxIdle: 200 jedis.maxTotal: 200 The two values ​​are set too small, the original is 4

2. In the first-tier bolt, set 100,000 records to be submitted to the second-tier bolt once, resulting in a huge amount of data submitted each time

3. Spout into the first layer of bolts adopts the fieldsGrouping grouping method, which directly results in the results generated by the first layer of bolts. The content of a single key is larger.

4. In the second layer of bolt, there are two types of results written to redis, one is long type, which is directly superimposed in redis, and the second is the deduplication data type of hll, which needs to obtain the existing hll from redis After calculating the new value and putting it back in redis, it consumes more performance and connection time of redis

Based on the situation in the above 4, it appears: the long-type writing problem is not big, but the hll problem is more serious and cannot be written at all