分布式缓存技术之Redis_04Redis的应用实战

1 Redis Java客户端的使用

 Redis的Java客户端: Jedis，Redisson，Lettuce
Jedis : java的客户端，提供了比较全面的redis命令支持
Redisson : 在基础上进行封装，直接提供了分布式锁，栈队列等功能 ；和Jedis相比，功能较为简单，不支持字符串操作，不支持排序、事务、管道、分区等Redis特性。Redisson主要是促进使用者对Redis的关注分离，从而让使用者能够将精力更集中地放在处理业务逻辑上。
Lettuce : 基于Netty构建的线程安全的高级Reds客户端，用于线程安全同步，异步和响应使用，支持集群，Sentinel，管道和编码器

Jedis 单点连接

  当是单点服务时，Java 连接Redis的客户端:

  Jedis jedis = null;

        try {
           jedis = new Jedis("192.168.237.130", 6379);
           jedis.hset("hashzz", "k1", "v1");
        } catch (Exception e) {
            System.out.println(e);
            e.printStackTrace();
        } finally {
            if (null != jedis) {
                jedis.disconnect();
            }
        }

 或者

        JedisPool pool = null;
        try {
            pool = new JedisPool("192.168.237.130", 6379);
            pool.getResource().hset("hashzz", "k2", "v2");
        } catch (Exception e) {
            System.out.println(e);
            e.printStackTrace();
        } finally {
            if (null != pool) {
                pool.close();
            }
        }

Jedis sentinel连接哨兵集群

 // sentinel 哨兵
        // sentinel.conf 中配置的master名称
        String masterName = "mymaster";
        // sentinel 集群环境
        Set<String> sentinelIps = new HashSet<>();
        sentinelIps.add("192.168.237.129:26370");
        sentinelIps.add("192.168.237.130:26370");
        JedisSentinelPool sentinelPool = null;

        Jedis jedis = null;

        try {
            sentinelPool = new JedisSentinelPool(masterName, sentinelIps);
            jedis = sentinelPool.getResource();

            for (int i = 0; i< 10; i++) {
                jedis.lpush("javaRedisClientList" + jedis.getClient().getHost(), new Integer(i).toString());
            }

        } catch (Exception e) {
            System.out.println(e);
            e.printStackTrace();
        } finally {
            if (jedis != null) {
                jedis.disconnect();
            }
            if (sentinelPool != null) {
                sentinelPool.close();
            }
        }

Jedis sentinel源码分析

• 首先查看JedisSentinelPool的构造方法，最终都会进入如下构造方法

  public JedisSentinelPool(String masterName, Set<String> sentinels,
      final GenericObjectPoolConfig poolConfig, final int connectionTimeout, final int soTimeout,
      final String password, final int database, final String clientName) {
    this.poolConfig = poolConfig;
    this.connectionTimeout = connectionTimeout;
    this.soTimeout = soTimeout;
    this.password = password;
    this.database = database;
    this.clientName = clientName;
    // 初始化sentinel监听列表, 并返回当前master节点
    HostAndPort master = initSentinels(sentinels, masterName);
    // 初始化
    initPool(master);
  }

• initSentinels()方法

private HostAndPort initSentinels(Set<String> sentinels, final String masterName) {

    HostAndPort master = null;
    boolean sentinelAvailable = false;

    log.info("Trying to find master from available Sentinels...");
    // 首先遍历sentinel哨兵节点 
    for (String sentinel : sentinels) {
      final HostAndPort hap = HostAndPort.parseString(sentinel);

      log.debug("Connecting to Sentinel {}", hap);

      Jedis jedis = null;
      try {
        jedis = new Jedis(hap);
        // 从当前哨兵节点获取当前master
        List<String> masterAddr = jedis.sentinelGetMasterAddrByName(masterName);

        // connected to sentinel...
        sentinelAvailable = true;
        
        if (masterAddr == null || masterAddr.size() != 2) {
          log.warn("Can not get master addr, master name: {}. Sentinel: {}", masterName, hap);
          continue;
        }

        master = toHostAndPort(masterAddr);
        log.debug("Found Redis master at {}", master);
        // 找到master,跳出循环
        break;
      } catch (JedisException e) {
        // resolves #1036, it should handle JedisException there's another chance
        // of raising JedisDataException
        log.warn(
          "Cannot get master address from sentinel running @ {}. Reason: {}. Trying next one.", hap,
          e.toString());
      } finally {
        if (jedis != null) {
          jedis.close();
        }
      }
    }

    if (master == null) {
      // 从哨兵列表中不能获取master 
      if (sentinelAvailable) {
        // 哨兵是可用的,则可能是master有问题
        // can connect to sentinel, but master name seems to not
        // monitored
        throw new JedisException("Can connect to sentinel, but " + masterName
            + " seems to be not monitored...");
      } else {
       //  所有哨兵可能都宕机不可用
        throw new JedisConnectionException("All sentinels down, cannot determine where is "
            + masterName + " master is running...");
      }
    }
   
    // 进行到这里，master和sentinel都是可用的
    log.info("Redis master running at " + master + ", starting Sentinel listeners...");

    for (String sentinel : sentinels) {
     // 为每个sentinel启动一个后台线程监听
      final HostAndPort hap = HostAndPort.parseString(sentinel);
      MasterListener masterListener = new MasterListener(masterName, hap.getHost(), hap.getPort());
      // whether MasterListener threads are alive or not, process can be stopped
      masterListener.setDaemon(true);
      masterListeners.add(masterListener);
      masterListener.start();
    }

    return master;
  }

• 内部类 MasterListener

 protected class MasterListener extends Thread {

    protected String masterName;
    protected String host;
    protected int port;
    protected long subscribeRetryWaitTimeMillis = 5000;
    protected volatile Jedis j;
    protected AtomicBoolean running = new AtomicBoolean(false);

    protected MasterListener() {
    }

    public MasterListener(String masterName, String host, int port) {
      super(String.format("MasterListener-%s-[%s:%d]", masterName, host, port));
      this.masterName = masterName;
      this.host = host;
      this.port = port;
    }

    public MasterListener(String masterName, String host, int port,
        long subscribeRetryWaitTimeMillis) {
      this(masterName, host, port);
      this.subscribeRetryWaitTimeMillis = subscribeRetryWaitTimeMillis;
    }

    @Override
    public void run() {

      running.set(true);

      while (running.get()) {

        j = new Jedis(host, port);

        try {
          // double check that it is not being shutdown
          if (!running.get()) {
            break;
          }
          
          /*
           * Added code for active refresh
           */
          // 根据哨兵sentinel连接获取当前master
          List<String> masterAddr = j.sentinelGetMasterAddrByName(masterName);  
          if (masterAddr == null || masterAddr.size() != 2) {
            log.warn("Can not get master addr, master name: {}. Sentinel: {}：{}.",masterName,host,port);
          }else{
              initPool(toHostAndPort(masterAddr)); 
          }
          // 基于redis 频道发布订阅(channel pub/sub)实现的java内部master选举的监听
          j.subscribe(new JedisPubSub() {
            @Override
            public void onMessage(String channel, String message) {
              log.debug("Sentinel {}:{} published: {}.", host, port, message);
              // 订阅的频道发来消息
              String[] switchMasterMsg = message.split(" ");

              if (switchMasterMsg.length > 3) {

                if (masterName.equals(switchMasterMsg[0])) {
                  // 收到的消息 第一节数据 是 sentinel.conf 内配置的 master名称
                  // 根据订阅到的新的master信息, 重新初始化 master
                  initPool(toHostAndPort(Arrays.asList(switchMasterMsg[3], switchMasterMsg[4])));
                } else {
                  log.debug(
                    "Ignoring message on +switch-master for master name {}, our master name is {}",
                    switchMasterMsg[0], masterName);
                }

              } else {
                log.error(
                  "Invalid message received on Sentinel {}:{} on channel +switch-master: {}", host,
                  port, message);
              }
            }
          // 订阅选举频道 :  +switch-master  
          }, "+switch-master");

        } catch (JedisException e) {

          if (running.get()) {
            log.error("Lost connection to Sentinel at {}:{}. Sleeping 5000ms and retrying.", host,
              port, e);
            try {
              Thread.sleep(subscribeRetryWaitTimeMillis);
            } catch (InterruptedException e1) {
              log.error("Sleep interrupted: ", e1);
            }
          } else {
            log.debug("Unsubscribing from Sentinel at {}:{}", host, port);
          }
        } finally {
          j.close();
        }
      }
    }

    public void shutdown() {
      try {
        log.debug("Shutting down listener on {}:{}", host, port);
        running.set(false);
        // This isn't good, the Jedis object is not thread safe
        if (j != null) {
          j.disconnect();
        }
      } catch (Exception e) {
        log.error("Caught exception while shutting down: ", e);
      }
    }
  }

 可见Java Jedis连接Redis集群是基于哨兵集群的监听，首先传入哨兵的地址，根据哨兵信息及哨兵的内部通信得到当前的master连接redis客户端，然后后台为每个哨兵分配线程，线程内基于redis channel pub/sub来设立监听，如果有新的master选举，java内部订阅到消息之后重新对master进行初始化。

Jedis Cluster分片环境连接

JedisCluster cluster = null;

        try {
            Set<HostAndPort> clusterIps = new HashSet();
            clusterIps.add(new HostAndPort("192.168.237.129", 7000));
            clusterIps.add(new HostAndPort("192.168.237.129", 7001));
            cluster = new JedisCluster(clusterIps);

            cluster.set("k1", "v1");
        } catch (Exception e) {
            System.out.println(e.toString());
            e.printStackTrace();
        } finally {
            cluster.close();
        }

Jedis Cluster源码分析

 首先new JedisCluster(clusterIps)构造方法作为入口,会进入BinaryJedisCluster(..)构造方法

  public BinaryJedisCluster(Set<HostAndPort> jedisClusterNode, int timeout, int maxAttempts,
      final GenericObjectPoolConfig poolConfig) {
    this.connectionHandler = new JedisSlotBasedConnectionHandler(jedisClusterNode, poolConfig,
        timeout);
    this.maxAttempts = maxAttempts;
  }

 继续new JedisSlotBasedConnectionHandler(...)构造方法，会进入JedisClusterConnectionHandler(...)构造方法

  public JedisClusterConnectionHandler(Set<HostAndPort> nodes,
          final GenericObjectPoolConfig poolConfig, int connectionTimeout, int soTimeout, String password, String clientName) {
    this.cache = new JedisClusterInfoCache(poolConfig, connectionTimeout, soTimeout, password, clientName);
    initializeSlotsCache(nodes, poolConfig, connectionTimeout, soTimeout, password, clientName);
}

 initializeSlotsCache(..)方法

  for (HostAndPort hostAndPort : startNodes) {
      // 遍历传入的node节点
      Jedis jedis = null;
      try {
        jedis = new Jedis(hostAndPort.getHost(), hostAndPort.getPort(), connectionTimeout, soTimeout);
        if (password != null) {
          jedis.auth(password);
        }
        if (clientName != null) {
          jedis.clientSetname(clientName);
        }
        cache.discoverClusterNodesAndSlots(jedis);
        break;
      } catch (JedisConnectionException e) {
        // try next nodes
      } finally {
        if (jedis != null) {
          jedis.close();
        }
      }
    }

 核心discoverClusterNodesAndSlots(..)方法

  private final Map<String, JedisPool> nodes = new HashMap<String, JedisPool>();
  private final Map<Integer, JedisPool> slots = new HashMap<Integer, JedisPool>();
  private final ReentrantReadWriteLock rwl = new ReentrantReadWriteLock();
  private final Lock r = rwl.readLock();
  private final Lock w = rwl.writeLock();
  ...
 public void discoverClusterNodesAndSlots(Jedis jedis) {
   //重入读写锁的读锁加锁 
    w.lock();

    try {
      reset();
      
      /*
    redis 127.0.0.1:6379> cluster slots
1) 1) (integer) 0
   2) (integer) 4095
   3) 1) "127.0.0.1"
      2) (integer) 7000
   4) 1) "127.0.0.1"
      2) (integer) 7004
2) 1) (integer) 12288
   2) (integer) 16383
   3) 1) "127.0.0.1"
      2) (integer) 7003
   4) 1) "127.0.0.1"
      2) (integer) 7007
      */
      // jedis.clusterSlots(); 内部执行  `cluster slots`命令，返回一个数组，数组每个对象为 [hash slot的起始结束编号, master节点ip+port, slave节点ip+port] 
      List<Object> slots = jedis.clusterSlots();

      for (Object slotInfoObj : slots) {
        List<Object> slotInfo = (List<Object>) slotInfoObj;

        if (slotInfo.size() <= MASTER_NODE_INDEX) {
          continue;
        }
        // 取数组元素的第一二个节点，即hash slot的起始末尾，得出该区间所有槽节点
        List<Integer> slotNums = getAssignedSlotArray(slotInfo);

        // hostInfos
        int size = slotInfo.size();
        for (int i = MASTER_NODE_INDEX; i < size; i++) {
           // 取数组元素的第三及以后的数据，即master+slave的ip.port信息
          List<Object> hostInfos = (List<Object>) slotInfo.get(i);
          if (hostInfos.size() <= 0) {
            continue;
          }
          
          HostAndPort targetNode = generateHostAndPort(hostInfos);
          // 将master/slave节点信息维护于内存nodes 中
          setupNodeIfNotExist(targetNode);
          if (i == MASTER_NODE_INDEX) {
            // 如果是master节点下标位置，将前面得出的所有hash槽节点位置与master节点的关系维护与内存slots中
            assignSlotsToNode(slotNums, targetNode);
          }
        }
      }
    } finally {
      w.unlock();
    }
  }

 'getAssignedSlotArray()'方法

  private List<Integer> getAssignedSlotArray(List<Object> slotInfo) {
    List<Integer> slotNums = new ArrayList<Integer>();
    // 遍历 cluster slots命令返回节点信息; 根据每个节点对应hash 槽的起始开始位置，初始化为该区间内全部槽节点数组
    for (int slot = ((Long) slotInfo.get(0)).intValue(); slot <= ((Long) slotInfo.get(1))
        .intValue(); slot++) {
      slotNums.add(slot);
    }
    return slotNums;
  }

 setupNodeIfNotExist()方法

private final Map<String, JedisPool> nodes = new HashMap<String, JedisPool>();
...
  public JedisPool setupNodeIfNotExist(HostAndPort node) {
    w.lock();
    try {
      String nodeKey = getNodeKey(node);
      JedisPool existingPool = nodes.get(nodeKey);
      if (existingPool != null) return existingPool;
      // 针对内存nodes Map中不存在的节点，初始化该节点，并加入内存中
      JedisPool nodePool = new JedisPool(poolConfig, node.getHost(), node.getPort(),
          connectionTimeout, soTimeout, password, 0, clientName, false, null, null, null);
      nodes.put(nodeKey, nodePool);
      return nodePool;
    } finally {
      w.unlock();
    }
  }

  assignSlotsToNode()方法

  public void assignSlotsToNode(List<Integer> targetSlots, HostAndPort targetNode) {
    w.lock();
    try {
      JedisPool targetPool = setupNodeIfNotExist(targetNode);
      for (Integer slot : targetSlots) {
        //将之前遍历得出的所有hash槽与redis节点绑定,关系维护于map中
        slots.put(slot, targetPool);
      }
    } finally {
      w.unlock();
    }
  }

此时new JedisCluster(clusterIps)构造方法完成之后，已将Cluster环境的所有hash slot(16384个)对应应该存储于的master节点的映射关系维护于内存中，且将不同分区master节点信息以 key为 ip:port形式，value为节点的JedisPool 信息维护于内存nodes中;
 之后执行插入命令cluster.set("k1", "v1");时

  public String set(final String key, final String value) {
    return new JedisClusterCommand<String>(connectionHandler, maxAttempts) {
      @Override
      public String execute(Jedis connection) {
        return connection.set(key, value);
      }
    }.run(key);
  }

 先执行run方法

  public T run(String key) {
    //  JedisClusterCRC16算法, 将key根据 hash slot算法得到值与16383取与，得出对应slot值
    return runWithRetries(JedisClusterCRC16.getSlot(key), this.maxAttempts, false, null);
  }

 runWithRetries(...)内部

private T runWithRetries(final int slot, int attempts, boolean tryRandomNode, JedisRedirectionException redirect) {
    if (attempts <= 0) {
      throw new JedisClusterMaxAttemptsException("No more cluster attempts left.");
    }

    Jedis connection = null;
    try {

      if (redirect != null) {
        connection = this.connectionHandler.getConnectionFromNode(redirect.getTargetNode());
        if (redirect instanceof JedisAskDataException) {
          // TODO: Pipeline asking with the original command to make it faster....
          connection.asking();
        }
      } else {
        if (tryRandomNode) {
          connection = connectionHandler.getConnection();
        } else {
          //因为我们执行set方法,传入的 tryRandomNode为false, redirect为空,会执行此逻辑
          connection = connectionHandler.getConnectionFromSlot(slot);
        }
      }
   ...

  核心逻辑connectionHandler.getConnectionFromSlot(slot)

  public Jedis getConnectionFromSlot(int slot) {
     // 从cache内存中找出hash slot节点对应的redis 数据库连接信息，返回 执行 connection.set(key, value);在对应分片节点插入数据
    JedisPool connectionPool = cache.getSlotPool(slot);
    if (connectionPool != null) {
      // It can't guaranteed to get valid connection because of node
      // assignment
      return connectionPool.getResource();
    } else {
      renewSlotCache(); //It's abnormal situation for cluster mode, that we have just nothing for slot, try to rediscover state
      connectionPool = cache.getSlotPool(slot);
      if (connectionPool != null) {
        return connectionPool.getResource();
      } else {
        //no choice, fallback to new connection to random node
        return getConnection();
      }
    }
  }

总结Jedis连接Cluster原理如下:
 1. 首先JedisCluster构造方法执行cluster slots命令，得到16384个hash slot槽及对应master与slave节点信息;
然后遍历这些数据，将master,slave信息以key为ip:port形式，value为JedisPool的形式维护于内存nodes中；
之后将0 - 18383个hash slot槽对应的JedisPool维护于内存slots中；
 2.之后set方法执行插入数据时，先将key根据CRC16算法得出0-16383内的槽节点，然后从第一步构造方法内维护的关系中找出所要存储数据对应的JedisPool节点，执行对应插入命令。

2 基于Redis实现分布式锁

3 Redis的管道模式

4 Redis应用中的问题

Redis的缓存与数据一致性

 Redis经常被用过当做高速缓存，存放热点数据；一般在服务启动会从DB初始化数据至内存中，然后取数据先从内存中取，取不到再查找数据库；
如果此时执行更改操作，就会涉及到 1 先更新缓存在更新数据库 2先更新数据库再更新缓存的问题；这两个更新步骤不在同一个事物中，也会有数据库更新成功，redis更新不成功这种分布式事物的问题。
更新缓存还是让缓存失效
 1. 当修改数据库数据也需要更新缓存时，判断更新缓存的逻辑或者代价，如果逻辑简单代价很小可以直接更新缓存；
 2. 如果更新缓存需要调用很多接口，代价很大，可以直接让缓存key失效，下次读数据从DB加载，重新放入内存；
先更新数据库还是先更新缓存
 由于更新数据库和更新缓存两个步骤不能保证原子性，会出现临时不一致的情况，不过要保证最终数据一致性；至于先更新哪个可以根据实际业务需求来定；
最终一致性的实现方案
 上面提到的两个步骤，比如我们先更新数据库，执行成功了，然后更新Redis缓存，执行失败了，此时可以将失败信息放入MQ中间件里，让服务消费这些消息，保证最终一致性。

5 布隆过滤器