hadoop 的lease(一)

本文链接： https://blog.csdn.net/qq_36438618/article/details/102764744

“租约” ，何为租约，一个租约管理着一个client的所有锁，对于每个client 都有一个相应的租约，但一个client 定期 check in 时租约的时间戳就会更新，当一个client 停止的时候并且允许租约到期，则对应的所有锁都会释放。
大家来看看租约这个类：

class Lease implements Comparable<Lease> {
    // 租约的持有者，就是客户端名称 
    private final String holder;
   //最后一次检入时间
    private long lastUpdate;
    //持有的文件路径集
    private final Collection<String> paths = new TreeSet<String>();
  }

租约管理的超时限制：

软超时默认 60000ms=1min 当对一个文件进行创建或者append的时候会判断，配置：hdfs.regionserver.lease.period
硬超时默认时 60*60000ms=1h ,namenode 会监控硬链接是否超时，若超时会恢复租约

租约恢复算法

Namenode检索租赁信息
对于租约中的每个文件f，使用f的最后一个块b
- 获取包含b的datanode
- 指定其中一个datanode 作为主要的datanode 并命名其为p
- p 从namenode 获取一个新生成的时间戳
- p 从每个datanode 获取块信息
- p 计算最小块的长度 length
- p 使用新生成的时间戳更新和最小块长度更新具有有效的生成时间戳的datanode
- p 让namenode 更新信息
- namenode 更新块信息
- Namenode在删除所有文件后从租约中删除f并删除租约
  下面的这个方法会在客户端create或者append的时候执行该方法：

private void recoverLeaseInternal(INodeFile fileInode, 
      String src, String holder, String clientMachine, boolean force)
      throws IOException {
    assert hasWriteLock();
    if (fileInode != null && fileInode.isUnderConstruction()) {
      INodeFileUnderConstruction pendingFile = (INodeFileUnderConstruction) fileInode;
      /** 如果文件正在构建中，那么它必须在我们的租约中。找到合适的租赁记录。*/
      Lease lease = leaseManager.getLease(holder);
      //通过文件地址获取租约，若发现原创建该文件，会报异常
      if (!force && lease != null) {
        Lease leaseFile = leaseManager.getLeaseByPath(src);
        if ((leaseFile != null && leaseFile.equals(lease))) { 
          throw new AlreadyBeingCreatedException(
            "failed to create file " + src + " for " + holder +
            " on client " + clientMachine + 
            " because current leaseholder is trying to recreate file.");
        }
      }
      //
      // 获取原始的持有者      
      lease = leaseManager.getLease(pendingFile.getClientName());
      if (lease == null) {
        throw new AlreadyBeingCreatedException(
          "failed to create file " + src + " for " + holder +
          " on client " + clientMachine + 
          " because pendingCreates is non-null but no leases found.");
      }
      if (force) {
        //如果是强制的
        // 直接关闭：不需要 等待软超时过期
        // 只关闭当前的文件
        LOG.info("recoverLease: " + lease + ", src=" + src +
          " from client " + pendingFile.getClientName());
        internalReleaseLease(lease, src, holder);
      } else {
        assert lease.getHolder().equals(pendingFile.getClientName()) :
          "Current lease holder " + lease.getHolder() +
          " does not match file creator " + pendingFile.getClientName();
        //如果软超时已经超时 则尝试恢复租约
        if (lease.expiredSoftLimit()) {
          LOG.info("startFile: recover " + lease + ", src=" + src + " client "
              + pendingFile.getClientName());
          boolean isClosed = internalReleaseLease(lease, src, null);
          if(!isClosed)
            throw new RecoveryInProgressException(
                "Failed to close file " + src +
                ". Lease recovery is in progress. Try again later.");
        } else {
          final BlockInfo lastBlock = pendingFile.getLastBlock();
          if (lastBlock != null
              && lastBlock.getBlockUCState() == BlockUCState.UNDER_RECOVERY) {
            throw new RecoveryInProgressException("Recovery in progress, file ["
                + src + "], " + "lease owner [" + lease.getHolder() + "]");
          } else {
            throw new AlreadyBeingCreatedException("Failed to create file ["
                + src + "] for [" + holder + "] on client [" + clientMachine
                + "], because this file is already being created by ["
                + pendingFile.getClientName() + "] on ["
                + pendingFile.getClientMachine() + "]");
          }
        }
      }
    }
  }

租约恢复算法

猜你喜欢