hadoop 2.6 源码 解读之限流

DataTransferThrottler

hdfs Datanode是 IO密集型应用，网络IO、磁盘IO容易成为系统的瓶颈。
Datanode 有各种任务占用这些共同资源，为保证可用性，有必要对流量进行控制

//a class to throttle the data transfers.
实现限流作用的是类DataTransferThrottler ，其实现也相当简单：
周期period（毫秒）和bytesPerPeriod（每个周期发送的字节数），curReserve保存了当前周期剩余的数据字节数量。方法throttle()如果发送的数据量小于curReserve，立即执行，反之等待下一个周期。

DataTransferThrottler 还有一个子类BlockBalanceThrottler，除了正常的限流功能，还可以限制共享平衡器（BlockBalanceThrottler 对象）的实例（工作任务）的数量

throttle()

看下限流的方法实现

  public synchronized void throttle(long numOfBytes, Canceler canceler) {
    if ( numOfBytes <= 0 ) {
      return;
    }

    curReserve -= numOfBytes;
    bytesAlreadyUsed += numOfBytes;

    while (curReserve <= 0) {
      if (canceler != null && canceler.isCancelled()) {
        return;
      }
      long now = monotonicNow();
      long curPeriodEnd = curPeriodStart + period;

      if ( now < curPeriodEnd ) {
        // Wait for next period so that curReserve can be increased.
        try {
        //等待超时的方式来限流
          wait( curPeriodEnd - now );
        } catch (InterruptedException e) {
          // Abort throttle and reset interrupted status to make sure other
          // interrupt handling higher in the call stack executes.
          Thread.currentThread().interrupt();
          break;
        }
      } else if ( now <  (curPeriodStart + periodExtension)) {
        curPeriodStart = curPeriodEnd;
        curReserve += bytesPerPeriod;
      } else {
        // discard the prev period. Throttler might not have
        // been used for a long time.
        curPeriodStart = now;
        curReserve = bytesPerPeriod - bytesAlreadyUsed;
      }
    }

    bytesAlreadyUsed -= numOfBytes;
  }

使用方法

再来看看调用方
BlockSender类使用到了

  long sendBlock(DataOutputStream out, OutputStream baseStream, 
                 DataTransferThrottler throttler) throws IOException {
                 ...
                 while (endOffset > offset && !Thread.currentThread().isInterrupted()) {
        manageOsCache();
        long len = sendPacket(pktBuf, maxChunksPerPacket, streamForSendChunks,
            transferTo, throttler);
        offset += len;
        totalRead += len + (numberOfChunks(len) * checksumSize);
        seqno++;
      }
                 ...
                 }

进入 sendPacket

  private int sendPacket(ByteBuffer pkt, int maxChunks, OutputStream out,
      boolean transferTo, DataTransferThrottler throttler) throws IOException {

...

//先写后节流
  if (throttler != null) { // rebalancing so throttle
      throttler.throttle(packetLen);
    }

    return dataLen;
}

sendPacket 每发送一次数据调用一次throttler.throttle(),先写数据后限流。