netty resource leak detector: http://donald-draper.iteye.com/blog/2393940
netty abstract byte buf parsing: http ://donald-draper.iteye.com/blog/2394078
netty abstract byte buf reference counter: http://donald-draper.iteye.com/blog/2394109
netty composite buf concept: http://donald-draper. iteye.com/blog/2394408
Introduction
In the last article, we looked at the concept of composite buf, let's review it first:
composite byte buffer CompositeByteBuf, there is a byte buf array inside, used to store byte buf, each byte When buf is added to the composite buf set, it will be packaged into a buf component. If the buf is added and the composite buf set is full, then all bufs in the buf set will be integrated into a component buf, and the original buf set will be emptied. Add The integrated buf into the buf set. The read-write index of the composite buf is the start index and size of the byte buf set; each component buf Component internally records the start position and end position of the byte buf in the composite buf, and the readable data length of the buf.
Default channel configuration initialization in netty ( http://donald-draper.iteye.com/blog/2393504 )
In this article, we see that the byte buf is allocated by the byte allocator. The default byte allocator is defined in ByteBufUtil. Let's go back to the definition of the
channel configuration byte buf allocator:
//byte buf allocator
private volatile ByteBufAllocator allocator = ByteBufAllocator.DEFAULT;
public final class ByteBufUtil {
static final ByteBufAllocator DEFAULT_ALLOCATOR;//Default byte allocator
static {
String allocType = SystemPropertyUtil.get(
"io.netty.allocator.type", PlatformDependent.isAndroid() ? "unpooled" : "pooled");
allocType = allocType.toLowerCase(Locale.US).trim();
ByteBufAllocator alloc;
if ("unpooled".equals(allocType)) {
alloc = UnpooledByteBufAllocator.DEFAULT;
logger.debug("-Dio.netty.allocator.type: {}", allocType);
} else if ("pooled".equals(allocType)) {
alloc = PooledByteBufAllocator.DEFAULT;
logger.debug("-Dio.netty.allocator.type: {}", allocType);
} else {
alloc = PooledByteBufAllocator.DEFAULT;
logger.debug("-Dio.netty.allocator.type: pooled (unknown: {})", allocType);
}
DEFAULT_ALLOCATOR = alloc;
THREAD_LOCAL_BUFFER_SIZE = SystemPropertyUtil.getInt("io.netty.threadLocalDirectBufferSize", 64 * 1024);
logger.debug("-Dio.netty.threadLocalDirectBufferSize: {}", THREAD_LOCAL_BUFFER_SIZE);
MAX_CHAR_BUFFER_SIZE = SystemPropertyUtil.getInt("io.netty.maxThreadLocalCharBufferSize", 16 * 1024);
logger.debug("-Dio.netty.maxThreadLocalCharBufferSize: {}", MAX_CHAR_BUFFER_SIZE);
}
}
From the byte buf tool class, it can be seen that if the system property io.netty.allocator.type is configured as unpooled,
the default byte buf allocator is UnpooledByteBufAllocator, otherwise it is PooledByteBufAllocator.
For the Android platform, the default is UnpooledByteBufAllocator.
//UnpooledByteBufAllocator
public final class UnpooledByteBufAllocator extends AbstractByteBufAllocator implements ByteBufAllocatorMetricProvider {
private final UnpooledByteBufAllocatorMetric metric = new UnpooledByteBufAllocatorMetric();
private final boolean disableLeakDetector;
/**
* Default instance which uses leak-detection for direct buffers.
*/
public static final UnpooledByteBufAllocator DEFAULT =
new UnpooledByteBufAllocator(PlatformDependent.directBufferPreferred());
...
}
//PooledByteBufAllocator
public class PooledByteBufAllocator extends AbstractByteBufAllocator implements ByteBufAllocatorMetricProvider {
public static final PooledByteBufAllocator DEFAULT =
new PooledByteBufAllocator(PlatformDependent.directBufferPreferred());
...
}
Since the above two methods inherit AbstractByteBufAllocator, let's take a look at the abstract byte buf allocator first.
/**
* Skeletal {@link ByteBufAllocator} implementation to extend.
*/
public abstract class AbstractByteBufAllocator implements ByteBufAllocator {
static final int DEFAULT_INITIAL_CAPACITY = 256;//Initialize capacity
static final int DEFAULT_MAX_CAPACITY = Integer.MAX_VALUE;//最大容量
static final int DEFAULT_MAX_COMPONENTS = 16;//The maximum number of composite buf component buf
static final int CALCULATE_THRESHOLD = 1048576 * 4; // 4 MiB page capacity adjustment threshold
private final boolean directByDefault;//Whether the allocated buf is of direct type
private final ByteBuf emptyBuf;
/**
* Instance use heap buffers by default
*/
protected AbstractByteBufAllocator() {
this(false);
}
/**
* Create new instance
*
* @param preferDirect {@code true} if {@link #buffer(int)} should try to allocate a direct buffer rather than
* a heap buffer
*/
protected AbstractByteBufAllocator(boolean preferDirect) {
directByDefault = preferDirect && PlatformDependent.hasUnsafe();
emptyBuf = new EmptyByteBuf(this);
}
}
Let's look at creating a byte buf:
@Override
public ByteBuf buffer() {
if (directByDefault) {
return directBuffer();
}
return heapBuffer();
}
@Override
public ByteBuf buffer(int initialCapacity) {
if (directByDefault) {
return directBuffer(initialCapacity);
}
return heapBuffer(initialCapacity);
}
@Override
public ByteBuf buffer(int initialCapacity, int maxCapacity) {
if (directByDefault) {
return directBuffer(initialCapacity, maxCapacity);
}
return heapBuffer(initialCapacity, maxCapacity);
}
Creating a byte buf is mainly based on the directByDefault attribute of the byte buf allocator to determine whether the allocated buf is a direct type or a heap type.
Let's first look at the allocation of direct type buf
@Override
public ByteBuf directBuffer() {
return directBuffer(DEFAULT_INITIAL_CAPACITY, DEFAULT_MAX_CAPACITY);
}
@Override
public ByteBuf directBuffer(int initialCapacity) {
return directBuffer(initialCapacity, DEFAULT_MAX_CAPACITY);
}
@Override
public ByteBuf directBuffer(int initialCapacity, int maxCapacity) {
if (initialCapacity == 0 && maxCapacity == 0) {
return emptyBuf;
}
validate(initialCapacity, maxCapacity);
return newDirectBuffer(initialCapacity, maxCapacity);
}
/**
* Create a direct {@link ByteBuf} with the given initialCapacity and maxCapacity.
To be extended by subclasses
*/
protected abstract ByteBuf newDirectBuffer(int initialCapacity, int maxCapacity);
Let's look at the allocation of heap type buf:
@Override
public ByteBuf heapBuffer() {
return heapBuffer(DEFAULT_INITIAL_CAPACITY, DEFAULT_MAX_CAPACITY);
}
@Override
public ByteBuf heapBuffer(int initialCapacity) {
return heapBuffer(initialCapacity, DEFAULT_MAX_CAPACITY);
}
@Override
public ByteBuf heapBuffer(int initialCapacity, int maxCapacity) {
if (initialCapacity == 0 && maxCapacity == 0) {
return emptyBuf;
}
validate(initialCapacity, maxCapacity);
return newHeapBuffer(initialCapacity, maxCapacity);
}
/**
* Create a heap {@link ByteBuf} with the given initialCapacity and maxCapacity.
To be extended by subclasses
*/
protected abstract ByteBuf newHeapBuffer(int initialCapacity, int maxCapacity);
It can be seen from the above that the creation of direct and heap buf actually passes the newDirectBuffer and newHeapBuffer methods, which need to be extended by subclasses.
Let's create ioBufer
@Override
public ByteBuf ioBuffer() {
if (PlatformDependent.hasUnsafe()) {
return directBuffer(DEFAULT_INITIAL_CAPACITY);
}
return heapBuffer(DEFAULT_INITIAL_CAPACITY);
}
@Override
public ByteBuf ioBuffer(int initialCapacity) {
if (PlatformDependent.hasUnsafe()) {
return directBuffer(initialCapacity);
}
return heapBuffer(initialCapacity);
}
@Override
public ByteBuf ioBuffer(int initialCapacity, int maxCapacity) {
if (PlatformDependent.hasUnsafe()) {
return directBuffer(initialCapacity, maxCapacity);
}
return heapBuffer(initialCapacity, maxCapacity);
}
It can be seen from the above that the byte buf created by the ioBuffer method is the direct type first. When the system platform does not support Unsafe, it is the heap type
. Let's look at creating a composite buf:
@Override
public CompositeByteBuf compositeBuffer() {
if (directByDefault) {
return compositeDirectBuffer();
}
return compositeHeapBuffer();
}
@Override
public CompositeByteBuf compositeBuffer(int maxNumComponents) {
if (directByDefault) {
return compositeDirectBuffer(maxNumComponents);
}
return compositeHeapBuffer(maxNumComponents);
}
It can be seen from the above that creating a composite buf is mainly based on the directByDefault attribute of the byte buf allocator to determine whether the allocated buf is a direct type or a heap type; let's
look at creating a heap type composite buf
@Override
public CompositeByteBuf compositeHeapBuffer() {
return compositeHeapBuffer(DEFAULT_MAX_COMPONENTS);
}
@Override
public CompositeByteBuf compositeHeapBuffer(int maxNumComponents) {
return toLeakAwareBuffer(new CompositeByteBuf(this, false, maxNumComponents));
}
// track compound buf memory leaks
protected static CompositeByteBuf toLeakAwareBuffer(CompositeByteBuf buf) {
ResourceLeakTracker<ByteBuf> leak;
switch (ResourceLeakDetector.getLevel()) {
case SIMPLE:
leak = AbstractByteBuf.leakDetector.track(buf);
if (leak != null) {
buf = new SimpleLeakAwareCompositeByteBuf(buf, leak);
}
break;
case ADVANCED:
case PARANOID:
leak = AbstractByteBuf.leakDetector.track(buf);
if (leak != null) {
buf = new AdvancedLeakAwareCompositeByteBuf(buf, leak);
}
break;
default:
break;
}
return buf;
}
//SimpleLeakAwareCompositeByteBuf
class SimpleLeakAwareCompositeByteBuf extends WrappedCompositeByteBuf {
final ResourceLeakTracker<ByteBuf> leak;//Internal resource leak detector
SimpleLeakAwareCompositeByteBuf(CompositeByteBuf wrapped, ResourceLeakTracker<ByteBuf> leak) {
super(wrapped);
this.leak = ObjectUtil.checkNotNull(leak, "leak");
}
...
}
// trace bytes buf memory leak
protected static ByteBuf toLeakAwareBuffer(ByteBuf buf) {
ResourceLeakTracker<ByteBuf> leak;
switch (ResourceLeakDetector.getLevel()) {
case SIMPLE:
leak = AbstractByteBuf.leakDetector.track(buf);
if (leak != null) {
buf = new SimpleLeakAwareByteBuf(buf, leak);
}
break;
case ADVANCED:
case PARANOID:
leak = AbstractByteBuf.leakDetector.track(buf);
if (leak != null) {
buf = new AdvancedLeakAwareByteBuf(buf, leak);
}
break;
default:
break;
}
return buf;
}
Let's look at creating a direct type composite buf
@Override
public CompositeByteBuf compositeDirectBuffer() {
return compositeDirectBuffer(DEFAULT_MAX_COMPONENTS);
}
@Override
public CompositeByteBuf compositeDirectBuffer(int maxNumComponents) {
return toLeakAwareBuffer(new CompositeByteBuf(this, true, maxNumComponents));
}
It can be seen from the above that when creating a composite buf, if the resource leak detection function is enabled, the memory leak of the composite buf is tracked.
Let's look again at calculating the new capacity
@Override
public int calculateNewCapacity(int minNewCapacity, int maxCapacity) {
if (minNewCapacity < 0) {
throw new IllegalArgumentException("minNewCapacity: " + minNewCapacity + " (expected: 0+)");
}
if (minNewCapacity > maxCapacity) {
throw new IllegalArgumentException(String.format(
"minNewCapacity: %d (expected: not greater than maxCapacity(%d)",
minNewCapacity, maxCapacity));
}
final int threshold = CALCULATE_THRESHOLD; // 4 MiB page
if (minNewCapacity == threshold) {
return threshold;
}
// If over threshold, do not double but just increase by threshold.
//If the new capacity is greater than the threshold,
if (minNewCapacity > threshold) {
int newCapacity = minNewCapacity / threshold * threshold;
if (newCapacity > maxCapacity - threshold) {//New capacity + threshold is greater than maximum capacity
newCapacity = maxCapacity;//The capacity is the maximum capacity
} else {
newCapacity += threshold;//Otherwise the increment is the threshold
}
return newCapacity;
}
// Not over threshold. Double up to 4 MiB, starting from 64.
//Otherwise start from the capacity 64 and double each time until it is greater than the minimum new capacity
int newCapacity = 64;
while (newCapacity < minNewCapacity) {
newCapacity <<= 1;
}
return Math.min(newCapacity, maxCapacity);
}
Let's look at the byte buf allocator metric provider ByteBufAllocatorMetricProvider
public interface ByteBufAllocatorMetricProvider {
/**
* Returns a {@link ByteBufAllocatorMetric} for a {@link ByteBufAllocator}.
Returns a byte buf allocator metric
*/
ByteBufAllocatorMetric metric();
}
public interface ByteBufAllocatorMetric {
/**
* Returns the number of bytes of heap memory used by a {@link ByteBufAllocator} or {@code -1} if unknown.
Returns the heap memory used by byte buf allocation
*/
long usedHeapMemory();
/**
* Returns the number of bytes of direct memory used by a {@link ByteBufAllocator} or {@code -1} if unknown.
Returns the direct memory used by byte buf allocation
*/
long usedDirectMemory();
}
Summary:
The creation of byte buf is mainly based on the directByDefault attribute of the byte buf allocator to determine whether the allocated buf is of direct type or heap type; the creation of direct and heap buf actually passes the newDirectBuffer and newHeapBuffer methods to be extended by subclasses. It can be seen that the byte buf created by the ioBuffer method is of the direct type first. When the system platform does not support Unsafe, it is of the heap type. The creation of a composite buf is mainly based on the directByDefault attribute of the byte buf allocator to determine whether the allocated buf is a direct type. It is still a heap type; when creating a composite buf, if the resource leak detection function is enabled, the memory leak of the composite buf is tracked.