[Serie Netflix Hollow] Profundice en el motor subyacente de lectura y escritura de Hollow

prefacio

En el artículo anterior [Netflix Hollow Series] Análisis en profundidad del modelo de producción y consumo de Hollow, se analizó en detalle el modelo de producción y consumo de Hollow, pero ¿cómo maneja Hollow blobs y flujos de datos específicos?

En vista de los problemas anteriores, este artículo elabora y analiza el código fuente de las funciones básicas de lectura y escritura subyacentes de lectura y escritura de Hollow.

Incluye principalmente cuatro partes:

  1. Cómo la capa inferior de Hollow implementa la escritura de datos
  2. Cómo la capa inferior de Hollow implementa la lectura de datos
  3. Flujo de gotas de Hollow
  4. Implementación de los datos subyacentes de Hollow ByteBuff.

Este artículo irá acompañado de muchos códigos fuente clave. Espero que a través de este artículo, pueda comprender en detalle el flujo de datos subyacente de Hollow y cómo se realizan la producción y el consumo. Ir un paso más allá es poder tener más ideas a la hora de diseñar e implementar tu propio sistema.

@empty canción piedra blanca original.

ModoMemoria

Antes de presentar el flujo de datos, veamos primero la clasificación del modelo de datos en memoria de Hollow.

Hollow divide el modelo de memoria en dos tipos ON_HEAPy SHARED_MEMORY_LAZYreserva la tercera definición SHARED_MEMORY_EAGER, que actualmente no está implementada. La enumeración se define de la siguiente manera:

  1. ON_HEAP : carga en la memoria principal inmediatamente, en el montón de JVM.
  2. SHARED_MEMORY_LAZY : Asignar a la memoria virtual y carga diferida en la memoria principal, fuera del montón
  3. SHARED_MEMORY_EAGER : (futuro) asignado a la memoria virtual y cargado en la memoria principal inmediatamente, dejando el montón

Es una pequeña nota aquí que no todos los modos de memoria son compatibles TypeFilter, actualmente solo son ON_HEAPcompatibles. Sobre lo que es TypeFilter, se introducirá en artículos posteriores.

public enum MemoryMode {
    ON_HEAP,
    SHARED_MEMORY_LAZY;
    // SHARED_MEMORY_EAGER
    /*
     * 返回是否支持某个内存模式
     */
    public boolean consumerSupported() {
        return this.equals(ON_HEAP) || this.equals(SHARED_MEMORY_LAZY);
    }
    /*
     * 返回内存模式是否支持类型Filter
     */
    public boolean supportsFiltering() {
        return this.equals(ON_HEAP);
    }
}

Escribe

首先我们查看HollowProducer的核心方法runCycle的具体实现。runCycle是 Producer 循环执行生成全量和增量逻辑。

核心流程:

  1. prepare
  2. populate
  3. produce new state
  4. publish
  5. checksum
  6. validation
  7. announce
long runCycle(
        ProducerListeners listeners,
        HollowProducer.Incremental.IncrementalPopulator incrementalPopulator, HollowProducer.Populator populator,
        Status.StageWithStateBuilder cycleStatus, long toVersion) {
    // 1. Begin a new cycle
    Artifacts artifacts = new Artifacts();
    HollowWriteStateEngine writeEngine = getWriteEngine();

    try {
        // 1a. Prepare the write state
        writeEngine.prepareForNextCycle();

        // save timestamp in ms of when cycle starts
        writeEngine.addHeaderTag(HollowStateEngine.HEADER_TAG_METRIC_CYCLE_START, String.valueOf(System.currentTimeMillis()));

        // 2. Populate the state
        populate(listeners, incrementalPopulator, populator, toVersion);

        // 3. Produce a new state if there's work to do
        if (writeEngine.hasChangedSinceLastCycle()) {
            boolean schemaChangedFromPriorVersion = readStates.hasCurrent() &&
                    !writeEngine.hasIdenticalSchemas(readStates.current().getStateEngine());
            if (schemaChangedFromPriorVersion) {
                writeEngine.addHeaderTag(HollowStateEngine.HEADER_TAG_SCHEMA_CHANGE, Boolean.TRUE.toString());
            } else {
                writeEngine.getHeaderTags().remove(HollowStateEngine.HEADER_TAG_SCHEMA_CHANGE);
            }

            // 3a. Publish, run checks & validation, then announce new state consumers
            publish(listeners, toVersion, artifacts);

            ReadStateHelper candidate = readStates.roundtrip(toVersion);
            cycleStatus.readState(candidate.pending());
            candidate = doIntegrityCheck ?
                    checkIntegrity(listeners, candidate, artifacts, schemaChangedFromPriorVersion) :
                        noIntegrityCheck(candidate, artifacts);

            try {
                validate(listeners, candidate.pending());

                announce(listeners, candidate.pending());

                readStates = candidate.commit();
                cycleStatus.readState(readStates.current()).success();
            } catch (Throwable th) {
                if (artifacts.hasReverseDelta()) {
                    applyDelta(artifacts.reverseDelta, candidate.pending().getStateEngine());
                    readStates = candidate.rollback();
                }
                throw th;
            }
            lastSuccessfulCycle = toVersion;
        } else {
            // 3b. Nothing to do; reset the effects of Step 2
            // Return the lastSucessfulCycle to the caller thereby
            // the callee can track that version against consumers
            // without having to listen to events.
            // Consistently report the version that would be used if
            // data had been published for the events.  This
            // is for consistency in tracking
            writeEngine.resetToLastPrepareForNextCycle();
            cycleStatus.success();
            listeners.fireNoDelta(toVersion);

            log.info("Populate stage completed with no delta in output state; skipping publish, announce, etc.");
        }
    } catch (Throwable th) {
        writeEngine.resetToLastPrepareForNextCycle();
        cycleStatus.fail(th);

        if (th instanceof RuntimeException) {
            throw (RuntimeException) th;
        }
        throw new RuntimeException(th);
    } finally {
        artifacts.cleanup();
    }
    return lastSuccessfulCycle;
}

populate

Hollow写数据,主要由填充方法populate实现。由firePopulateStartpopulatesuccess三个阶段组成。

void populate(
        ProducerListeners listeners,
        HollowProducer.Incremental.IncrementalPopulator incrementalPopulator, HollowProducer.Populator populator,
        long toVersion) throws Exception {
    assert incrementalPopulator != null ^ populator != null;

    Status.StageBuilder populateStatus = listeners.firePopulateStart(toVersion);
    try {
        if (incrementalPopulator != null) {
            // Incremental population is a sub-stage of the population stage
            // This ensures good integration with existing population listeners if this sub-stage fails
            // then the population stage will fail
            populator = incrementalPopulate(listeners, incrementalPopulator, toVersion);
        }

        try (CloseableWriteState writeState = new CloseableWriteState(toVersion, objectMapper,
                readStates.current())) {
            populator.populate(writeState);
            populateStatus.success();
        }
    } catch (Throwable th) {
        populateStatus.fail(th);
        throw th;
    } finally {
        listeners.firePopulateComplete(populateStatus);
    }
}

HollowTypeMapper

HollowTypeMapper负责数据从特定的类型转换为bytes。HollowTypeMapper作为一个接口,针对不同的数据类型有不同的实现,具体的依赖关系如下图:

Asignador de tipo hueco.png

核心的记录写入方法如下

protected void addTypeState(HollowWriteStateEngine stateEngine) {
    if(stateEngine.getTypeState(getTypeName()) == null)
        stateEngine.addTypeState(getTypeWriteState());
}

protected HollowWriteRecord writeRecord() {
    HollowWriteRecord rec = writeRec.get();
    if(rec == null) {
        rec = newWriteRecord();
        writeRec.set(rec);
    }
    rec.reset();
    return rec;
}

protected ByteDataArray flatRecBuffer() {
    ByteDataArray buf = flatRecBuffer.get();
    if(buf == null) {
        buf = new ByteDataArray();
        flatRecBuffer.set(buf);
    }
    buf.reset();
    return buf;
}

以Object类的实现HollowObjectTypeMapper为例展开介绍下:

@Override
protected HollowWriteRecord newWriteRecord() {
    return new HollowObjectWriteRecord(schema);
}

HollowObjectWriteRecord的构造方法中,依据SMALL_ARRAY_RECYCLER初始化字段数据。

public HollowObjectWriteRecord(HollowObjectSchema schema) {
    this.schema = schema;
    this.fieldData = new ByteDataArray[schema.numFields()];
    this.isNonNull = new boolean[schema.numFields()];
    for (int i = 0; i < fieldData.length; i++) {
        fieldData[i] = new ByteDataArray(WastefulRecycler.SMALL_ARRAY_RECYCLER);
    }
}

当需要添加记录时,需要调用add方法,将Object写入到Mapper。

    /**
     * Adds the specified POJO to the state engine.
     * <p>
     * Unless previously initialized with {@link #initializeTypeState(Class)},
     * the first time an instance of a particular type is added
     * its schema is derived and added to the data model.
     *
     * @param o the POJO to add
     * @return the ordinal assigned to the newly added object
     */
    public int add(Object o) {
        HollowTypeMapper typeMapper = getTypeMapper(o.getClass(), null, null);
        return typeMapper.write(o);
    }

HollowWriteRecord

HollowWriteRecord定义了每一条数据记录写入到Blob的行为,包括writeDataToreset两个方法。继承关系如下图:

HollowWriteRecord.png

package com.netflix.hollow.core.write;

import com.netflix.hollow.core.memory.ByteDataArray;

public interface HollowWriteRecord {

    public void writeDataTo(ByteDataArray buf);

    public void reset();

}

针对不同的数据类型需要对HollowWriteRecord进行特殊的实现。我们以Object的实现HollowObjectWriteRecord说明。

private final HollowObjectSchema schema;

private final ByteDataArray fieldData[];
private final boolean isNonNull[];

public HollowObjectWriteRecord(HollowObjectSchema schema) {
    this.schema = schema;
    this.fieldData = new ByteDataArray[schema.numFields()];
    this.isNonNull = new boolean[schema.numFields()];
    for (int i = 0; i < fieldData.length; i++) {
        fieldData[i] = new ByteDataArray(WastefulRecycler.SMALL_ARRAY_RECYCLER);
    }
}

具体writeDataTo的实现,可以看出Hollow在写数据时,会将Object的每一个Filed根据字段具体数据类型依次写入到ByteBuff中。如果为NULL则会特殊处理。

public void writeDataTo(ByteDataArray buf) {
    for (int i = 0; i < fieldData.length; i++) {
        writeField(buf, i);
    }
}

public void writeDataTo(ByteDataArray buf, HollowObjectSchema translate) {
    for(int i=0; i < translate.numFields(); i++) {
        int fieldIndex = schema.getPosition(translate.getFieldName(i));

        if(fieldIndex != -1) {
            writeField(buf, fieldIndex);
        } else {
            writeNull(buf, translate.getFieldType(i));
        }
    }
}

private void writeField(ByteDataArray buf, int fieldIndex) {
    if (isNonNull[fieldIndex]) {
        if (getSchema().getFieldType(fieldIndex).isVariableLength())
            VarInt.writeVInt(buf, (int)fieldData[fieldIndex].length());
        fieldData[fieldIndex].copyTo(buf);
    } else {
        writeNull(buf, schema.getFieldType(fieldIndex));
    }
}

数据模型可以用以下图形为例,加以理解。有兴趣的朋友也可以查阅之前的文章:# 【Netflix Hollow系列】深入分析Hollow内存布局,可以有更详细的理解。

imagen.png

Read

接下来,将阐述Consumer是如何Read数据,首先让我们看下Read数据相关的核心类的依赖关系。

Sistema de Dependencia del Consumidor Hueco.drawio.png

HollowBlobReader

HollowBlobReader负责填充全量和消费增量数据,也可以称之为更新数据到HollowReadStateEngine。在方法中可以根据具体的MemoryMode选择不同的堆内模型。

private void applySnapshotTransition(HollowConsumer.Blob snapshotBlob,
        HollowConsumer.RefreshListener[] refreshListeners,
        Runnable apiInitCallback) throws Throwable {
    try (HollowBlobInput in = HollowBlobInput.modeBasedSelector(memoryMode, snapshotBlob);
            OptionalBlobPartInput optionalPartIn = snapshotBlob.getOptionalBlobPartInputs()) {
        applyStateEngineTransition(in, optionalPartIn, snapshotBlob, refreshListeners);
        initializeAPI(apiInitCallback);

        for (HollowConsumer.RefreshListener refreshListener : refreshListeners) {
            if (refreshListener instanceof TransitionAwareRefreshListener)
                ((TransitionAwareRefreshListener)refreshListener).snapshotApplied(currentAPI, stateEngine, snapshotBlob.getToVersion());
        }
    } catch (Throwable t) {
        failedTransitionTracker.markFailedTransition(snapshotBlob);
        throw t;
    }
}

private void applyDeltaTransition(HollowConsumer.Blob blob, boolean isSnapshotPlan, HollowConsumer.RefreshListener[] refreshListeners) throws Throwable {
    if (!memoryMode.equals(MemoryMode.ON_HEAP)) {
        LOG.warning("Skipping delta transition in shared-memory mode");
        return;
    }

    try (HollowBlobInput in = HollowBlobInput.modeBasedSelector(memoryMode, blob);
            OptionalBlobPartInput optionalPartIn = blob.getOptionalBlobPartInputs()) {
        applyStateEngineTransition(in, optionalPartIn, blob, refreshListeners);

        if(objLongevityConfig.enableLongLivedObjectSupport()) {
            HollowDataAccess previousDataAccess = currentAPI.getDataAccess();
            HollowHistoricalStateDataAccess priorState = new HollowHistoricalStateCreator(null).createBasedOnNewDelta(currentVersion, stateEngine);
            HollowProxyDataAccess newDataAccess = new HollowProxyDataAccess();
            newDataAccess.setDataAccess(stateEngine);
            currentAPI = apiFactory.createAPI(newDataAccess, currentAPI);

            if(previousDataAccess instanceof HollowProxyDataAccess)
                ((HollowProxyDataAccess)previousDataAccess).setDataAccess(priorState);

            wireHistoricalStateChain(priorState);
        } else {
            if(currentAPI.getDataAccess() != stateEngine)
                currentAPI = apiFactory.createAPI(stateEngine);

            priorHistoricalDataAccess = null;
        }

        if(!staleReferenceDetector.isKnownAPIHandle(currentAPI))
            staleReferenceDetector.newAPIHandle(currentAPI);

        for(HollowConsumer.RefreshListener refreshListener : refreshListeners) {
            if(!isSnapshotPlan)
                refreshListener.deltaUpdateOccurred(currentAPI, stateEngine, blob.getToVersion());
            if (refreshListener instanceof TransitionAwareRefreshListener)
                ((TransitionAwareRefreshListener)refreshListener).deltaApplied(currentAPI, stateEngine, blob.getToVersion());
        }

    } catch (Throwable t) {
        failedTransitionTracker.markFailedTransition(blob);
        throw t;
    }
}

private void applyStateEngineTransition(HollowBlobInput in, OptionalBlobPartInput optionalPartIn, HollowConsumer.Blob transition, HollowConsumer.RefreshListener[] refreshListeners) throws IOException {
    if (transition.isSnapshot()) {
        if (filter == null) {
            reader.readSnapshot(in, optionalPartIn);
        }
        else {
            reader.readSnapshot(in, optionalPartIn, filter);
        }
    } else {
        reader.applyDelta(in, optionalPartIn);
    }

    setVersion(transition.getToVersion());

    for (HollowConsumer.RefreshListener refreshListener : refreshListeners)
        refreshListener.blobLoaded(transition);
}

读取全量数据的方法。

public void readSnapshot(HollowBlobInput in, OptionalBlobPartInput optionalParts, TypeFilter filter) throws IOException {
    validateMemoryMode(in.getMemoryMode());
    Map<String, HollowBlobInput> optionalPartInputs = null;
    if (optionalParts != null)
        optionalPartInputs = optionalParts.getInputsByPartName(in.getMemoryMode());

    HollowBlobHeader header = readHeader(in, false);
    List<HollowBlobOptionalPartHeader> partHeaders = readPartHeaders(header, optionalPartInputs, in.getMemoryMode());
    List<HollowSchema> allSchemas = combineSchemas(header, partHeaders);

    filter = filter.resolve(allSchemas);

    notifyBeginUpdate();

    long startTime = System.currentTimeMillis();

    int numStates = VarInt.readVInt(in);

    Collection<String> typeNames = new TreeSet<>();
    for (int i=0;i<numStates;i++) {
        String typeName = readTypeStateSnapshot(in, filter);
        typeNames.add(typeName);
    }

    if (optionalPartInputs != null) {
        for (Map.Entry<String, HollowBlobInput> optionalPartEntry : optionalPartInputs.entrySet()) {
            numStates = VarInt.readVInt(optionalPartEntry.getValue());

            for (int i=0;i<numStates;i++) {
                String typeName = readTypeStateSnapshot(optionalPartEntry.getValue(), filter);
                typeNames.add(typeName);
            }
        }
    }

    stateEngine.wireTypeStatesToSchemas();

    long endTime = System.currentTimeMillis();

    log.info("SNAPSHOT COMPLETED IN " + (endTime - startTime) + "ms");
    log.info("TYPES: " + typeNames);

    notifyEndUpdate();

    stateEngine.afterInitialization();
}

private String readTypeStateSnapshot(HollowBlobInput in, TypeFilter filter) throws IOException {
    HollowSchema schema = HollowSchema.readFrom(in);
    int numShards = readNumShards(in);
    String typeName = schema.getName();

    if (schema instanceof HollowObjectSchema) {
        if (!filter.includes(typeName)) {
            HollowObjectTypeReadState.discardSnapshot(in, (HollowObjectSchema)schema, numShards);
        } else {
            HollowObjectSchema unfilteredSchema = (HollowObjectSchema)schema;
            HollowObjectSchema filteredSchema = unfilteredSchema.filterSchema(filter);
            populateTypeStateSnapshot(in, new HollowObjectTypeReadState(stateEngine, memoryMode, filteredSchema, unfilteredSchema, numShards));
        }
    } else if (schema instanceof HollowListSchema) {
        if (!filter.includes(typeName)) {
            HollowListTypeReadState.discardSnapshot(in, numShards);
        } else {
            populateTypeStateSnapshot(in, new HollowListTypeReadState(stateEngine, memoryMode, (HollowListSchema)schema, numShards));
        }
    } else if(schema instanceof HollowSetSchema) {
        if (!filter.includes(typeName)) {
            HollowSetTypeReadState.discardSnapshot(in, numShards);
        } else {
            populateTypeStateSnapshot(in, new HollowSetTypeReadState(stateEngine, memoryMode, (HollowSetSchema)schema, numShards));
        }
    } else if(schema instanceof HollowMapSchema) {
        if (!filter.includes(typeName)) {
            HollowMapTypeReadState.discardSnapshot(in, numShards);
        } else {
            populateTypeStateSnapshot(in, new HollowMapTypeReadState(stateEngine, memoryMode, (HollowMapSchema)schema, numShards));
        }
    }
    return typeName;
}

private void populateTypeStateSnapshot(HollowBlobInput in, HollowTypeReadState typeState) throws IOException {
    stateEngine.addTypeState(typeState);
    typeState.readSnapshot(in, stateEngine.getMemoryRecycler());
}

@Override
public void readSnapshot(HollowBlobInput in, ArraySegmentRecycler memoryRecycler) throws IOException {
    if (shards.length > 1)
        maxOrdinal = VarInt.readVInt(in);

    for (int i = 0; i < shards.length; i++) {
        HollowObjectTypeDataElements snapshotData = new HollowObjectTypeDataElements(getSchema(), memoryMode, memoryRecycler);
        snapshotData.readSnapshot(in, unfilteredSchema);
        shards[i].setCurrentData(snapshotData);
    }

    if (shards.length == 1)
        maxOrdinal = shards[0].currentDataElements().maxOrdinal;

    SnapshotPopulatedOrdinalsReader.readOrdinals(in, stateListeners);
}

从数据流HollowBlobInput读取数据,同时指定了当前数据流是否为增量,HollowObjectSchema定义了HollowBlobInput的数据格式。

void readFromInput(HollowBlobInput in, boolean isDelta, HollowObjectSchema unfilteredSchema) throws IOException {
    maxOrdinal = VarInt.readVInt(in);

    if (isDelta) {
        encodedRemovals = GapEncodedVariableLengthIntegerReader.readEncodedDeltaOrdinals(in, memoryRecycler);
        encodedAdditions = GapEncodedVariableLengthIntegerReader.readEncodedDeltaOrdinals(in, memoryRecycler);
    }

    readFieldStatistics(in, unfilteredSchema);

    fixedLengthData = FixedLengthDataFactory.get(in, memoryMode, memoryRecycler);
    removeExcludedFieldsFromFixedLengthData();

    readVarLengthData(in, unfilteredSchema);
}

public static GapEncodedVariableLengthIntegerReader readEncodedDeltaOrdinals(HollowBlobInput in, ArraySegmentRecycler memoryRecycler) throws IOException {
    SegmentedByteArray arr = new SegmentedByteArray(memoryRecycler);
    long numBytesEncodedOrdinals = VarInt.readVLong(in);
    arr.loadFrom(in, numBytesEncodedOrdinals);
    return new GapEncodedVariableLengthIntegerReader(arr, (int)numBytesEncodedOrdinals);
}

@Override
public void loadFrom(HollowBlobInput is, long length) throws IOException {
    int segmentSize = 1 << log2OfSegmentSize;
    int segment = 0;

    byte scratch[] = new byte[segmentSize];

    while (length > 0) {
        ensureCapacity(segment);
        long bytesToCopy = Math.min(segmentSize, length);
        long bytesCopied = 0;
        while (bytesCopied < bytesToCopy) {
            bytesCopied += is.read(scratch, (int)bytesCopied, (int)(bytesToCopy - bytesCopied));
        }
        orderedCopy(scratch, 0, segments[segment++], 0, (int)bytesCopied);
        length -= bytesCopied;
    }
}

private void orderedCopy(byte[] src, int srcPos, byte[] dest, int destPos, int length) {
    int endSrcPos = srcPos + length;
    destPos += Unsafe.ARRAY_BYTE_BASE_OFFSET;

    while(srcPos < endSrcPos) {
        unsafe.putByteVolatile(dest, destPos++, src[srcPos++]);
    }
}

/**
* Reads up to {@code len} bytes of data from the HollowBlobInput by relaying the call to the underlying
* {@code DataInputStream} or {@code RandomAccessFile} into an array of bytes. This method blocks until at
* least one byte of input is available.
*
* @return an integer in the range 0 to 255
* @throws IOException if underlying {@code DataInputStream} or {@code RandomAccessFile}
* @throws UnsupportedOperationException if the input type wasn't  one of {@code DataInputStream} or {@code RandomAccessFile}
*/
public int read(byte b[], int off, int len) throws IOException {
    if (input instanceof RandomAccessFile) {
        return ((RandomAccessFile) input).read(b, off, len);
    } else if (input instanceof DataInputStream) {
        return ((DataInputStream) input).read(b, off, len);
    } else {
        throw new UnsupportedOperationException("Unknown Hollow Blob Input type");
    }
}

HollowTypeReadState

HollowTypeReadState定义了两个抽象方法,readSnapshotapplyDelta。分别负责读取全量和消费增量数据。 HollowTypeReadState.png

public abstract void readSnapshot(HollowBlobInput in, ArraySegmentRecycler recycler) throws IOException;
public abstract void applyDelta(HollowBlobInput in, HollowSchema schema, ArraySegmentRecycler memoryRecycler) throws IOException;

从继承关系图中可以看出,HollowObjectTypeReadStateHollowListTypeReadStateHollowSetTypeReadStateHollowMapTypeReadState,分别实现了HollowTypeReadState。我们以为例查看下子类是如何实现增量和全量的消费的。

@Override
public void readSnapshot(HollowBlobInput in, ArraySegmentRecycler memoryRecycler) throws IOException {
    if(shards.length > 1)
        maxOrdinal = VarInt.readVInt(in);

    for(int i=0;i<shards.length;i++) {
        HollowObjectTypeDataElements snapshotData = new HollowObjectTypeDataElements(getSchema(), memoryMode, memoryRecycler);
        snapshotData.readSnapshot(in, unfilteredSchema);
        shards[i].setCurrentData(snapshotData);
    }

    if(shards.length == 1)
        maxOrdinal = shards[0].currentDataElements().maxOrdinal;

    SnapshotPopulatedOrdinalsReader.readOrdinals(in, stateListeners);
}

@Override
public void applyDelta(HollowBlobInput in, HollowSchema deltaSchema, ArraySegmentRecycler memoryRecycler) throws IOException {
    if(shards.length > 1)
        maxOrdinal = VarInt.readVInt(in);

    for(int i=0;i<shards.length;i++) {
        HollowObjectTypeDataElements deltaData = new HollowObjectTypeDataElements((HollowObjectSchema)deltaSchema, memoryMode, memoryRecycler);
        deltaData.readDelta(in);
        if(stateEngine.isSkipTypeShardUpdateWithNoAdditions() && deltaData.encodedAdditions.isEmpty()) {

            if(!deltaData.encodedRemovals.isEmpty())
                notifyListenerAboutDeltaChanges(deltaData.encodedRemovals, deltaData.encodedAdditions, i, shards.length);

            HollowObjectTypeDataElements currentData = shards[i].currentDataElements();
            GapEncodedVariableLengthIntegerReader oldRemovals = currentData.encodedRemovals == null ? GapEncodedVariableLengthIntegerReader.EMPTY_READER : currentData.encodedRemovals;
            if(oldRemovals.isEmpty()) {
                currentData.encodedRemovals = deltaData.encodedRemovals;
                oldRemovals.destroy();
            } else {
                if(!deltaData.encodedRemovals.isEmpty()) {
                    currentData.encodedRemovals = GapEncodedVariableLengthIntegerReader.combine(oldRemovals, deltaData.encodedRemovals, memoryRecycler);
                    oldRemovals.destroy();
                }
                deltaData.encodedRemovals.destroy();
            }

            deltaData.encodedAdditions.destroy();
        } else {
            HollowObjectTypeDataElements nextData = new HollowObjectTypeDataElements(getSchema(), memoryMode, memoryRecycler);
            HollowObjectTypeDataElements oldData = shards[i].currentDataElements();
            nextData.applyDelta(oldData, deltaData);
            shards[i].setCurrentData(nextData);
            notifyListenerAboutDeltaChanges(deltaData.encodedRemovals, deltaData.encodedAdditions, i, shards.length);
            oldData.destroy();
        }
        deltaData.destroy();
        stateEngine.getMemoryRecycler().swap();
    }

    if(shards.length == 1)
        maxOrdinal = shards[0].currentDataElements().maxOrdinal;
}

针对全量数据,我们继续查看SnapshotPopulatedOrdinalsReader.readOrdinals(in, stateListeners);的实现。

public static void readOrdinals(HollowBlobInput in, HollowTypeStateListener[] listeners) throws IOException {
    int numLongs = in.readInt();

    int currentOrdinal = 0;

    for(int i=0;i<numLongs;i++) {
        long l = in.readLong();
        notifyPopulatedOrdinals(l, currentOrdinal, listeners);
        currentOrdinal += 64;
    }
}

private static void notifyPopulatedOrdinals(long l, int ordinal, HollowTypeStateListener[] listeners) {
    if(l == 0)
        return;

    int stopOrdinal = ordinal + 64;

    while(ordinal < stopOrdinal) {
        long mask = 1L << ordinal;
        if((l & mask) != 0) {
            for(int i=0; i<listeners.length; i++) {
                listeners[i].addedOrdinal(ordinal);
            }
        }
        ordinal++;
    }
}

HollowTypeStateListener

最终可以看到在HollowTypeStateListeneraddedOrdinal方法中实现了数据的读取。HollowTypeStateListener是一个接口。下图展示了接口的实现情况。

HollowTypeStateListener注册到特定的HollowTypeReadState以在将增量应用于HollowReadStateEngine时接收回调通知。

HollowTypeStateListener.png

HollowObjectCacheProvider为例详细看下具体源码。

@Override
public void addedOrdinal(int ordinal) {
    // guard against being detached (or constructed without a HollowTypeReadState)
    if (factory == null)
        return;

    for (int i = cachedItems.size(); i <= ordinal; ++i)
        cachedItems.add(null);
    cachedItems.set(ordinal, instantiateCachedObject(factory, typeReadState, typeAPI, ordinal));
}

private T instantiateCachedObject(HollowFactory<T> factory, HollowTypeDataAccess typeDataAccess, HollowTypeAPI typeAPI, int ordinal) {
    try {
        return factory.newCachedHollowObject(typeDataAccess, typeAPI, ordinal);
    } catch(Throwable th) {
        log.log(Level.SEVERE, "Cached object instantiation failed", th);
        return null;
    }
}

factory.newCachedHollowObject就可以和MomoryMode小节的阐述相关联起来。

/**
 * A HollowFactory is responsible for returning objects in a generated Hollow Object API.  The HollowFactory for individual
 * types can be overridden to return hand-coded implementations of specific record types.
 */
public abstract class HollowFactory<T> {

    public abstract T newHollowObject(HollowTypeDataAccess dataAccess, HollowTypeAPI typeAPI, int ordinal);

    public T newCachedHollowObject(HollowTypeDataAccess dataAccess, HollowTypeAPI typeAPI, int ordinal) {
        return newHollowObject(dataAccess, typeAPI, ordinal);
    }
}

HollowFactory

HollowFactory的实现类继承关系如下图:

FábricaHueca.png

以String的实现为例看下源码。

public class StringHollowFactory extends HollowFactory<HString> {

    @Override
    public HString newHollowObject(HollowTypeDataAccess dataAccess, HollowTypeAPI typeAPI, int ordinal) {
        return new HString(((StringTypeAPI)typeAPI).getDelegateLookupImpl(), ordinal);
    }

    @Override
    public HString newCachedHollowObject(HollowTypeDataAccess dataAccess, HollowTypeAPI typeAPI, int ordinal) {
        return new HString(new StringDelegateCachedImpl((StringTypeAPI)typeAPI, ordinal), ordinal);
    }
}

HollowBlobInput

为支持不同的内存模式以及使用DataInputStreamRandomAccessFile作为Hollow Producer/Consumer Blob的底层资源之间进行传递,Hollow封装了HollowBlobInput。可以认为此类是Hollow数据传输的核心数据流。

Constructor

HollowBlobInput包含了三个属性:MemoryModeinputStreamBlobByteBuffer

private final MemoryMode memoryMode;

private Object input;
private BlobByteBuffer buffer;

private HollowBlobInput(MemoryMode memoryMode) {
    this.memoryMode = memoryMode;
}

public MemoryMode getMemoryMode() {
    return memoryMode;
}

Blob To HollowBlobInput

本小节将分析Hollow是如何将Blob转换为数据流HollowBlobInput的。

modeBasedSelector

modeBasedSelector方法将数据从Blob文件中读取到内存流中,此方法主要作为一个选择器,基于不同的MemoryMode选择randomAccess还是serial方式读取数据流。

/**
* Initialize the Hollow Blob Input object from the Hollow Consumer blob's Input Stream or Random Access File,
* depending on the configured memory mode. The returned HollowBlobInput object must be closed to free up resources.
*
* @param mode Configured memory mode
* @param blob Hollow Consumer blob
* @return the initialized Hollow Blob Input
* @throws IOException if the Hollow Blob Input couldn't be initialized
*/
public static HollowBlobInput modeBasedSelector(MemoryMode mode, HollowConsumer.Blob blob) throws IOException {
    if (mode.equals(ON_HEAP)) {
        return serial(blob.getInputStream());
    } else if (mode.equals(SHARED_MEMORY_LAZY)) {
        return randomAccess(blob.getFile());
    } else {
        throw new UnsupportedOperationException();
    }
}

/**
* Initialize the Hollow Blob Input object from the Hollow Consumer blob's Input Stream or Random Access File,
* depending on the configured memory mode. The returned HollowBlobInput object must be closed to free up resources.
*
* @param mode Configured memory mode
* @param input Hollow Consumer blob
* @param partName the name of the optional part
* @return the initialized Hollow Blob Input
* @throws IOException if the Hollow Blob Input couldn't be initialized
*/
public static HollowBlobInput modeBasedSelector(MemoryMode mode, OptionalBlobPartInput input, String partName) throws IOException {
    if (mode.equals(ON_HEAP)) {
        return serial(input.getInputStream(partName));
    } else if (mode.equals(SHARED_MEMORY_LAZY)) {
        return randomAccess(input.getFile(partName));
    } else {
        throw new UnsupportedOperationException();
    }
}

randomAccess

使用RandomAccessFile的方式读取Blob文件,在BlobByteBuffer.mmapBlob方法中基于JDK的MappedByteBuffer类实现mmap的数据读取。

/**
* Initialize a random access Hollow Blob input object from a file. The returned HollowBlobInput object must be
* closed to free up resources.
*
* @param f file containing the Hollow blob
* @return a random access HollowBlobInput object
* @throws IOException if the mmap operation reported an IOException
*/
public static HollowBlobInput randomAccess(File f) throws IOException {
    return randomAccess(f, MAX_SINGLE_BUFFER_CAPACITY);
}

/**
* Useful for testing with custom buffer capacity
*/
public static HollowBlobInput randomAccess(File f,int singleBufferCapacity) throws IOException {
    HollowBlobInput hbi = new HollowBlobInput(SHARED_MEMORY_LAZY);
    RandomAccessFile raf = new RandomAccessFile(f, "r");
    hbi.input = raf;
    FileChannel channel = ((RandomAccessFile) hbi.input).getChannel();
    hbi.buffer = BlobByteBuffer.mmapBlob(channel, singleBufferCapacity);
    return hbi;
}

serial

serial的实现方式,直接将InputStrem转换为DataInputStream,赋值到HollowBlobInputinput属性中。

/**
* Shorthand for calling {@link HollowBlobInput#serial(InputStream)} on a byte[]
*/
public static HollowBlobInput serial(byte[] bytes) {
    InputStream is = new ByteArrayInputStream(bytes);
    return serial(is);
}

/**
* Initialize a serial access Hollow Blob input object from an input stream. The returned HollowBlobInput object
* must be closed to free up resources.
*
* @param is input stream containing for Hollow blob data
* @return a serial access HollowBlobInput object
*/
public static HollowBlobInput serial(InputStream is) {
    HollowBlobInput hbi = new HollowBlobInput(ON_HEAP);
    hbi.input = new DataInputStream(is);
    return hbi;
}

read stream

HollowBlobInput借助于read开头的方法,加载来自于不同的数据流以及不同的数据类型的数据。借助于RandomAccessFileDataInputStream的底层数据read能力。

/**
* Reads the next byte of data from the input stream by relaying the call to the underlying {@code DataInputStream} or
* {@code RandomAccessFile}. The byte is returned as an integer in the range 0 to 255.
*
* @return an integer in the range 0 to 255
* @throws IOException if underlying {@code DataInputStream} or {@code RandomAccessFile}
* @throws UnsupportedOperationException if the input type wasn't  one of {@code DataInputStream} or {@code RandomAccessFile}
*/
public int read() throws IOException {
    if (input instanceof RandomAccessFile) {
        return ((RandomAccessFile) input).read();
    } else if (input instanceof DataInputStream) {
        return ((DataInputStream) input).read();
    } else {
        throw new UnsupportedOperationException("Unknown Hollow Blob Input type");
    }
}

/**
* Reads up to {@code len} bytes of data from the HollowBlobInput by relaying the call to the underlying
* {@code DataInputStream} or {@code RandomAccessFile} into an array of bytes. This method blocks until at
* least one byte of input is available.
*
* @return an integer in the range 0 to 255
* @throws IOException if underlying {@code DataInputStream} or {@code RandomAccessFile}
* @throws UnsupportedOperationException if the input type wasn't  one of {@code DataInputStream} or {@code RandomAccessFile}
*/
public int read(byte b[], int off, int len) throws IOException {
    if (input instanceof RandomAccessFile) {
        return ((RandomAccessFile) input).read(b, off, len);
    } else if (input instanceof DataInputStream) {
        return ((DataInputStream) input).read(b, off, len);
    } else {
        throw new UnsupportedOperationException("Unknown Hollow Blob Input type");
    }
}

/**
* Sets the file-pointer to the desired offset measured from the beginning of the file by relaying the call to the
* underlying {@code RandomAccessFile}. Operation not supported if the Hollow Blob Input is an {@code DataInputStream}.
*
* @param pos the position in bytes from the beginning of the file at which to set the file pointer to.
* @exception IOException if originated in the underlying {@code RandomAccessFile} implementation
* @exception UnsupportedOperationException if called when Hollow Blob Input is not a {@code RandomAccessFile}
*/
public void seek(long pos) throws IOException {
    if (input instanceof RandomAccessFile) {
        ((RandomAccessFile) input).seek(pos);
    } else if (input instanceof DataInputStream) {
        throw new UnsupportedOperationException("Can not seek on Hollow Blob Input of type DataInputStream");
    } else {
        throw new UnsupportedOperationException("Unknown Hollow Blob Input type");
    }
}
/**
* Reads two bytes from the input (at the current file pointer) into a signed 16-bit short, and advances the offset
* in input.
*
* @return short value read from current offset in input
* @exception IOException if an I/O error occurs.
*/
public final short readShort() throws IOException {
    if (input instanceof RandomAccessFile) {
        return ((RandomAccessFile) input).readShort();
    } else if (input instanceof DataInputStream) {
        return ((DataInputStream) input).readShort();
    } else {
        throw new UnsupportedOperationException("Unknown Hollow Blob Input type");
    }
}

/**
* Reads 4 bytes from the input (at the current file pointer) into a signed 32-bit int, and advances the offset
* in input.
*
* @return int value read from current offset in input
* @exception IOException if an I/O error occurs.
*/
public final int readInt() throws IOException {
    if (input instanceof RandomAccessFile) {
        return ((RandomAccessFile) input).readInt();
    } else if (input instanceof DataInputStream) {
        return ((DataInputStream) input).readInt();
    } else {
        throw new UnsupportedOperationException("Unknown Hollow Blob Input type");
    }
}

/**
* Reads 8 bytes from the input (at the current file pointer) into a signed 64-bit long, and advances the offset
* in input.
*
* @return long value read from current offset in input
* @exception IOException if an I/O error occurs.
*/
public final long readLong() throws IOException {
    if (input instanceof RandomAccessFile) {
        return ((RandomAccessFile) input).readLong();
    } else if (input instanceof DataInputStream) {
        return ((DataInputStream) input).readLong();
    } else {
        throw new UnsupportedOperationException("Unknown Hollow Blob Input type");
    }
}

/**
* Reads in a string from this file, encoded using <a href="DataInput.html#modified-utf-8">modified UTF-8</a>
* format, and advances the offset in input.
* @return UTF-8 string read from current offset in input
* @exception IOException if an I/O error occurs.
*/
public final String readUTF() throws IOException {
    if (input instanceof RandomAccessFile) {
        return ((RandomAccessFile) input).readUTF();
    } else if (input instanceof DataInputStream) {
        return ((DataInputStream) input).readUTF();
    } else {
        throw new UnsupportedOperationException("Unknown Hollow Blob Input type");
    }
}

close

close负责关闭数据流。

/**
* Closes underlying InputStream/RandomAccessFile and releases any system resources associated with the Hollow Blob Input.
* @throws IOException
*/
@Override
public void close() throws IOException {
    if (input instanceof RandomAccessFile) {
        ((RandomAccessFile) input).close();
    } else if (input instanceof DataInputStream) {
        ((DataInputStream) input).close();
    } else {
        throw new UnsupportedOperationException("Unknown Hollow Blob Input type");
    }
}

BlobByteBuffer

HollowBlobInput中可以看到BlobByteBuffer这个属性。本小节着重探讨下BlobByteBuffer的实现。

BlobByteBufferResponsable de todo el almacenamiento de datos en memoria, implementa un puente entre BlobByteBufferarchivos grandes como BLOB y se limita a la asignación de memoria de tamaño entero.MappedByteBufferMappedByteBuffer

Tenga en cuenta: JDK 14 introducirá API mejoradas para acceder a la memoria externa y reemplazar MappedByteBuffer.

BlobByteBufferNo es seguro para subprocesos, pero es seguro compartir el búfer de bytes subyacente para lecturas paralelas. El tamaño máximo admitido blobes de aproximadamente 2 elevado a 16. Se pueden alcanzar otros límites en Hollow o límites reales antes de encontrar este límite.

campos

BlobByteBufferLos datos finalmente se almacenan en spineuna ByteBuffermatriz denominada .

public static final int MAX_SINGLE_BUFFER_CAPACITY = 1 << 30;   // largest, positive power-of-two int

private final ByteBuffer[] spine;   // array of MappedByteBuffers
private final long capacity;        // in bytes
private final int shift;
private final int mask;

private long position;              // within index 0 to capacity-1 in the underlying ByteBuffer

mmapBlob

BlobByteBufferEl mmapBlobmétodo implementa la lectura de datos de Archivo a través de mmap.

/**
* mmap the entire contents of FileChannel into an array of {@code MappedByteBuffer}s, each of size singleBufferCapacity.
* @param channel FileChannel for file to be mmap-ed
* @param singleBufferCapacity Size of individual MappedByteBuffers in array of {@code MappedByteBuffer}s required
*                to map the entire file channel. It must be a power of 2, and due to {@code MappedByteBuffer}
*                constraints it is limited to the max integer that is a power of 2.
* @return BlobByteBuffer containing an array of {@code MappedByteBuffer}s that mmap-ed the entire file channel
* @throws IOException
*/
public static BlobByteBuffer mmapBlob(FileChannel channel, int singleBufferCapacity) throws IOException {
    long size = channel.size();
    if (size == 0) {
        throw new IllegalStateException("File to be mmap-ed has no data");
    }
    if ((singleBufferCapacity & (singleBufferCapacity - 1)) != 0) { // should be a power of 2
        throw new IllegalArgumentException("singleBufferCapacity must be a power of 2");
    }

    // divide into N buffers with an int capacity that is a power of 2
    final int bufferCapacity = size > (long) singleBufferCapacity
            ? singleBufferCapacity
            : Integer.highestOneBit((int) size);
    long bufferCount = size % bufferCapacity == 0
            ? size / (long)bufferCapacity
            : (size / (long)bufferCapacity) + 1;
    if (bufferCount > Integer.MAX_VALUE)
        throw new IllegalArgumentException("file too large; size=" + size);

    int shift = 31 - Integer.numberOfLeadingZeros(bufferCapacity); // log2
    int mask = (1 << shift) - 1;
    ByteBuffer[] spine = new MappedByteBuffer[(int)bufferCount];
    for (int i = 0; i < bufferCount; i++) {
        long pos = (long)i * bufferCapacity;
        int cap = i == (bufferCount - 1)
                ? (int)(size - pos)
                : bufferCapacity;
        ByteBuffer buffer = channel.map(READ_ONLY, pos, cap);
        /*
            * if (!((MappedByteBuffer) buffer).isLoaded()) // TODO(timt): make pre-fetching configurable
            *    ((MappedByteBuffer) buffer).load();
            */
        spine[i] = buffer;
    }

    return new BlobByteBuffer(size, shift, mask, spine);
}

Resumir

A partir del análisis detallado del código fuente, este artículo analiza en profundidad cómo Hollow realiza la lectura y escritura de registros de datos. Finalmente, hablé sobre la implementación de ByteBuff por parte de Hollow. Creo que no todo el mundo usa ByteBuff por defecto. Casi todos los frameworks lo usarán, lo que demuestra su importancia.

Observaciones finales

imagen.png

La lectura en profundidad del código fuente puede brindarle una comprensión más profunda de la arquitectura de diseño del sistema, no solo en el nivel de uso.

No lo dudes, no lo dudes, leamos juntos el código fuente.

Supongo que te gusta

Origin juejin.im/post/7122826206679924767
Recomendado
Clasificación