Modifier and Type | Method and Description |
---|---|
abstract void |
TypeComparator.putNormalizedKey(T record,
MemorySegment target,
int offset,
int numBytes)
Writes a normalized key for the given record into the target byte array, starting at the specified position
an writing exactly the given number of bytes.
|
Modifier and Type | Method and Description |
---|---|
void |
BooleanComparator.putNormalizedKey(Boolean value,
MemorySegment target,
int offset,
int numBytes) |
void |
ByteComparator.putNormalizedKey(Byte value,
MemorySegment target,
int offset,
int numBytes) |
void |
CharComparator.putNormalizedKey(Character value,
MemorySegment target,
int offset,
int numBytes) |
void |
DateComparator.putNormalizedKey(Date lValue,
MemorySegment target,
int offset,
int numBytes) |
void |
DoubleComparator.putNormalizedKey(Double value,
MemorySegment target,
int offset,
int numBytes) |
void |
FloatComparator.putNormalizedKey(Float value,
MemorySegment target,
int offset,
int numBytes) |
void |
IntComparator.putNormalizedKey(Integer iValue,
MemorySegment target,
int offset,
int numBytes) |
void |
LongComparator.putNormalizedKey(Long lValue,
MemorySegment target,
int offset,
int numBytes) |
void |
ShortComparator.putNormalizedKey(Short value,
MemorySegment target,
int offset,
int numBytes) |
void |
StringComparator.putNormalizedKey(String record,
MemorySegment target,
int offset,
int len) |
void |
EnumComparator.putNormalizedKey(T iValue,
MemorySegment target,
int offset,
int numBytes) |
Modifier and Type | Method and Description |
---|---|
void |
PrimitiveArrayComparator.putNormalizedKey(T record,
MemorySegment target,
int offset,
int numBytes) |
Modifier and Type | Method and Description |
---|---|
void |
WritableComparator.putNormalizedKey(T record,
MemorySegment target,
int offset,
int numBytes) |
void |
ValueComparator.putNormalizedKey(T record,
MemorySegment target,
int offset,
int numBytes) |
void |
TupleComparator.putNormalizedKey(T value,
MemorySegment target,
int offset,
int numBytes) |
void |
PojoComparator.putNormalizedKey(T value,
MemorySegment target,
int offset,
int numBytes) |
void |
GenericTypeComparator.putNormalizedKey(T record,
MemorySegment target,
int offset,
int numBytes) |
void |
CopyableValueComparator.putNormalizedKey(T record,
MemorySegment target,
int offset,
int numBytes) |
Modifier and Type | Method and Description |
---|---|
void |
EnumValueComparator.putNormalizedKey(scala.Enumeration.Value v,
MemorySegment target,
int offset,
int numBytes) |
void |
CaseClassComparator.putNormalizedKey(T value,
MemorySegment target,
int offsetParam,
int numBytesParam) |
Modifier and Type | Method and Description |
---|---|
void |
RowComparator.putNormalizedKey(Row record,
MemorySegment target,
int offset,
int numBytes) |
void |
NullAwareComparator.putNormalizedKey(T record,
MemorySegment target,
int offset,
int numBytes) |
Modifier and Type | Class and Description |
---|---|
class |
HeapMemorySegment
This class represents a piece of heap memory managed by Flink.
|
class |
HybridMemorySegment
This class represents a piece of memory managed by Flink.
|
Modifier and Type | Method and Description |
---|---|
static MemorySegment |
MemorySegmentFactory.allocateUnpooledSegment(int size)
Allocates some unpooled memory and creates a new memory segment that represents
that memory.
|
static MemorySegment |
MemorySegmentFactory.allocateUnpooledSegment(int size,
Object owner)
Allocates some unpooled memory and creates a new memory segment that represents
that memory.
|
MemorySegment |
MemorySegmentFactory.Factory.allocateUnpooledSegment(int size,
Object owner)
Allocates some unpooled memory and creates a new memory segment that represents
that memory.
|
MemorySegment |
MemorySegmentSource.nextSegment()
Gets the next memory segment.
|
static MemorySegment |
MemorySegmentFactory.wrap(byte[] buffer)
Creates a new memory segment that targets the given heap memory region.
|
MemorySegment |
MemorySegmentFactory.Factory.wrap(byte[] memory)
Creates a new memory segment that targets the given heap memory region.
|
static MemorySegment |
MemorySegmentFactory.wrapPooledHeapMemory(byte[] memory,
Object owner)
Creates a memory segment that wraps the given byte array.
|
MemorySegment |
MemorySegmentFactory.Factory.wrapPooledHeapMemory(byte[] memory,
Object owner)
Creates a memory segment that wraps the given byte array.
|
static MemorySegment |
MemorySegmentFactory.wrapPooledOffHeapMemory(ByteBuffer memory,
Object owner)
Creates a memory segment that wraps the off-heap memory backing the given ByteBuffer.
|
MemorySegment |
MemorySegmentFactory.Factory.wrapPooledOffHeapMemory(ByteBuffer memory,
Object owner)
Creates a memory segment that wraps the off-heap memory backing the given ByteBuffer.
|
Modifier and Type | Method and Description |
---|---|
int |
MemorySegment.compare(MemorySegment seg2,
int offset1,
int offset2,
int len)
Compares two memory segment regions.
|
void |
MemorySegment.copyTo(int offset,
MemorySegment target,
int targetOffset,
int numBytes)
Bulk copy method.
|
void |
MemorySegment.swapBytes(byte[] tempBuffer,
MemorySegment seg2,
int offset1,
int offset2,
int len)
Swaps bytes between two memory segments, using the given auxiliary buffer.
|
Modifier and Type | Method and Description |
---|---|
protected MemorySegment |
SeekableFileChannelInputView.nextSegment(MemorySegment current) |
protected MemorySegment |
RandomAccessInputView.nextSegment(MemorySegment current) |
protected MemorySegment |
FileChannelInputView.nextSegment(MemorySegment current) |
protected MemorySegment |
SpillingBuffer.nextSegment(MemorySegment current,
int positionInCurrent) |
protected MemorySegment |
SimpleCollectingOutputView.nextSegment(MemorySegment current,
int positionInCurrent) |
protected MemorySegment |
RandomAccessOutputView.nextSegment(MemorySegment current,
int positionInCurrent) |
protected MemorySegment |
FileChannelOutputView.nextSegment(MemorySegment current,
int posInSegment) |
Modifier and Type | Method and Description |
---|---|
List<MemorySegment> |
SpillingBuffer.close() |
Modifier and Type | Method and Description |
---|---|
protected int |
SeekableFileChannelInputView.getLimitForSegment(MemorySegment segment) |
protected int |
RandomAccessInputView.getLimitForSegment(MemorySegment segment) |
protected int |
FileChannelInputView.getLimitForSegment(MemorySegment segment) |
protected MemorySegment |
SeekableFileChannelInputView.nextSegment(MemorySegment current) |
protected MemorySegment |
RandomAccessInputView.nextSegment(MemorySegment current) |
protected MemorySegment |
FileChannelInputView.nextSegment(MemorySegment current) |
protected MemorySegment |
SpillingBuffer.nextSegment(MemorySegment current,
int positionInCurrent) |
protected MemorySegment |
SimpleCollectingOutputView.nextSegment(MemorySegment current,
int positionInCurrent) |
protected MemorySegment |
RandomAccessOutputView.nextSegment(MemorySegment current,
int positionInCurrent) |
protected MemorySegment |
FileChannelOutputView.nextSegment(MemorySegment current,
int posInSegment) |
Constructor and Description |
---|
RandomAccessOutputView(MemorySegment[] segments,
int segmentSize) |
RandomAccessOutputView(MemorySegment[] segments,
int segmentSize,
int segmentSizeBits) |
Modifier and Type | Field and Description |
---|---|
protected BlockChannelReader<MemorySegment> |
ChannelReaderInputView.reader |
Modifier and Type | Method and Description |
---|---|
MemorySegment |
AsynchronousBlockWriter.getNextReturnedBlock()
Gets the next memory segment that has been written and is available again.
|
MemorySegment |
AsynchronousBlockReader.getNextReturnedBlock()
Gets the next memory segment that has been filled with data by the reader.
|
protected MemorySegment |
HeaderlessChannelReaderInputView.nextSegment(MemorySegment current) |
protected MemorySegment |
ChannelReaderInputView.nextSegment(MemorySegment current)
Gets the next segment from the asynchronous block reader.
|
protected MemorySegment |
ChannelWriterOutputView.nextSegment(MemorySegment current,
int posInSegment) |
Modifier and Type | Method and Description |
---|---|
List<MemorySegment> |
ChannelWriterOutputView.close()
Closes this OutputView, closing the underlying writer and returning all memory segments.
|
List<MemorySegment> |
ChannelReaderInputView.close()
Closes this InputView, closing the underlying reader and returning all memory segments.
|
BlockChannelReader<MemorySegment> |
IOManager.createBlockChannelReader(FileIOChannel.ID channelID)
Creates a block channel reader that reads blocks from the given channel.
|
BlockChannelReader<MemorySegment> |
IOManagerAsync.createBlockChannelReader(FileIOChannel.ID channelID,
LinkedBlockingQueue<MemorySegment> returnQueue)
Creates a block channel reader that reads blocks from the given channel.
|
abstract BlockChannelReader<MemorySegment> |
IOManager.createBlockChannelReader(FileIOChannel.ID channelID,
LinkedBlockingQueue<MemorySegment> returnQueue)
Creates a block channel reader that reads blocks from the given channel.
|
BlockChannelWriter<MemorySegment> |
IOManager.createBlockChannelWriter(FileIOChannel.ID channelID)
Creates a block channel writer that writes to the given channel.
|
BlockChannelWriter<MemorySegment> |
IOManagerAsync.createBlockChannelWriter(FileIOChannel.ID channelID,
LinkedBlockingQueue<MemorySegment> returnQueue) |
abstract BlockChannelWriter<MemorySegment> |
IOManager.createBlockChannelWriter(FileIOChannel.ID channelID,
LinkedBlockingQueue<MemorySegment> returnQueue)
Creates a block channel writer that writes to the given channel.
|
BlockChannelWriterWithCallback<MemorySegment> |
IOManagerAsync.createBlockChannelWriter(FileIOChannel.ID channelID,
RequestDoneCallback<MemorySegment> callback) |
abstract BlockChannelWriterWithCallback<MemorySegment> |
IOManager.createBlockChannelWriter(FileIOChannel.ID channelID,
RequestDoneCallback<MemorySegment> callback)
Creates a block channel writer that writes to the given channel.
|
List<MemorySegment> |
BulkBlockChannelReader.getFullSegments() |
List<MemorySegment> |
AsynchronousBulkBlockReader.getFullSegments() |
LinkedBlockingQueue<MemorySegment> |
AsynchronousBlockWriter.getReturnQueue()
Gets the queue in which the memory segments are queued after the asynchronous write is completed.
|
LinkedBlockingQueue<MemorySegment> |
AsynchronousBlockReader.getReturnQueue()
Gets the queue in which the full memory segments are queued after the asynchronous read
is complete.
|
Modifier and Type | Method and Description |
---|---|
protected int |
HeaderlessChannelReaderInputView.getLimitForSegment(MemorySegment segment) |
protected int |
ChannelReaderInputView.getLimitForSegment(MemorySegment segment) |
protected MemorySegment |
HeaderlessChannelReaderInputView.nextSegment(MemorySegment current) |
protected MemorySegment |
ChannelReaderInputView.nextSegment(MemorySegment current)
Gets the next segment from the asynchronous block reader.
|
protected MemorySegment |
ChannelWriterOutputView.nextSegment(MemorySegment current,
int posInSegment) |
void |
AsynchronousBlockReader.readBlock(MemorySegment segment)
Issues a read request, which will asynchronously fill the given segment with the next block in the
underlying file channel.
|
protected void |
ChannelReaderInputView.sendReadRequest(MemorySegment seg)
Sends a new read requests, if further requests remain.
|
void |
AsynchronousBlockWriterWithCallback.writeBlock(MemorySegment segment)
Issues a asynchronous write request to the writer.
|
Modifier and Type | Method and Description |
---|---|
BlockChannelReader<MemorySegment> |
IOManagerAsync.createBlockChannelReader(FileIOChannel.ID channelID,
LinkedBlockingQueue<MemorySegment> returnQueue)
Creates a block channel reader that reads blocks from the given channel.
|
abstract BlockChannelReader<MemorySegment> |
IOManager.createBlockChannelReader(FileIOChannel.ID channelID,
LinkedBlockingQueue<MemorySegment> returnQueue)
Creates a block channel reader that reads blocks from the given channel.
|
BlockChannelWriter<MemorySegment> |
IOManagerAsync.createBlockChannelWriter(FileIOChannel.ID channelID,
LinkedBlockingQueue<MemorySegment> returnQueue) |
abstract BlockChannelWriter<MemorySegment> |
IOManager.createBlockChannelWriter(FileIOChannel.ID channelID,
LinkedBlockingQueue<MemorySegment> returnQueue)
Creates a block channel writer that writes to the given channel.
|
BlockChannelWriterWithCallback<MemorySegment> |
IOManagerAsync.createBlockChannelWriter(FileIOChannel.ID channelID,
RequestDoneCallback<MemorySegment> callback) |
abstract BlockChannelWriterWithCallback<MemorySegment> |
IOManager.createBlockChannelWriter(FileIOChannel.ID channelID,
RequestDoneCallback<MemorySegment> callback)
Creates a block channel writer that writes to the given channel.
|
BulkBlockChannelReader |
IOManagerAsync.createBulkBlockChannelReader(FileIOChannel.ID channelID,
List<MemorySegment> targetSegments,
int numBlocks)
Creates a block channel reader that reads all blocks from the given channel directly in one bulk.
|
abstract BulkBlockChannelReader |
IOManager.createBulkBlockChannelReader(FileIOChannel.ID channelID,
List<MemorySegment> targetSegments,
int numBlocks)
Creates a block channel reader that reads all blocks from the given channel directly in one bulk.
|
Constructor and Description |
---|
AsynchronousBlockReader(FileIOChannel.ID channelID,
RequestQueue<org.apache.flink.runtime.io.disk.iomanager.ReadRequest> requestQueue,
LinkedBlockingQueue<MemorySegment> returnSegments)
Creates a new block channel reader for the given channel.
|
AsynchronousBlockWriter(FileIOChannel.ID channelID,
RequestQueue<org.apache.flink.runtime.io.disk.iomanager.WriteRequest> requestQueue,
LinkedBlockingQueue<MemorySegment> returnSegments)
Creates a new block channel writer for the given channel.
|
AsynchronousBlockWriterWithCallback(FileIOChannel.ID channelID,
RequestQueue<org.apache.flink.runtime.io.disk.iomanager.WriteRequest> requestQueue,
RequestDoneCallback<MemorySegment> callback)
Creates a new asynchronous block writer for the given channel.
|
AsynchronousBulkBlockReader(FileIOChannel.ID channelID,
RequestQueue<org.apache.flink.runtime.io.disk.iomanager.ReadRequest> requestQueue,
List<MemorySegment> sourceSegments,
int numBlocks) |
ChannelReaderInputView(BlockChannelReader<MemorySegment> reader,
List<MemorySegment> memory,
boolean waitForFirstBlock)
Creates a new channel reader that reads from the given channel until the last block
(as marked by a
ChannelWriterOutputView ) is found. |
ChannelReaderInputView(BlockChannelReader<MemorySegment> reader,
List<MemorySegment> memory,
boolean waitForFirstBlock)
Creates a new channel reader that reads from the given channel until the last block
(as marked by a
ChannelWriterOutputView ) is found. |
ChannelReaderInputView(BlockChannelReader<MemorySegment> reader,
List<MemorySegment> memory,
int numBlocks,
boolean waitForFirstBlock)
Creates a new channel reader that reads from the given channel, expecting a specified
number of blocks in the channel.
|
ChannelReaderInputView(BlockChannelReader<MemorySegment> reader,
List<MemorySegment> memory,
int numBlocks,
boolean waitForFirstBlock)
Creates a new channel reader that reads from the given channel, expecting a specified
number of blocks in the channel.
|
ChannelWriterOutputView(BlockChannelWriter<MemorySegment> writer,
int segmentSize)
Creates an new ChannelWriterOutputView that writes to the given channel.
|
ChannelWriterOutputView(BlockChannelWriter<MemorySegment> writer,
List<MemorySegment> memory,
int segmentSize)
Creates an new ChannelWriterOutputView that writes to the given channel and buffers data
in the given memory segments.
|
ChannelWriterOutputView(BlockChannelWriter<MemorySegment> writer,
List<MemorySegment> memory,
int segmentSize)
Creates an new ChannelWriterOutputView that writes to the given channel and buffers data
in the given memory segments.
|
HeaderlessChannelReaderInputView(BlockChannelReader<MemorySegment> reader,
List<MemorySegment> memory,
int numBlocks,
int numBytesInLastBlock,
boolean waitForFirstBlock)
Creates a new channel reader that reads from the given channel, expecting a specified
number of blocks in the channel, and returns only a specified number of bytes from
the last block.
|
HeaderlessChannelReaderInputView(BlockChannelReader<MemorySegment> reader,
List<MemorySegment> memory,
int numBlocks,
int numBytesInLastBlock,
boolean waitForFirstBlock)
Creates a new channel reader that reads from the given channel, expecting a specified
number of blocks in the channel, and returns only a specified number of bytes from
the last block.
|
Modifier and Type | Method and Description |
---|---|
void |
SpillingAdaptiveSpanningRecordDeserializer.setNextMemorySegment(MemorySegment segment,
int numBytes) |
void |
RecordDeserializer.setNextMemorySegment(MemorySegment segment,
int numBytes) |
void |
AdaptiveSpanningRecordDeserializer.setNextMemorySegment(MemorySegment segment,
int numBytes) |
Modifier and Type | Method and Description |
---|---|
MemorySegment |
Buffer.getMemorySegment() |
MemorySegment |
NetworkBufferPool.requestMemorySegment() |
Modifier and Type | Method and Description |
---|---|
void |
NetworkBufferPool.recycle(MemorySegment segment) |
void |
FreeingBufferRecycler.recycle(MemorySegment memorySegment)
Frees the given memory segment.
|
void |
BufferRecycler.recycle(MemorySegment memorySegment)
Recycles the
MemorySegment to its original BufferPool
instance. |
Constructor and Description |
---|
Buffer(MemorySegment memorySegment,
BufferRecycler recycler) |
Buffer(MemorySegment memorySegment,
BufferRecycler recycler,
boolean isBuffer) |
Modifier and Type | Method and Description |
---|---|
protected MemorySegment |
SerializedUpdateBuffer.nextSegment(MemorySegment current,
int positionInCurrent) |
Modifier and Type | Method and Description |
---|---|
List<MemorySegment> |
SerializedUpdateBuffer.close() |
Modifier and Type | Method and Description |
---|---|
protected MemorySegment |
SerializedUpdateBuffer.nextSegment(MemorySegment current,
int positionInCurrent) |
Constructor and Description |
---|
SerializedUpdateBuffer(List<MemorySegment> memSegments,
int segmentSize,
IOManager ioManager) |
Modifier and Type | Method and Description |
---|---|
MemorySegment |
AbstractPagedOutputView.getCurrentSegment()
Gets the segment that the view currently writes to.
|
MemorySegment |
AbstractPagedInputView.getCurrentSegment()
Gets the memory segment that will be used to read the next bytes from.
|
MemorySegment |
ListMemorySegmentSource.nextSegment() |
protected abstract MemorySegment |
AbstractPagedInputView.nextSegment(MemorySegment current)
The method by which concrete subclasses realize page crossing.
|
protected abstract MemorySegment |
AbstractPagedOutputView.nextSegment(MemorySegment current,
int positionInCurrent)
This method must return a segment.
|
Modifier and Type | Method and Description |
---|---|
List<MemorySegment> |
MemoryManager.allocatePages(Object owner,
int numPages)
Allocates a set of memory segments from this memory manager.
|
Modifier and Type | Method and Description |
---|---|
protected abstract int |
AbstractPagedInputView.getLimitForSegment(MemorySegment segment)
Gets the limit for reading bytes from the given memory segment.
|
protected abstract MemorySegment |
AbstractPagedInputView.nextSegment(MemorySegment current)
The method by which concrete subclasses realize page crossing.
|
protected abstract MemorySegment |
AbstractPagedOutputView.nextSegment(MemorySegment current,
int positionInCurrent)
This method must return a segment.
|
void |
MemoryManager.release(MemorySegment segment)
Tries to release the memory for the specified segment.
|
protected void |
AbstractPagedInputView.seekInput(MemorySegment segment,
int positionInSegment,
int limitInSegment)
Sets the internal state of the view such that the next bytes will be read from the given memory segment,
starting at the given position.
|
protected void |
AbstractPagedOutputView.seekOutput(MemorySegment seg,
int position)
Sets the internal state to the given memory segment and the given position within the segment.
|
Modifier and Type | Method and Description |
---|---|
void |
MemoryManager.allocatePages(Object owner,
List<MemorySegment> target,
int numPages)
Allocates a set of memory segments from this memory manager.
|
void |
MemoryManager.release(Collection<MemorySegment> segments)
Tries to release many memory segments together.
|
Constructor and Description |
---|
AbstractPagedInputView(MemorySegment initialSegment,
int initialLimit,
int headerLength)
Creates a new view that starts with the given segment.
|
AbstractPagedOutputView(MemorySegment initialSegment,
int segmentSize,
int headerLength)
Creates a new output view that writes initially to the given initial segment.
|
Constructor and Description |
---|
ListMemorySegmentSource(List<MemorySegment> memorySegments) |
Modifier and Type | Field and Description |
---|---|
protected MemorySegment[] |
MutableHashTable.buckets
The array of memory segments that contain the buckets which form the actual hash-table
of hash-codes and pointers to the elements.
|
protected MemorySegment[] |
InMemoryPartition.overflowSegments |
protected MemorySegment[] |
HashPartition.overflowSegments |
protected MemorySegment[] |
HashPartition.partitionBuffers |
Modifier and Type | Field and Description |
---|---|
protected List<MemorySegment> |
MutableHashTable.availableMemory
The free memory segments currently available to the hash join.
|
protected BlockChannelWriter<MemorySegment> |
HashPartition.probeSideChannel |
protected LinkedBlockingQueue<MemorySegment> |
MutableHashTable.writeBehindBuffers
The queue of buffers that can be used for write-behind.
|
Modifier and Type | Method and Description |
---|---|
MemorySegment |
MutableHashTable.nextSegment()
This is the method called by the partitions to request memory to serialize records.
|
protected MemorySegment |
HashPartition.nextSegment(MemorySegment current) |
protected MemorySegment |
HashPartition.BuildSideBuffer.nextSegment(MemorySegment current,
int bytesUsed) |
Modifier and Type | Method and Description |
---|---|
BlockChannelWriter<MemorySegment> |
HashPartition.getBuildSideChannel() |
List<MemorySegment> |
MutableHashTable.getFreedMemory() |
List<MemorySegment> |
CompactingHashTable.getFreeMemory() |
abstract List<MemorySegment> |
AbstractMutableHashTable.getFreeMemory() |
BlockChannelWriter<MemorySegment> |
HashPartition.getProbeSideChannel() |
ArrayList<MemorySegment> |
InMemoryPartition.resetOverflowBuckets()
resets overflow bucket counters and returns freed memory and should only be used for resizing
|
Modifier and Type | Method and Description |
---|---|
protected int |
HashPartition.getLimitForSegment(MemorySegment segment) |
protected MemorySegment |
HashPartition.nextSegment(MemorySegment current) |
protected MemorySegment |
HashPartition.BuildSideBuffer.nextSegment(MemorySegment current,
int bytesUsed) |
Modifier and Type | Method and Description |
---|---|
void |
ReOpenableHashPartition.clearAllMemory(List<MemorySegment> target) |
void |
InMemoryPartition.clearAllMemory(List<MemorySegment> target)
releases all of the partition's segments (pages and overflow buckets)
|
void |
HashPartition.clearAllMemory(List<MemorySegment> target) |
void |
HashPartition.finalizeBuildPhase(IOManager ioAccess,
FileIOChannel.Enumerator probeChannelEnumerator,
LinkedBlockingQueue<MemorySegment> bufferReturnQueue) |
int |
ReOpenableHashPartition.finalizeProbePhase(List<MemorySegment> freeMemory,
List<HashPartition<BT,PT>> spilledPartitions,
boolean keepUnprobedSpilledPartitions) |
int |
HashPartition.finalizeProbePhase(List<MemorySegment> freeMemory,
List<HashPartition<BT,PT>> spilledPartitions,
boolean keepUnprobedSpilledPartitions) |
void |
HashPartition.prepareProbePhase(IOManager ioAccess,
FileIOChannel.Enumerator probeChannelEnumerator,
LinkedBlockingQueue<MemorySegment> bufferReturnQueue) |
int |
HashPartition.spillPartition(List<MemorySegment> target,
IOManager ioAccess,
FileIOChannel.ID targetChannel,
LinkedBlockingQueue<MemorySegment> bufferReturnQueue)
Spills this partition to disk and sets it up such that it continues spilling records that are added to
it.
|
int |
HashPartition.spillPartition(List<MemorySegment> target,
IOManager ioAccess,
FileIOChannel.ID targetChannel,
LinkedBlockingQueue<MemorySegment> bufferReturnQueue)
Spills this partition to disk and sets it up such that it continues spilling records that are added to
it.
|
Constructor and Description |
---|
CompactingHashTable(TypeSerializer<T> buildSideSerializer,
TypeComparator<T> buildSideComparator,
List<MemorySegment> memorySegments) |
CompactingHashTable(TypeSerializer<T> buildSideSerializer,
TypeComparator<T> buildSideComparator,
List<MemorySegment> memorySegments,
int avgRecordLen) |
MutableHashTable(TypeSerializer<BT> buildSideSerializer,
TypeSerializer<PT> probeSideSerializer,
TypeComparator<BT> buildSideComparator,
TypeComparator<PT> probeSideComparator,
TypePairComparator<PT,BT> comparator,
List<MemorySegment> memorySegments,
IOManager ioManager) |
MutableHashTable(TypeSerializer<BT> buildSideSerializer,
TypeSerializer<PT> probeSideSerializer,
TypeComparator<BT> buildSideComparator,
TypeComparator<PT> probeSideComparator,
TypePairComparator<PT,BT> comparator,
List<MemorySegment> memorySegments,
IOManager ioManager,
boolean useBloomFilters) |
MutableHashTable(TypeSerializer<BT> buildSideSerializer,
TypeSerializer<PT> probeSideSerializer,
TypeComparator<BT> buildSideComparator,
TypeComparator<PT> probeSideComparator,
TypePairComparator<PT,BT> comparator,
List<MemorySegment> memorySegments,
IOManager ioManager,
int avgRecordLen,
boolean useBloomFilters) |
ReOpenableMutableHashTable(TypeSerializer<BT> buildSideSerializer,
TypeSerializer<PT> probeSideSerializer,
TypeComparator<BT> buildSideComparator,
TypeComparator<PT> probeSideComparator,
TypePairComparator<PT,BT> comparator,
List<MemorySegment> memorySegments,
IOManager ioManager,
boolean useBitmapFilters) |
Modifier and Type | Method and Description |
---|---|
List<MemorySegment> |
SpillingResettableMutableObjectIterator.close() |
List<MemorySegment> |
SpillingResettableIterator.close() |
Constructor and Description |
---|
SpillingResettableIterator(Iterator<T> input,
TypeSerializer<T> serializer,
MemoryManager memoryManager,
IOManager ioManager,
List<MemorySegment> memory) |
SpillingResettableMutableObjectIterator(MutableObjectIterator<T> input,
TypeSerializer<T> serializer,
MemoryManager memoryManager,
IOManager ioManager,
List<MemorySegment> memory) |
Modifier and Type | Field and Description |
---|---|
protected List<MemorySegment> |
UnilateralSortMerger.SpillingThread.mergeReadMemory |
protected List<MemorySegment> |
UnilateralSortMerger.sortReadMemory
The memory segments used first for sorting and later for reading/pre-fetching
during the external merge.
|
protected List<MemorySegment> |
UnilateralSortMerger.writeMemory
The memory segments used to stage data to be written.
|
protected List<MemorySegment> |
UnilateralSortMerger.SpillingThread.writeMemory |
Modifier and Type | Method and Description |
---|---|
MutableObjectIterator<T> |
LargeRecordHandler.finishWriteAndSortKeys(List<MemorySegment> memory) |
protected MergeIterator<E> |
UnilateralSortMerger.SpillingThread.getMergingIterator(List<UnilateralSortMerger.ChannelWithBlockCount> channelIDs,
List<List<MemorySegment>> inputSegments,
List<FileIOChannel> readerList,
MutableObjectIterator<E> largeRecords)
Returns an iterator that iterates over the merged result from all given channels.
|
protected void |
UnilateralSortMerger.SpillingThread.getSegmentsForReaders(List<List<MemorySegment>> target,
List<MemorySegment> memory,
int numChannels)
Divides the given collection of memory buffers among
numChannels sublists. |
protected void |
UnilateralSortMerger.SpillingThread.getSegmentsForReaders(List<List<MemorySegment>> target,
List<MemorySegment> memory,
int numChannels)
Divides the given collection of memory buffers among
numChannels sublists. |
protected UnilateralSortMerger.ThreadBase<E> |
UnilateralSortMerger.getSpillingThread(ExceptionHandler<IOException> exceptionHandler,
UnilateralSortMerger.CircularQueues<E> queues,
AbstractInvokable parentTask,
MemoryManager memoryManager,
IOManager ioManager,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
List<MemorySegment> sortReadMemory,
List<MemorySegment> writeMemory,
int maxFileHandles) |
protected UnilateralSortMerger.ThreadBase<E> |
UnilateralSortMerger.getSpillingThread(ExceptionHandler<IOException> exceptionHandler,
UnilateralSortMerger.CircularQueues<E> queues,
AbstractInvokable parentTask,
MemoryManager memoryManager,
IOManager ioManager,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
List<MemorySegment> sortReadMemory,
List<MemorySegment> writeMemory,
int maxFileHandles) |
protected UnilateralSortMerger.ThreadBase<E> |
CombiningUnilateralSortMerger.getSpillingThread(ExceptionHandler<IOException> exceptionHandler,
UnilateralSortMerger.CircularQueues<E> queues,
AbstractInvokable parentTask,
MemoryManager memoryManager,
IOManager ioManager,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
List<MemorySegment> sortReadMemory,
List<MemorySegment> writeMemory,
int maxFileHandles) |
protected UnilateralSortMerger.ThreadBase<E> |
CombiningUnilateralSortMerger.getSpillingThread(ExceptionHandler<IOException> exceptionHandler,
UnilateralSortMerger.CircularQueues<E> queues,
AbstractInvokable parentTask,
MemoryManager memoryManager,
IOManager ioManager,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
List<MemorySegment> sortReadMemory,
List<MemorySegment> writeMemory,
int maxFileHandles) |
protected List<UnilateralSortMerger.ChannelWithBlockCount> |
UnilateralSortMerger.SpillingThread.mergeChannelList(List<UnilateralSortMerger.ChannelWithBlockCount> channelIDs,
List<MemorySegment> allReadBuffers,
List<MemorySegment> writeBuffers)
Merges the given sorted runs to a smaller number of sorted runs.
|
protected List<UnilateralSortMerger.ChannelWithBlockCount> |
UnilateralSortMerger.SpillingThread.mergeChannelList(List<UnilateralSortMerger.ChannelWithBlockCount> channelIDs,
List<MemorySegment> allReadBuffers,
List<MemorySegment> writeBuffers)
Merges the given sorted runs to a smaller number of sorted runs.
|
protected UnilateralSortMerger.ChannelWithBlockCount |
UnilateralSortMerger.SpillingThread.mergeChannels(List<UnilateralSortMerger.ChannelWithBlockCount> channelIDs,
List<List<MemorySegment>> readBuffers,
List<MemorySegment> writeBuffers)
Merges the sorted runs described by the given Channel IDs into a single sorted run.
|
protected UnilateralSortMerger.ChannelWithBlockCount |
UnilateralSortMerger.SpillingThread.mergeChannels(List<UnilateralSortMerger.ChannelWithBlockCount> channelIDs,
List<List<MemorySegment>> readBuffers,
List<MemorySegment> writeBuffers)
Merges the sorted runs described by the given Channel IDs into a single sorted run.
|
protected UnilateralSortMerger.ChannelWithBlockCount |
CombiningUnilateralSortMerger.CombiningSpillingThread.mergeChannels(List<UnilateralSortMerger.ChannelWithBlockCount> channelIDs,
List<List<MemorySegment>> readBuffers,
List<MemorySegment> writeBuffers)
Merges the sorted runs described by the given Channel IDs into a single sorted run.
|
protected UnilateralSortMerger.ChannelWithBlockCount |
CombiningUnilateralSortMerger.CombiningSpillingThread.mergeChannels(List<UnilateralSortMerger.ChannelWithBlockCount> channelIDs,
List<List<MemorySegment>> readBuffers,
List<MemorySegment> writeBuffers)
Merges the sorted runs described by the given Channel IDs into a single sorted run.
|
Constructor and Description |
---|
CircularElement(int id,
InMemorySorter<E> buffer,
List<MemorySegment> memory) |
CombiningSpillingThread(ExceptionHandler<IOException> exceptionHandler,
UnilateralSortMerger.CircularQueues<E> queues,
AbstractInvokable parentTask,
MemoryManager memManager,
IOManager ioManager,
TypeSerializer<E> serializer,
TypeComparator<E> comparator,
List<MemorySegment> sortReadMemory,
List<MemorySegment> writeMemory,
int maxNumFileHandles) |
CombiningSpillingThread(ExceptionHandler<IOException> exceptionHandler,
UnilateralSortMerger.CircularQueues<E> queues,
AbstractInvokable parentTask,
MemoryManager memManager,
IOManager ioManager,
TypeSerializer<E> serializer,
TypeComparator<E> comparator,
List<MemorySegment> sortReadMemory,
List<MemorySegment> writeMemory,
int maxNumFileHandles) |
FixedLengthRecordSorter(TypeSerializer<T> serializer,
TypeComparator<T> comparator,
List<MemorySegment> memory) |
LargeRecordHandler(TypeSerializer<T> serializer,
TypeComparator<T> comparator,
IOManager ioManager,
MemoryManager memManager,
List<MemorySegment> memory,
AbstractInvokable memoryOwner,
int maxFilehandles) |
NormalizedKeySorter(TypeSerializer<T> serializer,
TypeComparator<T> comparator,
List<MemorySegment> memory) |
NormalizedKeySorter(TypeSerializer<T> serializer,
TypeComparator<T> comparator,
List<MemorySegment> memory,
int maxNormalizedKeyBytes) |
SpillingThread(ExceptionHandler<IOException> exceptionHandler,
UnilateralSortMerger.CircularQueues<E> queues,
AbstractInvokable parentTask,
MemoryManager memManager,
IOManager ioManager,
TypeSerializer<E> serializer,
TypeComparator<E> comparator,
List<MemorySegment> sortReadMemory,
List<MemorySegment> writeMemory,
int maxNumFileHandles)
Creates the spilling thread.
|
SpillingThread(ExceptionHandler<IOException> exceptionHandler,
UnilateralSortMerger.CircularQueues<E> queues,
AbstractInvokable parentTask,
MemoryManager memManager,
IOManager ioManager,
TypeSerializer<E> serializer,
TypeComparator<E> comparator,
List<MemorySegment> sortReadMemory,
List<MemorySegment> writeMemory,
int maxNumFileHandles)
Creates the spilling thread.
|
UnilateralSortMerger(MemoryManager memoryManager,
List<MemorySegment> memory,
IOManager ioManager,
MutableObjectIterator<E> input,
AbstractInvokable parentTask,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
int numSortBuffers,
int maxNumFileHandles,
float startSpillingFraction,
boolean handleLargeRecords,
boolean objectReuseEnabled) |
UnilateralSortMerger(MemoryManager memoryManager,
List<MemorySegment> memory,
IOManager ioManager,
MutableObjectIterator<E> input,
AbstractInvokable parentTask,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
int numSortBuffers,
int maxNumFileHandles,
float startSpillingFraction,
boolean noSpillingMemory,
boolean handleLargeRecords,
boolean objectReuseEnabled) |
Modifier and Type | Method and Description |
---|---|
MemorySegment |
BloomFilter.BitSet.getMemorySegment() |
Modifier and Type | Method and Description |
---|---|
void |
BloomFilter.setBitsLocation(MemorySegment memorySegment,
int offset) |
void |
BloomFilter.BitSet.setMemorySegment(MemorySegment memorySegment,
int offset) |
void |
BitSet.setMemorySegment(MemorySegment memorySegment,
int offset) |
Modifier and Type | Method and Description |
---|---|
void |
StringValue.copyNormalizedKey(MemorySegment target,
int offset,
int len) |
void |
ShortValue.copyNormalizedKey(MemorySegment target,
int offset,
int len) |
void |
NullValue.copyNormalizedKey(MemorySegment target,
int offset,
int len) |
void |
NormalizableKey.copyNormalizedKey(MemorySegment memory,
int offset,
int len)
Writes a normalized key for the given record into the target byte array, starting at the specified position
an writing exactly the given number of bytes.
|
void |
LongValue.copyNormalizedKey(MemorySegment target,
int offset,
int len) |
void |
IntValue.copyNormalizedKey(MemorySegment target,
int offset,
int len) |
void |
CharValue.copyNormalizedKey(MemorySegment target,
int offset,
int len) |
void |
ByteValue.copyNormalizedKey(MemorySegment target,
int offset,
int len) |
void |
BooleanValue.copyNormalizedKey(MemorySegment target,
int offset,
int len) |
Copyright © 2014–2017 The Apache Software Foundation. All rights reserved.