public class BinaryExternalSorter extends Object implements Sorter<BinaryRow>
BinaryExternalSorter
is a full fledged sorter for binary format.
It implements a multi-way merge sort.
Internally, it has three asynchronous threads (sort, spill, merger) which communicate through
a set of blocking circularQueues, forming a closed loop. Memory is allocated using the
MemoryManager
interface. Thus the component will not exceed the provided memory limits.Constructor and Description |
---|
BinaryExternalSorter(Object owner,
MemoryManager memoryManager,
long reservedMemorySize,
IOManager ioManager,
AbstractRowSerializer<BaseRow> inputSerializer,
BinaryRowSerializer serializer,
NormalizedKeyComputer normalizedKeyComputer,
RecordComparator comparator,
Configuration conf) |
BinaryExternalSorter(Object owner,
MemoryManager memoryManager,
long reservedMemorySize,
IOManager ioManager,
AbstractRowSerializer<BaseRow> inputSerializer,
BinaryRowSerializer serializer,
NormalizedKeyComputer normalizedKeyComputer,
RecordComparator comparator,
Configuration conf,
float startSpillingFraction) |
Modifier and Type | Method and Description |
---|---|
void |
close()
Shuts down all the threads initiated by this sorter.
|
MutableObjectIterator<BinaryRow> |
getIterator()
Gets the iterator over this input.
|
long |
getNumSpillFiles() |
long |
getSpillInBytes() |
long |
getUsedMemoryInBytes() |
void |
startThreads()
Starts all the threads that are used by this sorter.
|
void |
write(BaseRow current) |
void |
write(MutableObjectIterator<BinaryRow> iterator) |
public BinaryExternalSorter(Object owner, MemoryManager memoryManager, long reservedMemorySize, IOManager ioManager, AbstractRowSerializer<BaseRow> inputSerializer, BinaryRowSerializer serializer, NormalizedKeyComputer normalizedKeyComputer, RecordComparator comparator, Configuration conf) throws IOException
IOException
public BinaryExternalSorter(Object owner, MemoryManager memoryManager, long reservedMemorySize, IOManager ioManager, AbstractRowSerializer<BaseRow> inputSerializer, BinaryRowSerializer serializer, NormalizedKeyComputer normalizedKeyComputer, RecordComparator comparator, Configuration conf, float startSpillingFraction) throws IOException
IOException
public void startThreads()
public void close()
The threads are set to exit directly, but depending on their operation, it may take a while to actually happen. The sorting thread will for example not finish before the current batch is sorted. This method attempts to wait for the working thread to exit. If it is however interrupted, the method exits immediately and is not guaranteed how long the threads continue to exist and occupy resources afterwards.
close
in interface Closeable
close
in interface AutoCloseable
public void write(BaseRow current) throws IOException
IOException
@VisibleForTesting public void write(MutableObjectIterator<BinaryRow> iterator) throws IOException
IOException
public MutableObjectIterator<BinaryRow> getIterator() throws InterruptedException
CloseableInputProvider
getIterator
in interface Sorter<BinaryRow>
getIterator
in interface CloseableInputProvider<BinaryRow>
InterruptedException
public long getUsedMemoryInBytes()
public long getNumSpillFiles()
public long getSpillInBytes()
Copyright © 2014–2020 The Apache Software Foundation. All rights reserved.