org.apache.tez.runtime.library.common.sort.impl.dflt

Class DefaultSorter

java.lang.Object

org.apache.tez.runtime.library.common.sort.impl.ExternalSorter

org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter

All Implemented Interfaces:

org.apache.hadoop.util.IndexedSortable

public final class DefaultSorter
extends ExternalSorter
implements org.apache.hadoop.util.IndexedSortable

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	DefaultSorter.BlockingBuffer Inner class managing the spill of serialized records to disk.
class	DefaultSorter.Buffer
protected class	DefaultSorter.MRResultIterator
protected class	DefaultSorter.SpillThread

Nested classes/interfaces inherited from class org.apache.tez.runtime.library.common.sort.impl.ExternalSorter

ExternalSorter.MapBufferTooSmallException

Field Summary

Fields

Modifier and Type	Field and Description
protected static int	KEYSTART
static int	MAP_OUTPUT_INDEX_RECORD_LENGTH The size of each record in the index file for the map-outputs.
static int	MAX_IO_SORT_MB
protected static int	METASIZE
protected static int	NMETA
protected static int	PARTITION
protected static int	VALLEN
protected static int	VALSTART

Fields inherited from class org.apache.tez.runtime.library.common.sort.impl.ExternalSorter

additionalSpillBytesRead, additionalSpillBytesWritten, availableMemoryMb, cleanup, codec, combiner, comparator, conf, fileOutputByteCounter, finalIndexFile, finalMergeEnabled, finalOutputFile, ifileBufferSize, ifileReadAhead, ifileReadAheadLength, keyClass, keySerializer, mapOutputByteCounter, mapOutputFile, mapOutputRecordCounter, numAdditionalSpills, numShuffleChunks, numSpills, outputBytesWithOverheadCounter, outputContext, partitioner, partitions, partitionStats, progressable, rfs, sendEmptyPartitionDetails, serializationFactory, sorter, spilledRecordsCounter, spillFileIndexPaths, spillFilePaths, statsReporter, valClass, valSerializer

Constructor Summary

Constructors

Constructor and Description
DefaultSorter(org.apache.tez.runtime.api.OutputContext outputContext, org.apache.hadoop.conf.Configuration conf, int numOutputs, long initialMemoryAvailable)

Method Summary

Methods

Modifier and Type	Method and Description
int	compare(int mi, int mj) Compare logical range, st i, j MOD offset capacity.
void	flush()
protected int	getInMemVBytesLength(int kvoff)
protected void	sortAndSpill(long sameKeyCount, long totalKeysCount)
protected void	spill(int mstart, int mend, long sameKeyCount, long totalKeysCount)
void	swap(int mi, int mj) Swap metadata for items i,j
void	write(Object key, Object value)

Methods inherited from class org.apache.tez.runtime.library.common.sort.impl.ExternalSorter

cleanup, cleanup, cleanup, close, getFinalIndexFile, getFinalOutputFile, getInitialMemoryRequirement, getMapOutput, getNumSpills, getPartitionStats, getShuffleHeader, getSortedStream, isFinalMergeEnabled, reportPartitionStats, reportStatistics, runCombineProcessor, sameVolRename, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAP_OUTPUT_INDEX_RECORD_LENGTH

public static final int MAP_OUTPUT_INDEX_RECORD_LENGTH

The size of each record in the index file for the map-outputs.

See Also:

Constant Field Values

VALSTART

protected static final int VALSTART

See Also:

Constant Field Values

KEYSTART

protected static final int KEYSTART

See Also:

Constant Field Values

PARTITION

protected static final int PARTITION

See Also:

Constant Field Values

VALLEN

protected static final int VALLEN

See Also:

Constant Field Values

NMETA

protected static final int NMETA

See Also:

Constant Field Values

METASIZE

protected static final int METASIZE

See Also:

Constant Field Values

MAX_IO_SORT_MB

public static final int MAX_IO_SORT_MB

See Also:

Constant Field Values

Constructor Detail

DefaultSorter

public DefaultSorter(org.apache.tez.runtime.api.OutputContext outputContext,
             org.apache.hadoop.conf.Configuration conf,
             int numOutputs,
             long initialMemoryAvailable)
              throws IOException

Throws:

IOException

Method Detail

write

public void write(Object key,
         Object value)
           throws IOException

Specified by:

write in class ExternalSorter

Throws:

IOException

compare

public int compare(int mi,
          int mj)

Compare logical range, st i, j MOD offset capacity. Compare by partition, then by key.

Specified by:

compare in interface org.apache.hadoop.util.IndexedSortable

See Also:

IndexedSortable.compare(int, int)

swap

public void swap(int mi,
        int mj)

Swap metadata for items i,j

Specified by:

swap in interface org.apache.hadoop.util.IndexedSortable

See Also:

IndexedSortable.swap(int, int)

flush

public void flush()
           throws IOException

Specified by:

flush in class ExternalSorter

Throws:

IOException

sortAndSpill

protected void sortAndSpill(long sameKeyCount,
                long totalKeysCount)
                     throws IOException,
                            InterruptedException

Throws:

IOException

InterruptedException

spill

protected void spill(int mstart,
         int mend,
         long sameKeyCount,
         long totalKeysCount)
              throws IOException,
                     InterruptedException

Throws:

IOException

InterruptedException

getInMemVBytesLength

protected int getInMemVBytesLength(int kvoff)