org.apache.tez.runtime.library.common.shuffle.orderedgrouped

Class MergeManager

java.lang.Object

org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager

All Implemented Interfaces:

FetchedInputAllocatorOrderedGrouped

@InterfaceAudience.Private
@InterfaceStability.Unstable
public class MergeManager
extends Object
implements FetchedInputAllocatorOrderedGrouped

Usage. Create instance. setInitialMemoryAvailable(long), configureAndStart()

Constructor Summary

Constructors

Constructor and Description

MergeManager(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem localFS, org.apache.hadoop.fs.LocalDirAllocator localDirAllocator, org.apache.tez.runtime.api.InputContext inputContext, Combiner combiner, org.apache.tez.common.counters.TezCounter spilledRecordsCounter, org.apache.tez.common.counters.TezCounter reduceCombineInputCounter, org.apache.tez.common.counters.TezCounter mergedMapOutputsCounter, org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ExceptionReporter exceptionReporter, long initialMemoryAvailable, org.apache.hadoop.io.compress.CompressionCodec codec, boolean ifileReadAheadEnabled, int ifileReadAheadLength) Construct the MergeManager.

Method Summary

Methods

Modifier and Type	Method and Description
TezRawKeyValueIterator	close(boolean tryFinalMerge)
void	closeInMemoryFile(org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput mapOutput)
void	closeInMemoryMergedFile(org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput mapOutput)
void	closeOnDiskFile(FileChunk file)
org.apache.hadoop.fs.FileSystem	getLocalFileSystem()
boolean	isMergeComplete() Should only be used after the Shuffle phaze is complete, otherwise can return an invalid state since a merge may not be in progress dur to inadequate inputs
boolean	isShutdown()
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput	reserve(InputAttemptIdentifier srcAttemptIdentifier, long requestedSize, long compressedLength, int fetcher)
void	unreserve(long size)
void	waitForInMemoryMerge()
void	waitForShuffleToMergeMemory()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MergeManager

public MergeManager(org.apache.hadoop.conf.Configuration conf,
            org.apache.hadoop.fs.FileSystem localFS,
            org.apache.hadoop.fs.LocalDirAllocator localDirAllocator,
            org.apache.tez.runtime.api.InputContext inputContext,
            Combiner combiner,
            org.apache.tez.common.counters.TezCounter spilledRecordsCounter,
            org.apache.tez.common.counters.TezCounter reduceCombineInputCounter,
            org.apache.tez.common.counters.TezCounter mergedMapOutputsCounter,
            org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ExceptionReporter exceptionReporter,
            long initialMemoryAvailable,
            org.apache.hadoop.io.compress.CompressionCodec codec,
            boolean ifileReadAheadEnabled,
            int ifileReadAheadLength)

Construct the MergeManager. Must call start before it becomes usable.

Method Detail

waitForInMemoryMerge

public void waitForInMemoryMerge()
                          throws InterruptedException

Throws:

InterruptedException

waitForShuffleToMergeMemory

public void waitForShuffleToMergeMemory()
                                 throws InterruptedException

Throws:

InterruptedException

reserve

public org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput reserve(InputAttemptIdentifier srcAttemptIdentifier,
                                                                             long requestedSize,
                                                                             long compressedLength,
                                                                             int fetcher)
                                                                               throws IOException

Specified by:

reserve in interface FetchedInputAllocatorOrderedGrouped

Throws:

IOException

unreserve

public void unreserve(long size)

Specified by:

unreserve in interface FetchedInputAllocatorOrderedGrouped

closeInMemoryFile

public void closeInMemoryFile(org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput mapOutput)

Specified by:

closeInMemoryFile in interface FetchedInputAllocatorOrderedGrouped

closeInMemoryMergedFile

public void closeInMemoryMergedFile(org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput mapOutput)

getLocalFileSystem

public org.apache.hadoop.fs.FileSystem getLocalFileSystem()

Specified by:

getLocalFileSystem in interface FetchedInputAllocatorOrderedGrouped

closeOnDiskFile

public void closeOnDiskFile(FileChunk file)

Specified by:

closeOnDiskFile in interface FetchedInputAllocatorOrderedGrouped

isMergeComplete

@InterfaceAudience.Private
public boolean isMergeComplete()

Should only be used after the Shuffle phaze is complete, otherwise can return an invalid state since a merge may not be in progress dur to inadequate inputs

Returns:

true if the merge process is complete, otherwise false

close

public TezRawKeyValueIterator close(boolean tryFinalMerge)
                             throws Throwable

Throws:

Throwable

isShutdown

public boolean isShutdown()