org.apache.tez.runtime.library.common.shuffle

Class ShuffleUtils

java.lang.Object

org.apache.tez.runtime.library.common.shuffle.ShuffleUtils

public class ShuffleUtils
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static String	SHUFFLE_HANDLER_SERVICE_ID

Constructor Summary

Constructors

Constructor and Description
ShuffleUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static StringBuilder	constructBaseURIForShuffleHandler(String host, int port, int partition, String appId, int dagIdentifier, boolean sslShuffle)
static URL	constructInputURL(String baseURI, Collection<InputAttemptIdentifier> inputs, boolean keepAlive)
static ByteBuffer	convertJobTokenToBytes(org.apache.hadoop.security.token.Token<org.apache.tez.common.security.JobTokenIdentifier> jobToken)
static int	deserializeShuffleProviderMetaData(ByteBuffer meta)
static void	generateEventOnSpill(List<org.apache.tez.runtime.api.Event> eventList, boolean finalMergeEnabled, boolean isLastEvent, org.apache.tez.runtime.api.OutputContext context, int spillId, TezSpillRecord spillRecord, int numPhysicalOutputs, boolean sendEmptyPartitionDetails, String pathComponent, long[] partitionStats) Generate events when spill happens
static void	generateEventsForNonStartedOutput(List<org.apache.tez.runtime.api.Event> eventList, int numPhysicalOutputs, org.apache.tez.runtime.api.OutputContext context, boolean generateVmEvent, boolean isCompositeEvent) Generate events for outputs which have not been started.
static BaseHttpConnection	getHttpConnection(boolean asyncHttp, URL url, HttpConnectionParams params, String logIdentifier, org.apache.tez.common.security.JobTokenSecretManager jobTokenSecretManager)
static HttpConnectionParams	getHttpConnectionParams(org.apache.hadoop.conf.Configuration conf) Build HttpConnectionParams from configuration
static SecretKey	getJobTokenSecretFromTokenBytes(ByteBuffer meta)
static org.roaringbitmap.RoaringBitmap	getPartitionStatsForPhysicalOutput(long[] sizes) Data size for the destinations
static void	ioCleanup(Closeable... closeables)
static void	logIndividualFetchComplete(org.slf4j.Logger log, long millis, long bytesCompressed, long bytesDecompressed, String outputType, InputAttemptIdentifier srcAttemptIdentifier) Log individual fetch complete event.
static void	shuffleToDisk(OutputStream output, String hostIdentifier, InputStream input, long compressedLength, long decompressedLength, org.slf4j.Logger LOG, String identifier)
static void	shuffleToMemory(byte[] shuffleData, InputStream input, int decompressedLength, int compressedLength, org.apache.hadoop.io.compress.CompressionCodec codec, boolean ifileReadAhead, int ifileReadAheadLength, org.slf4j.Logger LOG, String identifier)
static String	stringify(ShuffleUserPayloads.DataMovementEventPayloadProto dmProto)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SHUFFLE_HANDLER_SERVICE_ID

public static final String SHUFFLE_HANDLER_SERVICE_ID

See Also:

Constant Field Values

Constructor Detail

ShuffleUtils

public ShuffleUtils()

Method Detail

getJobTokenSecretFromTokenBytes

public static SecretKey getJobTokenSecretFromTokenBytes(ByteBuffer meta)
                                                 throws IOException

Throws:

IOException

convertJobTokenToBytes

public static ByteBuffer convertJobTokenToBytes(org.apache.hadoop.security.token.Token<org.apache.tez.common.security.JobTokenIdentifier> jobToken)
                                         throws IOException

Throws:

IOException

deserializeShuffleProviderMetaData

public static int deserializeShuffleProviderMetaData(ByteBuffer meta)
                                              throws IOException

Throws:

IOException

shuffleToMemory

public static void shuffleToMemory(byte[] shuffleData,
                   InputStream input,
                   int decompressedLength,
                   int compressedLength,
                   org.apache.hadoop.io.compress.CompressionCodec codec,
                   boolean ifileReadAhead,
                   int ifileReadAheadLength,
                   org.slf4j.Logger LOG,
                   String identifier)
                            throws IOException

Throws:

IOException

shuffleToDisk

public static void shuffleToDisk(OutputStream output,
                 String hostIdentifier,
                 InputStream input,
                 long compressedLength,
                 long decompressedLength,
                 org.slf4j.Logger LOG,
                 String identifier)
                          throws IOException

Throws:

IOException

ioCleanup

public static void ioCleanup(Closeable... closeables)

constructBaseURIForShuffleHandler

public static StringBuilder constructBaseURIForShuffleHandler(String host,
                                              int port,
                                              int partition,
                                              String appId,
                                              int dagIdentifier,
                                              boolean sslShuffle)

constructInputURL

public static URL constructInputURL(String baseURI,
                    Collection<InputAttemptIdentifier> inputs,
                    boolean keepAlive)
                             throws MalformedURLException

Throws:

MalformedURLException

getHttpConnection

public static BaseHttpConnection getHttpConnection(boolean asyncHttp,
                                   URL url,
                                   HttpConnectionParams params,
                                   String logIdentifier,
                                   org.apache.tez.common.security.JobTokenSecretManager jobTokenSecretManager)
                                            throws IOException

Throws:

IOException

stringify

public static String stringify(ShuffleUserPayloads.DataMovementEventPayloadProto dmProto)

generateEventsForNonStartedOutput

public static void generateEventsForNonStartedOutput(List<org.apache.tez.runtime.api.Event> eventList,
                                     int numPhysicalOutputs,
                                     org.apache.tez.runtime.api.OutputContext context,
                                     boolean generateVmEvent,
                                     boolean isCompositeEvent)
                                              throws IOException

Generate events for outputs which have not been started.

Parameters:

eventList -

numPhysicalOutputs -

context -

generateVmEvent - whether to generate a vm event or not

isCompositeEvent - whether to generate a CompositeDataMovementEvent or a DataMovementEvent

Throws:

IOException

generateEventOnSpill

public static void generateEventOnSpill(List<org.apache.tez.runtime.api.Event> eventList,
                        boolean finalMergeEnabled,
                        boolean isLastEvent,
                        org.apache.tez.runtime.api.OutputContext context,
                        int spillId,
                        TezSpillRecord spillRecord,
                        int numPhysicalOutputs,
                        boolean sendEmptyPartitionDetails,
                        String pathComponent,
                        @Nullable
                        long[] partitionStats)
                                 throws IOException

Generate events when spill happens

Parameters:

eventList - events would be added to this list

finalMergeEnabled -

isLastEvent -

context -

spillId -

spillRecord -

numPhysicalOutputs -

pathComponent -

partitionStats -

Throws:

IOException

getPartitionStatsForPhysicalOutput

public static org.roaringbitmap.RoaringBitmap getPartitionStatsForPhysicalOutput(long[] sizes)

Data size for the destinations

Parameters:

sizes - for physical outputs

logIndividualFetchComplete

public static void logIndividualFetchComplete(org.slf4j.Logger log,
                              long millis,
                              long bytesCompressed,
                              long bytesDecompressed,
                              String outputType,
                              InputAttemptIdentifier srcAttemptIdentifier)

Log individual fetch complete event. This log information would be used by tez-tool/perf-analzyer/shuffle tools for mining - amount of data transferred between source to destination machine - time taken to transfer data between source to destination machine - details on DISK/DISK_DIRECT/MEMORY based shuffles

Parameters:

log -

millis -

bytesCompressed -

bytesDecompressed -

outputType -

srcAttemptIdentifier -

getHttpConnectionParams

public static HttpConnectionParams getHttpConnectionParams(org.apache.hadoop.conf.Configuration conf)

Build HttpConnectionParams from configuration

Parameters:

conf -

Returns:

HttpConnectionParams