org.apache.tez.mapreduce.hadoop

Class MRInputHelpers

java.lang.Object

org.apache.tez.mapreduce.hadoop.MRInputHelpers

@InterfaceAudience.Public
@InterfaceStability.Unstable
public class MRInputHelpers
extends Object

Constructor Summary

Constructors

Constructor and Description
MRInputHelpers()

Method Summary

Methods

Modifier and Type	Method and Description
static org.apache.tez.dag.api.DataSourceDescriptor	configureMRInputWithLegacySplitGeneration(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path splitsDir, boolean useLegacyInput) Setup split generation on the client, with splits being distributed via the traditional MapReduce mechanism of distributing splits via the Distributed Cache.
protected static org.apache.tez.dag.api.UserPayload	createMRInputPayload(org.apache.hadoop.conf.Configuration conf, MRRuntimeProtos.MRSplitsProto mrSplitsProto)
protected static org.apache.tez.dag.api.UserPayload	createMRInputPayloadWithGrouping(org.apache.hadoop.conf.Configuration conf) Called to specify that grouping of input splits be performed by Tez The conf should have the input format class configuration set to the TezGroupedSplitsInputFormat.
static org.apache.hadoop.mapreduce.InputSplit	createNewFormatSplitFromUserPayload(MRRuntimeProtos.MRSplitProto splitProto, org.apache.hadoop.io.serializer.SerializationFactory serializationFactory) Create an instance of InputSplit from the MRInput representation of a split.
static org.apache.hadoop.mapred.InputSplit	createOldFormatSplitFromUserPayload(MRRuntimeProtos.MRSplitProto splitProto, org.apache.hadoop.io.serializer.SerializationFactory serializationFactory) Create an instance of InputSplit from the MRInput representation of a split.
static MRRuntimeProtos.MRSplitProto	createSplitProto(org.apache.hadoop.mapred.InputSplit oldSplit)
static <T extends org.apache.hadoop.mapreduce.InputSplit> MRRuntimeProtos.MRSplitProto	createSplitProto(T newSplit, org.apache.hadoop.io.serializer.SerializationFactory serializationFactory)
static InputSplitInfoMem	generateInputSplitsToMem(org.apache.hadoop.conf.Configuration conf, boolean groupSplits, int targetTasks) Generates Input splits and stores them in a MRRuntimeProtos.MRSplitsProto instance.
static String	getApplicationIdString(org.apache.hadoop.conf.Configuration conf)
static int	getDagAttemptNumber(org.apache.hadoop.conf.Configuration conf)
static String	getDagIdString(org.apache.hadoop.conf.Configuration conf) Returns string representation of full DAG identifier
static int	getDagIndex(org.apache.hadoop.conf.Configuration conf)
static String	getDagName(org.apache.hadoop.conf.Configuration conf)
static int	getInputIndex(org.apache.hadoop.conf.Configuration conf)
static String	getInputName(org.apache.hadoop.conf.Configuration conf)
static String	getTaskAttemptIdString(org.apache.hadoop.conf.Configuration conf) Returns string representation of full task attempt identifier
static int	getTaskAttemptIndex(org.apache.hadoop.conf.Configuration conf)
static String	getTaskIdString(org.apache.hadoop.conf.Configuration conf) Returns string representation of full task identifier
static int	getTaskIndex(org.apache.hadoop.conf.Configuration conf)
static String	getUniqueIdentifier(org.apache.hadoop.conf.Configuration conf)
static String	getVertexIdString(org.apache.hadoop.conf.Configuration conf) Returns string representation of full vertex identifier
static int	getVertexIndex(org.apache.hadoop.conf.Configuration conf) * @see TaskContext.getTaskVertexIndex()
static String	getVertexName(org.apache.hadoop.conf.Configuration conf)
static MRRuntimeProtos.MRInputUserPayloadProto	parseMRInputPayload(org.apache.tez.dag.api.UserPayload payload) Parse the payload used by MRInputPayload

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MRInputHelpers

public MRInputHelpers()

Method Detail

configureMRInputWithLegacySplitGeneration

@InterfaceStability.Unstable
public static org.apache.tez.dag.api.DataSourceDescriptor configureMRInputWithLegacySplitGeneration(org.apache.hadoop.conf.Configuration conf,
                                                                                                                org.apache.hadoop.fs.Path splitsDir,
                                                                                                                boolean useLegacyInput)

Setup split generation on the client, with splits being distributed via the traditional MapReduce mechanism of distributing splits via the Distributed Cache.

Usage of this technique for handling splits is not advised. Instead, splits should be either generated in the AM, or generated in the client and distributed via the AM. See MRInput.MRInputConfigBuilder

Note: Attempting to use this method to add multiple Inputs to a Vertex is not supported. This mechanism of propagating splits may be removed in a subsequent release, and is not recommended.

Parameters:

conf - configuration to be used by MRInput. This is expected to be fully configured.

splitsDir - the path to which splits will be generated.

useLegacyInput - whether to use MRInputLegacy or MRInput

Returns:

an instance of DataSourceDescriptor which can be added as a data source to a Vertex

parseMRInputPayload

@InterfaceStability.Evolving
public static MRRuntimeProtos.MRInputUserPayloadProto parseMRInputPayload(org.apache.tez.dag.api.UserPayload payload)
                                                                   throws IOException

Parse the payload used by MRInputPayload

Parameters:

payload - the UserPayload instance

Returns:

an instance of MRRuntimeProtos.MRInputUserPayloadProto, which provides access to the underlying configuration bytes

Throws:

IOException

createOldFormatSplitFromUserPayload

@InterfaceStability.Evolving
public static org.apache.hadoop.mapred.InputSplit createOldFormatSplitFromUserPayload(MRRuntimeProtos.MRSplitProto splitProto,
                                                                                                  org.apache.hadoop.io.serializer.SerializationFactory serializationFactory)
                                                                               throws IOException

Create an instance of InputSplit from the MRInput representation of a split.

Parameters:

splitProto - The MRRuntimeProtos.MRSplitProto instance representing the split

serializationFactory - the serialization mechanism used to write out the split

Returns:

an instance of the split

Throws:

IOException

createNewFormatSplitFromUserPayload

@InterfaceStability.Evolving
public static org.apache.hadoop.mapreduce.InputSplit createNewFormatSplitFromUserPayload(MRRuntimeProtos.MRSplitProto splitProto,
                                                                                                     org.apache.hadoop.io.serializer.SerializationFactory serializationFactory)
                                                                                  throws IOException

Create an instance of InputSplit from the MRInput representation of a split.

Parameters:

splitProto - The MRRuntimeProtos.MRSplitProto instance representing the split

serializationFactory - the serialization mechanism used to write out the split

Returns:

an instance of the split

Throws:

IOException

createSplitProto

@InterfaceStability.Evolving
public static <T extends org.apache.hadoop.mapreduce.InputSplit> MRRuntimeProtos.MRSplitProto createSplitProto(T newSplit,
                                                                                                                           org.apache.hadoop.io.serializer.SerializationFactory serializationFactory)
                                                     throws IOException,
                                                            InterruptedException

Throws:

IOException

InterruptedException

createSplitProto

@InterfaceStability.Evolving
public static MRRuntimeProtos.MRSplitProto createSplitProto(org.apache.hadoop.mapred.InputSplit oldSplit)
                                                     throws IOException

Throws:

IOException

generateInputSplitsToMem

@InterfaceStability.Unstable
public static InputSplitInfoMem generateInputSplitsToMem(org.apache.hadoop.conf.Configuration conf,
                                                                     boolean groupSplits,
                                                                     int targetTasks)
                                                  throws IOException,
                                                         ClassNotFoundException,
                                                         InterruptedException

Generates Input splits and stores them in a MRRuntimeProtos.MRSplitsProto instance. Returns an instance of InputSplitInfoMem With grouping enabled, the eventual configuration used by the tasks, will have the user-specified InputFormat replaced by either TezGroupedSplitsInputFormat or TezGroupedSplitsInputFormat

Parameters:

conf - an instance of Configuration which is used to determine whether the mapred of mapreduce API is being used. This Configuration instance should also contain adequate information to be able to generate splits - like the InputFormat being used and related configuration.

groupSplits - whether to group the splits or not

targetTasks - the number of target tasks if grouping is enabled. Specify as 0 otherwise.

Returns:

an instance of InputSplitInfoMem which supports a subset of the APIs defined on InputSplitInfo

Throws:

IOException

ClassNotFoundException

InterruptedException

createMRInputPayloadWithGrouping

@InterfaceAudience.Private
protected static org.apache.tez.dag.api.UserPayload createMRInputPayloadWithGrouping(org.apache.hadoop.conf.Configuration conf)
                                                                              throws IOException

Called to specify that grouping of input splits be performed by Tez The conf should have the input format class configuration set to the TezGroupedSplitsInputFormat. The real input format class name should be passed as an argument to this method.

With grouping enabled, the eventual configuration used by the tasks, will have the user-specified InputFormat replaced by either TezGroupedSplitsInputFormat or TezGroupedSplitsInputFormat

Throws:

IOException

createMRInputPayload

@InterfaceAudience.Private
protected static org.apache.tez.dag.api.UserPayload createMRInputPayload(org.apache.hadoop.conf.Configuration conf,
                                                                                MRRuntimeProtos.MRSplitsProto mrSplitsProto)
                                                                  throws IOException

Throws:

IOException

getDagIndex

@InterfaceAudience.Public
public static int getDagIndex(org.apache.hadoop.conf.Configuration conf)

Parameters:

conf - configuration instance

Returns:

dag index

See Also:

InputContext#getDagIdentifier}

getDagIdString

@InterfaceAudience.Public
public static String getDagIdString(org.apache.hadoop.conf.Configuration conf)

Returns string representation of full DAG identifier

Parameters:

conf - configuration instance

Returns:

dag identifier

getVertexIndex

@InterfaceAudience.Public
public static int getVertexIndex(org.apache.hadoop.conf.Configuration conf)

* @see TaskContext.getTaskVertexIndex()

Parameters:

conf - configuration instance

Returns:

vertex index

getVertexIdString

@InterfaceAudience.Public
public static String getVertexIdString(org.apache.hadoop.conf.Configuration conf)

Returns string representation of full vertex identifier

Parameters:

conf - configuration instance

Returns:

vertex identifier

getTaskIndex

@InterfaceAudience.Public
public static int getTaskIndex(org.apache.hadoop.conf.Configuration conf)

Parameters:

conf - configuration instance

Returns:

task index

See Also:

InputContext#getTaskIndex}

getTaskIdString

@InterfaceAudience.Public
public static String getTaskIdString(org.apache.hadoop.conf.Configuration conf)

Returns string representation of full task identifier

Parameters:

conf - configuration instance

Returns:

task identifier

getTaskAttemptIndex

@InterfaceAudience.Public
public static int getTaskAttemptIndex(org.apache.hadoop.conf.Configuration conf)

Parameters:

conf - configuration instance

Returns:

task attempt index

See Also:

InputContext#getTaskAttemptNumber}

getTaskAttemptIdString

@InterfaceAudience.Public
public static String getTaskAttemptIdString(org.apache.hadoop.conf.Configuration conf)

Returns string representation of full task attempt identifier

Parameters:

conf - configuration instance

Returns:

task attempt identifier

getInputIndex

@InterfaceAudience.Public
public static int getInputIndex(org.apache.hadoop.conf.Configuration conf)

Parameters:

conf - configuration instance

Returns:

input index

See Also:

InputContext#getInputIndex}

getDagName

@InterfaceAudience.Public
public static String getDagName(org.apache.hadoop.conf.Configuration conf)

Parameters:

conf - configuration instance

Returns:

dag name

See Also:

InputContext#getDAGName}

getVertexName

@InterfaceAudience.Public
public static String getVertexName(org.apache.hadoop.conf.Configuration conf)

Parameters:

conf - configuration instance

Returns:

vertex name

See Also:

InputContext#getTaskVertexName}

getInputName

@InterfaceAudience.Public
public static String getInputName(org.apache.hadoop.conf.Configuration conf)

Parameters:

conf - configuration instance

Returns:

source name

See Also:

InputContext#getSourceVertexName}

getApplicationIdString

@InterfaceAudience.Public
public static String getApplicationIdString(org.apache.hadoop.conf.Configuration conf)

Parameters:

conf - configuration instance

Returns:

applicationId as a string

See Also:

InputContext#getApplicationId}

getUniqueIdentifier

@InterfaceAudience.Public
public static String getUniqueIdentifier(org.apache.hadoop.conf.Configuration conf)

Parameters:

conf - configuration instance

Returns:

unique identifier for the input

See Also:

InputContext#getUniqueIdentifier}

getDagAttemptNumber

@InterfaceAudience.Public
public static int getDagAttemptNumber(org.apache.hadoop.conf.Configuration conf)

Parameters:

conf - configuration instance

Returns:

attempt number

See Also:

InputContext#getDAGAttemptNumber}