org.apache.tez.mapreduce.input

Class MRInput

java.lang.Object

org.apache.tez.runtime.api.AbstractLogicalInput

org.apache.tez.mapreduce.input.base.MRInputBase

org.apache.tez.mapreduce.input.MRInput

All Implemented Interfaces:

org.apache.tez.runtime.api.Input, org.apache.tez.runtime.api.InputFrameworkInterface, org.apache.tez.runtime.api.LogicalInput, org.apache.tez.runtime.api.LogicalInputFrameworkInterface

Direct Known Subclasses:

MRInputLegacy

@InterfaceAudience.Public
public class MRInput
extends MRInputBase

MRInput is an Input which provides key/values pairs for the consumer. It is compatible with all standard Apache Hadoop MapReduce InputFormat implementations. This class is not meant to be extended by external projects.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	MRInput.MRInputConfigBuilder Helper class to configure MRInput

Field Summary

Fields

Modifier and Type	Field and Description
protected MRReader	mrReader
protected org.apache.hadoop.mapreduce.split.JobSplit.TaskSplitIndex	splitMetaInfo
static String	TEZ_MAPREDUCE_APPLICATION_ID
static String	TEZ_MAPREDUCE_DAG_ATTEMPT_NUMBER
static String	TEZ_MAPREDUCE_DAG_INDEX
static String	TEZ_MAPREDUCE_DAG_NAME
static String	TEZ_MAPREDUCE_INPUT_INDEX
static String	TEZ_MAPREDUCE_INPUT_NAME
static String	TEZ_MAPREDUCE_TASK_ATTEMPT_INDEX
static String	TEZ_MAPREDUCE_TASK_INDEX
static String	TEZ_MAPREDUCE_UNIQUE_IDENTIFIER
static String	TEZ_MAPREDUCE_VERTEX_INDEX
static String	TEZ_MAPREDUCE_VERTEX_NAME

Fields inherited from class org.apache.tez.mapreduce.input.base.MRInputBase

inputRecordCounter, jobConf, useNewApi

Constructor Summary

Constructors

Constructor and Description
MRInput(org.apache.tez.runtime.api.InputContext inputContext, int numPhysicalInputs)

Method Summary

Methods

Modifier and Type	Method and Description
List<org.apache.tez.runtime.api.Event>	close()
static MRInput.MRInputConfigBuilder	createConfigBuilder(org.apache.hadoop.conf.Configuration conf, Class<?> inputFormat) Create an MRInput.MRInputConfigBuilder
static MRInput.MRInputConfigBuilder	createConfigBuilder(org.apache.hadoop.conf.Configuration conf, Class<?> inputFormat, String inputPaths) Create an MRInput.MRInputConfigBuilder for FileInputFormat or FileInputFormat format based InputFormats.
org.apache.hadoop.conf.Configuration	getConfigUpdates() MRInput sets some additional parameters like split location when using the new API.
float	getProgress()
org.apache.tez.runtime.library.api.KeyValueReader	getReader() Returns a KeyValueReader that can be used to read Map Reduce compatible key value data.
void	handleEvents(List<org.apache.tez.runtime.api.Event> inputEvents)
List<org.apache.tez.runtime.api.Event>	initialize()
void	start()

Methods inherited from class org.apache.tez.runtime.api.AbstractLogicalInput

getContext, getNumPhysicalInputs

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEZ_MAPREDUCE_DAG_INDEX

@InterfaceAudience.Private
public static final String TEZ_MAPREDUCE_DAG_INDEX

See Also:

Constant Field Values

TEZ_MAPREDUCE_DAG_NAME

@InterfaceAudience.Private
public static final String TEZ_MAPREDUCE_DAG_NAME

See Also:

Constant Field Values

TEZ_MAPREDUCE_VERTEX_INDEX

@InterfaceAudience.Private
public static final String TEZ_MAPREDUCE_VERTEX_INDEX

See Also:

Constant Field Values

TEZ_MAPREDUCE_VERTEX_NAME

@InterfaceAudience.Private
public static final String TEZ_MAPREDUCE_VERTEX_NAME

See Also:

Constant Field Values

TEZ_MAPREDUCE_TASK_INDEX

@InterfaceAudience.Private
public static final String TEZ_MAPREDUCE_TASK_INDEX

See Also:

Constant Field Values

TEZ_MAPREDUCE_TASK_ATTEMPT_INDEX

@InterfaceAudience.Private
public static final String TEZ_MAPREDUCE_TASK_ATTEMPT_INDEX

See Also:

Constant Field Values

TEZ_MAPREDUCE_INPUT_INDEX

@InterfaceAudience.Private
public static final String TEZ_MAPREDUCE_INPUT_INDEX

See Also:

Constant Field Values

TEZ_MAPREDUCE_INPUT_NAME

@InterfaceAudience.Private
public static final String TEZ_MAPREDUCE_INPUT_NAME

See Also:

Constant Field Values

TEZ_MAPREDUCE_APPLICATION_ID

@InterfaceAudience.Private
public static final String TEZ_MAPREDUCE_APPLICATION_ID

See Also:

Constant Field Values

TEZ_MAPREDUCE_UNIQUE_IDENTIFIER

@InterfaceAudience.Private
public static final String TEZ_MAPREDUCE_UNIQUE_IDENTIFIER

See Also:

Constant Field Values

TEZ_MAPREDUCE_DAG_ATTEMPT_NUMBER

@InterfaceAudience.Private
public static final String TEZ_MAPREDUCE_DAG_ATTEMPT_NUMBER

See Also:

Constant Field Values

mrReader

protected MRReader mrReader

splitMetaInfo

protected org.apache.hadoop.mapreduce.split.JobSplit.TaskSplitIndex splitMetaInfo

Constructor Detail

MRInput

public MRInput(org.apache.tez.runtime.api.InputContext inputContext,
       int numPhysicalInputs)

Method Detail

createConfigBuilder

public static MRInput.MRInputConfigBuilder createConfigBuilder(org.apache.hadoop.conf.Configuration conf,
                                               @Nullable
                                               Class<?> inputFormat)

Create an MRInput.MRInputConfigBuilder

The preferred usage model is to provide all of the parameters, and use methods to configure the Input.

For legacy applications, which may already have a fully configured Configuration instance, the inputFormat can be specified as null

Parameters:

conf - Configuration for the MRInput. This configuration instance will be modified in place

inputFormat - InputFormat derived class. This can be null. If the InputFormat specified is null, the provided configuration should be complete.

Returns:

MRInput.MRInputConfigBuilder

createConfigBuilder

public static MRInput.MRInputConfigBuilder createConfigBuilder(org.apache.hadoop.conf.Configuration conf,
                                               @Nullable
                                               Class<?> inputFormat,
                                               @Nullable
                                               String inputPaths)

Create an MRInput.MRInputConfigBuilder for FileInputFormat or FileInputFormat format based InputFormats.

The preferred usage model is to provide all of the parameters, and use methods to configure the Input.

For legacy applications, which may already have a fully configured Configuration instance, the inputFormat and inputPath can be specified as null

Parameters:

conf - Configuration for the MRInput. This configuration instance will be modified in place

inputFormat - InputFormat derived class. This can be null. If the InputFormat specified is null, the provided configuration should be complete.

inputPaths - Comma separated input paths

Returns:

MRInput.MRInputConfigBuilder

initialize

public List<org.apache.tez.runtime.api.Event> initialize()
                                                  throws IOException

Specified by:

initialize in interface org.apache.tez.runtime.api.InputFrameworkInterface

Overrides:

initialize in class MRInputBase

Throws:

IOException

start

public void start()

getReader

public org.apache.tez.runtime.library.api.KeyValueReader getReader()
                                                            throws IOException

Returns a KeyValueReader that can be used to read Map Reduce compatible key value data. An exception will be thrown if next() is invoked after false, either from the framework or from the underlying InputFormat

Specified by:

getReader in interface org.apache.tez.runtime.api.Input

Overrides:

getReader in class MRInputBase

Throws:

IOException

handleEvents

public void handleEvents(List<org.apache.tez.runtime.api.Event> inputEvents)
                  throws Exception

Throws:

Exception

close

public List<org.apache.tez.runtime.api.Event> close()
                                             throws IOException

Throws:

IOException

getConfigUpdates

public org.apache.hadoop.conf.Configuration getConfigUpdates()

MRInput sets some additional parameters like split location when using the new API. This methods returns the list of additional updates, and should be used by Processors using the old MapReduce API with MRInput.

Returns:

the additional fields set by MRInput

getProgress

public float getProgress()
                  throws IOException,
                         InterruptedException

Throws:

IOException

InterruptedException