org.apache.tez.mapreduce.hadoop

Class InputSplitInfoDisk

java.lang.Object

org.apache.tez.mapreduce.hadoop.InputSplitInfoDisk

All Implemented Interfaces:

InputSplitInfo

public class InputSplitInfoDisk
extends Object
implements InputSplitInfo

Information obtained by using the InputFormat class to generate the required InputSplit information files that in turn can be used to setup a DAG. The splitsFile and splitsMetaInfoFile need to be provided as LocalResources to the vertex in question. The numTasks represents the parallelism for the vertex and the taskLocationHints define the possible nodes on which the tasks should be run based on the location of the splits that will be processed by each task. The getSplitsProto method is not supported by this implementation.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.tez.mapreduce.hadoop.InputSplitInfo

InputSplitInfo.Type

Constructor Summary

Constructors

Constructor and Description
InputSplitInfoDisk(org.apache.hadoop.fs.Path splitsFile, org.apache.hadoop.fs.Path splitsMetaInfoFile, int numTasks, List<org.apache.tez.dag.api.TaskLocationHint> taskLocationHints, org.apache.hadoop.security.Credentials credentials)

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.security.Credentials	getCredentials() Get Credentials which may be required to access the splits.
org.apache.hadoop.mapreduce.InputSplit[]	getNewFormatSplits() Get new format splits.
int	getNumTasks() Get the number of splits that were generated.
org.apache.hadoop.mapred.InputSplit[]	getOldFormatSplits() Get old format splits.
org.apache.hadoop.fs.Path	getSplitsFile() Get the path to the splits file
org.apache.hadoop.fs.Path	getSplitsMetaInfoFile() Get the path to the splits meta info file
MRRuntimeProtos.MRSplitsProto	getSplitsProto() Get the splits proto
List<org.apache.tez.dag.api.TaskLocationHint>	getTaskLocationHints() Get the TaskLocationHints for each task
InputSplitInfo.Type	getType() Get the InputSplitInfo.Type of the InputSplitInfo
boolean	holdsNewFormatSplits() Check whether the current instance is using old / new format splits

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

InputSplitInfoDisk

public InputSplitInfoDisk(org.apache.hadoop.fs.Path splitsFile,
                  org.apache.hadoop.fs.Path splitsMetaInfoFile,
                  int numTasks,
                  List<org.apache.tez.dag.api.TaskLocationHint> taskLocationHints,
                  org.apache.hadoop.security.Credentials credentials)

Method Detail

getTaskLocationHints

public List<org.apache.tez.dag.api.TaskLocationHint> getTaskLocationHints()

Description copied from interface: InputSplitInfo

Get the TaskLocationHints for each task

Specified by:

getTaskLocationHints in interface InputSplitInfo

getSplitsMetaInfoFile

public org.apache.hadoop.fs.Path getSplitsMetaInfoFile()

Description copied from interface: InputSplitInfo

Get the path to the splits meta info file

Specified by:

getSplitsMetaInfoFile in interface InputSplitInfo

getSplitsFile

public org.apache.hadoop.fs.Path getSplitsFile()

Description copied from interface: InputSplitInfo

Get the path to the splits file

Specified by:

getSplitsFile in interface InputSplitInfo

getNumTasks

public int getNumTasks()

Description copied from interface: InputSplitInfo

Get the number of splits that were generated. Same as number of tasks that should be run for the vertex processing these set of splits.

Specified by:

getNumTasks in interface InputSplitInfo

getType

public InputSplitInfo.Type getType()

Description copied from interface: InputSplitInfo

Get the InputSplitInfo.Type of the InputSplitInfo

Specified by:

getType in interface InputSplitInfo

getSplitsProto

public MRRuntimeProtos.MRSplitsProto getSplitsProto()

Description copied from interface: InputSplitInfo

Get the splits proto

Specified by:

getSplitsProto in interface InputSplitInfo

getCredentials

public org.apache.hadoop.security.Credentials getCredentials()

Description copied from interface: InputSplitInfo

Get Credentials which may be required to access the splits.

Specified by:

getCredentials in interface InputSplitInfo

Returns:

Credentials which may be required to access the splits.

holdsNewFormatSplits

public boolean holdsNewFormatSplits()

Description copied from interface: InputSplitInfo

Check whether the current instance is using old / new format splits

Specified by:

holdsNewFormatSplits in interface InputSplitInfo

Returns:

true if using new format splits, false otherwise

getNewFormatSplits

public org.apache.hadoop.mapreduce.InputSplit[] getNewFormatSplits()

Description copied from interface: InputSplitInfo

Get new format splits. Should only be used if the mapreduce API is being used

Specified by:

getNewFormatSplits in interface InputSplitInfo

Returns:

array of mapreduce format splits

getOldFormatSplits

public org.apache.hadoop.mapred.InputSplit[] getOldFormatSplits()

Description copied from interface: InputSplitInfo

Get old format splits. Should only be used if the mapred API is being used

Specified by:

getOldFormatSplits in interface InputSplitInfo

Returns:

array of mapred format splits