org.apache.hadoop.mapreduce.split

Class TezMapReduceSplitsGrouper

java.lang.Object

org.apache.hadoop.mapreduce.split.TezMapReduceSplitsGrouper

@InterfaceAudience.Public
@InterfaceStability.Evolving
public class TezMapReduceSplitsGrouper
extends Object

Helper that provides a grouping of input splits based on multiple parameters. It creates TezGroupedSplit to wrap the each group of real InputSplits

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TezMapReduceSplitsGrouper.TezMRSplitsGrouperConfigBuilder

Field Summary

Fields

Modifier and Type	Field and Description
static String	TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION This factor is used to decrease the per group desired (length and count) limits for groups created by combining splits within a rack.
static float	TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION_DEFAULT
static String	TEZ_GROUPING_REPEATABLE Repeated invocations of grouping on the same splits with the same parameters will produce the same groups.
static boolean	TEZ_GROUPING_REPEATABLE_DEFAULT
static String	TEZ_GROUPING_SPLIT_BY_COUNT Limit the number of splits in a group by the number of splits in the group
static boolean	TEZ_GROUPING_SPLIT_BY_COUNT_DEFAULT
static String	TEZ_GROUPING_SPLIT_BY_LENGTH Limit the number of splits in a group by the total length of the splits in the group
static boolean	TEZ_GROUPING_SPLIT_BY_LENGTH_DEFAULT
static String	TEZ_GROUPING_SPLIT_COUNT Specify the number of splits desired to be created
static String	TEZ_GROUPING_SPLIT_MAX_SIZE Upper bound on the size (in bytes) of a grouped split, to avoid generating excessively large splits.
static long	TEZ_GROUPING_SPLIT_MAX_SIZE_DEFAULT
static String	TEZ_GROUPING_SPLIT_MIN_SIZE Lower bound on the size (in bytes) of a grouped split, to avoid generating too many small splits.
static long	TEZ_GROUPING_SPLIT_MIN_SIZE_DEFAULT
static String	TEZ_GROUPING_SPLIT_WAVES The multiplier for available queue capacity when determining number of tasks for a Vertex.
static float	TEZ_GROUPING_SPLIT_WAVES_DEFAULT

Constructor Summary

Constructors

Constructor and Description
TezMapReduceSplitsGrouper()

Method Summary

Methods

Modifier and Type	Method and Description
static TezMapReduceSplitsGrouper.TezMRSplitsGrouperConfigBuilder	createConfigBuilder(org.apache.hadoop.conf.Configuration conf) Builder that can be used to configure grouping in Tez
List<org.apache.hadoop.mapreduce.InputSplit>	getGroupedSplits(org.apache.hadoop.conf.Configuration conf, List<org.apache.hadoop.mapreduce.InputSplit> originalSplits, int desiredNumSplits, String wrappedInputFormatName)
List<org.apache.hadoop.mapreduce.InputSplit>	getGroupedSplits(org.apache.hadoop.conf.Configuration conf, List<org.apache.hadoop.mapreduce.InputSplit> originalSplits, int desiredNumSplits, String wrappedInputFormatName, SplitSizeEstimator estimator)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEZ_GROUPING_SPLIT_COUNT

public static final String TEZ_GROUPING_SPLIT_COUNT

Specify the number of splits desired to be created

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_BY_LENGTH

public static final String TEZ_GROUPING_SPLIT_BY_LENGTH

Limit the number of splits in a group by the total length of the splits in the group

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_BY_LENGTH_DEFAULT

public static final boolean TEZ_GROUPING_SPLIT_BY_LENGTH_DEFAULT

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_BY_COUNT

public static final String TEZ_GROUPING_SPLIT_BY_COUNT

Limit the number of splits in a group by the number of splits in the group

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_BY_COUNT_DEFAULT

public static final boolean TEZ_GROUPING_SPLIT_BY_COUNT_DEFAULT

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_WAVES

public static final String TEZ_GROUPING_SPLIT_WAVES

The multiplier for available queue capacity when determining number of tasks for a Vertex. 1.7 with 100% queue available implies generating a number of tasks roughly equal to 170% of the available containers on the queue. This enables multiple waves of mappers where the final wave is slightly smaller than the remaining waves. The gap helps overlap the final wave with any slower tasks from previous waves and tries to hide the delays from the slower tasks. Good values for this are 1.7, 2.7, 3.7 etc. Increase the number of waves to make the tasks smaller or shorter.

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_WAVES_DEFAULT

public static final float TEZ_GROUPING_SPLIT_WAVES_DEFAULT

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_MAX_SIZE

public static final String TEZ_GROUPING_SPLIT_MAX_SIZE

Upper bound on the size (in bytes) of a grouped split, to avoid generating excessively large splits.

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_MAX_SIZE_DEFAULT

public static final long TEZ_GROUPING_SPLIT_MAX_SIZE_DEFAULT

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_MIN_SIZE

public static final String TEZ_GROUPING_SPLIT_MIN_SIZE

Lower bound on the size (in bytes) of a grouped split, to avoid generating too many small splits.

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_MIN_SIZE_DEFAULT

public static final long TEZ_GROUPING_SPLIT_MIN_SIZE_DEFAULT

See Also:

Constant Field Values

TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION

public static final String TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION

This factor is used to decrease the per group desired (length and count) limits for groups created by combining splits within a rack. Since reading this split involves reading data intra rack, the group is made smaller to cover up for the increased latencies in doing intra rack reads. The value should be a fraction <= 1.

See Also:

Constant Field Values

TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION_DEFAULT

public static final float TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION_DEFAULT

See Also:

Constant Field Values

TEZ_GROUPING_REPEATABLE

public static final String TEZ_GROUPING_REPEATABLE

Repeated invocations of grouping on the same splits with the same parameters will produce the same groups. This may help in cache reuse but may cause hot-spotting on nodes when there are a large number of jobs reading the same hot data. True by default.

See Also:

Constant Field Values

TEZ_GROUPING_REPEATABLE_DEFAULT

public static final boolean TEZ_GROUPING_REPEATABLE_DEFAULT

See Also:

Constant Field Values

Constructor Detail

TezMapReduceSplitsGrouper

public TezMapReduceSplitsGrouper()

Method Detail

getGroupedSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getGroupedSplits(org.apache.hadoop.conf.Configuration conf,
                                                            List<org.apache.hadoop.mapreduce.InputSplit> originalSplits,
                                                            int desiredNumSplits,
                                                            String wrappedInputFormatName)
                                                              throws IOException,
                                                                     InterruptedException

Throws:

IOException

InterruptedException

getGroupedSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getGroupedSplits(org.apache.hadoop.conf.Configuration conf,
                                                            List<org.apache.hadoop.mapreduce.InputSplit> originalSplits,
                                                            int desiredNumSplits,
                                                            String wrappedInputFormatName,
                                                            SplitSizeEstimator estimator)
                                                              throws IOException,
                                                                     InterruptedException

Throws:

IOException

InterruptedException

createConfigBuilder

public static TezMapReduceSplitsGrouper.TezMRSplitsGrouperConfigBuilder createConfigBuilder(org.apache.hadoop.conf.Configuration conf)

Builder that can be used to configure grouping in Tez

Parameters:

conf - Configuration This will be modified in place. If configuration values may be changed at runtime via a config file then pass in a Configuration that is initialized from a config file. The parameters that are not overridden in code will be derived from the Configuration object.

Returns:

TezMapReduceSplitsGrouper.TezMRSplitsGrouperConfigBuilder