org.apache.tez.dag.library.vertexmanager

Class FairShuffleVertexManager

java.lang.Object

org.apache.tez.dag.api.VertexManagerPlugin

org.apache.tez.dag.library.vertexmanager.FairShuffleVertexManager

@InterfaceAudience.Public
 @InterfaceStability.Evolving
public class FairShuffleVertexManager
extends org.apache.tez.dag.api.VertexManagerPlugin

Fair routing based on partition size distribution to achieve optimal input size for any destination task thus reduce data skewness. By default the feature is turned off and it supports the regular shuffle like ShuffleVertexManager. When the feature is turned on, there are two routing types as defined in FairShuffleVertexManager.FairRoutingType. One is FairShuffleVertexManager.FairRoutingType.REDUCE_PARALLELISM which is similar to ShuffleVertexManager's auto reduce functionality. Another one is FairShuffleVertexManager.FairRoutingType.FAIR_PARALLELISM where each destination task can process a range of consecutive partitions from a range of consecutive source tasks.

Field Summary

Fields

Modifier and Type	Field and Description
static String	TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE The desired size of input per task.
static long	TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE_DEFAULT
static String	TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL Enables automatic parallelism determination for the vertex.
static String	TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL_DEFAULT
static String	TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION In case of a ScatterGather connection, once this fraction of source tasks have completed, all tasks on the current vertex can be scheduled.
static float	TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION_DEFAULT
static String	TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION In case of a ScatterGather connection, the fraction of source tasks which should complete before tasks for the current vertex are scheduled
static float	TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION_DEFAULT

Constructor Summary

Constructors

Constructor and Description
FairShuffleVertexManager(org.apache.tez.dag.api.VertexManagerPluginContext context)

Method Summary

Modifier and Type	Method and Description
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManagerBase.ReconfigVertexParams	computeRouting() Subclass might return null to indicate there is no new routing.
static org.apache.tez.dag.library.vertexmanager.FairShuffleVertexManager.FairShuffleVertexManagerConfigBuilder	createConfigBuilder(org.apache.hadoop.conf.Configuration conf) Create a VertexManagerPluginDescriptor builder that can be used to configure the plugin.
long[]	estimatePartitionSize()
void	initialize()
void	onRootVertexInitialized(String inputName, org.apache.tez.dag.api.InputDescriptor inputDescriptor, List<org.apache.tez.runtime.api.Event> events)
void	onSourceTaskCompleted(org.apache.tez.runtime.api.TaskAttemptIdentifier attempt)
void	onVertexManagerEventReceived(org.apache.tez.runtime.api.events.VertexManagerEvent vmEvent)
void	onVertexStarted(List<org.apache.tez.runtime.api.TaskAttemptIdentifier> completions)
protected void	onVertexStartedCheck()
void	onVertexStateUpdated(org.apache.tez.dag.api.event.VertexStateUpdate stateUpdate)

Methods inherited from class org.apache.tez.dag.api.VertexManagerPlugin

getContext, onSourceTaskCompleted, onVertexStarted

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE

public static final String TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE

The desired size of input per task. Parallelism will be changed to meet this criteria.

See Also:

Constant Field Values

TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE_DEFAULT

public static final long TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE_DEFAULT

TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL

public static final String TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL

Enables automatic parallelism determination for the vertex. Based on input data statistics the parallelism is adjusted to a desired level.

See Also:

Constant Field Values

TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL_DEFAULT

public static final String TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL_DEFAULT

TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION

public static final String TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION

In case of a ScatterGather connection, the fraction of source tasks which should complete before tasks for the current vertex are scheduled

See Also:

Constant Field Values

TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION_DEFAULT

public static final float TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION_DEFAULT

See Also:

Constant Field Values

TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION

public static final String TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION

In case of a ScatterGather connection, once this fraction of source tasks have completed, all tasks on the current vertex can be scheduled. Number of tasks ready for scheduling on the current vertex scales linearly between min-fraction and max-fraction. Defaults to the greater of the default value or tez.fair-shuffle-vertex-manager.min-src-fraction.

See Also:

Constant Field Values

TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION_DEFAULT

public static final float TEZ_FAIR_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION_DEFAULT

See Also:

Constant Field Values

Constructor Detail

FairShuffleVertexManager

public FairShuffleVertexManager(org.apache.tez.dag.api.VertexManagerPluginContext context)

Method Detail

onVertexStartedCheck

protected void onVertexStartedCheck()

estimatePartitionSize

public long[] estimatePartitionSize()

computeRouting

public org.apache.tez.dag.library.vertexmanager.ShuffleVertexManagerBase.ReconfigVertexParams computeRouting()

Subclass might return null to indicate there is no new routing.

createConfigBuilder

public static org.apache.tez.dag.library.vertexmanager.FairShuffleVertexManager.FairShuffleVertexManagerConfigBuilder createConfigBuilder(@Nullable
                                                                                                                                          org.apache.hadoop.conf.Configuration conf)

Create a VertexManagerPluginDescriptor builder that can be used to configure the plugin.

Parameters:

conf - Configuration May be modified in place. May be null if the configuration parameters are to be set only via code. If configuration values may be changed at runtime via a config file then pass in a Configuration that is initialized from a config file. The parameters that are not overridden in code will be derived from the Configuration object.

Returns:

FairShuffleVertexManager.FairShuffleVertexManagerConfigBuilder

onVertexStarted

public void onVertexStarted(List<org.apache.tez.runtime.api.TaskAttemptIdentifier> completions)

Overrides:

onVertexStarted in class org.apache.tez.dag.api.VertexManagerPlugin

onSourceTaskCompleted

public void onSourceTaskCompleted(org.apache.tez.runtime.api.TaskAttemptIdentifier attempt)

Overrides:

onSourceTaskCompleted in class org.apache.tez.dag.api.VertexManagerPlugin

onVertexManagerEventReceived

public void onVertexManagerEventReceived(org.apache.tez.runtime.api.events.VertexManagerEvent vmEvent)

Specified by:

onVertexManagerEventReceived in class org.apache.tez.dag.api.VertexManagerPlugin

initialize

public void initialize()

Specified by:

initialize in class org.apache.tez.dag.api.VertexManagerPlugin

onVertexStateUpdated

public void onVertexStateUpdated(org.apache.tez.dag.api.event.VertexStateUpdate stateUpdate)

Overrides:

onVertexStateUpdated in class org.apache.tez.dag.api.VertexManagerPlugin

onRootVertexInitialized

public void onRootVertexInitialized(String inputName,
                                    org.apache.tez.dag.api.InputDescriptor inputDescriptor,
                                    List<org.apache.tez.runtime.api.Event> events)

Specified by:

onRootVertexInitialized in class org.apache.tez.dag.api.VertexManagerPlugin