org.apache.tez.dag.library.vertexmanager

Class ShuffleVertexManager

java.lang.Object

org.apache.tez.dag.api.VertexManagerPlugin

org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager

@InterfaceAudience.Public
@InterfaceStability.Evolving
public class ShuffleVertexManager
extends org.apache.tez.dag.api.VertexManagerPlugin

Starts scheduling tasks when number of completed source tasks crosses slowStartMinSrcCompletionFraction and schedules all tasks when slowStartMaxSrcCompletionFraction is reached

Field Summary

Fields

Modifier and Type	Field and Description
static String	TEZ_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE The desired size of input per task.
static long	TEZ_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE_DEFAULT
static String	TEZ_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL Enables automatic parallelism determination for the vertex.
static boolean	TEZ_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL_DEFAULT
static String	TEZ_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION In case of a ScatterGather connection, once this fraction of source tasks have completed, all tasks on the current vertex can be scheduled.
static float	TEZ_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION_DEFAULT
static String	TEZ_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION In case of a ScatterGather connection, the fraction of source tasks which should complete before tasks for the current vertex are scheduled
static float	TEZ_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION_DEFAULT
static String	TEZ_SHUFFLE_VERTEX_MANAGER_MIN_TASK_PARALLELISM Automatic parallelism determination will not decrease parallelism below this value
static int	TEZ_SHUFFLE_VERTEX_MANAGER_MIN_TASK_PARALLELISM_DEFAULT

Constructor Summary

Constructors

Constructor and Description
ShuffleVertexManager(org.apache.tez.dag.api.VertexManagerPluginContext context)

Method Summary

Methods

Modifier and Type	Method and Description
static org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.ShuffleVertexManagerConfigBuilder	createConfigBuilder(org.apache.hadoop.conf.Configuration conf) Create a VertexManagerPluginDescriptor builder that can be used to configure the plugin.
void	initialize()
void	onRootVertexInitialized(String inputName, org.apache.tez.dag.api.InputDescriptor inputDescriptor, List<org.apache.tez.runtime.api.Event> events)
void	onSourceTaskCompleted(org.apache.tez.runtime.api.TaskAttemptIdentifier attempt)
void	onVertexManagerEventReceived(org.apache.tez.runtime.api.events.VertexManagerEvent vmEvent)
void	onVertexStarted(List<org.apache.tez.runtime.api.TaskAttemptIdentifier> completions)
void	onVertexStateUpdated(org.apache.tez.dag.api.event.VertexStateUpdate stateUpdate)

Methods inherited from class org.apache.tez.dag.api.VertexManagerPlugin

getContext, onSourceTaskCompleted, onVertexStarted

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEZ_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION

public static final String TEZ_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION

In case of a ScatterGather connection, the fraction of source tasks which should complete before tasks for the current vertex are scheduled

See Also:

Constant Field Values

TEZ_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION_DEFAULT

public static final float TEZ_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION_DEFAULT

See Also:

Constant Field Values

TEZ_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION

public static final String TEZ_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION

In case of a ScatterGather connection, once this fraction of source tasks have completed, all tasks on the current vertex can be scheduled. Number of tasks ready for scheduling on the current vertex scales linearly between min-fraction and max-fraction. Defaults to the greater of the default value or tez.shuffle-vertex-manager.min-src-fraction.

See Also:

Constant Field Values

TEZ_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION_DEFAULT

public static final float TEZ_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION_DEFAULT

See Also:

Constant Field Values

TEZ_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL

public static final String TEZ_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL

Enables automatic parallelism determination for the vertex. Based on input data statisitics the parallelism is decreased to a desired level.

See Also:

Constant Field Values

TEZ_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL_DEFAULT

public static final boolean TEZ_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL_DEFAULT

See Also:

Constant Field Values

TEZ_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE

public static final String TEZ_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE

The desired size of input per task. Parallelism will be changed to meet this criteria

See Also:

Constant Field Values

TEZ_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE_DEFAULT

public static final long TEZ_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE_DEFAULT

See Also:

Constant Field Values

TEZ_SHUFFLE_VERTEX_MANAGER_MIN_TASK_PARALLELISM

public static final String TEZ_SHUFFLE_VERTEX_MANAGER_MIN_TASK_PARALLELISM

Automatic parallelism determination will not decrease parallelism below this value

See Also:

Constant Field Values

TEZ_SHUFFLE_VERTEX_MANAGER_MIN_TASK_PARALLELISM_DEFAULT

public static final int TEZ_SHUFFLE_VERTEX_MANAGER_MIN_TASK_PARALLELISM_DEFAULT

See Also:

Constant Field Values

Constructor Detail

ShuffleVertexManager

public ShuffleVertexManager(org.apache.tez.dag.api.VertexManagerPluginContext context)

Method Detail

onVertexStarted

public void onVertexStarted(List<org.apache.tez.runtime.api.TaskAttemptIdentifier> completions)

Overrides:

onVertexStarted in class org.apache.tez.dag.api.VertexManagerPlugin

onSourceTaskCompleted

public void onSourceTaskCompleted(org.apache.tez.runtime.api.TaskAttemptIdentifier attempt)

Overrides:

onSourceTaskCompleted in class org.apache.tez.dag.api.VertexManagerPlugin

onVertexManagerEventReceived

public void onVertexManagerEventReceived(org.apache.tez.runtime.api.events.VertexManagerEvent vmEvent)

Specified by:

onVertexManagerEventReceived in class org.apache.tez.dag.api.VertexManagerPlugin

initialize

public void initialize()

Specified by:

initialize in class org.apache.tez.dag.api.VertexManagerPlugin

onVertexStateUpdated

public void onVertexStateUpdated(org.apache.tez.dag.api.event.VertexStateUpdate stateUpdate)

Overrides:

onVertexStateUpdated in class org.apache.tez.dag.api.VertexManagerPlugin

onRootVertexInitialized

public void onRootVertexInitialized(String inputName,
                           org.apache.tez.dag.api.InputDescriptor inputDescriptor,
                           List<org.apache.tez.runtime.api.Event> events)

Specified by:

onRootVertexInitialized in class org.apache.tez.dag.api.VertexManagerPlugin

createConfigBuilder

public static org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.ShuffleVertexManagerConfigBuilder createConfigBuilder(@Nullable
                                                                                                                  org.apache.hadoop.conf.Configuration conf)

Create a VertexManagerPluginDescriptor builder that can be used to configure the plugin.

Parameters:

conf - Configuration May be modified in place. May be null if the configuration parameters are to be set only via code. If configuration values may be changed at runtime via a config file then pass in a Configuration that is initialized from a config file. The parameters that are not overridden in code will be derived from the Configuration object.

Returns:

ShuffleVertexManager.ShuffleVertexManagerConfigBuilder