org.apache.tez.dag.api

Class EdgeManagerPluginOnDemand

java.lang.Object

org.apache.tez.dag.api.EdgeManagerPlugin

org.apache.tez.dag.api.EdgeManagerPluginOnDemand

@InterfaceAudience.Public
@InterfaceStability.Unstable
public abstract class EdgeManagerPluginOnDemand
extends EdgeManagerPlugin

This interface defines the routing of the event between tasks of producer and consumer vertices. The routing is bi-directional. Users can customize the routing by providing an implementation of this interface.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	EdgeManagerPluginOnDemand.EventRouteMetadata Class to provide routing metadata for Events to be routed between producer and consumer tasks.

Constructor Summary

Constructors

Constructor and Description
EdgeManagerPluginOnDemand(EdgeManagerPluginContext context) Create an instance of the EdgeManagerPluginOnDemand.

Method Summary

Methods

Modifier and Type	Method and Description
EdgeManagerPluginContext	getContext() Return the EdgeManagerPluginContext for this specific instance of the vertex manager.
abstract int	getNumDestinationConsumerTasks(int sourceTaskIndex) Get the number of destination tasks that consume data from the source task
abstract int	getNumDestinationTaskPhysicalInputs(int destinationTaskIndex) Get the number of physical inputs on the destination task
abstract int	getNumSourceTaskPhysicalOutputs(int sourceTaskIndex) Get the number of physical outputs on the source task
abstract void	initialize() Initializes the EdgeManagerPlugin.
abstract void	prepareForRouting() This method will be invoked just before routing of events will begin.
abstract EdgeManagerPluginOnDemand.EventRouteMetadata	routeCompositeDataMovementEventToDestination(int sourceTaskIndex, int destinationTaskIndex) The method provides the EdgeManagerPluginOnDemand.EventRouteMetadata to route a CompositeDataMovementEvent produced by the given source task to the given destination task.
void	routeDataMovementEventToDestination(DataMovementEvent event, int sourceTaskIndex, int sourceOutputIndex, Map<Integer,List<Integer>> destinationTaskAndInputIndices) Return the routing information to inform consumers about the source task output that is now available.
abstract EdgeManagerPluginOnDemand.EventRouteMetadata	routeDataMovementEventToDestination(int sourceTaskIndex, int sourceOutputIndex, int destinationTaskIndex) The method provides the EdgeManagerPluginOnDemand.EventRouteMetadata to route a DataMovementEvent produced by the given source task to the given destination task.
int	routeInputErrorEventToSource(InputReadErrorEvent event, int destinationTaskIndex, int destinationFailedInputIndex) Return the source task index to which to send the input error event
abstract int	routeInputErrorEventToSource(int destinationTaskIndex, int destinationFailedInputIndex) Return the source task index to which to send the input error event
abstract EdgeManagerPluginOnDemand.EventRouteMetadata	routeInputSourceTaskFailedEventToDestination(int sourceTaskIndex, int destinationTaskIndex) The method provides the EdgeManagerPluginOnDemand.EventRouteMetadata to route an InputFailedEvent produced by the given source task to the given destination task.
void	routeInputSourceTaskFailedEventToDestination(int sourceTaskIndex, Map<Integer,List<Integer>> destinationTaskAndInputIndices) Return the routing information to inform consumers about the failure of a source task whose outputs have been potentially lost.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EdgeManagerPluginOnDemand

public EdgeManagerPluginOnDemand(EdgeManagerPluginContext context)

Create an instance of the EdgeManagerPluginOnDemand. Classes extending this to create a EdgeManagerPluginOnDemand, must provide the same constructor so that Tez can create an instance of the class at runtime.

Parameters:

context - the context within which this EdgeManagerPluginOnDemand will run. Includes information like configuration which the user may have specified while setting up the edge.

Method Detail

initialize

public abstract void initialize()
                         throws Exception

Initializes the EdgeManagerPlugin. This method is called in the following circumstances

1. when initializing an EdgeManagerPlugin for the first time.

2. When an EdgeManagerPlugin is replaced at runtime. At this point, an EdgeManagerPlugin instance is created and setup by the user. The initialize method will be called with the original EdgeManagerPluginContext when the EdgeManagerPlugin is replaced.

Specified by:

initialize in class EdgeManagerPlugin

Throws:

Exception

prepareForRouting

public abstract void prepareForRouting()
                                throws Exception

This method will be invoked just before routing of events will begin. The plugin can use this opportunity to make any runtime initialization's that depend on the actual state of the DAG or vertices.

Throws:

Exception

getNumDestinationTaskPhysicalInputs

public abstract int getNumDestinationTaskPhysicalInputs(int destinationTaskIndex)
                                                 throws Exception

Get the number of physical inputs on the destination task

Specified by:

getNumDestinationTaskPhysicalInputs in class EdgeManagerPlugin

Parameters:

destinationTaskIndex - Index of destination task for which number of inputs is needed

Returns:

Number of physical inputs on the destination task

Throws:

Exception

getNumSourceTaskPhysicalOutputs

public abstract int getNumSourceTaskPhysicalOutputs(int sourceTaskIndex)
                                             throws Exception

Get the number of physical outputs on the source task

Specified by:

getNumSourceTaskPhysicalOutputs in class EdgeManagerPlugin

Parameters:

sourceTaskIndex - Index of the source task for which number of outputs is needed

Returns:

Number of physical outputs on the source task

Throws:

Exception

getNumDestinationConsumerTasks

public abstract int getNumDestinationConsumerTasks(int sourceTaskIndex)
                                            throws Exception

Get the number of destination tasks that consume data from the source task

Specified by:

getNumDestinationConsumerTasks in class EdgeManagerPlugin

Parameters:

sourceTaskIndex - Source task index

Throws:

Exception

routeInputErrorEventToSource

public abstract int routeInputErrorEventToSource(int destinationTaskIndex,
                               int destinationFailedInputIndex)
                                          throws Exception

Return the source task index to which to send the input error event

Parameters:

destinationTaskIndex - Destination task that reported the error

destinationFailedInputIndex - Index of the physical input on the destination task that reported the error

Returns:

Index of the source task that created the unavailable input

Throws:

Exception

routeDataMovementEventToDestination

@Nullable
public abstract EdgeManagerPluginOnDemand.EventRouteMetadata routeDataMovementEventToDestination(int sourceTaskIndex,
                                                                                        int sourceOutputIndex,
                                                                                        int destinationTaskIndex)
                                                                                          throws Exception

The method provides the EdgeManagerPluginOnDemand.EventRouteMetadata to route a DataMovementEvent produced by the given source task to the given destination task. The returned EdgeManagerPluginOnDemand.EventRouteMetadata should have the target input indices set to enable the routing. If the routing metadata is common across different events then the plugin can cache and reuse the same object.

Parameters:

sourceTaskIndex - The index of the task in the source vertex of this edge that produced a DataMovementEvent

sourceOutputIndex - Index of the physical output on the source task that produced the event

destinationTaskIndex -

Returns:

EdgeManagerPluginOnDemand.EventRouteMetadata with target indices set. Maybe null if the given destination task does not read input from the given source task.

Throws:

Exception

routeCompositeDataMovementEventToDestination

@Nullable
public abstract EdgeManagerPluginOnDemand.EventRouteMetadata routeCompositeDataMovementEventToDestination(int sourceTaskIndex,
                                                                                                 int destinationTaskIndex)
                                                                                                   throws Exception

The method provides the EdgeManagerPluginOnDemand.EventRouteMetadata to route a CompositeDataMovementEvent produced by the given source task to the given destination task. The returned EdgeManagerPluginOnDemand.EventRouteMetadata should have the target input indices and source output indices set to enable the routing. If the routing metadata is common across different events then the plugin can cache and reuse the same object.

Parameters:

sourceTaskIndex - The index of the task in the source vertex of this edge that produced a CompositeDataMovementEvent

destinationTaskIndex - The index of the task in the destination vertex of this edge

Returns:

EdgeManagerPluginOnDemand.EventRouteMetadata with source and target indices set. This may be null if the destination task does not read data from the source task.

Throws:

Exception

routeInputSourceTaskFailedEventToDestination

@Nullable
public abstract EdgeManagerPluginOnDemand.EventRouteMetadata routeInputSourceTaskFailedEventToDestination(int sourceTaskIndex,
                                                                                                 int destinationTaskIndex)
                                                                                                   throws Exception

The method provides the EdgeManagerPluginOnDemand.EventRouteMetadata to route an InputFailedEvent produced by the given source task to the given destination task. The returned EdgeManagerPluginOnDemand.EventRouteMetadata should have the target input indices set to enable the routing. If the routing metadata is common across different events then the plugin can cache and reuse the same object.

Parameters:

sourceTaskIndex - The index of the failed task in the source vertex of this edge.

destinationTaskIndex - The index of a task in the destination vertex of this edge.

Returns:

EdgeManagerPluginOnDemand.EventRouteMetadata with target indices set. Maybe null if the given destination task does not read input from the given source task.

Throws:

Exception

getContext

public EdgeManagerPluginContext getContext()

Return the EdgeManagerPluginContext for this specific instance of the vertex manager.

Overrides:

getContext in class EdgeManagerPlugin

Returns:

the EdgeManagerPluginContext for the input

routeDataMovementEventToDestination

public void routeDataMovementEventToDestination(DataMovementEvent event,
                                       int sourceTaskIndex,
                                       int sourceOutputIndex,
                                       Map<Integer,List<Integer>> destinationTaskAndInputIndices)
                                         throws Exception

Return the routing information to inform consumers about the source task output that is now available. The return map has the routing information. The event will be routed to every destination task index in the key of the map. Every physical input in the value for that task key will receive the input.

Specified by:

routeDataMovementEventToDestination in class EdgeManagerPlugin

Parameters:

event - Data movement event that contains the output information

sourceTaskIndex - Source task that produced the event

sourceOutputIndex - Index of the physical output on the source task that produced the event

destinationTaskAndInputIndices - Map via which the routing information is returned

Throws:

Exception

routeInputSourceTaskFailedEventToDestination

public void routeInputSourceTaskFailedEventToDestination(int sourceTaskIndex,
                                                Map<Integer,List<Integer>> destinationTaskAndInputIndices)
                                                  throws Exception

Return the routing information to inform consumers about the failure of a source task whose outputs have been potentially lost. The return map has the routing information. The failure notification event will be sent to every task index in the key of the map. Every physical input in the value for that task key will receive the failure notification. This method will be called once for every source task failure and information for all affected destinations must be provided in that invocation.

Specified by:

routeInputSourceTaskFailedEventToDestination in class EdgeManagerPlugin

Parameters:

sourceTaskIndex - Source task

destinationTaskAndInputIndices - Map via which the routing information is returned

Throws:

Exception

routeInputErrorEventToSource

public int routeInputErrorEventToSource(InputReadErrorEvent event,
                               int destinationTaskIndex,
                               int destinationFailedInputIndex)
                                 throws Exception

Return the source task index to which to send the input error event

Specified by:

routeInputErrorEventToSource in class EdgeManagerPlugin

Parameters:

event - Input read error event. Has more information about the error

destinationTaskIndex - Destination task that reported the error

destinationFailedInputIndex - Index of the physical input on the destination task that reported the error

Returns:

Index of the source task that created the unavailable input

Throws:

Exception