org.apache.tez.dag.api

Class EdgeManagerPlugin

java.lang.Object

org.apache.tez.dag.api.EdgeManagerPlugin

Direct Known Subclasses:

EdgeManagerPluginOnDemand

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public abstract class EdgeManagerPlugin
extends Object

This interface defines the routing of the event between tasks of producer and consumer vertices. The routing is bi-directional. Users can customize the routing by providing an implementation of this interface.

Constructor Summary

Constructors

Constructor and Description
EdgeManagerPlugin(EdgeManagerPluginContext context) Create an instance of the EdgeManagerPlugin.

Method Summary

Modifier and Type	Method and Description
EdgeManagerPluginContext	getContext() Return the EdgeManagerPluginContext for this specific instance of the vertex manager.
abstract int	getNumDestinationConsumerTasks(int sourceTaskIndex) Get the number of destination tasks that consume data from the source task
abstract int	getNumDestinationTaskPhysicalInputs(int destinationTaskIndex) Get the number of physical inputs on the destination task
abstract int	getNumSourceTaskPhysicalOutputs(int sourceTaskIndex) Get the number of physical outputs on the source task
abstract void	initialize() Initializes the EdgeManagerPlugin.
abstract void	routeDataMovementEventToDestination(DataMovementEvent event, int sourceTaskIndex, int sourceOutputIndex, Map<Integer,List<Integer>> destinationTaskAndInputIndices) Return the routing information to inform consumers about the source task output that is now available.
abstract int	routeInputErrorEventToSource(InputReadErrorEvent event, int destinationTaskIndex, int destinationFailedInputIndex) Return the source task index to which to send the input error event
abstract void	routeInputSourceTaskFailedEventToDestination(int sourceTaskIndex, Map<Integer,List<Integer>> destinationTaskAndInputIndices) Return the routing information to inform consumers about the failure of a source task whose outputs have been potentially lost.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EdgeManagerPlugin

public EdgeManagerPlugin(EdgeManagerPluginContext context)

Create an instance of the EdgeManagerPlugin. Classes extending this to create a EdgeManagerPlugin, must provide the same constructor so that Tez can create an instance of the class at runtime.

Parameters:

context - the context within which this EdgeManagerPlugin will run. Includes information like configuration which the user may have specified while setting up the edge.

Method Detail

initialize

public abstract void initialize()
                         throws Exception

Initializes the EdgeManagerPlugin. This method is called in the following circumstances

1. when initializing an EdgeManagerPlugin for the first time.

2. When an EdgeManagerPlugin is replaced at runtime. At this point, an EdgeManagerPlugin instance is created and setup by the user. The initialize method will be called with the original EdgeManagerPluginContext when the EdgeManagerPlugin is replaced.

Throws:

Exception

getNumDestinationTaskPhysicalInputs

public abstract int getNumDestinationTaskPhysicalInputs(int destinationTaskIndex)
                                                 throws Exception

Get the number of physical inputs on the destination task

Parameters:

destinationTaskIndex - Index of destination task for which number of inputs is needed

Returns:

Number of physical inputs on the destination task

Throws:

Exception

getNumSourceTaskPhysicalOutputs

public abstract int getNumSourceTaskPhysicalOutputs(int sourceTaskIndex)
                                             throws Exception

Get the number of physical outputs on the source task

Parameters:

sourceTaskIndex - Index of the source task for which number of outputs is needed

Returns:

Number of physical outputs on the source task

Throws:

Exception

routeDataMovementEventToDestination

public abstract void routeDataMovementEventToDestination(DataMovementEvent event,
                                                         int sourceTaskIndex,
                                                         int sourceOutputIndex,
                                                         Map<Integer,List<Integer>> destinationTaskAndInputIndices)
                                                  throws Exception

Return the routing information to inform consumers about the source task output that is now available. The return map has the routing information. The event will be routed to every destination task index in the key of the map. Every physical input in the value for that task key will receive the input.

Parameters:

event - Data movement event that contains the output information

sourceTaskIndex - Source task that produced the event

sourceOutputIndex - Index of the physical output on the source task that produced the event

destinationTaskAndInputIndices - Map via which the routing information is returned

Throws:

Exception

routeInputSourceTaskFailedEventToDestination

public abstract void routeInputSourceTaskFailedEventToDestination(int sourceTaskIndex,
                                                                  Map<Integer,List<Integer>> destinationTaskAndInputIndices)
                                                           throws Exception

Return the routing information to inform consumers about the failure of a source task whose outputs have been potentially lost. The return map has the routing information. The failure notification event will be sent to every task index in the key of the map. Every physical input in the value for that task key will receive the failure notification. This method will be called once for every source task failure and information for all affected destinations must be provided in that invocation.

Parameters:

sourceTaskIndex - Source task

destinationTaskAndInputIndices - Map via which the routing information is returned

Throws:

Exception

getNumDestinationConsumerTasks

public abstract int getNumDestinationConsumerTasks(int sourceTaskIndex)
                                            throws Exception

Get the number of destination tasks that consume data from the source task

Parameters:

sourceTaskIndex - Source task index

Throws:

Exception

routeInputErrorEventToSource

public abstract int routeInputErrorEventToSource(InputReadErrorEvent event,
                                                 int destinationTaskIndex,
                                                 int destinationFailedInputIndex)
                                          throws Exception

Return the source task index to which to send the input error event

Parameters:

event - Input read error event. Has more information about the error

destinationTaskIndex - Destination task that reported the error

destinationFailedInputIndex - Index of the physical input on the destination task that reported the error

Returns:

Index of the source task that created the unavailable input

Throws:

Exception

getContext

public EdgeManagerPluginContext getContext()

Return the EdgeManagerPluginContext for this specific instance of the vertex manager.

Returns:

the EdgeManagerPluginContext for the input