org.apache.tez.dag.api

Interface VertexManagerPluginContext

@InterfaceStability.Unstable
 @InterfaceAudience.Public
public interface VertexManagerPluginContext

Object with API's to interact with the Tez execution engine

Method Summary

Modifier and Type	Method and Description
void	addRootInputEvents(String inputName, Collection<InputDataInformationEvent> events) Allows a VertexManagerPlugin to assign Events for Root Inputs For regular Event Routing changes - the EdgeManager should be configured via the setVertexParallelism method
void	doneReconfiguringVertex() Optional API.
int	getDAGAttemptNumber()
Map<String,EdgeProperty>	getInputVertexEdgeProperties() Get the edge properties on the input edges of this vertex.
Map<String,List<String>>	getInputVertexGroups() Get input vertex groups of this vertex, including vertex group name and all members vertex name
int	getNumClusterNodes() Get the number of nodes in the cluster
Map<String,EdgeProperty>	getOutputVertexEdgeProperties() Get the edge properties on the output edges of this vertex.
org.apache.hadoop.yarn.api.records.Resource	getTotalAvailableResource() Get the total resource allocated to this vertex.
UserPayload	getUserPayload() Get the payload set for the plugin
Set<String>	getVertexInputNames() Get the names of the non-vertex inputs of this vertex.
String	getVertexName() Get the name of the vertex
int	getVertexNumTasks(String vertexName) Get the number of tasks in the given vertex
VertexStatistics	getVertexStatistics(String vertexName) Get a VertexStatistics object to find out execution statistics about the given Vertex.
org.apache.hadoop.yarn.api.records.Resource	getVertexTaskResource() Get the resource allocated to a task of this vertex
void	reconfigureVertex(int parallelism, VertexLocationHint locationHint, Map<String,EdgeProperty> sourceEdgeProperties) API to reconfigure a Vertex by changing its task parallelism.
void	reconfigureVertex(int parallelism, VertexLocationHint locationHint, Map<String,EdgeProperty> sourceEdgeProperties, Map<String,InputSpecUpdate> rootInputSpecUpdate) API to reconfigure a Vertex that is reading root inputs based on the data read from the root inputs.
void	reconfigureVertex(Map<String,InputSpecUpdate> rootInputSpecUpdate, VertexLocationHint locationHint, int parallelism) API to reconfigure a Vertex that is reading root inputs based on the data read from the root inputs.
void	registerForVertexStateUpdates(String vertexName, Set<VertexState> stateSet) Register to get notifications on updates to the specified vertex.
void	scheduleTasks(List<org.apache.tez.dag.api.VertexManagerPluginContext.ScheduleTaskRequest> tasks) Notify the vertex to schedule the given tasks
void	scheduleVertexTasks(List<org.apache.tez.dag.api.VertexManagerPluginContext.TaskWithLocationHint> tasks) Deprecated.
void	sendEventToProcessor(Collection<org.apache.tez.runtime.api.events.CustomProcessorEvent> events, int taskId) Allows a VertexManagerPlugin to send events of custom payload to processor of a specific task of managed vertex It's up to user to make sure taskId is valid
void	setVertexLocationHint(VertexLocationHint locationHint) Set the placement hint for tasks in this vertex
void	setVertexParallelism(int parallelism, VertexLocationHint locationHint, Map<String,EdgeManagerPluginDescriptor> sourceEdgeManagers, Map<String,InputSpecUpdate> rootInputSpecUpdate) Deprecated.
void	vertexReconfigurationPlanned() Optional API.

Method Detail

getInputVertexEdgeProperties

Map<String,EdgeProperty> getInputVertexEdgeProperties()

Get the edge properties on the input edges of this vertex. The input edge is represented by the source vertex name

Returns:

Map of source vertex name and edge property

getOutputVertexEdgeProperties

Map<String,EdgeProperty> getOutputVertexEdgeProperties()

Get the edge properties on the output edges of this vertex. The output edge is represented by the destination vertex name

Returns:

Map of destination vertex name and edge property

getVertexStatistics

VertexStatistics getVertexStatistics(String vertexName)

Get a VertexStatistics object to find out execution statistics about the given Vertex.
This only provides point in time values for statistics (completed tasks) and must be called again to get updated values.

Parameters:

vertexName - Name of the Vertex

Returns:

VertexStatistics for the given vertex

getVertexName

String getVertexName()

Get the name of the vertex

Returns:

Vertex name

getUserPayload

UserPayload getUserPayload()

Get the payload set for the plugin

Returns:

user payload

getVertexNumTasks

int getVertexNumTasks(String vertexName)

Get the number of tasks in the given vertex

Parameters:

vertexName -

Returns:

Total number of tasks in this vertex

getVertexTaskResource

org.apache.hadoop.yarn.api.records.Resource getVertexTaskResource()

Get the resource allocated to a task of this vertex

Returns:

Resource

getTotalAvailableResource

org.apache.hadoop.yarn.api.records.Resource getTotalAvailableResource()

Get the total resource allocated to this vertex. If the DAG is running in a busy cluster then it may have no resources available dedicated to it. The DAG may divide its available resource among member vertices.

Returns:

Resource

getNumClusterNodes

int getNumClusterNodes()

Get the number of nodes in the cluster

Returns:

Number of nodes

setVertexParallelism

@Deprecated
void setVertexParallelism(int parallelism,
                                      @Nullable
                                      VertexLocationHint locationHint,
                                      @Nullable
                                      Map<String,EdgeManagerPluginDescriptor> sourceEdgeManagers,
                                      @Nullable
                                      Map<String,InputSpecUpdate> rootInputSpecUpdate)

Deprecated.

Set the new parallelism (number of tasks) of this vertex, Map of source (input) vertices and edge managers to change the event routing between the source tasks and the new destination tasks and the number of physical inputs for root inputs. This API can change the parallelism only once. Subsequent attempts will be disallowed

Parameters:

parallelism - New number of tasks in the vertex

locationHint - the placement policy for tasks.

sourceEdgeManagers - Edge Managers to be updated

rootInputSpecUpdate - Updated Root Input specifications, if any. If none specified, a default of 1 physical input is used

reconfigureVertex

void reconfigureVertex(int parallelism,
                       @Nullable
                       VertexLocationHint locationHint,
                       @Nullable
                       Map<String,EdgeProperty> sourceEdgeProperties,
                       @Nullable
                       Map<String,InputSpecUpdate> rootInputSpecUpdate)

API to reconfigure a Vertex that is reading root inputs based on the data read from the root inputs. Root inputs are external data sources that provide the initial data for the DAG and are added to the Vertex using the Vertex.addDataSource(String, DataSourceDescriptor) API. Typically, the parallelism of such vertices is determined at runtime by gathering information about the data source. This API may be used to set the parallelism of the vertex at runtime based on the data sources, as well as changing the specification for those inputs. In addition, changing parallelism is often accompanied by changing the EdgeProperty of the source Edge because event routing between source and destination tasks may need to be updated to account for the new task parallelism. This method can be called to update the parallelism multiple times until any of the tasks of the vertex have been scheduled (by invoking scheduleTasks(List). If needed, the original source edge properties may be obtained via getInputVertexEdgeProperties()

Parameters:

parallelism - New number of tasks in the vertex

locationHint - the placement policy for tasks specified at VertexLocationHints

sourceEdgeProperties - Map with Key=name of Edge to be updated and Value= EdgeProperty. The name of the Edge will be the corresponding source vertex name.

rootInputSpecUpdate - The key of the map is the name of the data source and the value is the updated InputSpecUpdate for that data source. If none specified, a default value is used. See InputSpecUpdate for details.

reconfigureVertex

void reconfigureVertex(int parallelism,
                       @Nullable
                       VertexLocationHint locationHint,
                       @Nullable
                       Map<String,EdgeProperty> sourceEdgeProperties)

API to reconfigure a Vertex by changing its task parallelism. Task parallelism is often accompanied by changing the EdgeProperty of the source Edge because event routing between source and destination tasks may need to be updated to account for the new task parallelism. This method can be called to update the parallelism multiple times until any of the tasks of the vertex have been scheduled (by invoking scheduleTasks(List). If needed, the original source edge properties may be obtained via getInputVertexEdgeProperties()

Parameters:

parallelism - New number of tasks in the vertex

locationHint - the placement policy for tasks specified at VertexLocationHints

sourceEdgeProperties - Map with Key=name of Edge to be updated and Value= EdgeProperty. The name of the Edge will be the corresponding source vertex name.

reconfigureVertex

void reconfigureVertex(@Nullable
                       Map<String,InputSpecUpdate> rootInputSpecUpdate,
                       @Nullable
                       VertexLocationHint locationHint,
                       int parallelism)

API to reconfigure a Vertex that is reading root inputs based on the data read from the root inputs. Root inputs are external data sources that provide the initial data for the DAG and are added to the Vertex using the Vertex.addDataSource(String, DataSourceDescriptor) API. Typically, the parallelism of such vertices is determined at runtime by gathering information about the data source. This API may be used to set the parallelism of the vertex at runtime based on the data sources, as well as changing the specification for those inputs.

Parameters:

rootInputSpecUpdate - The key of the map is the name of the data source and the value is the updated InputSpecUpdate for that data source. If none specified, a default value is used. See InputSpecUpdate for details.

locationHint - the placement policy for tasks specified at VertexLocationHints

parallelism - New number of tasks in the vertex

addRootInputEvents

void addRootInputEvents(String inputName,
                        Collection<InputDataInformationEvent> events)

Allows a VertexManagerPlugin to assign Events for Root Inputs For regular Event Routing changes - the EdgeManager should be configured via the setVertexParallelism method

Parameters:

inputName - The input name associated with the event

events - The list of Events to be assigned to various tasks belonging to the Vertex. The target index on individual events represents the task to which events need to be sent.

sendEventToProcessor

void sendEventToProcessor(Collection<org.apache.tez.runtime.api.events.CustomProcessorEvent> events,
                          int taskId)

Allows a VertexManagerPlugin to send events of custom payload to processor of a specific task of managed vertex It's up to user to make sure taskId is valid

Parameters:

events - events to be sent

taskId - id of a task of managed vertex

scheduleVertexTasks

@Deprecated
void scheduleVertexTasks(List<org.apache.tez.dag.api.VertexManagerPluginContext.TaskWithLocationHint> tasks)

Deprecated.

scheduleTasks

void scheduleTasks(List<org.apache.tez.dag.api.VertexManagerPluginContext.ScheduleTaskRequest> tasks)

Notify the vertex to schedule the given tasks

Parameters:

tasks - Identifier and metadata for the tasks to schedule

getVertexInputNames

@Nullable
Set<String> getVertexInputNames()

Get the names of the non-vertex inputs of this vertex. These are primary sources of data.

Returns:

Names of inputs to this vertex. Maybe null if there are no inputs

setVertexLocationHint

void setVertexLocationHint(VertexLocationHint locationHint)

Set the placement hint for tasks in this vertex

Parameters:

locationHint -

getDAGAttemptNumber

int getDAGAttemptNumber()

Returns:

DAG Attempt number

registerForVertexStateUpdates

void registerForVertexStateUpdates(String vertexName,
                                   @Nullable
                                   Set<VertexState> stateSet)

Register to get notifications on updates to the specified vertex. Notifications will be sent via VertexManagerPlugin.onVertexStateUpdated(org.apache.tez.dag.api.event.VertexStateUpdate) This method can only be invoked once. Duplicate invocations will result in an error.

Parameters:

vertexName - the vertex name for which notifications are required.

stateSet - the set of states for which notifications are required. null implies all

vertexReconfigurationPlanned

void vertexReconfigurationPlanned()

Optional API. No need to call this when the vertex is not fully defined to start with. E.g. vertex parallelism is not defined, or edges are not configured. In that case, Tez will assume that the vertex needs reconfiguration. If the vertex is already fully defined, but the VertexManagerPlugin wants to reconfigure the vertex, then it must use this API to inform Tez about its intention. Without invoking this method, it is invalid to re-configure the vertex if the vertex is already fully defined. This can be invoked at any time until VertexManagerPlugin.initialize() has completed. Its invalid to invoke this method after VertexManagerPlugin.initialize() has completed
If this API is invoked, then doneReconfiguringVertex() must be invoked after the VertexManagerPlugin is done reconfiguring the vertex, . Actions like scheduling tasks or sending events do not count as reconfiguration.

doneReconfiguringVertex

void doneReconfiguringVertex()

Optional API. This needs to be called only if vertexReconfigurationPlanned() has been invoked. This must be called after vertexReconfigurationPlanned() is called.

getInputVertexGroups

Map<String,List<String>> getInputVertexGroups()

Get input vertex groups of this vertex, including vertex group name and all members vertex name

Returns:

map whose key is vertex group name and value is list of members' name, or empty map if there is no input vertex group.