org.apache.tez.dag.api

Class Vertex

java.lang.Object

org.apache.tez.dag.api.Vertex

Direct Known Subclasses:

PreWarmVertex

@InterfaceAudience.Public
public class Vertex
extends Object

Defines a vertex in the DAG. It represents the application logic that processes and transforms the input data to create the output data. The vertex represents the template from which tasks are created to execute the application in parallel across a distributed execution environment.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Vertex.VertexExecutionContext The execution context for a running vertex.

Method Summary

Methods

Modifier and Type	Method and Description
Vertex	addDataSink(String outputName, DataSinkDescriptor dataSinkDescriptor) Specifies an external data sink for a Vertex.
Vertex	addDataSource(String inputName, DataSourceDescriptor dataSourceDescriptor) Specifies an external data source for a Vertex.
Vertex	addTaskLocalFiles(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localFiles) Set the files etc that must be provided to the tasks of this vertex
static Vertex	create(String vertexName, ProcessorDescriptor processorDescriptor) Create a new vertex with the given name.
static Vertex	create(String vertexName, ProcessorDescriptor processorDescriptor, int parallelism) Create a new vertex with the given name and parallelism.
static Vertex	create(String vertexName, ProcessorDescriptor processorDescriptor, int parallelism, org.apache.hadoop.yarn.api.records.Resource taskResource) Create a new vertex with the given name.
boolean	equals(Object obj)
Map<String,String>	getConf()
List<DataSinkDescriptor>	getDataSinks()
List<DataSourceDescriptor>	getDataSources()
List<Vertex>	getInputVertices() Get the input vertices for this vertex
String	getName() Get the vertex name
List<Vertex>	getOutputVertices() Get the output vertices for this vertex
int	getParallelism() Get the specified number of tasks specified to run in this vertex.
ProcessorDescriptor	getProcessorDescriptor() Get the vertex task processor descriptor
Map<String,String>	getTaskEnvironment() Get the environment variables of the tasks
String	getTaskLaunchCmdOpts() Get the launch command opts for tasks in this vertex
Map<String,org.apache.hadoop.yarn.api.records.LocalResource>	getTaskLocalFiles() Get the files etc that must be provided by the tasks of this vertex
org.apache.hadoop.yarn.api.records.Resource	getTaskResource() Get the resources for the vertex
Vertex.VertexExecutionContext	getVertexExecutionContext()
int	hashCode()
Vertex	setConf(String property, String value) This is currently used to setup additional configuration parameters which will be available in the Vertex specific configuration used in the AppMaster.
Vertex	setExecutionContext(Vertex.VertexExecutionContext vertexExecutionContext) Sets the execution context for this Vertex - i.e.
Vertex	setLocationHint(VertexLocationHint locationHint) Specify location hints for the tasks of this vertex.
Vertex	setTaskEnvironment(Map<String,String> environment) Set the Key-Value pairs of environment variables for tasks of this vertex.
Vertex	setTaskLaunchCmdOpts(String cmdOpts) Set the command opts for tasks of this vertex.
Vertex	setVertexManagerPlugin(VertexManagerPluginDescriptor vertexManagerPluginDescriptor) Specifies a VertexManagerPlugin for the vertex.
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Method Detail

create

public static Vertex create(String vertexName,
            ProcessorDescriptor processorDescriptor,
            int parallelism,
            org.apache.hadoop.yarn.api.records.Resource taskResource)

Create a new vertex with the given name.

Parameters:

vertexName - Name of the vertex

processorDescriptor - Description of the processor that is executed in every task of this vertex

parallelism - Number of tasks in this vertex. Set to -1 if this is going to be decided at runtime. Parallelism may change at runtime due to graph reconfigurations.

taskResource - Physical resources like memory/cpu thats used by each task of this vertex.

Returns:

a new Vertex with the given parameters

create

public static Vertex create(String vertexName,
            ProcessorDescriptor processorDescriptor)

Create a new vertex with the given name.
The vertex task resource will be picked from configuration
The vertex parallelism will be inferred. If it cannot be inferred then an error will be reported. This constructor may be used for vertices that have data sources, or connected via 1-1 edges or have runtime parallelism estimation via data source initializers or vertex managers. Calling this constructor is equivalent to calling Vertex(String, ProcessorDescriptor, int) with the parallelism set to -1.

Parameters:

vertexName - Name of the vertex

processorDescriptor - Description of the processor that is executed in every task of this vertex

Returns:

a new Vertex with the given parameters

create

public static Vertex create(String vertexName,
            ProcessorDescriptor processorDescriptor,
            int parallelism)

Create a new vertex with the given name and parallelism.
The vertex task resource will be picked from configuration TezConfiguration.TEZ_TASK_RESOURCE_MEMORY_MB & TezConfiguration.TEZ_TASK_RESOURCE_CPU_VCORES Applications that want more control over their task resource specification may create their own logic to determine task resources and use Vertex(String, ProcessorDescriptor, int, Resource) to create the Vertex.

Parameters:

vertexName - Name of the vertex

processorDescriptor - Description of the processor that is executed in every task of this vertex

parallelism - Number of tasks in this vertex. Set to -1 if this is going to be decided at runtime. Parallelism may change at runtime due to graph reconfigurations.

Returns:

a new Vertex with the given parameters

getName

public String getName()

Get the vertex name

Returns:

vertex name

getProcessorDescriptor

public ProcessorDescriptor getProcessorDescriptor()

Get the vertex task processor descriptor

Returns:

process descriptor

getParallelism

public int getParallelism()

Get the specified number of tasks specified to run in this vertex. It may be -1 if the parallelism is defined at runtime. Parallelism may change at runtime

Returns:

vertex parallelism

getTaskResource

public org.apache.hadoop.yarn.api.records.Resource getTaskResource()

Get the resources for the vertex

Returns:

the physical resources like pcu/memory of each vertex task

setLocationHint

public Vertex setLocationHint(VertexLocationHint locationHint)

Specify location hints for the tasks of this vertex. Hints must be specified for all tasks as defined by the parallelism

Parameters:

locationHint - list of locations for each task in the vertex

Returns:

this Vertex

addTaskLocalFiles

public Vertex addTaskLocalFiles(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localFiles)

Set the files etc that must be provided to the tasks of this vertex

Parameters:

localFiles - files that must be available locally for each task. These files may be regular files, archives etc. as specified by the value elements of the map.

Returns:

this Vertex

getTaskLocalFiles

public Map<String,org.apache.hadoop.yarn.api.records.LocalResource> getTaskLocalFiles()

Get the files etc that must be provided by the tasks of this vertex

Returns:

local files of the vertex. Key is the file name.

setTaskEnvironment

public Vertex setTaskEnvironment(Map<String,String> environment)

Set the Key-Value pairs of environment variables for tasks of this vertex. This method should be used if different vertices need different env. Else, set environment for all vertices via Tezconfiguration#TEZ_TASK_LAUNCH_ENV

Parameters:

environment -

Returns:

this Vertex

getTaskEnvironment

public Map<String,String> getTaskEnvironment()

Get the environment variables of the tasks

Returns:

environment variable map

getConf

public Map<String,String> getConf()

setTaskLaunchCmdOpts

public Vertex setTaskLaunchCmdOpts(String cmdOpts)

Set the command opts for tasks of this vertex. This method should be used when different vertices have different opts. Else, set the launch opts for ' all vertices via Tezconfiguration#TEZ_TASK_LAUNCH_CMD_OPTS

Parameters:

cmdOpts -

Returns:

this Vertex

addDataSource

public Vertex addDataSource(String inputName,
                   DataSourceDescriptor dataSourceDescriptor)

Specifies an external data source for a Vertex. This is meant to be used when a Vertex reads Input directly from an external source

For vertices which read data generated by another vertex - use the addEdge method. If a vertex needs to use data generated by another vertex in the DAG and also from an external source, a combination of this API and the DAG.addEdge API can be used.

Note: If more than one RootInput exists on a vertex, which generates events which need to be routed, or generates information to set parallelism, a custom vertex manager should be setup to handle this. Not using a custom vertex manager for such a scenario will lead to a runtime failure.

Parameters:

inputName - the name of the input. This will be used when accessing the input in the LogicalIOProcessor

dataSourceDescriptor - the @{link DataSourceDescriptor} for this input.

Returns:

this Vertex

addDataSink

public Vertex addDataSink(String outputName,
                 DataSinkDescriptor dataSinkDescriptor)

Specifies an external data sink for a Vertex. This is meant to be used when a Vertex writes Output directly to an external destination.

If an output of the vertex is meant to be consumed by another Vertex in the DAG - use the addEdge method. If a vertex needs generate data to an external source as well as for another Vertex in the DAG, a combination of this API and the DAG.addEdge API can be used.

Parameters:

outputName - the name of the output. This will be used when accessing the output in the LogicalIOProcessor

dataSinkDescriptor - the DataSinkDescriptor for this output

Returns:

this Vertex

setVertexManagerPlugin

public Vertex setVertexManagerPlugin(VertexManagerPluginDescriptor vertexManagerPluginDescriptor)

Specifies a VertexManagerPlugin for the vertex. This plugin can be used to modify the parallelism or reconfigure the vertex at runtime using user defined code embedded in the plugin

Parameters:

vertexManagerPluginDescriptor -

Returns:

this Vertex

getTaskLaunchCmdOpts

public String getTaskLaunchCmdOpts()

Get the launch command opts for tasks in this vertex

Returns:

launch command opts

setConf

@InterfaceStability.Unstable
public Vertex setConf(String property,
                                         String value)

This is currently used to setup additional configuration parameters which will be available in the Vertex specific configuration used in the AppMaster. This API would be used for properties which are used by the Tez framework while executing this vertex as part of a larger DAG. As an example, the number of attempts for a task.

A vertex inherits it's Configuration from the DAG, and can override properties for this Vertex only using this method

Currently, properties which are used by the task runtime, such as the task to AM heartbeat interval, cannot be changed using this method.

Note: This API does not add any configuration to runtime components such as InputInitializers, OutputCommitters, Inputs and Outputs.

Parameters:

property - the property name

value - the value for the property

Returns:

the current DAG being constructed

setExecutionContext

public Vertex setExecutionContext(Vertex.VertexExecutionContext vertexExecutionContext)

Sets the execution context for this Vertex - i.e. the Task Scheduler, ContainerLauncher and TaskCommunicator to be used. Also whether the vertex will be executed within the AM. If partially specified, the default components in Tez will be used - which may or may not work with the custom context.

Parameters:

vertexExecutionContext - the execution context for the vertex.

Returns:

this Vertex

toString

public String toString()

Overrides:

toString in class Object

getInputVertices

public List<Vertex> getInputVertices()

Get the input vertices for this vertex

Returns:

List of input vertices

getOutputVertices

public List<Vertex> getOutputVertices()

Get the output vertices for this vertex

Returns:

List of output vertices

getDataSources

@InterfaceAudience.Private
public List<DataSourceDescriptor> getDataSources()

getDataSinks

@InterfaceAudience.Private
public List<DataSinkDescriptor> getDataSinks()

getVertexExecutionContext

@InterfaceAudience.Private
public Vertex.VertexExecutionContext getVertexExecutionContext()

hashCode

public int hashCode()

Overrides:

hashCode in class Object

equals

public boolean equals(Object obj)

Overrides:

equals in class Object