org.apache.tez.dag.api

Class DAG

java.lang.Object

org.apache.tez.dag.api.DAG

@InterfaceAudience.Public
public class DAG
extends Object

Top level entity that defines the DAG (Directed Acyclic Graph) representing the data flow graph. Consists of a set of Vertices and Edges connecting the vertices. Vertices represent transformations of data and edges represent movement of data between vertices.

Method Summary

Methods

Modifier and Type	Method and Description
DAG	addEdge(Edge edge) Add an Edge connecting vertices in the DAG
DAG	addEdge(GroupInputEdge edge) Add a GroupInputEdge to the DAG.
DAG	addTaskLocalFiles(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localFiles) Set the files etc that must be provided to the tasks of this DAG
DAG	addURIsForCredentials(Collection<URI> uris) One of the methods that can be used to provide information about required Credentials when running on a secure cluster.
DAG	addVertex(Vertex vertex)
static DAG	create(String name) Create a DAG with the specified name.
DAGProtos.DAGPlan	createDag(org.apache.hadoop.conf.Configuration tezConf, org.apache.hadoop.security.Credentials extraCredentials, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> tezJarResources, org.apache.hadoop.yarn.api.records.LocalResource binaryConfig, boolean tezLrsAsArchive)
DAGProtos.DAGPlan	createDag(org.apache.hadoop.conf.Configuration tezConf, org.apache.hadoop.security.Credentials extraCredentials, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> tezJarResources, org.apache.hadoop.yarn.api.records.LocalResource binaryConfig, boolean tezLrsAsArchive, Map<String,String> additionalConfigs, ServicePluginsDescriptor servicePluginsDescriptor, JavaOptsChecker javaOptsChecker)
VertexGroup	createVertexGroup(String name, Vertex... members) Create a group of vertices that share a common output.
CallerContext	getCallerContext()
org.apache.hadoop.security.Credentials	getCredentials()
DAGAccessControls	getDagAccessControls()
Map<String,String>	getDagConf()
String	getName() Get the DAG name
Map<String,org.apache.hadoop.yarn.api.records.LocalResource>	getTaskLocalFiles()
Collection<URI>	getURIsForCredentials()
Vertex	getVertex(String vertexName)
Set<Vertex>	getVertices()
DAG	setAccessControls(DAGAccessControls accessControls) Set Access controls for the DAG.
DAG	setCallerContext(CallerContext callerContext) Set the Context in which Tez is being called.
DAG	setConf(String property, String value) This is currently used to setup additional configuration parameters which will be available in the DAG configuration used in the AppMaster.
DAG	setCredentials(org.apache.hadoop.security.Credentials credentials) One of the methods that can be used to provide information about required Credentials when running on a secure cluster.
DAG	setDAGInfo(String dagInfo) Deprecated.
DAG	setExecutionContext(Vertex.VertexExecutionContext vertexExecutionContext) Sets the default execution context for the DAG.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

create

public static DAG create(String name)

Create a DAG with the specified name.

Parameters:

name - the name of the DAG

Returns:

this DAG

addTaskLocalFiles

public DAG addTaskLocalFiles(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localFiles)

Set the files etc that must be provided to the tasks of this DAG

Parameters:

localFiles - files that must be available locally for each task. These files may be regular files, archives etc. as specified by the value elements of the map.

Returns:

DAG

addVertex

public DAG addVertex(Vertex vertex)

getVertex

public Vertex getVertex(String vertexName)

setCredentials

public DAG setCredentials(org.apache.hadoop.security.Credentials credentials)

One of the methods that can be used to provide information about required Credentials when running on a secure cluster. A combination of this and addURIsForCredentials should be used to specify information about all credentials required by a DAG. AM specific credentials are not used when executing a DAG. Set credentials which will be required to run this dag. This method can be used if the client has already obtained some or all of the required credentials.

Parameters:

credentials - Credentials for the DAG

Returns:

DAG

setDAGInfo

@Deprecated
public DAG setDAGInfo(String dagInfo)

Deprecated.

Set description info for this DAG that can be used for visualization purposes.

Parameters:

dagInfo - JSON blob as a serialized string. Recognized keys by the UI are: "context" - The application context in which this DAG is being used. For example, this could be set to "Hive" or "Pig" if this is being run as part of a Hive or Pig script. "description" - General description on what this DAG is going to do. In the case of Hive, this could be the SQL query text.

Returns:

DAG

setCallerContext

public DAG setCallerContext(CallerContext callerContext)

Set the Context in which Tez is being called.

Parameters:

callerContext - Caller Context

Returns:

DAG

createVertexGroup

public VertexGroup createVertexGroup(String name,
                            Vertex... members)

Create a group of vertices that share a common output. This can be used to implement unions efficiently.

Parameters:

name - Name of the group.

members - Vertex members of the group

Returns:

DAG

getCredentials

@InterfaceAudience.Private
public org.apache.hadoop.security.Credentials getCredentials()

setAccessControls

public DAG setAccessControls(DAGAccessControls accessControls)

Set Access controls for the DAG. Which user/groups can view the DAG progess/history and who can modify the DAG i.e. kill the DAG. The owner of the Tez Session and the user submitting the DAG are super-users and have access to all operations on the DAG.

Parameters:

accessControls - Access Controls

Returns:

DAG

getDagAccessControls

@InterfaceAudience.Private
public DAGAccessControls getDagAccessControls()

addURIsForCredentials

public DAG addURIsForCredentials(Collection<URI> uris)

One of the methods that can be used to provide information about required Credentials when running on a secure cluster. A combination of this and setCredentials should be used to specify information about all credentials required by a DAG. AM specific credentials are not used when executing a DAG. This method can be used to specify a list of URIs for which Credentials need to be obtained so that the job can run. An incremental list of URIs can be provided by making multiple calls to the method. Currently, @{link credentials} can only be fetched for HDFS and other FileSystem implementations that support credentials.

Parameters:

uris - a list of URIs

Returns:

DAG

getURIsForCredentials

@InterfaceAudience.Private
public Collection<URI> getURIsForCredentials()

Returns:

an unmodifiable list representing the URIs for which credentials are required.

getVertices

@InterfaceAudience.Private
public Set<Vertex> getVertices()

addEdge

public DAG addEdge(Edge edge)

Add an Edge connecting vertices in the DAG

Parameters:

edge - The edge to be added

Returns:

DAG

addEdge

public DAG addEdge(GroupInputEdge edge)

Add a GroupInputEdge to the DAG.

Parameters:

edge - GroupInputEdge

Returns:

DAG

getName

public String getName()

Get the DAG name

Returns:

DAG name

setConf

@InterfaceStability.Unstable
public DAG setConf(String property,
                                      String value)

This is currently used to setup additional configuration parameters which will be available in the DAG configuration used in the AppMaster. This API would be used for properties which are used by the Tez framework while executing the DAG. As an example, the number of attempts for a task.

A DAG inherits it's base properties from the ApplicationMaster within which it's running. This method allows for these properties to be overridden. Currently, properties which are used by the task runtime, such as the task to AM heartbeat interval, cannot be changed using this method.

Note: This API does not add any configuration to runtime components such as InputInitializers, OutputCommitters, Inputs and Outputs.

Parameters:

property - the property name

value - the value for the property

Returns:

the current DAG being constructed

setExecutionContext

@InterfaceAudience.Public
@InterfaceStability.Unstable
public DAG setExecutionContext(Vertex.VertexExecutionContext vertexExecutionContext)

Sets the default execution context for the DAG. This can be overridden at a per Vertex level. See Vertex.setExecutionContext(VertexExecutionContext)

Parameters:

vertexExecutionContext - the default execution context for the DAG

Returns:

this DAG

getDagConf

@InterfaceAudience.Private
public Map<String,String> getDagConf()

getTaskLocalFiles

@InterfaceAudience.Private
public Map<String,org.apache.hadoop.yarn.api.records.LocalResource> getTaskLocalFiles()

createDag

@InterfaceAudience.Private
public DAGProtos.DAGPlan createDag(org.apache.hadoop.conf.Configuration tezConf,
                                                    org.apache.hadoop.security.Credentials extraCredentials,
                                                    Map<String,org.apache.hadoop.yarn.api.records.LocalResource> tezJarResources,
                                                    org.apache.hadoop.yarn.api.records.LocalResource binaryConfig,
                                                    boolean tezLrsAsArchive)

createDag

@InterfaceAudience.Private
public DAGProtos.DAGPlan createDag(org.apache.hadoop.conf.Configuration tezConf,
                                                    org.apache.hadoop.security.Credentials extraCredentials,
                                                    Map<String,org.apache.hadoop.yarn.api.records.LocalResource> tezJarResources,
                                                    org.apache.hadoop.yarn.api.records.LocalResource binaryConfig,
                                                    boolean tezLrsAsArchive,
                                                    Map<String,String> additionalConfigs,
                                                    ServicePluginsDescriptor servicePluginsDescriptor,
                                                    JavaOptsChecker javaOptsChecker)

getCallerContext

public CallerContext getCallerContext()