org.apache.tez.dag.api

Class DAG

java.lang.Object

org.apache.tez.dag.api.DAG

@InterfaceAudience.Public
public class DAG
extends Object

Top level entity that defines the DAG (Directed Acyclic Graph) representing the data flow graph. Consists of a set of Vertices and Edges connecting the vertices. Vertices represent transformations of data and edges represent movement of data between vertices.

Method Summary

Methods

Modifier and Type	Method and Description
DAG	addEdge(Edge edge) Add an Edge connecting vertices in the DAG
DAG	addEdge(GroupInputEdge edge) Add a GroupInputEdge to the DAG.
DAG	addTaskLocalFiles(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localFiles) Set the files etc that must be provided to the tasks of this DAG
DAG	addURIsForCredentials(Collection<URI> uris) One of the methods that can be used to provide information about required Credentials when running on a secure cluster.
DAG	addVertex(Vertex vertex)
static DAG	create(String name) Create a DAG with the specified name.
org.apache.tez.dag.api.records.DAGProtos.DAGPlan	createDag(org.apache.hadoop.conf.Configuration dagConf, org.apache.hadoop.security.Credentials extraCredentials, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> tezJarResources, org.apache.hadoop.yarn.api.records.LocalResource binaryConfig, boolean tezLrsAsArchive)
VertexGroup	createVertexGroup(String name, Vertex... members) Create a group of vertices that share a common output.
String	getName() Get the DAG name
Vertex	getVertex(String vertexName)
DAG	setAccessControls(DAGAccessControls accessControls) Set Access controls for the DAG.
DAG	setCredentials(org.apache.hadoop.security.Credentials credentials) One of the methods that can be used to provide information about required Credentials when running on a secure cluster.
DAG	setDAGInfo(String dagInfo) Set description info for this DAG that can be used for visualization purposes.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

create

public static DAG create(String name)

Create a DAG with the specified name.

Parameters:

name - the name of the DAG

Returns:

this DAG

addTaskLocalFiles

public DAG addTaskLocalFiles(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localFiles)

Set the files etc that must be provided to the tasks of this DAG

Parameters:

localFiles - files that must be available locally for each task. These files may be regular files, archives etc. as specified by the value elements of the map.

Returns:

DAG

addVertex

public DAG addVertex(Vertex vertex)

getVertex

public Vertex getVertex(String vertexName)

setCredentials

public DAG setCredentials(org.apache.hadoop.security.Credentials credentials)

One of the methods that can be used to provide information about required Credentials when running on a secure cluster. A combination of this and addURIsForCredentials should be used to specify information about all credentials required by a DAG. AM specific credentials are not used when executing a DAG. Set credentials which will be required to run this dag. This method can be used if the client has already obtained some or all of the required credentials.

Parameters:

credentials - Credentials for the DAG

Returns:

DAG

setDAGInfo

public DAG setDAGInfo(String dagInfo)

Set description info for this DAG that can be used for visualization purposes.

Parameters:

dagInfo - JSON blob as a serialized string. Recognized keys by the UI are: "context" - The application context in which this DAG is being used. For example, this could be set to "Hive" or "Pig" if this is being run as part of a Hive or Pig script. "description" - General description on what this DAG is going to do. In the case of Hive, this could be the SQL query text.

Returns:

DAG

createVertexGroup

public VertexGroup createVertexGroup(String name,
                            Vertex... members)

Create a group of vertices that share a common output. This can be used to implement unions efficiently.

Parameters:

name - Name of the group.

members - Vertex members of the group

Returns:

DAG

setAccessControls

public DAG setAccessControls(DAGAccessControls accessControls)

Set Access controls for the DAG. Which user/groups can view the DAG progess/history and who can modify the DAG i.e. kill the DAG. The owner of the Tez Session and the user submitting the DAG are super-users and have access to all operations on the DAG.

Parameters:

accessControls - Access Controls

Returns:

DAG

addURIsForCredentials

public DAG addURIsForCredentials(Collection<URI> uris)

One of the methods that can be used to provide information about required Credentials when running on a secure cluster. A combination of this and setCredentials should be used to specify information about all credentials required by a DAG. AM specific credentials are not used when executing a DAG. This method can be used to specify a list of URIs for which Credentials need to be obtained so that the job can run. An incremental list of URIs can be provided by making multiple calls to the method. Currently, @{link credentials} can only be fetched for HDFS and other FileSystem implementations that support credentials.

Parameters:

uris - a list of URIs

Returns:

DAG

addEdge

public DAG addEdge(Edge edge)

Add an Edge connecting vertices in the DAG

Parameters:

edge - The edge to be added

Returns:

DAG

addEdge

public DAG addEdge(GroupInputEdge edge)

Add a GroupInputEdge to the DAG.

Parameters:

edge - GroupInputEdge

Returns:

DAG

getName

public String getName()

Get the DAG name

Returns:

DAG name

createDag

public org.apache.tez.dag.api.records.DAGProtos.DAGPlan createDag(org.apache.hadoop.conf.Configuration dagConf,
                                                         org.apache.hadoop.security.Credentials extraCredentials,
                                                         Map<String,org.apache.hadoop.yarn.api.records.LocalResource> tezJarResources,
                                                         org.apache.hadoop.yarn.api.records.LocalResource binaryConfig,
                                                         boolean tezLrsAsArchive)