org.apache.tez.dag.api

Class DataSinkDescriptor

java.lang.Object

org.apache.tez.dag.api.DataSinkDescriptor

@InterfaceAudience.Public
public class DataSinkDescriptor
extends Object

Defines the output and output committer for a data sink

Constructor Summary

Constructors

Constructor and Description
DataSinkDescriptor(OutputDescriptor outputDescriptor, OutputCommitterDescriptor committerDescriptor, org.apache.hadoop.security.Credentials credentials) Deprecated.

Method Summary

Modifier and Type	Method and Description
DataSinkDescriptor	addURIsForCredentials(Collection<URI> uris) This method can be used to specify a list of URIs for which Credentials need to be obtained so that the job can run.
static DataSinkDescriptor	create(OutputDescriptor outputDescriptor, OutputCommitterDescriptor committerDescriptor, org.apache.hadoop.security.Credentials credentials) Create a DataSinkDescriptor
org.apache.hadoop.security.Credentials	getCredentials() Get the Credentials for this DataSinkDescriptor
OutputCommitterDescriptor	getOutputCommitterDescriptor() Get the OutputCommitterDescriptor for this DataSinkDescriptor
OutputDescriptor	getOutputDescriptor() Get the OutputDescriptor for this DataSinkDescriptor
Collection<URI>	getURIsForCredentials() Get the URIs for which credentials will be obtained

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataSinkDescriptor

@Deprecated
public DataSinkDescriptor(OutputDescriptor outputDescriptor,
                                      @Nullable
                                      OutputCommitterDescriptor committerDescriptor,
                                      @Nullable
                                      org.apache.hadoop.security.Credentials credentials)

Deprecated.

Create a DataSinkDescriptor

Parameters:

outputDescriptor - An OutputDescriptor for the output

committerDescriptor - Specify a committer to be used for the output. Can be null. After all tasks in the vertex (or in the DAG) have completed, the committer (if specified) is invoked to commit the outputs. Commit is a data sink specific operation that usually determines the visibility of the output to external observers. E.g. moving output files from temporary dirs to the real output dir. When there are multiple executions of a task, the commit process also helps decide which execution will be included in the final output. Users should consider whether their application or data sink need a commit operation.

credentials - Credentials needs to access the data sink

Method Detail

create

public static DataSinkDescriptor create(OutputDescriptor outputDescriptor,
                                        @Nullable
                                        OutputCommitterDescriptor committerDescriptor,
                                        @Nullable
                                        org.apache.hadoop.security.Credentials credentials)

Create a DataSinkDescriptor

Parameters:

outputDescriptor - An OutputDescriptor for the output

committerDescriptor - Specify a committer to be used for the output. Can be null. After all tasks in the vertex (or in the DAG) have completed, the committer (if specified) is invoked to commit the outputs. Commit is a data sink specific operation that usually determines the visibility of the output to external observers. E.g. moving output files from temporary dirs to the real output dir. When there are multiple executions of a task, the commit process also helps decide which execution will be included in the final output. Users should consider whether their application or data sink need a commit operation.

credentials - Credentials needs to access the data sink

getOutputDescriptor

public OutputDescriptor getOutputDescriptor()

Get the OutputDescriptor for this DataSinkDescriptor

Returns:

OutputDescriptor

getOutputCommitterDescriptor

@Nullable
public OutputCommitterDescriptor getOutputCommitterDescriptor()

Get the OutputCommitterDescriptor for this DataSinkDescriptor

Returns:

OutputCommitterDescriptor

addURIsForCredentials

public DataSinkDescriptor addURIsForCredentials(Collection<URI> uris)

This method can be used to specify a list of URIs for which Credentials need to be obtained so that the job can run. An incremental list of URIs can be provided by making multiple calls to the method. Currently, @{link credentials} can only be fetched for HDFS and other FileSystem implementations that support credentials.

Parameters:

uris - a list of URIs

Returns:

this

getURIsForCredentials

public Collection<URI> getURIsForCredentials()

Get the URIs for which credentials will be obtained

Returns:

an unmodifiable list representing the URIs for which credentials are required.

getCredentials

@Nullable
public org.apache.hadoop.security.Credentials getCredentials()

Get the Credentials for this DataSinkDescriptor

Returns:

Credentials