org.apache.tez.runtime.api

Class OutputCommitter

java.lang.Object

org.apache.tez.runtime.api.OutputCommitter

@InterfaceStability.Evolving
 @InterfaceAudience.Public
public abstract class OutputCommitter
extends Object

OutputCommitter to "finalize" the output and make it user-visible if needed. The OutputCommitter is allowed only on a terminal Output.

Constructor Summary

Constructors

Constructor and Description
OutputCommitter(OutputCommitterContext committerContext) Constructor an instance of the OutputCommitter.

Method Summary

Modifier and Type	Method and Description
abstract void	abortOutput(org.apache.tez.dag.api.client.VertexStatus.State finalState) For aborting an output.
abstract void	commitOutput() For committing the output after successful completion of tasks that write the output.
OutputCommitterContext	getContext() Return ahe OutputCommitterContext for this specific instance of the Committer.
abstract void	initialize() Setup up the Output committer.
boolean	isTaskRecoverySupported() Whether the OutputCommitter supports recovery of output from a Task that completed in a previous DAG attempt
void	recoverTask(int taskIndex, int previousDAGAttempt) Recover task output from a previous DAG attempt
abstract void	setupOutput() For the framework to setup the output during initialization.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

OutputCommitter

public OutputCommitter(OutputCommitterContext committerContext)

Constructor an instance of the OutputCommitter. Classes extending this to create a OutputCommitter, must provide the same constructor so that Tez can create an instance of the class at runtime.

Parameters:

committerContext - committer context which can be used to access the payload, vertex properties, etc

Method Detail

initialize

public abstract void initialize()
                         throws Exception

Setup up the Output committer.

Throws:

Exception

setupOutput

public abstract void setupOutput()
                          throws Exception

For the framework to setup the output during initialization. This is called from the application master process for the vertex. This will be called multiple times, once per dag attempt.

Throws:

Exception - if setup fails

commitOutput

public abstract void commitOutput()
                           throws Exception

For committing the output after successful completion of tasks that write the output. Note that this is invoked for the outputs of vertices whose tasks have successfully completed. This is called from the application master process. Based on user configuration, commit is called at the end of the DAG execution for all outputs or immediately upon completion of all the tasks that produced the output. This is guaranteed to only be called once.

Throws:

Exception

abortOutput

public abstract void abortOutput(org.apache.tez.dag.api.client.VertexStatus.State finalState)
                          throws Exception

For aborting an output. Note that this is invoked for vertices with a final non-successful state. This is also called to abort a previously committed output in the case of a post-commit failure. This is called from the application master process. This may be called multiple times.

Parameters:

finalState - final run-state of the vertex

Throws:

Exception

isTaskRecoverySupported

public boolean isTaskRecoverySupported()

Whether the OutputCommitter supports recovery of output from a Task that completed in a previous DAG attempt

Returns:

True if recovery supported

recoverTask

public void recoverTask(int taskIndex,
                        int previousDAGAttempt)
                 throws Exception

Recover task output from a previous DAG attempt

Parameters:

taskIndex - Index of task to be recovered

previousDAGAttempt - Previous DAG Attempt Number

Throws:

Exception

getContext

public final OutputCommitterContext getContext()

Return ahe OutputCommitterContext for this specific instance of the Committer.

Returns:

the OutputCommitterContext for the input