org.apache.tez.mapreduce.output

Class MROutput

java.lang.Object

org.apache.tez.runtime.api.AbstractLogicalOutput

org.apache.tez.mapreduce.output.MROutput

All Implemented Interfaces:

org.apache.tez.runtime.api.LogicalOutput, org.apache.tez.runtime.api.LogicalOutputFrameworkInterface, org.apache.tez.runtime.api.Output, org.apache.tez.runtime.api.OutputFrameworkInterface

Direct Known Subclasses:

MultiMROutput

@InterfaceAudience.Public
public class MROutput
extends org.apache.tez.runtime.api.AbstractLogicalOutput

MROutput is an Output which allows key/values pairs to be written by a processor. It is compatible with all standard Apache Hadoop MapReduce OutputFormat implementations. This class is not meant to be extended by external projects.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.mapreduce.OutputCommitter	committer
protected AtomicBoolean	flushed
protected org.apache.hadoop.mapred.JobConf	jobConf
protected org.apache.tez.common.counters.TezCounter	outputRecordCounter

Constructor Summary

Constructors

Constructor and Description
MROutput(org.apache.tez.runtime.api.OutputContext outputContext, int numPhysicalOutputs)

Method Summary

Modifier and Type	Method and Description
void	abort() MROutput expects that a Processor call abort in case of any error ( including an error during commit ) prior to the Processor's completion
List<org.apache.tez.runtime.api.Event>	close()
void	commit() MROutput expects that a Processor call commit prior to the Processor's completion
static org.apache.tez.mapreduce.output.MROutput.MROutputConfigBuilder	createConfigBuilder(org.apache.hadoop.conf.Configuration conf, Class<?> outputFormat) Create an MROutput.MROutputConfigBuilder
static org.apache.tez.mapreduce.output.MROutput.MROutputConfigBuilder	createConfigBuilder(org.apache.hadoop.conf.Configuration conf, Class<?> outputFormat, boolean useLazyOutputFormat)
static org.apache.tez.mapreduce.output.MROutput.MROutputConfigBuilder	createConfigBuilder(org.apache.hadoop.conf.Configuration conf, Class<?> outputFormat, String outputPath) Create an MROutput.MROutputConfigBuilder for a FileOutputFormat or FileOutputFormat based OutputFormats.
static org.apache.tez.mapreduce.output.MROutput.MROutputConfigBuilder	createConfigBuilder(org.apache.hadoop.conf.Configuration conf, Class<?> outputFormat, String outputPath, boolean useLazyOutputFormat)
void	flush() Call this in the processor before finishing to ensure outputs that outputs have been flushed.
protected String	getOutputFileNamePrefix()
protected String	getOutputName(String prefix)
org.apache.tez.runtime.library.api.KeyValueWriter	getWriter() Get a key value write to write Map Reduce compatible output
void	handleEvents(List<org.apache.tez.runtime.api.Event> outputEvents)
void	initCommitter(org.apache.hadoop.mapred.JobConf job, boolean useNewApi)
List<org.apache.tez.runtime.api.Event>	initialize()
protected List<org.apache.tez.runtime.api.Event>	initializeBase()
boolean	isCommitRequired()
void	start()

Methods inherited from class org.apache.tez.runtime.api.AbstractLogicalOutput

getContext, getNumPhysicalOutputs

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

jobConf

protected org.apache.hadoop.mapred.JobConf jobConf

flushed

protected AtomicBoolean flushed

outputRecordCounter

protected org.apache.tez.common.counters.TezCounter outputRecordCounter

committer

protected org.apache.hadoop.mapreduce.OutputCommitter committer

Constructor Detail

MROutput

public MROutput(org.apache.tez.runtime.api.OutputContext outputContext,
                int numPhysicalOutputs)

Method Detail

createConfigBuilder

public static org.apache.tez.mapreduce.output.MROutput.MROutputConfigBuilder createConfigBuilder(org.apache.hadoop.conf.Configuration conf,
                                                                                                 @Nullable
                                                                                                 Class<?> outputFormat)

Create an MROutput.MROutputConfigBuilder

The preferred usage model is to provide all of the parameters, and use methods to configure the Output.

For legacy applications, which may already have a fully configured Configuration instance, the outputFormat can be specified as null

Parameters:

conf - Configuration for the MROutput. This configuration instance will be modified in place

outputFormat - OutputFormat derived class. If the OutputFormat specified is null, the provided configuration should be complete.

Returns:

MROutput.MROutputConfigBuilder

createConfigBuilder

public static org.apache.tez.mapreduce.output.MROutput.MROutputConfigBuilder createConfigBuilder(org.apache.hadoop.conf.Configuration conf,
                                                                                                 @Nullable
                                                                                                 Class<?> outputFormat,
                                                                                                 boolean useLazyOutputFormat)

createConfigBuilder

public static org.apache.tez.mapreduce.output.MROutput.MROutputConfigBuilder createConfigBuilder(org.apache.hadoop.conf.Configuration conf,
                                                                                                 @Nullable
                                                                                                 Class<?> outputFormat,
                                                                                                 @Nullable
                                                                                                 String outputPath)

Create an MROutput.MROutputConfigBuilder for a FileOutputFormat or FileOutputFormat based OutputFormats.

The preferred usage model is to provide all of the parameters, and use methods to configure the Output.

For legacy applications, which may already have a fully configured Configuration instance, the outputFormat and outputPath can be specified as null

Parameters:

conf - Configuration for the MROutput. This configuration instance will be modified in place

outputFormat - FileInputFormat derived class. If the InputFormat specified is null, the provided configuration should be complete.

outputPath - Output path. This can be null if already setup in the configuration

Returns:

MROutput.MROutputConfigBuilder

createConfigBuilder

public static org.apache.tez.mapreduce.output.MROutput.MROutputConfigBuilder createConfigBuilder(org.apache.hadoop.conf.Configuration conf,
                                                                                                 @Nullable
                                                                                                 Class<?> outputFormat,
                                                                                                 @Nullable
                                                                                                 String outputPath,
                                                                                                 boolean useLazyOutputFormat)

initialize

public List<org.apache.tez.runtime.api.Event> initialize()
                                                  throws IOException,
                                                         InterruptedException

Specified by:

initialize in interface org.apache.tez.runtime.api.OutputFrameworkInterface

Specified by:

initialize in class org.apache.tez.runtime.api.AbstractLogicalOutput

Throws:

IOException

InterruptedException

initializeBase

protected List<org.apache.tez.runtime.api.Event> initializeBase()
                                                         throws IOException,
                                                                InterruptedException

Throws:

IOException

InterruptedException

start

public void start()

initCommitter

public void initCommitter(org.apache.hadoop.mapred.JobConf job,
                          boolean useNewApi)
                   throws IOException,
                          InterruptedException

Throws:

IOException

InterruptedException

isCommitRequired

public boolean isCommitRequired()
                         throws IOException

Throws:

IOException

getOutputFileNamePrefix

protected String getOutputFileNamePrefix()

getOutputName

protected String getOutputName(String prefix)

getWriter

public org.apache.tez.runtime.library.api.KeyValueWriter getWriter()
                                                            throws IOException

Get a key value write to write Map Reduce compatible output

Throws:

IOException

handleEvents

public void handleEvents(List<org.apache.tez.runtime.api.Event> outputEvents)

close

public List<org.apache.tez.runtime.api.Event> close()
                                             throws IOException

Throws:

IOException

flush

public void flush()
           throws IOException

Call this in the processor before finishing to ensure outputs that outputs have been flushed. Must be called before commit.

Throws:

IOException

commit

public void commit()
            throws IOException

MROutput expects that a Processor call commit prior to the Processor's completion

Throws:

IOException

abort

public void abort()
           throws IOException

MROutput expects that a Processor call abort in case of any error ( including an error during commit ) prior to the Processor's completion

Throws:

IOException