org.apache.tez.runtime.library.common.writers

Class BaseUnorderedPartitionedKVWriter

java.lang.Object

org.apache.tez.runtime.api.Writer

org.apache.tez.runtime.library.api.KeyValueWriter

org.apache.tez.runtime.library.api.KeyValuesWriter

org.apache.tez.runtime.library.common.writers.BaseUnorderedPartitionedKVWriter

Direct Known Subclasses:

UnorderedPartitionedKVWriter

public abstract class BaseUnorderedPartitionedKVWriter
extends KeyValuesWriter

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.tez.common.counters.TezCounter	additionalSpillBytesReadCounter Represents additional bytes read from disk to merge all the previous spills into a single file.
protected org.apache.tez.common.counters.TezCounter	additionalSpillBytesWritternCounter Represents additional bytes written to disk as a result of spills, excluding the final spill.
protected org.apache.hadoop.io.compress.CompressionCodec	codec
protected org.apache.hadoop.conf.Configuration	conf
protected org.apache.tez.common.counters.TezCounter	fileOutputBytesCounter Represents the final output size, with file format overheads and compression factored in.
protected int	ifileBufferSize
protected boolean	ifileReadAhead
protected int	ifileReadAheadLength
protected Class	keyClass
protected org.apache.hadoop.io.serializer.Serializer	keySerializer
protected org.apache.tez.common.counters.TezCounter	numAdditionalSpillsCounter Represents the number of additional spills.
protected int	numPartitions
protected org.apache.tez.common.counters.TezCounter	outputBytesWithOverheadCounter Represents the size of the final output - with any overheads introduced by meta-information.
protected org.apache.tez.runtime.api.OutputContext	outputContext
protected TezTaskOutput	outputFileHandler
protected org.apache.tez.common.counters.TezCounter	outputRecordBytesCounter Represents the serialized size of the output records.
protected org.apache.tez.common.counters.TezCounter	outputRecordsCounter Represents final number of records written (spills are not counted)
protected Partitioner	partitioner
protected org.apache.hadoop.io.serializer.SerializationFactory	serializationFactory
protected org.apache.tez.common.counters.TezCounter	spilledRecordsCounter Represents the additional records written to disk due to spills.
protected Class	valClass
protected org.apache.hadoop.io.serializer.Serializer	valSerializer

Constructor Summary

Constructors

Constructor and Description
BaseUnorderedPartitionedKVWriter(org.apache.tez.runtime.api.OutputContext outputContext, org.apache.hadoop.conf.Configuration conf, int numOutputs)

Method Summary

Methods

Modifier and Type	Method and Description
abstract List<org.apache.tez.runtime.api.Event>	close()
void	write(Object key, Iterable<Object> values) Writes a key and its associated values
abstract void	write(Object key, Object value) Writes a key/value pair.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

outputContext

protected final org.apache.tez.runtime.api.OutputContext outputContext

conf

protected final org.apache.hadoop.conf.Configuration conf

partitioner

protected final Partitioner partitioner

keyClass

protected final Class keyClass

valClass

protected final Class valClass

keySerializer

protected final org.apache.hadoop.io.serializer.Serializer keySerializer

valSerializer

protected final org.apache.hadoop.io.serializer.Serializer valSerializer

serializationFactory

protected final org.apache.hadoop.io.serializer.SerializationFactory serializationFactory

numPartitions

protected final int numPartitions

codec

protected final org.apache.hadoop.io.compress.CompressionCodec codec

outputFileHandler

protected final TezTaskOutput outputFileHandler

ifileReadAhead

protected final boolean ifileReadAhead

ifileReadAheadLength

protected final int ifileReadAheadLength

ifileBufferSize

protected final int ifileBufferSize

outputRecordBytesCounter

protected final org.apache.tez.common.counters.TezCounter outputRecordBytesCounter

Represents the serialized size of the output records. Does not consider overheads from the buffer meta-information, storage format, or compression if it is enabled.

outputRecordsCounter

protected final org.apache.tez.common.counters.TezCounter outputRecordsCounter

Represents final number of records written (spills are not counted)

outputBytesWithOverheadCounter

protected final org.apache.tez.common.counters.TezCounter outputBytesWithOverheadCounter

Represents the size of the final output - with any overheads introduced by meta-information.

fileOutputBytesCounter

protected final org.apache.tez.common.counters.TezCounter fileOutputBytesCounter

Represents the final output size, with file format overheads and compression factored in. Does not consider spills.

spilledRecordsCounter

protected final org.apache.tez.common.counters.TezCounter spilledRecordsCounter

Represents the additional records written to disk due to spills. Does not count the final write to disk.

additionalSpillBytesWritternCounter

protected final org.apache.tez.common.counters.TezCounter additionalSpillBytesWritternCounter

Represents additional bytes written to disk as a result of spills, excluding the final spill.

additionalSpillBytesReadCounter

protected final org.apache.tez.common.counters.TezCounter additionalSpillBytesReadCounter

Represents additional bytes read from disk to merge all the previous spills into a single file.

numAdditionalSpillsCounter

protected final org.apache.tez.common.counters.TezCounter numAdditionalSpillsCounter

Represents the number of additional spills. The final spill is not counted towards this.

Constructor Detail

BaseUnorderedPartitionedKVWriter

public BaseUnorderedPartitionedKVWriter(org.apache.tez.runtime.api.OutputContext outputContext,
                                org.apache.hadoop.conf.Configuration conf,
                                int numOutputs)

Method Detail

write

public abstract void write(Object key,
         Object value)
                    throws IOException

Description copied from class: KeyValueWriter

Writes a key/value pair.

Specified by:

write in class KeyValueWriter

Parameters:

key - the key to write

value - the value to write

Throws:

IOException - if an error occurs

write

public void write(Object key,
         Iterable<Object> values)
           throws IOException

Description copied from class: KeyValuesWriter

Writes a key and its associated values

Specified by:

write in class KeyValuesWriter

Parameters:

key - the key to write

values - values to write

Throws:

IOException

close

public abstract List<org.apache.tez.runtime.api.Event> close()
                                                      throws IOException,
                                                             InterruptedException

Throws:

IOException

InterruptedException