org.apache.tez.runtime.library.input

Class OrderedGroupedKVInput

java.lang.Object

org.apache.tez.runtime.api.AbstractLogicalInput

org.apache.tez.runtime.library.input.OrderedGroupedKVInput

All Implemented Interfaces:

org.apache.tez.runtime.api.Input, org.apache.tez.runtime.api.InputFrameworkInterface, org.apache.tez.runtime.api.LogicalInput, org.apache.tez.runtime.api.LogicalInputFrameworkInterface

@InterfaceAudience.Public
public class OrderedGroupedKVInput
extends org.apache.tez.runtime.api.AbstractLogicalInput

OrderedGroupedKVInput in a AbstractLogicalInput which shuffles intermediate sorted data, merges them and provides key/ to the consumer. This is typically used to bring one partition of a set of partitioned distributed data to one consumer. The shuffle operation brings all partitions to one place. These partitions are assumed to be sorted and are merged sorted to merge them into a single input view. The Copy and Merge will be triggered by the initialization - which is handled by the Tez framework. Input is not consumable until the Copy and Merge are complete. Methods are provided to check for this, as well as to wait for completion. Attempting to get a reader on a non-complete input will block.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	conf
protected MemoryUpdateCallbackHandler	memoryUpdateCallbackHandler
protected org.apache.tez.runtime.library.common.sort.impl.TezRawKeyValueIterator	rawIter
protected org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle	shuffle
protected org.apache.tez.runtime.library.common.ValuesIterator	vIter

Constructor Summary

Constructors

Constructor and Description
OrderedGroupedKVInput(org.apache.tez.runtime.api.InputContext inputContext, int numPhysicalInputs)

Method Summary

Methods

Modifier and Type	Method and Description
List<org.apache.tez.runtime.api.Event>	close()
protected void	createValuesIterator()
org.apache.hadoop.io.RawComparator	getInputKeyComparator()
float	getProgress()
KeyValuesReader	getReader() Get a KVReader for the Input.
void	handleEvents(List<org.apache.tez.runtime.api.Event> inputEvents)
List<org.apache.tez.runtime.api.Event>	initialize()
boolean	isInputReady() Check if the input is ready for consumption
void	start()
void	waitForInputReady() Waits for the input to become ready for consumption

Methods inherited from class org.apache.tez.runtime.api.AbstractLogicalInput

getContext, getNumPhysicalInputs

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

rawIter

protected org.apache.tez.runtime.library.common.sort.impl.TezRawKeyValueIterator rawIter

conf

protected org.apache.hadoop.conf.Configuration conf

shuffle

protected org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle shuffle

memoryUpdateCallbackHandler

protected MemoryUpdateCallbackHandler memoryUpdateCallbackHandler

vIter

protected org.apache.tez.runtime.library.common.ValuesIterator vIter

Constructor Detail

OrderedGroupedKVInput

public OrderedGroupedKVInput(org.apache.tez.runtime.api.InputContext inputContext,
                     int numPhysicalInputs)

Method Detail

initialize

public List<org.apache.tez.runtime.api.Event> initialize()
                                                  throws IOException

Specified by:

initialize in interface org.apache.tez.runtime.api.InputFrameworkInterface

Specified by:

initialize in class org.apache.tez.runtime.api.AbstractLogicalInput

Throws:

IOException

start

public void start()
           throws IOException

Throws:

IOException

isInputReady

public boolean isInputReady()
                     throws IOException,
                            InterruptedException,
                            org.apache.tez.dag.api.TezException

Check if the input is ready for consumption

Returns:

true if the input is ready for consumption, or if an error occurred processing fetching the input. false if the shuffle and merge are still in progress

Throws:

InterruptedException

IOException

org.apache.tez.dag.api.TezException

waitForInputReady

public void waitForInputReady()
                       throws IOException,
                              InterruptedException,
                              org.apache.tez.dag.api.TezException

Waits for the input to become ready for consumption

Throws:

IOException

InterruptedException

org.apache.tez.dag.api.TezException

close

public List<org.apache.tez.runtime.api.Event> close()
                                             throws IOException

Throws:

IOException

getReader

public KeyValuesReader getReader()
                          throws IOException,
                                 org.apache.tez.dag.api.TezException

Get a KVReader for the Input.

This method will block until the input is ready - i.e. the copy and merge stages are complete. Users can use the isInputReady method to check if the input is ready, which gives an indication of whether this method will block or not. NOTE: All values for the current K-V pair must be read prior to invoking moveToNext. Once moveToNext() is called, the valueIterator from the previous K-V pair will throw an Exception

Returns:

a KVReader over the sorted input.

Throws:

{@link - IOInterruptedException} if IO was performing a blocking operation and was interrupted

IOException

org.apache.tez.dag.api.TezException

getProgress

public float getProgress()
                  throws org.apache.tez.runtime.api.ProgressFailedException,
                         InterruptedException

Overrides:

getProgress in class org.apache.tez.runtime.api.AbstractLogicalInput

Throws:

org.apache.tez.runtime.api.ProgressFailedException

InterruptedException

handleEvents

public void handleEvents(List<org.apache.tez.runtime.api.Event> inputEvents)
                  throws IOException

Throws:

IOException

createValuesIterator

protected void createValuesIterator()
                             throws IOException

Throws:

IOException

getInputKeyComparator

public org.apache.hadoop.io.RawComparator getInputKeyComparator()