org.apache.tez.runtime.library.conf

Class OrderedGroupedKVInputConfig.Builder

java.lang.Object

org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.Builder

All Implemented Interfaces:

org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder<OrderedGroupedKVInputConfig.Builder>

Enclosing class:

OrderedGroupedKVInputConfig

@InterfaceAudience.Public
@InterfaceStability.Evolving
public static class OrderedGroupedKVInputConfig.Builder
extends Object
implements org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder<OrderedGroupedKVInputConfig.Builder>

Method Summary

Methods

Modifier and Type	Method and Description
OrderedGroupedKVInputConfig	build() Create the actual configuration instance.
OrderedGroupedKVInputConfig.Builder	setAdditionalConfiguration(Map<String,String> confMap) Used to set additional configuration parameters which are not set via API methods.
OrderedGroupedKVInputConfig.Builder	setAdditionalConfiguration(String key, String value) Used to set additional configuration parameters which are not set via API methods.
OrderedGroupedKVInputConfig.Builder	setCombiner(String combinerClassName) Configure the combiner class
OrderedGroupedKVInputConfig.Builder	setCombiner(String combinerClassName, Map<String,String> combinerConf) Configure the combiner class and it's associated configuration (specified as key-value pairs).
OrderedGroupedKVInputConfig.Builder	setCompression(boolean enabled, String compressionCodec, Map<String,String> codecConf)
OrderedGroupedKVInputConfig.Builder	setFromConfiguration(org.apache.hadoop.conf.Configuration conf) Used to build out a configuration from an existing Hadoop Configuration.
OrderedGroupedKVInputConfig.Builder	setFromConfigurationUnfiltered(org.apache.hadoop.conf.Configuration conf)
OrderedGroupedKVInputConfig.Builder	setKeyComparatorClass(String comparatorClassName) Set the key comparator class
OrderedGroupedKVInputConfig.Builder	setKeyComparatorClass(String comparatorClassName, Map<String,String> comparatorConf) Set the key comparator class and it's associated configuration.
OrderedGroupedKVInputConfig.Builder	setKeySerializationClass(String serializationClassName, String comparatorClassName, Map<String,String> serializerConf) Set serialization class and the relevant comparator to be used for sorting.
OrderedGroupedKVInputConfig.Builder	setMaxSingleMemorySegmentFraction(float maxSingleSegmentFraction) Sets a size limit on the maximum segment size to be shuffled to disk.
OrderedGroupedKVInputConfig.Builder	setMemToMemMerger(boolean enable) Enable the memory to memory merger
OrderedGroupedKVInputConfig.Builder	setMergeFraction(float mergeFraction) Configure the point at which in memory segments will be merged.
OrderedGroupedKVInputConfig.Builder	setPostMergeBufferFraction(float postMergeBufferFraction) Sets the buffer fraction, as a fraction of container size, to be used after the fetch and merge are complete.
OrderedGroupedKVInputConfig.Builder	setShuffleBufferFraction(float shuffleBufferFraction) Sets the buffer fraction, as a fraction of container size, to be used while fetching remote data.
OrderedGroupedKVInputConfig.Builder	setValueSerializationClass(String serializationClassName, Map<String,String> serializerConf) Serialization class to be used for serializing values.
OrderedGroupedKVInputConfig.Builder	useLegacyInput() Specifies whether the legacy version of this input should be used.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

useLegacyInput

public OrderedGroupedKVInputConfig.Builder useLegacyInput()

Description copied from interface: org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder

Specifies whether the legacy version of this input should be used.

Specified by:

useLegacyInput in interface org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder<OrderedGroupedKVInputConfig.Builder>

Returns:

instance of the current builder

setShuffleBufferFraction

public OrderedGroupedKVInputConfig.Builder setShuffleBufferFraction(float shuffleBufferFraction)

Description copied from interface: org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder

Sets the buffer fraction, as a fraction of container size, to be used while fetching remote data.

Specified by:

setShuffleBufferFraction in interface org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder<OrderedGroupedKVInputConfig.Builder>

Parameters:

shuffleBufferFraction - fraction of container size

Returns:

instance of the current builder

setPostMergeBufferFraction

public OrderedGroupedKVInputConfig.Builder setPostMergeBufferFraction(float postMergeBufferFraction)

Description copied from interface: org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder

Sets the buffer fraction, as a fraction of container size, to be used after the fetch and merge are complete. This buffer is used to cache merged data and avoids writing it out to disk.

Specified by:

setPostMergeBufferFraction in interface org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder<OrderedGroupedKVInputConfig.Builder>

Parameters:

postMergeBufferFraction - fraction of container size

Returns:

instance of the current builder

setMaxSingleMemorySegmentFraction

public OrderedGroupedKVInputConfig.Builder setMaxSingleMemorySegmentFraction(float maxSingleSegmentFraction)

Description copied from interface: org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder

Sets a size limit on the maximum segment size to be shuffled to disk. This is a fraction of the shuffle buffer.

Specified by:

setMaxSingleMemorySegmentFraction in interface org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder<OrderedGroupedKVInputConfig.Builder>

Parameters:

maxSingleSegmentFraction - fraction of memory determined by ShuffleBufferFraction

Returns:

instance of the current builder

setMemToMemMerger

public OrderedGroupedKVInputConfig.Builder setMemToMemMerger(boolean enable)

Description copied from interface: org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder

Enable the memory to memory merger

Specified by:

setMemToMemMerger in interface org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder<OrderedGroupedKVInputConfig.Builder>

Parameters:

enable - whether to enable the memory to memory merger

Returns:

instance of the current builder

setMergeFraction

public OrderedGroupedKVInputConfig.Builder setMergeFraction(float mergeFraction)

Description copied from interface: org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder

Configure the point at which in memory segments will be merged. This is specified as a fraction of the shuffle buffer.

Specified by:

setMergeFraction in interface org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder<OrderedGroupedKVInputConfig.Builder>

Parameters:

mergeFraction - fraction of memory determined by ShuffleBufferFraction, which when filled, will trigger a merge

Returns:

instance of the current builder

setCombiner

public OrderedGroupedKVInputConfig.Builder setCombiner(String combinerClassName)

Description copied from interface: org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder

Configure the combiner class

Specified by:

setCombiner in interface org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder<OrderedGroupedKVInputConfig.Builder>

Parameters:

combinerClassName - the combiner class name

Returns:

instance of the current builder

setCombiner

public OrderedGroupedKVInputConfig.Builder setCombiner(String combinerClassName,
                                              Map<String,String> combinerConf)

Description copied from interface: org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder

Configure the combiner class and it's associated configuration (specified as key-value pairs). This method should only be used if the combiner requires some specific configuration. OrderedGroupedKVInputConfig.SpecificConfigBuilder.setCombiner(String) is the preferred method for setting a combiner.

Specified by:

setCombiner in interface org.apache.tez.runtime.library.conf.OrderedGroupedKVInputConfig.SpecificConfigBuilder<OrderedGroupedKVInputConfig.Builder>

Parameters:

combinerClassName - the combiner class name

combinerConf - the combiner configuration. This can be null, and otherwise is a Map of key-value pairs. The keys should be limited to the ones required by the combiner.

Returns:

instance of the current builder

setKeyComparatorClass

public OrderedGroupedKVInputConfig.Builder setKeyComparatorClass(String comparatorClassName)

Set the key comparator class

Parameters:

comparatorClassName - the key comparator class name

Returns:

instance of the current builder

setKeyComparatorClass

public OrderedGroupedKVInputConfig.Builder setKeyComparatorClass(String comparatorClassName,
                                                        @Nullable
                                                        Map<String,String> comparatorConf)

Set the key comparator class and it's associated configuration. This method should only be used if the comparator requires some specific configuration, which is typically not the case. setKeyComparatorClass(String) is the preferred method for setting a comparator.

Parameters:

comparatorClassName - the key comparator class name

comparatorConf - the comparator configuration. This can be null, and is a Map of key-value pairs. The keys should be limited to the ones required by the comparator.

Returns:

instance of the current builder

setAdditionalConfiguration

public OrderedGroupedKVInputConfig.Builder setAdditionalConfiguration(String key,
                                                             String value)

Used to set additional configuration parameters which are not set via API methods. This is primarily meant for rarely used configuration options such as IFile read-ahead, configuring the number of parallel files to merge etc.

Parameters:

key - the key to set

value - the corresponding value

Returns:

this object for further chained method calls

setAdditionalConfiguration

public OrderedGroupedKVInputConfig.Builder setAdditionalConfiguration(Map<String,String> confMap)

Used to set additional configuration parameters which are not set via API methods. This is primarily meant for rarely used configuration options such as IFile read-ahead, configuring the number of parallel files to merge etc.

Additionally keys set via this method are made available to the combiner.

Parameters:

confMap - map of configuration key-value pairs

Returns:

this object for further chained method calls

setFromConfiguration

public OrderedGroupedKVInputConfig.Builder setFromConfiguration(org.apache.hadoop.conf.Configuration conf)

Used to build out a configuration from an existing Hadoop Configuration. This is a private API is present only for compatibility and ease of use for existing systems which rely heavily on Configuration.

Returns:

this object for further chained method calls

setFromConfigurationUnfiltered

public OrderedGroupedKVInputConfig.Builder setFromConfigurationUnfiltered(org.apache.hadoop.conf.Configuration conf)

setCompression

public OrderedGroupedKVInputConfig.Builder setCompression(boolean enabled,
                                                 @Nullable
                                                 String compressionCodec,
                                                 @Nullable
                                                 Map<String,String> codecConf)

setKeySerializationClass

public OrderedGroupedKVInputConfig.Builder setKeySerializationClass(String serializationClassName,
                                                           String comparatorClassName,
                                                           @Nullable
                                                           Map<String,String> serializerConf)

Set serialization class and the relevant comparator to be used for sorting. Providing custom serialization class could change the way, keys needs to be compared in sorting. Providing invalid comparator here could create invalid results.

Parameters:

serializationClassName -

comparatorClassName -

serializerConf - the serializer configuration. This can be null, and is a Map of key-value pairs. The keys should be limited to the ones required by the comparator.

Returns:

this object for further chained method calls

setValueSerializationClass

public OrderedGroupedKVInputConfig.Builder setValueSerializationClass(String serializationClassName,
                                                             @Nullable
                                                             Map<String,String> serializerConf)

Serialization class to be used for serializing values.

Parameters:

serializationClassName -

serializerConf - the serializer configuration. This can be null, and is a Map of key-value pairs. The keys should be limited to the ones required by the comparator.

Returns:

this object for further chained method calls

build

public OrderedGroupedKVInputConfig build()

Create the actual configuration instance.

Returns:

an instance of the Configuration