org.apache.tez.runtime.library.conf

Class UnorderedKVInputConfig.Builder

java.lang.Object

org.apache.tez.runtime.library.conf.UnorderedKVInputConfig.Builder

All Implemented Interfaces:

org.apache.tez.runtime.library.conf.UnorderedKVInputConfig.SpecificConfigBuilder<UnorderedKVInputConfig.Builder>

Enclosing class:

UnorderedKVInputConfig

@InterfaceAudience.Public
@InterfaceStability.Evolving
public static class UnorderedKVInputConfig.Builder
extends Object
implements org.apache.tez.runtime.library.conf.UnorderedKVInputConfig.SpecificConfigBuilder<UnorderedKVInputConfig.Builder>

Method Summary

Methods

Modifier and Type	Method and Description
UnorderedKVInputConfig	build() Create the actual configuration instance.
UnorderedKVInputConfig.Builder	setAdditionalConfiguration(Map<String,String> confMap) Used to set additional configuration parameters which are not set via API methods.
UnorderedKVInputConfig.Builder	setAdditionalConfiguration(String key, String value) Used to set additional configuration parameters which are not set via API methods.
UnorderedKVInputConfig.Builder	setCompression(boolean enabled, String compressionCodec, Map<String,String> codecConf)
UnorderedKVInputConfig.Builder	setFromConfiguration(org.apache.hadoop.conf.Configuration conf) Used to build out a configuration from an existing Hadoop Configuration.
UnorderedKVInputConfig.Builder	setKeySerializationClass(String serializationClassName, Map<String,String> serializerConf) Set serialization class responsible for providing serializer/deserializer for key/value and the corresponding comparator class to be used as key comparator.
UnorderedKVInputConfig.Builder	setMaxSingleMemorySegmentFraction(float maxSingleSegmentFraction) Sets a size limit on the maximum segment size to be shuffled to disk.
UnorderedKVInputConfig.Builder	setMergeFraction(float mergeFraction) Configure the point at which in memory segments will be merged and written out to a single large disk segment.
UnorderedKVInputConfig.Builder	setShuffleBufferFraction(float shuffleBufferFraction) Sets the buffer fraction, as a fraction of container size, to be used while fetching remote data.
UnorderedKVInputConfig.Builder	setValueSerializationClass(String serializationClassName, Map<String,String> serializerConf) Set serialization class responsible for providing serializer/deserializer for values.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

setShuffleBufferFraction

public UnorderedKVInputConfig.Builder setShuffleBufferFraction(float shuffleBufferFraction)

Description copied from interface: org.apache.tez.runtime.library.conf.UnorderedKVInputConfig.SpecificConfigBuilder

Sets the buffer fraction, as a fraction of container size, to be used while fetching remote data.

Specified by:

setShuffleBufferFraction in interface org.apache.tez.runtime.library.conf.UnorderedKVInputConfig.SpecificConfigBuilder<UnorderedKVInputConfig.Builder>

Parameters:

shuffleBufferFraction - fraction of container size

Returns:

instance of the current builder

setMaxSingleMemorySegmentFraction

public UnorderedKVInputConfig.Builder setMaxSingleMemorySegmentFraction(float maxSingleSegmentFraction)

Description copied from interface: org.apache.tez.runtime.library.conf.UnorderedKVInputConfig.SpecificConfigBuilder

Sets a size limit on the maximum segment size to be shuffled to disk. This is a fraction of the shuffle buffer.

Specified by:

setMaxSingleMemorySegmentFraction in interface org.apache.tez.runtime.library.conf.UnorderedKVInputConfig.SpecificConfigBuilder<UnorderedKVInputConfig.Builder>

Parameters:

maxSingleSegmentFraction - fraction of memory determined by ShuffleBufferFraction

Returns:

instance of the current builder

setMergeFraction

public UnorderedKVInputConfig.Builder setMergeFraction(float mergeFraction)

Description copied from interface: org.apache.tez.runtime.library.conf.UnorderedKVInputConfig.SpecificConfigBuilder

Configure the point at which in memory segments will be merged and written out to a single large disk segment. This is specified as a fraction of the shuffle buffer.

Has no affect at the moment.

Specified by:

setMergeFraction in interface org.apache.tez.runtime.library.conf.UnorderedKVInputConfig.SpecificConfigBuilder<UnorderedKVInputConfig.Builder>

Parameters:

mergeFraction - fraction of memory determined by ShuffleBufferFraction, which when filled, will trigger a merge

Returns:

instance of the current builder

setAdditionalConfiguration

public UnorderedKVInputConfig.Builder setAdditionalConfiguration(String key,
                                                        String value)

Used to set additional configuration parameters which are not set via API methods. This is primarily meant for rarely used configuration options such as IFile read-ahead, configuring the number of parallel files to merge etc.

Parameters:

key - the key to set

value - the corresponding value

Returns:

this object for further chained method calls

setAdditionalConfiguration

public UnorderedKVInputConfig.Builder setAdditionalConfiguration(Map<String,String> confMap)

Used to set additional configuration parameters which are not set via API methods. This is primarily meant for rarely used configuration options such as IFile read-ahead, configuring the number of parallel files to merge etc.

Additionally keys set via this method are made available to the combiner.

Parameters:

confMap - map of configuration key-value pairs

Returns:

this object for further chained method calls

setFromConfiguration

public UnorderedKVInputConfig.Builder setFromConfiguration(org.apache.hadoop.conf.Configuration conf)

Used to build out a configuration from an existing Hadoop Configuration. This is a private API is present only for compatibility and ease of use for existing systems which rely heavily on Configuration.

Returns:

this object for further chained method calls

setCompression

public UnorderedKVInputConfig.Builder setCompression(boolean enabled,
                                            @Nullable
                                            String compressionCodec,
                                            @Nullable
                                            Map<String,String> codecConf)

setKeySerializationClass

public UnorderedKVInputConfig.Builder setKeySerializationClass(String serializationClassName,
                                                      @Nullable
                                                      Map<String,String> serializerConf)

Set serialization class responsible for providing serializer/deserializer for key/value and the corresponding comparator class to be used as key comparator.

Parameters:

serializationClassName -

serializerConf - the serializer configuration. This can be null, and is a Map of key-value pairs. The keys should be limited to the ones required by the comparator.

Returns:

this object for further chained method calls

setValueSerializationClass

public UnorderedKVInputConfig.Builder setValueSerializationClass(String serializationClassName,
                                                        @Nullable
                                                        Map<String,String> serializerConf)

Set serialization class responsible for providing serializer/deserializer for values.

Parameters:

serializationClassName -

serializerConf - the serializer configuration. This can be null, and is a Map of key-value pairs. The keys should be limited to the ones required by the comparator.

Returns:

this object for further chained method calls

build

public UnorderedKVInputConfig build()

Create the actual configuration instance.

Returns:

an instance of the Configuration