org.apache.tez.mapreduce.hadoop

Class MRHelpers

java.lang.Object

org.apache.tez.mapreduce.hadoop.MRHelpers

@InterfaceAudience.Public
@InterfaceStability.Evolving
public class MRHelpers
extends Object

This class contains helper methods for frameworks which migrate from MapReduce to Tez, and need to continue to work with existing MapReduce configurations.

Constructor Summary

Constructors

Constructor and Description
MRHelpers()

Method Summary

Methods

Modifier and Type	Method and Description
static void	configureMRApiUsage(org.apache.hadoop.conf.Configuration conf) Update the provided configuration to use the new API (mapreduce) or the old API (mapred) based on the configured InputFormat, OutputFormat, Partitioner etc.
static String	getJavaOptsForMRAM(org.apache.hadoop.conf.Configuration conf) Generate JVM options based on MapReduce AM java options.
static String	getJavaOptsForMRMapper(org.apache.hadoop.conf.Configuration conf) Generate JVM options based on MapReduce mapper java options.
static String	getJavaOptsForMRReducer(org.apache.hadoop.conf.Configuration conf) Generate JVM options based on MapReduce reducer java options.
static org.apache.hadoop.yarn.api.records.Resource	getResourceForMRMapper(org.apache.hadoop.conf.Configuration conf) Extract the container resource requirements from the provided configuration, which would otherwise have been used when running a Hadoop MapReduce mapper.
static org.apache.hadoop.yarn.api.records.Resource	getResourceForMRReducer(org.apache.hadoop.conf.Configuration conf) Extract the container resource requirements from the provided configuration, which would otherwise have been used when running a Hadoop MapReduce reducer.
static void	translateMRConfToTez(org.apache.hadoop.conf.Configuration conf) Translate MapReduce configuration keys to the equivalent Tez keys in the provided configuration.
static void	translateMRConfToTez(org.apache.hadoop.conf.Configuration conf, boolean preferTez) Translate MapReduce configuration keys to the equivalent Tez keys in the provided configuration.
static void	updateEnvBasedOnMRAMEnv(org.apache.hadoop.conf.Configuration conf, Map<String,String> environment) Setup environment variables based on the configured values for the MR AM
static void	updateEnvBasedOnMRTaskEnv(org.apache.hadoop.conf.Configuration conf, Map<String,String> environment, boolean isMap) Setup classpath and other environment variables based on the configured values for MR Mappers or Reducers

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MRHelpers

public MRHelpers()

Method Detail

translateMRConfToTez

public static void translateMRConfToTez(org.apache.hadoop.conf.Configuration conf)

Translate MapReduce configuration keys to the equivalent Tez keys in the provided configuration. The translation is done in place.

This method is meant to be used by frameworks which rely upon existing MapReduce configuration instead of setting up their own.

Parameters:

conf - mr based configuration to be translated to tez

translateMRConfToTez

public static void translateMRConfToTez(org.apache.hadoop.conf.Configuration conf,
                        boolean preferTez)

Translate MapReduce configuration keys to the equivalent Tez keys in the provided configuration. The translation is done in place.

This method is meant to be used by frameworks which rely upon existing MapReduce configuration instead of setting up their own.

Parameters:

conf - mr based configuration to be translated to tez

preferTez - If the tez setting already exists and is set, use the Tez setting

configureMRApiUsage

public static void configureMRApiUsage(org.apache.hadoop.conf.Configuration conf)

Update the provided configuration to use the new API (mapreduce) or the old API (mapred) based on the configured InputFormat, OutputFormat, Partitioner etc. Also ensures that keys not required by a particular mode are not present.

This method should be invoked after completely setting up the configuration.

Defaults to using the new API if relevant keys are not present.

getJavaOptsForMRAM

public static String getJavaOptsForMRAM(org.apache.hadoop.conf.Configuration conf)

Generate JVM options based on MapReduce AM java options.

This is only meant to be used if frameworks are not setting up their own java options or relying on the defaults specified by Tez, and instead want to use the options which may already have been configured for an MR AppMaster.

Parameters:

conf - Configuration to be used to extract JVM opts specific info

Returns:

JAVA_OPTS string to be used in launching the JVM

getJavaOptsForMRMapper

public static String getJavaOptsForMRMapper(org.apache.hadoop.conf.Configuration conf)

Generate JVM options based on MapReduce mapper java options.

This is only meant to be used if frameworks are not setting up their own java options, and would like to fallback to using java options which may already be configured for Hadoop MapReduce mappers. Uses mapreduce.admin.map.child.java.opts, mapreduce.map.java.opts and mapreduce.map.log.level from config to generate the opts.

Parameters:

conf - Configuration to be used to extract JVM opts specific info

Returns:

JAVA_OPTS string to be used in launching the JVM

getJavaOptsForMRReducer

public static String getJavaOptsForMRReducer(org.apache.hadoop.conf.Configuration conf)

Generate JVM options based on MapReduce reducer java options.

This is only meant to be used if frameworks are not setting up their own java options, and would like to fallback to using java options which may already be configured for Hadoop MapReduce reducers. Uses mapreduce.admin.reduce.child.java.opts, mapreduce.reduce.java.opts and mapreduce.reduce.log.level from config to generate the opts.

Parameters:

conf - Configuration to be used to extract JVM opts specific info

Returns:

JAVA_OPTS string to be used in launching the JVM

getResourceForMRMapper

public static org.apache.hadoop.yarn.api.records.Resource getResourceForMRMapper(org.apache.hadoop.conf.Configuration conf)

Extract the container resource requirements from the provided configuration, which would otherwise have been used when running a Hadoop MapReduce mapper.

This is only meant to be used if frameworks are not setting up their own Resource and would like to fallback to using resources which may already be configured for Hadoop MapReduce mappers.

Parameters:

conf - Configuration with MR specific settings used to extract information from

Returns:

Resource object used to define requirements for containers running Map tasks

getResourceForMRReducer

public static org.apache.hadoop.yarn.api.records.Resource getResourceForMRReducer(org.apache.hadoop.conf.Configuration conf)

Extract the container resource requirements from the provided configuration, which would otherwise have been used when running a Hadoop MapReduce reducer.

This is only meant to be used if frameworks are not setting up their own Resource and would like to fallback to using resources which may already be configured for Hadoop MapReduce reducers.

Uses mapreduce.reduce.memory.mb and mapreduce.reduce.cpu.vcores from the provided configuration.

Parameters:

conf - Configuration with MR specific settings used to extract information from

Returns:

Resource object used to define requirements for containers running Reduce tasks

updateEnvBasedOnMRTaskEnv

public static void updateEnvBasedOnMRTaskEnv(org.apache.hadoop.conf.Configuration conf,
                             Map<String,String> environment,
                             boolean isMap)

Setup classpath and other environment variables based on the configured values for MR Mappers or Reducers

Parameters:

conf - Configuration to retrieve settings from

environment - Environment to update

isMap - Whether task is a map or reduce task

updateEnvBasedOnMRAMEnv

public static void updateEnvBasedOnMRAMEnv(org.apache.hadoop.conf.Configuration conf,
                           Map<String,String> environment)

Setup environment variables based on the configured values for the MR AM

Parameters:

conf - Configuration from which to extract information

environment - Environment map to update