org.apache.tez.runtime.library.resources

Class WeightedScalingMemoryDistributor

java.lang.Object

org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.tez.runtime.common.resources.InitialMemoryAllocator

@InterfaceAudience.Public
@InterfaceStability.Unstable
public class WeightedScalingMemoryDistributor
extends Object
implements org.apache.tez.runtime.common.resources.InitialMemoryAllocator

Distributes memory between various requesting components by applying a weighted scaling function. Overall, ensures that all requestors stay within the JVM limits. Configuration involves specifying weights for the different Inputs available in the tez-runtime-library. As an example, SortedShuffle : SortedOutput : UnsortedShuffle could be configured to be 20:10:1. In this case, if both SortedShuffle and UnsortedShuffle ask for the same amount of initial memory, SortedShuffle will be given 20 times more; both may be scaled down to fit within the JVM though.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	WeightedScalingMemoryDistributor.RequestType

Constructor Summary

Constructors

Constructor and Description
WeightedScalingMemoryDistributor()

Method Summary

Methods

Modifier and Type	Method and Description
Iterable<Long>	assignMemory(long availableForAllocation, int numTotalInputs, int numTotalOutputs, Iterable<org.apache.tez.runtime.common.resources.InitialMemoryRequestContext> initialRequests)
static String[]	generateWeightStrings(int unsortedPartitioned, int unsorted, int broadcastIn, int sortedOut, int scatterGatherShuffleIn, int proc, int other)
org.apache.hadoop.conf.Configuration	getConf()
void	setConf(org.apache.hadoop.conf.Configuration conf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WeightedScalingMemoryDistributor

public WeightedScalingMemoryDistributor()

Method Detail

assignMemory

public Iterable<Long> assignMemory(long availableForAllocation,
                          int numTotalInputs,
                          int numTotalOutputs,
                          Iterable<org.apache.tez.runtime.common.resources.InitialMemoryRequestContext> initialRequests)

Specified by:

assignMemory in interface org.apache.tez.runtime.common.resources.InitialMemoryAllocator

generateWeightStrings

public static String[] generateWeightStrings(int unsortedPartitioned,
                             int unsorted,
                             int broadcastIn,
                             int sortedOut,
                             int scatterGatherShuffleIn,
                             int proc,
                             int other)

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable