org.apache.tez.dag.api

Class PreWarmVertex

java.lang.Object

org.apache.tez.dag.api.Vertex

org.apache.tez.dag.api.PreWarmVertex

@InterfaceStability.Unstable
@InterfaceAudience.Public
public class PreWarmVertex
extends Vertex

A PreWarmVertex is used to specify parameters to be used to setup prewarmed containers for Tez session mode. Sessions allow re-use of execution slots (containers) across DAG's. Pre- warming allows pre-allocation of containers so that the first DAG has some execution resources already available to re-use. In order to get re-use containers they must be setup identically. So the prewarm vertex must be setup identically to the real DAG vertex (typically the first vertex to execute in the read DAG). Identical settings include same execution resources, same task local files etc. This works best in use cases where all DAGs share the same files/jars/resource settings from a common template
The parallelism of the pre-warm vertex determines the number of containers to be pre-warmed. This would ideally ensures a viable number of containers to provide performance while sharing resources with other applications. Typically the session would also hold onto the same number of containers in-between DAGs in session mode via the TezConfiguration.TEZ_AM_SESSION_MIN_HELD_CONTAINERS property. The prewarm vertex by default runs the PreWarmProcessor from the Tez runtime library. This processor can be overridden to get the default behavior along with any app specific customizations. Alternatively, the application can provide any Processor to prewarm the containers. Pre-warming processors can be used to initialize classes etc. and setup the environment for the actual processing to reduce latency.

Method Summary

Methods

Modifier and Type	Method and Description
static PreWarmVertex	create(String vertexName, int parallelism, org.apache.hadoop.yarn.api.records.Resource taskResource) Create a PreWarmVertex to be used in @link TezClient.preWarm(PreWarmVertex) This uses a built in pre-warm processor that implements common functionality.
static PreWarmVertex	create(String vertexName, ProcessorDescriptor processorDescriptor, int parallelism, org.apache.hadoop.yarn.api.records.Resource taskResource) Create a PreWarmVertex to be used in TezClient.preWarm(PreWarmVertex) It may be necessary to call more methods to add local files etc on the pre-warm vertex post creation so that it matches the real DAG vertices.
static org.apache.tez.dag.api.PreWarmVertex.PreWarmVertexConfigBuilder	createConfigBuilder(org.apache.hadoop.conf.Configuration conf) Create a config builder for the @link PreWarmVertex.

Methods inherited from class org.apache.tez.dag.api.Vertex

addDataSink, addDataSource, addTaskLocalFiles, create, create, getInputVertices, getName, getOutputVertices, getParallelism, getProcessorDescriptor, getTaskEnvironment, getTaskLaunchCmdOpts, getTaskLocalFiles, getTaskResource, setLocationHint, setTaskEnvironment, setTaskLaunchCmdOpts, setVertexManagerPlugin, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

createConfigBuilder

public static org.apache.tez.dag.api.PreWarmVertex.PreWarmVertexConfigBuilder createConfigBuilder(org.apache.hadoop.conf.Configuration conf)

Create a config builder for the @link PreWarmVertex. This may be used to construct the pre-warm vertex more flexibly.

Parameters:

conf -

Returns:

a new config builder for PreWarmVertex

create

public static PreWarmVertex create(String vertexName,
                   ProcessorDescriptor processorDescriptor,
                   int parallelism,
                   org.apache.hadoop.yarn.api.records.Resource taskResource)

Create a PreWarmVertex to be used in TezClient.preWarm(PreWarmVertex) It may be necessary to call more methods to add local files etc on the pre-warm vertex post creation so that it matches the real DAG vertices.

Parameters:

vertexName - Name of the vertex

processorDescriptor - Descriptor of the processor to be run

parallelism - Number of containers to be pre-warmed

taskResource - Execution cpu/memory resources etc needed

create

public static PreWarmVertex create(String vertexName,
                   int parallelism,
                   org.apache.hadoop.yarn.api.records.Resource taskResource)

Create a PreWarmVertex to be used in @link TezClient.preWarm(PreWarmVertex) This uses a built in pre-warm processor that implements common functionality. Users may derive from this processor to add custom functionality but then they must add the jar for that class to the prewarm vertex and other vertices in their DAG for which they want the containers to be reused. It may be necessary to call more methods to add local files etc on the pre-warm vertex post creation so that it matches the real DAG vertices.

Parameters:

vertexName - Name of the vertex

parallelism - Number of containers to be pre-warmed

taskResource - Execution cpu/memory resources etc needed