@InterfaceStability.Unstable @InterfaceAudience.Public public class PreWarmVertex extends Vertex
PreWarmVertex
is used to specify parameters to be used to setup
prewarmed containers for Tez session mode. Sessions allow re-use of execution
slots (containers) across DAG's. Pre- warming allows pre-allocation of
containers so that the first DAG has some execution resources already
available to re-use. In order to get re-use containers they must be setup
identically. So the prewarm vertex must be setup identically to the real DAG
vertex (typically the first vertex to execute in the read DAG). Identical
settings include same execution resources, same task local files etc. This
works best in use cases where all DAGs share the same files/jars/resource
settings from a common templateTezConfiguration.TEZ_AM_SESSION_MIN_HELD_CONTAINERS
property. The
prewarm vertex by default runs the PreWarmProcessor from the Tez runtime
library. This processor can be overridden to get the default behavior along
with any app specific customizations. Alternatively, the application can
provide any Processor
to prewarm the containers. Pre-warming
processors can be used to initialize classes etc. and setup the environment
for the actual processing to reduce latency.Vertex.VertexExecutionContext
Modifier and Type | Method and Description |
---|---|
static PreWarmVertex |
create(String vertexName,
int parallelism,
org.apache.hadoop.yarn.api.records.Resource taskResource)
Create a
PreWarmVertex to be used in @link
TezClient.preWarm(PreWarmVertex) This uses a built in pre-warm
processor that implements common functionality. |
static PreWarmVertex |
create(String vertexName,
ProcessorDescriptor processorDescriptor,
int parallelism,
org.apache.hadoop.yarn.api.records.Resource taskResource)
Create a
PreWarmVertex to be used in
TezClient.preWarm(PreWarmVertex) It may be necessary to call more
methods to add local files etc on the pre-warm vertex post creation so that
it matches the real DAG vertices. |
static org.apache.tez.dag.api.PreWarmVertex.PreWarmVertexConfigBuilder |
createConfigBuilder(org.apache.hadoop.conf.Configuration conf)
Create a config builder for the @link
PreWarmVertex . |
addDataSink, addDataSource, addTaskLocalFiles, create, create, equals, getConf, getInputVertices, getName, getOutputVertices, getParallelism, getProcessorDescriptor, getTaskEnvironment, getTaskLaunchCmdOpts, getTaskLocalFiles, getTaskResource, hashCode, setConf, setExecutionContext, setLocationHint, setTaskEnvironment, setTaskLaunchCmdOpts, setVertexManagerPlugin, toString
public static org.apache.tez.dag.api.PreWarmVertex.PreWarmVertexConfigBuilder createConfigBuilder(org.apache.hadoop.conf.Configuration conf)
PreWarmVertex
. This may be used to construct the
pre-warm vertex more flexibly.conf
- PreWarmVertex
public static PreWarmVertex create(String vertexName, ProcessorDescriptor processorDescriptor, int parallelism, org.apache.hadoop.yarn.api.records.Resource taskResource)
PreWarmVertex
to be used in
TezClient.preWarm(PreWarmVertex)
It may be necessary to call more
methods to add local files etc on the pre-warm vertex post creation so that
it matches the real DAG vertices.vertexName
- Name of the vertexprocessorDescriptor
- Descriptor of the processor to be runparallelism
- Number of containers to be pre-warmedtaskResource
- Execution cpu/memory resources etc neededpublic static PreWarmVertex create(String vertexName, int parallelism, org.apache.hadoop.yarn.api.records.Resource taskResource)
PreWarmVertex
to be used in @link
TezClient.preWarm(PreWarmVertex)
This uses a built in pre-warm
processor that implements common functionality. Users may derive from this
processor to add custom functionality but then they must add the jar for
that class to the prewarm vertex and other vertices in their DAG for which
they want the containers to be reused. It may be necessary to call more
methods to add local files etc on the pre-warm vertex post creation so that
it matches the real DAG vertices.vertexName
- Name of the vertexparallelism
- Number of containers to be pre-warmedtaskResource
- Execution cpu/memory resources etc neededCopyright © 2024 Apache Software Foundation. All rights reserved.