org.apache.tez.client

Class TezClient

java.lang.Object

org.apache.tez.client.TezClient

@InterfaceAudience.Public
public class TezClient
extends Object

TezClient is used to submit Tez DAGs for execution. DAG's are executed via a Tez App Master. TezClient can run the App Master in session or non-session mode.
In non-session mode, each DAG is executed in a different App Master that exits after the DAG execution completes.
In session mode, the TezClient creates a single instance of the App Master and all DAG's are submitted to the same App Master.
Session mode may give better performance when a series of DAGs need to executed because it enables resource re-use across those DAGs. Non-session mode should be used when the user wants to submit a single DAG or wants to disconnect from the cluster after submitting a set of unrelated DAGs.
If API recommendations are followed, then the choice of running in session or non-session mode is transparent to writing the application. By changing the session mode configuration, the same application can be running in session or non-session mode.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TezClient.TezClientBuilder A builder for setting up an instance of TezClient

Field Summary

Fields

Modifier and Type	Field and Description
static String	DAG

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	TezClient(String name, TezConfiguration tezConf, boolean isSession, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources, org.apache.hadoop.security.Credentials credentials)
protected	TezClient(String name, TezConfiguration tezConf, boolean isSession, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources, org.apache.hadoop.security.Credentials credentials, ServicePluginsDescriptor servicePluginsDescriptor)

Method Summary

Modifier and Type	Method and Description
void	addAppMasterLocalFiles(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localFiles) Add local files for the DAG App Master.
static org.apache.hadoop.yarn.api.records.ApplicationId	appIdfromString(String appIdStr)
void	clearAppMasterLocalFiles() If the next DAG App Master needs different local files, then use this method to clear the local files and then add the new local files using addAppMasterLocalFiles(Map).
static TezClient	create(String name, TezConfiguration tezConf) Create a new TezClient.
static TezClient	create(String name, TezConfiguration tezConf, boolean isSession) Create a new TezClient with AM session mode set explicitly.
static TezClient	create(String name, TezConfiguration tezConf, boolean isSession, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localFiles, org.apache.hadoop.security.Credentials credentials) Create a new TezClient with AM session mode set explicitly.
static TezClient	create(String name, TezConfiguration tezConf, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localFiles, org.apache.hadoop.security.Credentials credentials) Create a new TezClient.
String	getAmHost()
protected ScheduledExecutorService	getAMKeepAliveService()
int	getAmPort()
TezAppMasterStatus	getAppMasterStatus() Get the status of the App Master executing the DAG In non-session mode it returns the status of the last submitted DAG App Master In session mode, it returns the status of the App Master hosting the session
TezClient	getClient(org.apache.hadoop.yarn.api.records.ApplicationId appId) Alternative to start() that explicitly sets sessionAppId and doesn't start a new AM.
TezClient	getClient(String appIdStr)
String	getClientName() Get the name of the client
protected long	getPrewarmWaitTimeMs()
static TezClient.TezClientBuilder	newBuilder(String name, TezConfiguration tezConf) Create a new TezClientBuilder.
void	preWarm(PreWarmVertex preWarmVertex) API to help pre-allocate containers in session mode.
void	preWarm(PreWarmVertex preWarmVertex, long timeout, TimeUnit unit) API to help pre-allocate containers in session mode.
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB	sendAMHeartbeat(org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB proxy)
void	setAppMasterCredentials(org.apache.hadoop.security.Credentials credentials) Set security credentials to be used inside the app master, if needed.
void	setHistoryLogLevel(HistoryLogLevel historyLogLevel) Sets the history log level for this session.
void	start() Start the client.
void	stop() Stop the client.
DAGClient	submitDAG(DAG dag) Submit a DAG.
void	waitTillReady() Wait till the DAG is ready to be submitted.
boolean	waitTillReady(long timeout, TimeUnit unit) Wait till the DAG is ready to be submitted.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DAG

public static final String DAG

See Also:

Constant Field Values

Constructor Detail

TezClient

@InterfaceAudience.Private
protected TezClient(String name,
                                               TezConfiguration tezConf,
                                               boolean isSession,
                                               @Nullable
                                               Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources,
                                               @Nullable
                                               org.apache.hadoop.security.Credentials credentials)

TezClient

@InterfaceAudience.Private
protected TezClient(String name,
                                               TezConfiguration tezConf,
                                               boolean isSession,
                                               @Nullable
                                               Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources,
                                               @Nullable
                                               org.apache.hadoop.security.Credentials credentials,
                                               ServicePluginsDescriptor servicePluginsDescriptor)

Method Detail

newBuilder

public static TezClient.TezClientBuilder newBuilder(String name,
                                                    TezConfiguration tezConf)

Create a new TezClientBuilder. This can be used to setup additional parameters like session mode, local resources, credentials, servicePlugins, etc.

If session mode is not specified in the builder, this will be inferred from the provided TezConfiguration.

Parameters:

name - Name of the client. Used for logging etc. This will also be used as app master name is session mode

tezConf - Configuration for the framework

Returns:

An instance of TezClient.TezClientBuilder which can be used to construct the final TezClient.

create

public static TezClient create(String name,
                               TezConfiguration tezConf)

Create a new TezClient. Session or non-session execution mode will be inferred from configuration.

Parameters:

name - Name of the client. Used for logging etc. This will also be used as app master name is session mode

tezConf - Configuration for the framework

create

public static TezClient create(String name,
                               TezConfiguration tezConf,
                               @Nullable
                               Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localFiles,
                               @Nullable
                               org.apache.hadoop.security.Credentials credentials)

Create a new TezClient. Session or non-session execution mode will be inferred from configuration. Set the initial resources and security credentials for the App Master. If app master resources/credentials are needed then this is the recommended method for session mode execution.

Parameters:

name - Name of the client. Used for logging etc. This will also be used as app master name is session mode

tezConf - Configuration for the framework

localFiles - local files for the App Master

credentials - Set security credentials to be used inside the app master, if needed. Tez App Master needs credentials to access the staging directory and for most HDFS cases these are automatically obtained by Tez client. If the staging directory is on a file system for which credentials cannot be obtained or for any credentials needed by user code running inside the App Master, credentials must be supplied by the user. These will be used by the App Master for the next DAG.
In session mode, credentials, if needed, must be set before calling start()

create

public static TezClient create(String name,
                               TezConfiguration tezConf,
                               boolean isSession)

Create a new TezClient with AM session mode set explicitly. This overrides the setting from configuration.

Parameters:

name - Name of the client. Used for logging etc. This will also be used as app master name is session mode

tezConf - Configuration for the framework

isSession - The AM will run in session mode or not

create

public static TezClient create(String name,
                               TezConfiguration tezConf,
                               boolean isSession,
                               @Nullable
                               Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localFiles,
                               @Nullable
                               org.apache.hadoop.security.Credentials credentials)

Create a new TezClient with AM session mode set explicitly. This overrides the setting from configuration. Set the initial files and security credentials for the App Master.

Parameters:

name - Name of the client. Used for logging etc. This will also be used as app master name is session mode

tezConf - Configuration for the framework

isSession - The AM will run in session mode or not

localFiles - local files for the App Master

credentials - credentials for the App Master

addAppMasterLocalFiles

public void addAppMasterLocalFiles(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localFiles)

Add local files for the DAG App Master. These may be files, archives, jars etc.

In non-session mode these will be added to the files of the App Master to be launched for the next DAG. Files added via this method will accumulate and be used for every new App Master until clearAppMasterLocalFiles() is invoked.

In session mode, the recommended usage is to add all files before calling start() so that all needed files are available to the app master before it starts. When called after start(), these local files will be re-localized to the running session DAG App Master and will be added to its classpath for execution of this DAG.

Caveats for invoking this method after start() in session mode: files accumulate across DAG submissions and are never removed from the classpath. Only LocalResourceType.FILE is supported. All files will be treated as private.

Parameters:

localFiles - the files to be made available in the AM

clearAppMasterLocalFiles

public void clearAppMasterLocalFiles()

If the next DAG App Master needs different local files, then use this method to clear the local files and then add the new local files using addAppMasterLocalFiles(Map). This method is a no-op in session mode, after start() is called.

setAppMasterCredentials

public void setAppMasterCredentials(org.apache.hadoop.security.Credentials credentials)

Set security credentials to be used inside the app master, if needed. Tez App Master needs credentials to access the staging directory and for most HDFS cases these are automatically obtained by Tez client. If the staging directory is on a file system for which credentials cannot be obtained or for any credentials needed by user code running inside the App Master, credentials must be supplied by the user. These will be used by the App Master for the next DAG.
In session mode, credentials, if needed, must be set before calling start()

Parameters:

credentials - credentials

setHistoryLogLevel

public void setHistoryLogLevel(HistoryLogLevel historyLogLevel)

Sets the history log level for this session. It will be in effect for DAGs submitted after this call.

Parameters:

historyLogLevel - The log level to be used.

start

public void start()
           throws TezException,
                  IOException

Start the client. This establishes a connection to the YARN cluster. In session mode, this start the App Master thats runs all the DAGs in the session.

Throws:

TezException

IOException

getClient

public TezClient getClient(String appIdStr)
                    throws IOException,
                           TezException

Throws:

IOException

TezException

getClient

public TezClient getClient(org.apache.hadoop.yarn.api.records.ApplicationId appId)
                    throws TezException,
                           IOException

Alternative to start() that explicitly sets sessionAppId and doesn't start a new AM. The caller of getClient is responsible for initializing the new TezClient with a Configuration compatible with the existing AM. It is expected the caller has cached the original Configuration (e.g. in Zookeeper). In contrast to "start", no resources are localized. It is the responsibility of the caller to ensure that existing localized resources and staging dirs are still valid.

Parameters:

appId -

Returns:

'this' just as a convenience for fluent style chaining

Throws:

TezException

IOException

sendAMHeartbeat

public org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB sendAMHeartbeat(org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB proxy)

submitDAG

public DAGClient submitDAG(DAG dag)
                    throws TezException,
                           IOException

Submit a DAG.
In non-session mode, it submits a new App Master to the cluster.
In session mode, it submits the DAG to the session App Master. It blocks until either the DAG is submitted to the session or configured timeout period expires. Cleans up session if the submission timed out.

Parameters:

dag - DAG to be submitted to Session

Returns:

DAGClient to monitor the DAG

Throws:

TezException

IOException

DAGSubmissionTimedOut - if submission timed out

getPrewarmWaitTimeMs

protected long getPrewarmWaitTimeMs()

stop

public void stop()
          throws TezException,
                 IOException

Stop the client. This terminates the connection to the YARN cluster. In session mode, this shuts down the session DAG App Master

Throws:

TezException

IOException

getClientName

public String getClientName()

Get the name of the client

Returns:

name

getAppMasterStatus

public TezAppMasterStatus getAppMasterStatus()
                                      throws TezException,
                                             IOException

Get the status of the App Master executing the DAG In non-session mode it returns the status of the last submitted DAG App Master In session mode, it returns the status of the App Master hosting the session

Returns:

State of the session

Throws:

TezException

IOException

preWarm

@InterfaceStability.Unstable
public void preWarm(PreWarmVertex preWarmVertex)
                                          throws TezException,
                                                 IOException

API to help pre-allocate containers in session mode. In non-session mode this is ignored. The pre-allocated containers may be re-used by subsequent job DAGs to improve performance. The preWarm vertex should be configured and setup exactly like the other vertices in the job DAGs so that the pre-allocated containers may be re-used by the subsequent DAGs to improve performance. The processor for the preWarmVertex may be used to pre-warm the containers by pre-loading classes etc. It should be short-running so that pre-warming does not block real execution. Users can specify their custom processors or use the PreWarmProcessor from the runtime library. The parallelism of the preWarmVertex will determine the number of preWarmed containers. Pre-warming is best efforts and among other factors is limited by the free resources on the cluster.

Parameters:

preWarmVertex -

Throws:

TezException

IOException

preWarm

@InterfaceStability.Unstable
public void preWarm(PreWarmVertex preWarmVertex,
                                                 long timeout,
                                                 TimeUnit unit)
                                          throws TezException,
                                                 IOException

API to help pre-allocate containers in session mode. In non-session mode this is ignored. The pre-allocated containers may be re-used by subsequent job DAGs to improve performance. The preWarm vertex should be configured and setup exactly like the other vertices in the job DAGs so that the pre-allocated containers may be re-used by the subsequent DAGs to improve performance. The processor for the preWarmVertex may be used to pre-warm the containers by pre-loading classes etc. It should be short-running so that pre-warming does not block real execution. Users can specify their custom processors or use the PreWarmProcessor from the runtime library. The parallelism of the preWarmVertex will determine the number of preWarmed containers. Pre-warming is best efforts and among other factors is limited by the free resources on the cluster. Based on the specified timeout value it returns false if the status is not READY after the wait period.

Parameters:

preWarmVertex -

timeout -

unit -

Throws:

TezException

IOException

waitTillReady

@InterfaceStability.Evolving
public void waitTillReady()
                                                throws IOException,
                                                       TezException,
                                                       InterruptedException

Wait till the DAG is ready to be submitted. In non-session mode this is a no-op since the application can be immediately submitted. In session mode, this waits for the session host to be ready to accept a DAG

Throws:

IOException

TezException

InterruptedException

waitTillReady

@InterfaceStability.Evolving
public boolean waitTillReady(long timeout,
                                                          TimeUnit unit)
                                                   throws IOException,
                                                          TezException,
                                                          InterruptedException

Wait till the DAG is ready to be submitted. In non-session mode this is a no-op since the application can be immediately submitted. In session mode, this waits for the session host to be ready to accept a DAG and returns false if not ready after a configured time wait period.

Parameters:

timeout -

unit -

Returns:

true if READY or is not in session mode, false otherwise.

Throws:

IOException

TezException

InterruptedException

getAMKeepAliveService

protected ScheduledExecutorService getAMKeepAliveService()

appIdfromString

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public static org.apache.hadoop.yarn.api.records.ApplicationId appIdfromString(String appIdStr)

getAmHost

public String getAmHost()

getAmPort

public int getAmPort()