org.apache.tez.serviceplugins.api

Class TaskScheduler

java.lang.Object

org.apache.tez.serviceplugins.api.TaskScheduler

All Implemented Interfaces:

org.apache.tez.common.ServicePluginLifecycle

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public abstract class TaskScheduler
extends Object
implements org.apache.tez.common.ServicePluginLifecycle

This class represents the API for a custom TaskScheduler which can be run within the Tez AM. This can be used to source resources from different sources, as well as control the logic of how these resources get allocated to the different tasks within a DAG which needs resources.

The plugin is initialized with an instance of TaskSchedulerContext - which provides a mechanism to notify the system about allocation decisions and resources to the Tez framework.

Constructor Summary

Constructors

Constructor and Description
TaskScheduler(TaskSchedulerContext taskSchedulerContext)

Method Summary

Modifier and Type	Method and Description
abstract void	allocateTask(Object task, org.apache.hadoop.yarn.api.records.Resource capability, org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.Priority priority, Object containerSignature, Object clientCookie) A request to the source to allocate resources for a requesting task, based on a previously used container
abstract void	allocateTask(Object task, org.apache.hadoop.yarn.api.records.Resource capability, String[] hosts, String[] racks, org.apache.hadoop.yarn.api.records.Priority priority, Object containerSignature, Object clientCookie) A request to the source to allocate resources for a requesting task, with location information optionally specified
abstract void	blacklistNode(org.apache.hadoop.yarn.api.records.NodeId nodeId) Indication to a source that a node has been blacklisted, and should not be used for subsequent allocations.
abstract void	dagComplete() Indicates to the scheduler that the currently running dag has completed.
abstract Object	deallocateContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) A request to de-allocate a previously allocated container.
abstract boolean	deallocateTask(Object task, boolean taskSucceeded, TaskAttemptEndReason endReason, String diagnostics) A request to deallocate a task.
abstract org.apache.hadoop.yarn.api.records.Resource	getAvailableResources() Get the currently available resources from this source
abstract int	getClusterNodeCount() Get the number of nodes available from the source
TaskSchedulerContext	getContext() Get the TaskSchedulerContext associated with this instance of the scheduler, which is used to communicate with the rest of the system
int	getHeldContainersCount() Get the number of held containers.
abstract org.apache.hadoop.yarn.api.records.Resource	getTotalResources() Get the total available resources from this source
abstract boolean	hasUnregistered() Checks with the scheduler whether it has unregistered.
void	initialize() An entry point for initialization.
void	initiateStop() The first step of stopping the task scheduler service.
protected void	onContainersAllocated(List<org.apache.hadoop.yarn.api.records.Container> containers) Callback to be used in the event of a container allocation.
abstract void	setShouldUnregister() Inform the scheduler that it should unregister.
void	shutdown() Stop the service.
void	start() An entry point for starting the service.
void	taskStateUpdated(Object task, org.apache.tez.serviceplugins.api.TaskScheduler.SchedulerTaskState state) Information about the state of a previously allocated task.
abstract void	unblacklistNode(org.apache.hadoop.yarn.api.records.NodeId nodeId) Indication to a source that a node has been un-blacklisted, and can be used from subsequent allocations

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TaskScheduler

public TaskScheduler(TaskSchedulerContext taskSchedulerContext)

Method Detail

initialize

public void initialize()
                throws Exception

An entry point for initialization. Order of service setup. Constructor, initialize(), start() - when starting a service.

Specified by:

initialize in interface org.apache.tez.common.ServicePluginLifecycle

Throws:

Exception

start

public void start()
           throws Exception

An entry point for starting the service. Order of service setup. Constructor, initialize(), start() - when starting a service.

Specified by:

start in interface org.apache.tez.common.ServicePluginLifecycle

Throws:

Exception

shutdown

public void shutdown()
              throws Exception

Stop the service. This could be invoked at any point, when the service is no longer required - including in case of errors.

Specified by:

shutdown in interface org.apache.tez.common.ServicePluginLifecycle

Throws:

Exception

initiateStop

public void initiateStop()

The first step of stopping the task scheduler service. This would typically be used to stop allocating new resources. shutdown() will typically be used to unregister from external services - especially YARN for instance, so that the app is not killed

getContext

public final TaskSchedulerContext getContext()

Get the TaskSchedulerContext associated with this instance of the scheduler, which is used to communicate with the rest of the system

Returns:

an instance of TaskSchedulerContext

getAvailableResources

public abstract org.apache.hadoop.yarn.api.records.Resource getAvailableResources()
                                                                           throws org.apache.tez.serviceplugins.api.ServicePluginException

Get the currently available resources from this source

Returns:

the resources available at the time of invocation

Throws:

org.apache.tez.serviceplugins.api.ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

getTotalResources

public abstract org.apache.hadoop.yarn.api.records.Resource getTotalResources()
                                                                       throws org.apache.tez.serviceplugins.api.ServicePluginException

Get the total available resources from this source

Returns:

the total available resources from the source

Throws:

org.apache.tez.serviceplugins.api.ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

getClusterNodeCount

public abstract int getClusterNodeCount()
                                 throws org.apache.tez.serviceplugins.api.ServicePluginException

Get the number of nodes available from the source

Returns:

the number of nodes

Throws:

org.apache.tez.serviceplugins.api.ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

blacklistNode

public abstract void blacklistNode(org.apache.hadoop.yarn.api.records.NodeId nodeId)
                            throws org.apache.tez.serviceplugins.api.ServicePluginException

Indication to a source that a node has been blacklisted, and should not be used for subsequent allocations.

Parameters:

nodeId - te nodeId to be blacklisted

Throws:

org.apache.tez.serviceplugins.api.ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

unblacklistNode

public abstract void unblacklistNode(org.apache.hadoop.yarn.api.records.NodeId nodeId)
                              throws org.apache.tez.serviceplugins.api.ServicePluginException

Indication to a source that a node has been un-blacklisted, and can be used from subsequent allocations

Parameters:

nodeId - the nodeId to be unblacklisted

Throws:

org.apache.tez.serviceplugins.api.ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

allocateTask

public abstract void allocateTask(Object task,
                                  org.apache.hadoop.yarn.api.records.Resource capability,
                                  String[] hosts,
                                  String[] racks,
                                  org.apache.hadoop.yarn.api.records.Priority priority,
                                  Object containerSignature,
                                  Object clientCookie)
                           throws org.apache.tez.serviceplugins.api.ServicePluginException

A request to the source to allocate resources for a requesting task, with location information optionally specified

Parameters:

task - the task for which resources are being accepted.

capability - the required resources to run this task

hosts - the preferred host locations for the task

racks - the preferred rack locations for the task

priority - the priority of the request for this allocation. A lower value implies a higher priority

containerSignature - the specifications for the container (environment, etc) which will be used for this task - if applicable

clientCookie - a cookie associated with this request. This should be returned back via the TaskSchedulerContext.taskAllocated(Object, Object, Container) method when a task is assigned to a resource

Throws:

org.apache.tez.serviceplugins.api.ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

allocateTask

public abstract void allocateTask(Object task,
                                  org.apache.hadoop.yarn.api.records.Resource capability,
                                  org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                  org.apache.hadoop.yarn.api.records.Priority priority,
                                  Object containerSignature,
                                  Object clientCookie)
                           throws org.apache.tez.serviceplugins.api.ServicePluginException

A request to the source to allocate resources for a requesting task, based on a previously used container

Parameters:

task - the task for which resources are being accepted.

capability - the required resources to run this task

containerId - a previous container which is used as an indication as to where this task should be placed

priority - the priority of the request for this allocation. A lower value implies a higher priority

containerSignature - the specifications for the container (environment, etc) which will be used for this task - if applicable

clientCookie - a cookie associated with this request. This should be returned back via the TaskSchedulerContext.taskAllocated(Object, Object, Container) method when a task is assigned to a resource

Throws:

org.apache.tez.serviceplugins.api.ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

taskStateUpdated

public void taskStateUpdated(Object task,
                             org.apache.tez.serviceplugins.api.TaskScheduler.SchedulerTaskState state)
                      throws org.apache.tez.serviceplugins.api.ServicePluginException

Information about the state of a previously allocated task.

Parameters:

task - the task for which an update is being provided

state - the updated state

Throws:

org.apache.tez.serviceplugins.api.ServicePluginException

deallocateTask

public abstract boolean deallocateTask(Object task,
                                       boolean taskSucceeded,
                                       TaskAttemptEndReason endReason,
                                       @Nullable
                                       String diagnostics)
                                throws org.apache.tez.serviceplugins.api.ServicePluginException

A request to deallocate a task. This is typically a result of a task completing - with success or failure. It could also be the result of a decision to not run the task, before it is allocated or started.

Plugin writers need to de-allocate containers via the context once it's no longer required, for correct book-keeping

Parameters:

task - the task being de-allocated.

taskSucceeded - whether the task succeeded or not

endReason - the reason for the task failure

diagnostics - additional diagnostics information which may be relevant

Returns:

true if the task was associated with a container, false if the task was not associated

Throws:

org.apache.tez.serviceplugins.api.ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown. with a container

deallocateContainer

public abstract Object deallocateContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
                                    throws org.apache.tez.serviceplugins.api.ServicePluginException

A request to de-allocate a previously allocated container.

Parameters:

containerId - the containerId to de-allocate

Returns:

the task which was previously associated with this container, null otherwise

Throws:

org.apache.tez.serviceplugins.api.ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

setShouldUnregister

public abstract void setShouldUnregister()
                                  throws org.apache.tez.serviceplugins.api.ServicePluginException

Inform the scheduler that it should unregister. This is primarily valid for schedulers which require registration (YARN a.t.m)

Throws:

org.apache.tez.serviceplugins.api.ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

hasUnregistered

public abstract boolean hasUnregistered()
                                 throws org.apache.tez.serviceplugins.api.ServicePluginException

Checks with the scheduler whether it has unregistered.

Returns:

true if the scheduler has unregistered. False otherwise.

Throws:

org.apache.tez.serviceplugins.api.ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

dagComplete

public abstract void dagComplete()
                          throws org.apache.tez.serviceplugins.api.ServicePluginException

Indicates to the scheduler that the currently running dag has completed. This can be used to reset dag specific statistics, potentially release resources and prepare for a new DAG.

Throws:

org.apache.tez.serviceplugins.api.ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

getHeldContainersCount

public int getHeldContainersCount()

Get the number of held containers.

onContainersAllocated

protected void onContainersAllocated(List<org.apache.hadoop.yarn.api.records.Container> containers)

Callback to be used in the event of a container allocation.