org.apache.tez.common

Class TezCommonUtils

java.lang.Object

org.apache.tez.common.TezCommonUtils

@InterfaceAudience.Private
public class TezCommonUtils
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.permission.FsPermission	TEZ_AM_DIR_PERMISSION
static org.apache.hadoop.fs.permission.FsPermission	TEZ_AM_FILE_PERMISSION
static String	TEZ_SYSTEM_SUB_DIR

Constructor Summary

Constructors

Constructor and Description
TezCommonUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static void	addAdditionalLocalResources(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> additionalLrs, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> originalLRs, String logContext)
static com.google.protobuf.ByteString	compressByteArrayToByteString(byte[] inBytes)
static ByteBuffer	convertJobTokenToBytes(org.apache.hadoop.security.token.Token<JobTokenIdentifier> jobToken)
static org.apache.hadoop.fs.FSDataOutputStream	createFileForAM(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path filePath) Create a file with TEZ_AM_FILE_PERMISSION permission and returns OutputStream
static org.apache.hadoop.fs.Path	createTezSystemStagingPath(org.apache.hadoop.conf.Configuration conf, String strAppId) This function returns the staging directory for TEZ system.
static byte[]	decompressByteStringToByteArray(com.google.protobuf.ByteString byteString)
static org.apache.hadoop.fs.Path	getAttemptRecoveryPath(org.apache.hadoop.fs.Path recoveryPath, int attemptID) Returns a path to store app attempt specific recovery details
static String	getCredentialsInfo(org.apache.hadoop.security.Credentials credentials, String identifier)
static org.apache.hadoop.fs.Path	getDAGRecoveryPath(org.apache.hadoop.fs.Path attemptRecoverPath, String dagID) Returns a path to store DAG specific recovery info
static org.apache.hadoop.fs.Path	getRecoveryPath(org.apache.hadoop.fs.Path tezSysStagingPath, org.apache.hadoop.conf.Configuration conf) Returns a path to store recovery information
static org.apache.hadoop.fs.Path	getSummaryRecoveryPath(org.apache.hadoop.fs.Path attemptRecoverPath) Returns a path to store summary info for recovery
static org.apache.hadoop.fs.Path	getTezAMJarStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath) Returns a path to store local resources/session jars
static org.apache.hadoop.fs.Path	getTezBaseStagingPath(org.apache.hadoop.conf.Configuration conf) This function returns the staging directory defined in the config with property name TezConfiguration.TEZ_AM_STAGING_DIR.
static org.apache.hadoop.fs.Path	getTezBinPlanStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath) Returns a path to store binary plan
static org.apache.hadoop.fs.Path	getTezConfStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath) Returns a path to store binary configuration
static org.apache.hadoop.fs.Path	getTezSystemStagingPath(org.apache.hadoop.conf.Configuration conf, String strAppId) This function returns the staging directory for TEZ system.
static org.apache.hadoop.fs.Path	getTezTextPlanStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath, String strAppId, String dagPBName) Returns a path to store text plan
static String[]	getTrimmedStrings(String str) Splits a comma separated value String, trimming leading and trailing whitespace on each value.
static void	logCredentials(org.slf4j.Logger log, org.apache.hadoop.security.Credentials credentials, String identifier)
static void	mkDirForAM(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir) Create a directory with predefined directory permission
static org.apache.hadoop.security.Credentials	parseCredentialsBytes(byte[] credentialsBytes)
static ByteBuffer	serializeServiceData(org.apache.hadoop.security.token.Token<JobTokenIdentifier> jobToken) A helper function to serialize the JobTokenIdentifier to be sent to the ShuffleHandler as ServiceData.
static Collection<String>	tokenizeString(String str, String delim)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEZ_AM_DIR_PERMISSION

public static final org.apache.hadoop.fs.permission.FsPermission TEZ_AM_DIR_PERMISSION

TEZ_AM_FILE_PERMISSION

public static final org.apache.hadoop.fs.permission.FsPermission TEZ_AM_FILE_PERMISSION

TEZ_SYSTEM_SUB_DIR

public static final String TEZ_SYSTEM_SUB_DIR

See Also:

Constant Field Values

Constructor Detail

TezCommonUtils

public TezCommonUtils()

Method Detail

getTezBaseStagingPath

public static org.apache.hadoop.fs.Path getTezBaseStagingPath(org.apache.hadoop.conf.Configuration conf)

This function returns the staging directory defined in the config with property name TezConfiguration.TEZ_AM_STAGING_DIR. If the property is not defined in the conf, Tez uses the value defined as TezConfiguration.TEZ_AM_STAGING_DIR_DEFAULT. In addition, the function makes sure if the staging directory exists. If not, it creates the directory with permission TEZ_AM_DIR_PERMISSION.

Parameters:

conf - TEZ configuration

Returns:

Fully qualified staging directory

createTezSystemStagingPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path createTezSystemStagingPath(org.apache.hadoop.conf.Configuration conf,
                                                                             String strAppId)

This function returns the staging directory for TEZ system. Tez creates all its temporary files under this sub-directory. The function creates a sub-directory (TEZ_SYSTEM_SUB_DIR/) under the base staging directory, often provided by user.

Parameters:

conf - Tez configuration

strAppId - Application ID as string

Returns:

TEZ system level staging directory used for Tez internals

getTezSystemStagingPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getTezSystemStagingPath(org.apache.hadoop.conf.Configuration conf,
                                                                          String strAppId)

This function returns the staging directory for TEZ system. Tez creates all its temporary files under this sub-directory. The function normally doesn't creates any sub-directory under the base staging directory.

Parameters:

conf - Tez configuration

strAppId - Application ID as string

Returns:

TEZ system level staging directory used for Tez internals

getTezConfStagingPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getTezConfStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath)

Returns a path to store binary configuration

Parameters:

tezSysStagingPath - TEZ system level staging directory used for Tez internals

Returns:

path to configuration

getTezAMJarStagingPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getTezAMJarStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath)

Returns a path to store local resources/session jars

Parameters:

tezSysStagingPath - TEZ system level staging directory used for Tez internals

Returns:

path to store the session jars

getTezBinPlanStagingPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getTezBinPlanStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath)

Returns a path to store binary plan

Parameters:

tezSysStagingPath - TEZ system level staging directory used for Tez internals

Returns:

path to store the plan in binary

getTezTextPlanStagingPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getTezTextPlanStagingPath(org.apache.hadoop.fs.Path tezSysStagingPath,
                                                                            String strAppId,
                                                                            String dagPBName)

Returns a path to store text plan

Parameters:

tezSysStagingPath - TEZ system level staging directory used for Tez internals

strAppId - Application ID

dagPBName - DAG PB Name

Returns:

path to store the plan in text

getRecoveryPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getRecoveryPath(org.apache.hadoop.fs.Path tezSysStagingPath,
                                                                  org.apache.hadoop.conf.Configuration conf)
                                                 throws IOException

Returns a path to store recovery information

Parameters:

tezSysStagingPath - TEZ system level staging directory used for Tez internals

conf - Tez configuration

Returns:

App recovery path

Throws:

IOException

getAttemptRecoveryPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getAttemptRecoveryPath(org.apache.hadoop.fs.Path recoveryPath,
                                                                         int attemptID)

Returns a path to store app attempt specific recovery details

Parameters:

recoveryPath - TEZ recovery directory used for Tez internals

attemptID - Application Attempt Id

Returns:

App attempt specific recovery path

getDAGRecoveryPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getDAGRecoveryPath(org.apache.hadoop.fs.Path attemptRecoverPath,
                                                                     String dagID)

Returns a path to store DAG specific recovery info

Parameters:

attemptRecoverPath - :TEZ system level staging directory used for Tez internals

dagID - DagID as string

Returns:

DAG specific recovery path

getSummaryRecoveryPath

@InterfaceAudience.Private
public static org.apache.hadoop.fs.Path getSummaryRecoveryPath(org.apache.hadoop.fs.Path attemptRecoverPath)

Returns a path to store summary info for recovery

Parameters:

attemptRecoverPath - TEZ system level staging directory used for Tez internals

Returns:

Summary event path used in recovery

mkDirForAM

public static void mkDirForAM(org.apache.hadoop.fs.FileSystem fs,
              org.apache.hadoop.fs.Path dir)
                       throws IOException

Create a directory with predefined directory permission

Parameters:

fs - Filesystem

dir - directory to be created

Throws:

IOException

createFileForAM

public static org.apache.hadoop.fs.FSDataOutputStream createFileForAM(org.apache.hadoop.fs.FileSystem fs,
                                                      org.apache.hadoop.fs.Path filePath)
                                                               throws IOException

Create a file with TEZ_AM_FILE_PERMISSION permission and returns OutputStream

Parameters:

fs - Filesystem

filePath - file path to create the file

Returns:

FSDataOutputStream

Throws:

IOException

addAdditionalLocalResources

public static void addAdditionalLocalResources(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> additionalLrs,
                               Map<String,org.apache.hadoop.yarn.api.records.LocalResource> originalLRs,
                               String logContext)

compressByteArrayToByteString

@InterfaceAudience.Private
public static com.google.protobuf.ByteString compressByteArrayToByteString(byte[] inBytes)
                                                                    throws IOException

Throws:

IOException

decompressByteStringToByteArray

@InterfaceAudience.Private
public static byte[] decompressByteStringToByteArray(com.google.protobuf.ByteString byteString)
                                              throws IOException

Throws:

IOException

getCredentialsInfo

public static String getCredentialsInfo(org.apache.hadoop.security.Credentials credentials,
                        String identifier)

convertJobTokenToBytes

public static ByteBuffer convertJobTokenToBytes(org.apache.hadoop.security.token.Token<JobTokenIdentifier> jobToken)
                                         throws IOException

Throws:

IOException

parseCredentialsBytes

public static org.apache.hadoop.security.Credentials parseCredentialsBytes(byte[] credentialsBytes)
                                                                    throws IOException

Throws:

IOException

logCredentials

public static void logCredentials(org.slf4j.Logger log,
                  org.apache.hadoop.security.Credentials credentials,
                  String identifier)

tokenizeString

public static Collection<String> tokenizeString(String str,
                                String delim)

getTrimmedStrings

public static String[] getTrimmedStrings(String str)

Splits a comma separated value String, trimming leading and trailing whitespace on each value.

Parameters:

str - a comma separated with values

Returns:

an array of String values

serializeServiceData

public static ByteBuffer serializeServiceData(org.apache.hadoop.security.token.Token<JobTokenIdentifier> jobToken)
                                       throws IOException

A helper function to serialize the JobTokenIdentifier to be sent to the ShuffleHandler as ServiceData. *NOTE* This is a copy of what is done by the MapReduce ShuffleHandler. Not using that directly to avoid a dependency on mapreduce.

Parameters:

jobToken - the job token to be used for authentication of shuffle data requests.

Returns:

the serialized version of the jobToken.

Throws:

IOException