Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/
AWS Glue is a serverless data-preparation service for extract, transform, and load (ETL) operations. It makes it easy for data engineers, data analysts, data scientists, and ETL developers to extract, clean, enrich, normalize, and load data.
This Universal Extension provides the capability to submit a new AWS Glue Job.
Template Name | Extension Name | Extension Version | Status |
---|---|---|---|
AWS Glue | ue-aws-glue | 2 | Fixes and new Features are introduced. |
AWS Glue | ue-aws-glue | 1 | Hot Fixes Only (Until UAC 7.2 and 7.3 are End of Support) |
Refer to Changelog for version history information.
This is a major release and introduces breaking changes that might affect some customers depending on their setup. Administrators are strongly advised to refer to Changelog for more information on the changes introduced in this release. |
This integration requires a Universal Agent and a Python runtime to execute the Universal Task.
Tested with Python version 3.7.6 and 3.11.6 and with the Universal Agent bundled Python distribution.
Both Windows and Linux agents are supported.
Universal Controller Version >=7.4.0.0
Extension's Universal Agent host should be able to reach AWS Glue REST endpoints. The AWS Credentials provided in the AWS Glue Universal Task, should have sufficient permissions on AWS to invoke Glue Jobs.
This Universal Extension provides the following key features.
To use the Universal Template, you first must perform the following steps.
This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
To import the Universal Template into your Controller, follow the instructions here.
When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.
Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.
For a new Universal Task, create a new task, and enter the required input fields.
The input fields for this Universal Extension are described in the following table.
Field | Input type | Default value | Type | Description | |
---|---|---|---|---|---|
Action | Required | Start Job Run | Choice | The action performed upon the task execution. The available actions are as follows.
| |
AWS Region Optional since version 1.1.0 | Optional | - | Text | Region for the Amazon Web Service. Find more information about the AWS Service endpoints and quotas here. When AWS Region is not populated as part of the task definition, during task execution the integration will look for credentials on the task execution environment. Refer to configuration options for more information. | |
AWS Credentials Optional since version 1.1.0 | Optional | - | Credentials | The Credentials definition should be as follows.
| |
Role Based Access | Optional | False | Boolean | Special type of authorization is provided by Role Assumption where the client sends his own credentials and the role he wants to assume from another user. If allowed, the client receives temporary credentials with limited time access to some resources. | |
Role ARN | Optional | - | Text | Role Arn: Amazon Role, which is applied for the connection. Role ARN format: Example RoleArn: arn:aws:iam::119322085622:role .Required when Role Based Access="True". | |
Job Name | Required | - | Text | Name of the Glue job that will be invoked. | |
Job Run ID | Optional | - | Text | ID of a previous Job Run to retry. | |
Security Configuration | Optional | - | Text | Name of the Security Configuration structure to be used with the Job Run. | |
Worker Type | Optional | -- None -- | Choice | Type of predefined worker that is allocated when a job runs. Available options are the following.
Introduced in version 2.0.0: G.4X, G.8X, G.025X | |
Number Of Workers | Optional | - | Integer | Number of workers of a defined Worker Type that are allocated when a job is executed. The maximum number of workers that can be defined are as follows.
| |
Job Timeout | Optional | 2880 | Integer | Job Run timeout in minutes.
| |
Notify Delay Period | Optional | - | Integer | After a job run starts, the number of minutes to wait before sending a job run delay notification. | |
Input Arguments Source | Required | Array Field | Choice | Source of job arguments with possible choices: “Array Field” or “Script”. Job arguments replace the default arguments set in the job definition, for the current run. More info here. | |
Input Arguments Script Introduced in version 1.2.0 | Optional | - | Script | 150078796 in UAC Script in JSON format. Used to pass arguments from UAC environment variables or UAC Functions. Data Type of arguments must be string and character escaping actions to be performed where needed. Check the 150078796 for more information. Visible when Input Arguments Source is configured as "Script". | |
Input Arguments | Optional | - | Array | 150078796 in array format. Visible when Input Arguments Source is configured as "Array Field". | |
Wait for Success or Failure Introduced in version 1.2.0 | Optional | False | Boolean | If selected, the task will continue running until Job reaches the "SUCCEDED" or "FAILED" state."STOPPED", "TIMEOUT","ERROR' are considered "FAILED" states. | |
Polling Interval Introduced in version 1.2.0 | Optional | 60 | Integer | The polling interval in seconds between checking for the Job status. Required when Wait for Success or Failure ="True". | |
Proxy Type | Optional | HTTP | Choice | Type of proxy connection to be used. Available options are the following.
| |
Proxy | Optional | - | Text | Comma separated list of Proxy servers. Valid formats are the following.
| |
Proxy CA Bundle File | Optional | - | Text | The path to a custom certificate bundle to use when establishing SSL/TLS connections with proxy. Used when Proxy Type is configured for "HTTPS" or "HTTPS With Credentials". | |
Proxy Credentials | Optional | - | Credentials | Credentials to be used for the proxy communication. The credential definition should be as follows.
Required when "Proxy Type" is configured for "HTTPS With Credentials". | |
Execution Class Introduced in version 2.0.0 | Optional | -- None -- | Choice | Indicates what execution class is used when the job is run. Available options are the following.
|
Start a new job run.
Start a new Job Run for a given Run ID (retries a previous execution), with all optional input argument.
Start a new Job Run for a given Run ID (retries a previous execution), with all optional input argument as above but use "Script" as Input Arguments Source.
Job arguments in UAC Script in JSON format can pass arguments from UAC Variables or UAC Functions as shown below. More information about escaping characters for json format here.
Start a new Job Run assuming a provided ARN Role, and also using a Proxy configuration.
Start a new job run, providing no AWS Credentials in task definition and providing AWS Region as Environment Variable, leaving the respective input fields empty. AWS Credentials are expected in this case to be configured on the task execution environment. Please refer to AWS Credentials input field for more information.
The output fields for this Universal Extension are described below.
Field | Type | Description |
---|---|---|
Job Run ID | text | ID of the started job run |
Job Run Status | text | Status of the job run. Generated for Action "Start Job Run" and Wait for Success or Failure = "True", updating live during execution. |
The exit codes for the Extension are described below.
Exit Code | Status Classification Code | Status Classification Description | Status Description |
---|---|---|---|
0 | SUCCESS | Successful Execution | SUCCESS: AWS Glue Job started successfully. |
0 | SUCCESS | Successful Execution with Wait for Success or Failure="True" | SUCCESS: AWS Glue Job started successfully and resulted in status SUCCEEDED. |
1 | FAIL | Failed Execution | FAIL: < Error Description >. |
1 | FAIL | Failed Execution with Wait for Success or Failure="True" | FAIL: Job Run started successfully but resulted in status < STATUS > Available values for are listed below.
|
2 | AUTHENTICATION_ERROR | Bad credentials | AUTHENTICATION_ERROR: Account cannot be authenticated. |
3 | AUTHORIZATION_ERROR | Insufficient Permissions | AUTHORIZATION_ERROR: Account is not authorized to perform the requested action. |
10 | CONNECTION_ERROR | Bad connection data or connection timed out | CONNECTION_ERROR: < Error Description >. |
11 | CONNECTION_ERROR | Extension specific connection error | CONNECTION_ERROR: ProxyConnectionError: Failed to connect to proxy URL <url> . |
20 | DATA_VALIDATION_ERROR | Input fields validation error | DATA_VALIDATION_ERROR: Some of the input fields cannot be validated. See STDERR for more details. |
21 | FAIL | User Stopped the execution | FAIL: Job Run started successfully but resulted in status STOPPED. |
In the context of a workflow, subsequent tasks can rely on the information provided by this integration as Extension Output.
Attribute changed
is populated as follows.
result
section includes the following attributes.
Attribute | Type | Description |
---|---|---|
out_job_run_id | string | ID of the started job run |
job_run_status Introduced in version 1.2.0 | text | Status of the job run. Generated for Action "Start Job Run" with Wait for Success or Failure = "True". |
started_on Introduced in version 1.2.0 | text | The date and time at which this job run was started. Generated for Action "Start Job Run" with Wait for Success or Failure = "True". |
last_modified_on Introduced in version 1.2.0 | text | The last time that this job run was modified. Generated for Action "Start Job Run" with Wait for Success or Failure = "True". |
completed_on Introduced in version 1.2.0 | text | The date and time that this job run completed. Generated for Action "Start Job Run" with Wait for Success or Failure = "True". |
error_message Introduced in version 1.2.0 | text | An error message associated with this job run. Generated for Action "Start Job Run" with Wait for Success or Failure = "True". |
An example of the Extension Output with Wait for Success or Failure = "False" for a successful triggering job is presented below.
{ "exit_code": 0, "status_description": "SUCCESS: AWS Glue Job started successfully.", "changed": true, "invocation": { "extension": "ue-aws-glue", "version": "2.0.0", "fields": { "action": "Start Job Run", "aws_credentials_user": "****", "aws_credentials_password": "****", "region": "us-east-1", "role_based_access": false, "role_arn": null, "job_name": "AWS_Glue_pythonJob", "job_run_id": null, "security_configuration": null, "worker_type": -- None --, "num_workers": null, "job_timeout": 2880, "notify_delay_period": null, "use_proxy": false, "proxy": null, "proxy_type": null, "proxy_ca_bundle_file": null, "proxy_credentials_user": null, "proxy_credentials_password": null, "wait_for_success_or_failure": false, "polling_interval": 60, "input_arguments_source": "array_field", "input_arguments_script": null, "input_arguments": [] "execution_class": -- None -- } }, "result": { "out_job_run_id": "jr_123456789" } } |
An example of the Extension Output with Wait for Success or Failure = "True" for a successful triggering job is presented below.
{ "exit_code": 0, "status_description": "SUCCESS: AWS Glue Job started successfully and resulted in status SUCCEEDED.", "changed": true, "invocation": { "extension": "ue-aws-glue", "version": "2.0.0", "fields": { "action": "Start Job Run", "aws_credentials_user": "****", "aws_credentials_password": "****", "region": "us-east-1", "role_based_access": false, "role_arn": null, "job_name": "AWS_Glue_pythonJob", "job_run_id": null, "security_configuration": null, "worker_type": -- None --, "num_workers": null, "job_timeout": 2880, "notify_delay_period": null, "use_proxy": false, "proxy": null, "proxy_type": null, "proxy_ca_bundle_file": null, "proxy_credentials_user": null, "proxy_credentials_password": null, "wait_for_success_or_failure": true, "polling_interval": 5, "input_arguments_source": "array_field", "input_arguments_script": null, "input_arguments": [], "execution_class": -- None -- } }, "result": { "job_run_id": "jr_57133f7bb82f13a29fa8813d95e2b941a3c6f5f67475227e1bb8d213e888478c", "job_run_status": "SUCCEEDED", "started_on": "2024-03-27 15:26:30.998000+02:00", "last_modified_on": "2024-03-27 15:26:58.791000+02:00", "completed_on": "2024-03-27 15:26:58.791000+02:00", "error_message": null } } |
STDOUT and STDERR provide additional information to the user.
Backward compatibility is not guaranteed for the content of STDOUT/STDERR and can be changed in future versions without notice |
Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.
Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.
This document references the following documents:
Document Link | Description |
---|---|
Universal Templates | User documentation for creating, working with and understanding Universal Templates and Integrations. |
Universal Tasks | User documentation for creating Universal Tasks in the Universal Controller user interface. |
Credentials | User documentation for creating and working with credentials. |
Resolvable Credentials Permitted Property | User documentation for Resolvable Credentials Permitted Property. |
Breaking Change:
drop support for agent 7.3.X or lower, agent version 7.4.X or higher is requiredAdded:
new input field - Execution ClassAdded:
newer worker types compatible with AWS Glue version 3, are now supported Fixed
: auto-renew AWS temporary credentials before expiration when using ARN based accessFixed
: fixed issue with polling logic where task would get stuck with status Running (#35135)Added
: Support Start Glue Job and Wait until Job Reaches status "Succeeded" or "Failed" (#30157)Added
: Larger set of output fields (#30157)Added
: Log payload response for Job Run Status and Start Glue Job Run Action on debug mode.Added
: Option to pass Input Arguments as UAC script supporting UAC environment variables and UAC Functions.Added
: Allow AWS Credentials and AWS Region as optional fields enabling their configuration on the task execution environment. (#28312)Initial Version