IBM InfoSphere DataStage Jobs
Disclaimer
Your use of this download is governed by Stonebranch’s Terms of Use, which are available at Stonebranch Integration Hub - Terms of Use.
Overview
IBM DataStage is a powerful data integration and ETL (Extract, Transform, Load) tool by IBM and it helps organizations extract data from various sources, transform it to meet specific business requirements, and load it into target systems such as data warehouses, data marts, and operational data stores. This Universal Task allows Stonebranch users to schedule, trigger, monitor, and orchestrate the InfoSphere DataStage jobs directly from Universal Controller via the InfoSphere DataStage Command-line Interface(CLI). It also helps to set up dependencies between Datastage jobs and other applications or platforms using UAC workflows and trigger DataStage Jobs from UAC either time-based or event-based.
Version Information
| Template Name | Version |
|---|---|
| CS Datastage CLI | 1.0.0 |
Refer to Changelog for version history information.
Software Requirements
This integration requires a Universal Agent and a Python runtime to execute the Universal Task.
Software Requirements for Universal Template and Universal Task
Requires Python 3.7.6 or higher. Tested with the Universal Agent bundled Python distribution.
Software Requirements for Universal Agent
Both Windows and Linux agents are supported.
- Universal Agent for Windows x64 Version 7.1.0.0 and later with Python options installed.
- Universal Agent for Linux Version 7.1.0.0 and later with Python options installed.
- IBM InfoSphere DataStage Client (CLI) installed in the agent host and accessible to the UAC execution User.
- IBM InfoSphere DataStage Client supported versions - 8.5,9.1,11.3,11.5 & 11.7
Software Requirements for Universal Controller
Universal Controller Version 7.1.0.0 and later.
Network and Connectivity Requirements
IBM InfoSphere DataStage Client (CLI) installed in the agent host is configured to connect with the DataStage Server.
Key Features
This Universal Task provides the following features:
- Ability to schedule, trigger & orchestrate the execution of Datastage jobs.
- Monitor the DataStage job execution.
- Resumes a Restartable DataStage Job from UAC ( “-mode RESTART” to the parameters).
- Print DataStage job logs post-execution.
Import Universal Template
To use the Universal Template, you first must perform the following steps.
This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
To import the Universal Template into your Controller, follow these instructions.
When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.
Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.
Configure Universal Task
For a new Universal Task, create a new task, and enter the required input fields.
Input Fields
| Field | Input type | Default value | Type | Description |
|---|---|---|---|---|
| Server | Required | - | Text | The DataStage server hostname or the IP address |
| Datastage Credentials | Required | - | Credentials | The Datastage user credentials for job execution, The Credentials definition should be as follows.
|
| Domain | Required | - | Text | The Domain of the DataStage server. |
| Project | Required | - | Text | The Project from where the DataStage job needs to be identified. |
| Job Name | Required | - | Text | The IBM Infosphere DataStage Job Name |
| Parameters | Optional | - | Large Text | The parameters for the execution of InfoSphere DataStage jobs Example : -param FI_CODE=002 -param DS_DT=2022-05-06 -param ENV_PARAM=DEV -mode RESTART Refer here for the valid dsjob run command parameters. |
| Wait for Success or Failure | Required | True | Boolean | If selected, the task will continue running until Job reaches the "SUCCEDED" or "FAILED" state. |
| Print Job Logs | Required | False | Boolean | Prints the entire log details by using CLI "dsjob -logdetail" |
| Reset Mode | Required | False | Boolean | Resets the IBM Infosphere Datastage Job using the CLI option - "dsjob -run -mode RESET" |
Task Examples
IBM Infosphere DataStage Job Definition
IBM DataStage Orchestrated using UAC Workflow
Task Output
Exit Codes
The exit codes for this Universal Extension are described below.
| Exit Code | Status Classification Code | Status Classification Description | Status Description |
|---|---|---|---|
| 0 | SUCCESS | Successful Execution | SUCCESS: Successful Task execution |
| 1 | FAILED | Failed Execution | Failed: DataStage job execution Failed |
STDOUT and STDERR
STDOUT and STDERR provide additional information to User. The populated content can be changed in future versions of this extension without notice. Backward compatibility is not guaranteed.
Document References
This document references the following documents.
| Document Link | Description |
|---|---|
| Universal Templates | User documentation for creating, working with and understanding Universal Templates and Integrations. |
| Universal Tasks | User documentation for creating Universal Tasks in the Universal Controller user interface. |
| Credentials | User documentation for creating and working with credentials. |
| Resolvable Credentials Permitted Property | User documentation for Resolvable Credentials Permitted Property. |
Integration Modifications
Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.
- Python code modifications should not be done.
- Template Modifications
- General Section
- "Name", "Extension", "Variable Prefix", "Icon" should not be changed.
- Universal Template Details Section
- "Template Type", "Agent Type", "Send Extension Variables", "Always Cancel on Force Finish" should not be changed.
- Result Processing Defaults Section
- Success and Failure Exit codes should not be changed.
- Success and Failure Output processing should not be changed.
- Fields Restriction Section
The setup of the template does not impose any restrictions, However with respect to "Exit Code Processing Fields" section.- Success/Failure exit codes need to be respected.
- In principle, as STDERR and STDOUT outputs can change in follow-up releases of this integration, they should not be considered as a reliable source for determining success or failure of a task.
- General Section
Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.
Changelog
ut-cs-datastage-1.0.0 (2023-07-28)
Initial Version