Skip to main content

IBM InfoSphere DataStage Jobs

Disclaimer​

Your use of this download is governed by Stonebranch's Terms of Use, which are available at Stonebranch Integration Hub - Terms of Use.

Overview​

IBM DataStage is a powerful data integration and ETL (Extract, Transform, Load) tool by IBM and it helps organizations extract data from various sources, transform it to meet specific business requirements, and load it into target systems such as data warehouses, data marts, and operational data stores. This Universal Task allows Stonebranch users to schedule, trigger, monitor, and orchestrate the InfoSphere DataStage jobs directly from Universal Controller via the InfoSphere DataStage Command-line Interface(CLI). It also helps to set up dependencies between Datastage jobs and other applications or platforms using UAC workflows and trigger DataStage Jobs from UAC either time-based or event-based.

Version Information​

Template Name

Version

CS Datastage CLI

1.0.0

Refer to Changelog for version history information.

Software Requirements​

This integration requires a Universal Agent and a Python runtime to execute the Universal Task.

Software Requirements for Universal Template and Universal Task​

Requires Python 3.7.6 or higher. Tested with the Universal Agent bundled Python distribution.

Software Requirements for Universal Agent​

Both Windows and Linux agents are supported.

  • Universal Agent for Windows x64 Version 7.1.0.0 and later with Python options installed.
  • Universal Agent for Linux Version 7.1.0.0 and later with Python options installed.
  • IBM InfoSphere DataStage Client (CLI) installed in the agent host and accessible to the UAC execution User.
  • IBM InfoSphere DataStage Client supported versions - 8.5,9.1,11.3,11.5 & 11.7

Software Requirements for Universal Controller​

Universal Controller Version 7.1.0.0 and later.

Network and Connectivity Requirements​

IBM InfoSphere DataStage Client (CLI) installed in the agent host is configured to connect with the DataStage Server.

Key Features​

This Universal Task provides the following features:

  • Ability to schedule, trigger & orchestrate the execution of Datastage jobs.
  • Monitor the DataStage job execution.
  • Resumes a Restartable DataStage Job from UAC ( "-mode RESTART" to the parameters).
  • Print DataStage job logs post-execution.

Import Universal Template​

To use the Universal Template, you first must perform the following steps.

  1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
  2. To import the Universal Template into your Controller, follow these instructions.
  3. When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Universal Task​

For a new Universal Task, create a new task, and enter the required input fields.

Input Fields​

Field

Input type

Default value

Type

Description

Server

Required

-

Text

The DataStage server hostname or the IP address

Datastage Credentials

Required

-

Credentials

The Datastage user credentials for job execution, The Credentials definition should be as follows.

  • User as "Runtime User".
  • User Password as "Runtime Password".

Domain

Required

-

Text

The Domain of the DataStage server.

Project

Required

-

Text

The Project from where the DataStage job needs to be identified.

Job Name

Required

-

Text

The IBM Infosphere DataStage Job Name

Parameters

Optional

-

Large Text

The parameters for the execution of InfoSphere DataStage jobs

Example : -param FI_CODE=002 -param DS_DT=2022-05-06 -param ENV_PARAM=DEV -mode RESTART

Refer here for the valid dsjob run command parameters.

Wait for Success or Failure

Required

True

Boolean

If selected, the task will continue running until Job reaches the "SUCCEDED" or "FAILED" state.

Print Job Logs

Required

False

Boolean

Prints the entire log details by using CLI "dsjob -logdetail"

Reset Mode

Required

False

Boolean

Resets the IBM Infosphere Datastage Job using the CLI option - "dsjob -run -mode RESET"

Task Examples​

IBM Infosphere DataStage Job Definition

IBM DataStage Orchestrated using UAC Workflow

Task Output​

Exit Codes​

The exit codes for this Universal Extension are described below.

Exit Code

Status Classification Code

Status Classification Description

Status Description

0

SUCCESS

Successful Execution

SUCCESS: Successful Task execution

1

FAILED

Failed Execution

Failed: DataStage job execution Failed

STDOUT and STDERR​

STDOUT and STDERR provide additional information to User. The populated content can be changed in future versions of this extension without notice. Backward compatibility is not guaranteed.

Document References​

This document references the following documents.

Document Link

Description

Universal Templates

User documentation for creating, working with and understanding Universal Templates and Integrations.

Universal Tasks

User documentation for creating Universal Tasks in the Universal Controller user interface.

Credentials

User documentation for creating and working with credentials.

Resolvable Credentials Permitted Property

User documentation for Resolvable Credentials Permitted Property.

Integration Modifications​

Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.

  • Python code modifications should not be done.
  • Template Modifications
    • General Section
      • "Name", "Extension", "Variable Prefix", "Icon" should not be changed.
    • Universal Template Details Section
      • "Template Type", "Agent Type", "Send Extension Variables", "Always Cancel on Force Finish" should not be changed.
    • Result Processing Defaults Section
      • Success and Failure Exit codes should not be changed.
      • Success and Failure Output processing should not be changed.
    • Fields Restriction Section
      The setup of the template does not impose any restrictions, However with respect to "Exit Code Processing Fields" section.
      1. Success/Failure exit codes need to be respected.
      2. In principle, as STDERR and STDOUT outputs can change in follow-up releases of this integration, they should not be considered as a reliable source for determining success or failure of a task.

Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.

Changelog​

ut-cs-datastage-1.0.0 (2023-07-28)​

Initial Version