Pentaho Integration
Disclaimer
Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/
Overview
Pentaho Data Integration provides powerful ETL (Extract, Transform and Load) capabilities. Universal Controller is integrated to orchestrate the jobs and transformations within Pentaho Data Integration platform via Carte webservice calls.
As an alternative approach, if there is no Carte webserver set-up, it is possible to use the Kitchen & Pan command line utility to execute the jobs & Transformations within the Pentaho Data Integration platform via Windows or Linux Task type in Universal Controller.
The current integration with Pentaho is developed using the Universal Extension functionality in Universal Controller.
The following is the overview of this integration.
UAC communicates with Pentaho Data Integration through the REST-API methods available for the Carte Server.
Universal Controller offers the following functionalities that can be orchestrated within the Pentaho Data Integration Platform:
Run a Pentaho Data Integration Job where the repository details are already supplied in Carte configuration files.
Run a Pentaho Data Integration Job from any of the available repository Via Carte web service calls.
Run a Pentaho Data Integration Job from a *.KJB file.
Define & Run a Pentaho Integration Job from Universal Controller Script library.
Execute a Pentaho Data Integration Transformation where the repository details are already supplied in Carte configuration files.
Run a Pentaho Data Integration Transformation from a *.KTB file.
Define & Run a Pentaho Integration Transformation from Universal Controller Script library.
Software Requirements
This integration requires a Universal Agent 7.0 and a Python runtime to execute the Universal Task against a Pentaho Data Integration.
Software Requirements for Universal Template and Universal Task
Universal Agent from V7.0 supporting Universal Extension
Requires Python 3.6 or higher. Tested with the Universal Agent bundled Python distribution.
Python modules required:
requests
base64
ElementTree
Software Requirements for Universal Agent
Universal Agent for Linux or Windows Version 7.0.0.0 or later is required.
Universal Agent needs to be installed with python option (--python yes).
Software Requirements for Universal Controller
Universal Controller Version 7.0.0.0 and later.
Software Requirements for the Application to be Scheduled
This Universal Task has been tested with the following Pentaho Data Integration versions:
9.1
Versions from 8.0 can also be supported as it have the same Carte REST-API
Technical Considerations
Universal Agent and Universal Controller from Version 7.0 and later.
Installation of a Universal Agent V7.0 in Pentaho Server.
Pentaho tool must be configured with the Carte Server configuration.
Necessary Python modules installed in the Universal Agent Python library as described in the section above.
Key Features
Feature | Description |
Run a Job from Carte configured repository | Trigger a Pentaho Data Integration Job execution ,Where the repository details are already supplied in Carte configuration files. |
Run a Job from a Repository | Trigger a Pentaho Data Integration Job execution , Where the Pentaho Repository details can be supplied as a Input in Universal Controller Job definition. |
Run Job from file | Run a Pentaho Data Integration Job from a *.KJB file located in the remote host. |
Run a Job from UAC Script Library | Define & Run a Pentaho Integration Job from Universal Controller Script library. |
Execute a Transformation from a Repository | Trigger a Pentaho Data Integration Transformation execution ,Where the repository details are already supplied in Carte configuration files. |
Execute a Transformation from a file | Run a Pentaho Data Integration Job from a *.KTB file located in the remote host. |
Run a Transformation from UAC Library | Define & Run a Pentaho Integration Job from Universal Controller Script library. |
Import Pentaho Integration Universal Template
To use the Pentaho Universal Extension, you first must perform the following steps:
- This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
- Download the provided ZIP file.
- In the Universal Controller UI, select Configuration > Universal Templates to display the current list of Universal Templates.
- Click Import Template.
- Select the template ZIP file and Import.
When the template has been imported successfully, the Universal Template will appear on the list. Refresh your Navigation Tree to see these tasks in the Automation Center Menu.
Configure Pentaho Integration Universal Task
For the new Universal Task type, create a new task, and enter the task-specific details that were created in the Universal Template.
Field Descriptions for Pentaho Integration Universal Task
Field | Description |
Select a Pentaho Function | Select the desired function you would need to perform in Pentaho |
Carte server URL | Provide the Carte webserver URL configured in the Pentaho Platform |
Carte Credentials | Provide the Carte webserver Credentials for Jobs/Transformation webservice calls |
SSL Verify(Carte API Call) | Check if carte web server URL needs to be SSL Verified |
Path to SSL Certificate | Provide the path and file name of the SSL Certificate (Needs to be in the utility agent file system) |
Is Agent Installed in Carte Server | If you need run a transformation or Job from UAC library then Universal agent needs to be installed in the Pentaho Carte server |
Select Pentaho Job or Transformation Script | Select a XML script for Pentaho Job or Transformation from UAC script library |
Repository Name | Provide the Pentaho Repository for the Job to be executed |
Pentaho Repository Credentials | Provide the Pentaho Repository credentials for the Job Execution REST-API call |
Pentaho Job Name | Provide the exact Pentaho job name with path incase of repository and just the job name incase of a file |
Job File(Path & Name) | Provide the Pentaho job file name with path and this file should reside in the carte server |
Transformation Name | Provide the Pentaho transformation name (*Ensure a unique transformation name when Execute a Transformation from a file) |
Transformation File Path & Name | Provide the Pentaho Transformation file name and path |
Input Parameters(if amy) | Add if any input parameters to be used for job execution starting with '&' and every parameters to be separated with ampersand symbol(&) |
Log Level | Select either INFO or DEBUG OR WARN OR ERROR |
Execution ID | Output only field : Displays the Carte execution ID of transformation triggered from Universal Controller |
Status | Output only field : Status of the Job/Transformation executing in Carte Web Server |
Print Job Log | Check Box if the Job/Transformation execution logs to be printed in Universal Controller STDERR |
Examples for Pentaho Integration Universal Tasks
Trigger a Pentaho Data Integration (Carte Server Configured with Repository)
Trigger a Pentaho Data Integration with Repository Details as Input
Launch a Pentaho Job using *.kjb files
Define and Trigger Pentaho Job from Universal Controller Script Library
Pentaho Job Definition Script
Trigger Pentaho Transformation with Input Parameters
Document References
This document references the following documents:
Name | Location | Description |
|---|---|---|
Universal Templates | User documentation for creating Universal Templates in the Universal Controller user interface. | |
Universal Tasks | User documentation for creating Universal Tasks in the Universal Controller user interface. | |
Universal Extension | User documentation for creating Universal Extension Tasks in the Universal Controller user interface. | |
Pentaho Rest-API Reference | https://help.hitachivantara.com/Documentation/Pentaho/9.1/Developer_center/REST_API_Reference | Pentaho Developer REST-API Reference |
Disclaimer
Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/
Overview
Pentaho Data Integration provides powerful ETL (Extract, Transform and Load) capabilities. Universal Controller is integrated to orchestrate the jobs and transformations within Pentaho Data Integration platform via Carte webservice calls.
As an alternative approach, if there is no Carte webserver set-up, it is possible to use the Kitchen & Pan command line utility to execute the jobs & Transformations within the Pentaho Data Integration platform via Windows or Linux Task type in Universal Controller.
The current integration with Pentaho is developed using the Universal Extension functionality in Universal Controller.
The following is the overview of this integration.
UAC communicates with Pentaho Data Integration through the REST-API methods available for the Carte Server.
Universal Controller offers the following functionalities that can be orchestrated within the Pentaho Data Integration Platform:
Run a Pentaho Data Integration Job where the repository details are already supplied in Carte configuration files.
Run a Pentaho Data Integration Job from any of the available repository Via Carte web service calls.
Run a Pentaho Data Integration Job from a *.KJB file.
Define & Run a Pentaho Integration Job from Universal Controller Script library.
Execute a Pentaho Data Integration Transformation where the repository details are already supplied in Carte configuration files.
Run a Pentaho Data Integration Transformation from a *.KTB file.
Define & Run a Pentaho Integration Transformation from Universal Controller Script library.
Software Requirements
This integration requires a Universal Agent 7.0 and a Python runtime to execute the Universal Task against a Pentaho Data Integration.
Software Requirements for Universal Template and Universal Task
Universal Agent from V7.0 supporting Universal Extension
Requires Python 3.6 or higher. Tested with the Universal Agent bundled Python distribution.
Python modules required:
requests
base64
ElementTree
Software Requirements for Universal Agent
Universal Agent for Linux or Windows Version 7.0.0.0 or later is required.
Universal Agent needs to be installed with python option (--python yes).
Software Requirements for Universal Controller
Universal Controller Version 7.0.0.0 and later.
Software Requirements for the Application to be Scheduled
This Universal Task has been tested with the following Pentaho Data Integration versions:
9.1
Versions from 8.0 can also be supported as it have the same Carte REST-API
Technical Considerations
Universal Agent and Universal Controller from Version 7.0 and later.
Installation of a Universal Agent V7.0 in Pentaho Server.
Pentaho tool must be configured with the Carte Server configuration.
Necessary Python modules installed in the Universal Agent Python library as described in the section above.
Key Features
Feature | Description |
Run a Job from Carte configured repository | Trigger a Pentaho Data Integration Job execution ,Where the repository details are already supplied in Carte configuration files. |
Run a Job from a Repository | Trigger a Pentaho Data Integration Job execution , Where the Pentaho Repository details can be supplied as a Input in Universal Controller Job definition. |
Run Job from file | Run a Pentaho Data Integration Job from a *.KJB file located in the remote host. |
Run a Job from UAC Script Library | Define & Run a Pentaho Integration Job from Universal Controller Script library. |
Execute a Transformation from a Repository | Trigger a Pentaho Data Integration Transformation execution ,Where the repository details are already supplied in Carte configuration files. |
Execute a Transformation from a file | Run a Pentaho Data Integration Job from a *.KTB file located in the remote host. |
Run a Transformation from UAC Library | Define & Run a Pentaho Integration Job from Universal Controller Script library. |
Import Pentaho Integration Universal Template
To use the Pentaho Universal Extension, you first must perform the following steps:
- This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
- Download the provided ZIP file.
- In the Universal Controller UI, select Configuration > Universal Templates to display the current list of Universal Templates.
- Click Import Template.
- Select the template ZIP file and Import.
When the template has been imported successfully, the Universal Template will appear on the list. Refresh your Navigation Tree to see these tasks in the Automation Center Menu.
Configure Pentaho Integration Universal Task
For the new Universal Task type, create a new task, and enter the task-specific details that were created in the Universal Template.
Field Descriptions for Pentaho Integration Universal Task
Field | Description |
Select a Pentaho Function | Select the desired function you would need to perform in Pentaho |
Carte server URL | Provide the Carte webserver URL configured in the Pentaho Platform |
Carte Credentials | Provide the Carte webserver Credentials for Jobs/Transformation webservice calls |
SSL Verify(Carte API Call) | Check if carte web server URL needs to be SSL Verified |
Path to SSL Certificate | Provide the path and file name of the SSL Certificate (Needs to be in the utility agent file system) |
Is Agent Installed in Carte Server | If you need run a transformation or Job from UAC library then Universal agent needs to be installed in the Pentaho Carte server |
Select Pentaho Job or Transformation Script | Select a XML script for Pentaho Job or Transformation from UAC script library |
Repository Name | Provide the Pentaho Repository for the Job to be executed |
Pentaho Repository Credentials | Provide the Pentaho Repository credentials for the Job Execution REST-API call |
Pentaho Job Name | Provide the exact Pentaho job name with path incase of repository and just the job name incase of a file |
Job File(Path & Name) | Provide the Pentaho job file name with path and this file should reside in the carte server |
Transformation Name | Provide the Pentaho transformation name (*Ensure a unique transformation name when Execute a Transformation from a file) |
Transformation File Path & Name | Provide the Pentaho Transformation file name and path |
Input Parameters(if amy) | Add if any input parameters to be used for job execution starting with '&' and every parameters to be separated with ampersand symbol(&) |
Log Level | Select either INFO or DEBUG OR WARN OR ERROR |
Execution ID | Output only field : Displays the Carte execution ID of transformation triggered from Universal Controller |
Status | Output only field : Status of the Job/Transformation executing in Carte Web Server |
Print Job Log | Check Box if the Job/Transformation execution logs to be printed in Universal Controller STDERR |
Examples for Pentaho Integration Universal Tasks
Trigger a Pentaho Data Integration (Carte Server Configured with Repository)
Trigger a Pentaho Data Integration with Repository Details as Input
Launch a Pentaho Job using *.kjb files
Define and Trigger Pentaho Job from Universal Controller Script Library
Pentaho Job Definition Script
Trigger Pentaho Transformation with Input Parameters
Document References
This document references the following documents:
Name | Location | Description |
|---|---|---|
Universal Templates | User documentation for creating Universal Templates in the Universal Controller user interface. | |
Universal Tasks | User documentation for creating Universal Tasks in the Universal Controller user interface. | |
Universal Extension | User documentation for creating Universal Extension Tasks in the Universal Controller user interface. | |
Pentaho Rest-API Reference | https://help.hitachivantara.com/Documentation/Pentaho/9.1/Developer_center/REST_API_Reference | Pentaho Developer REST-API Reference |