Azure Data Factory: Schedule, Trigger, and Monitor
Disclaimer
Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/
Introduction
This Integration allows Stonebranch users to schedule, trigger, and monitor the Azure Data Factory pipeline process directly from Universal Controller.
Overview
This integration uses Python modules azure-mgmt-resource and azure-mgmt-datafactory to make REST-API calls to Azure Data Factory.
This integration will use the Azure Tenant id , Subscription id , client id ,client secret, Resource group, and location for authenticating the REST-API calls to Azure Data Factory.
User can perform the following Azure Data Factory operations:
Run a Pipeline.
Get information on a Pipeline.
List all Pipelines.
Cancel Pipeline run.
List factory by resource group.
Also, with respect to Azure Data Factory triggers, users can perform the following operations from UAC:
Start Trigger.
Stop Trigger.
List Trigger by Factory.
UAC also can restart a failed pipeline either from the failed step or from any activity name in the failed pipeline.
Software Requirements
This integration requires a Universal Agent and a Python runtime to execute the Universal Task against an Azure Data Factory.
Software Requirements for Universal Template and Universal Task
Requires Python 3.4 or higher. Tested with the Universal Agent bundled Python distribution.
Python modules required:
azure-mgmt-resource
azure-mgmt-datafactory
- azure.identity
Software Requirements for Universal Agent
Either:
Universal Agent for Windows x64 Version 6.9 and later with Python options installed.
Universal Agent for Linux Version 6.9 and later with Python options installed.
Software Requirements for Universal Controller
Universal Controller Version 6.9.0.0 and later.
Software Requirements for the Application to be Scheduled
This Universal Task has been tested with the Azure Data Factory Version 2.
Technical Considerations
This integration uses the Python modules Azure resource Management Module and Azure Data Factory management to make REST-API calls with Azure Data Factory.
Use Azure App Registration services to create an app, client ID, and client and associate it with Data Factory to provide appropriate roles (for example, Contributor).
Key Features
Feature | Description |
Run a Pipeline | Execute a pipeline defined in Azure Data Factory. Schedule, trigger, and monitor the execution of a pipeline. |
Get Information on a Pipeline | Get a information about the pipeline name . |
List all Pipelines | List all pipelines that belongs to a Data Factory. |
Cancel a Pipeline Run | Abort a Pipeline execution from UAC by providing the run ID for the Pipeline execution. |
List Factory by resource group | List the available factory name that belongs to a resource group. |
Start a Trigger | Start a trigger that will manage pipeline execution schedule in the Azure Data Factory. |
Stop a Trigger | Stop a trigger that is in the Azure Data Factory. |
List Trigger by Factory | List all the triggers that belongs to the Data Factory. |
Import Azure Data Factory Integration Downloadable Universal Template
To use this downloadable Universal Template, you first must perform the following steps:
- This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
- To import the Universal Template into your Controller, follow the instructions here.
- When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.
Configure Azure Data Factory Integration Universal Task
For the new Universal Task type, create a new task, and enter the task-specific details that were created in the Universal Template.
Field Descriptions for Azure Data Factory Integration Universal Task
Field | Description |
Azure Location | Provide the Azure location. |
Tenant ID | Provide the Azure Tenant ID or Directory ID. |
Azure Client ID and Client Secret | Provide the Azure client ID and client secret (Client ID as user name and client secret as the password) - This comes from your Azure App registration services. |
Subscription | Azure Subscription ID. |
Azure Resource Group | Azure Resource group responsible for the Data Factory. |
Data factory Operation | Select the required Azure Data Factory function for the task. |
Data Factory Name | Provide the Azure Data Factory Name. |
Pipeline Name | Pipeline name that needs to be executed in Azure Data factory. |
Pipeline Parameters | Provide the pipeline run time parameters that needs to be passed for the pipeline execution. |
Run ID | Run id for the Azure Data Factory Pipeline execution - Required in case of a restart. |
Restart Pipeline | Check if the pipeline execution needs to be restarted for a failure. |
Start From Failure | Pipeline needs to be restarted from a Failure step. |
Start Activity Name | If Start from Failure is not checked, specify if a failed pipeline needs to be restarted from a specific activity. |
Run ID for Restart | Provide the Pipeline run ID for the restart. |
Trigger Name | Trigger Name in the Azure Data Factory. |
Polling | True for ARMPolling, False for no polling, or a polling object for personal polling strategy. |
Examples for Azure Data Factory Integration Universal Tasks
Run a Pipeline
Pipeline Execution Logs
Restart a Failed Pipeline Execution
Cancel a Pipeline Run
Other Azure Data Factory Operation from UAC
Document References
This document references the following documents:
Name | Description |
|---|---|
User documentation for creating Universal Templates in the Universal Controller user interface. | |
User documentation for creating Universal Tasks in the Universal Controller user interface. |