Azure Data Factory: Schedule, Trigger, and Monitor
Disclaimer
Your use of this download is governed by Stonebranch's Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/
Introduction
This Integration allows Stonebranch users to schedule, trigger, and monitor the Azure Data Factory pipeline process directly from Universal Controller.
Overview
- This integration uses Python modules azure-mgmt-resource and azure-mgmt-datafactory to make REST-API calls to Azure Data Factory.
- This integration will use the Azure Tenant id , Subscription id , client id ,client secret, Resource group, and location for authenticating the REST-API calls to Azure Data Factory.
- User can perform the following Azure Data Factory operations:
- Run a Pipeline.
- Get information on a Pipeline.
- List all Pipelines.
- Cancel Pipeline run.
- List factory by resource group.
- Also, with respect to Azure Data Factory triggers, users can perform the following operations from UAC:
- Start Trigger.
- Stop Trigger.
- List Trigger by Factory.
- UAC also can restart a failed pipeline either from the failed step or from any activity name in the failed pipeline.
Software Requirements
This integration requires a Universal Agent and a Python runtime to execute the Universal Task against an Azure Data Factory.
Software Requirements for Universal Template and Universal Task
- Requires Python 3.11 or higher. Tested with the Universal Agent bundled Python distribution.
- Python modules required:
- azure-mgmt-resource
- azure-mgmt-datafactory
Software Requirements for Universal Agent
Either:
- Universal Agent for Windows x64 Version 7.4 and later with Python options installed.
- Universal Agent for Linux Version 7.4 and later with Python options installed.
Software Requirements for Universal Controller
- Universal Controller Version 7.4.0.0 and later.
Software Requirements for the Application to be Scheduled
This Universal Task has been tested with the Azure Data Factory Version 2.
Technical Considerations
- This integration uses the Python modules Azure resource Management Module and Azure Data Factory management to make REST-API calls with Azure Data Factory.
- Use Azure App Registration services to create an app, client ID, and client and associate it with Data Factory to provide appropriate roles (for example, Contributor).
Key Features
Feature | Description |
|---|---|
Run a Pipeline | Execute a pipeline defined in Azure Data Factory. Schedule, trigger, and monitor the execution of a pipeline. |
Get Information on a Pipeline | Get information about the pipeline name. |
List all Pipelines | List all pipelines that belongs to a Data Factory. |
Cancel a Pipeline Run | Abort a pipeline execution from UAC by providing the run ID for the pipeline execution. |
List Factory by resource group | List the available factory name that belongs to a resource group. |
Start a Trigger | Start a trigger that will manage pipeline execution schedule in the Azure Data Factory. |
Stop a Trigger | Stop a trigger that is in the Azure Data Factory. |
List Trigger by Factory | List all the triggers that belongs to the Data Factory. |
Import Azure Data Factory Integration Downloadable Universal Template
To use this downloadable Universal Template, you first must perform the following steps:
- This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
- To import the Universal Template into your Controller, follow the instructions here.
- When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.
Configure Azure Data Factory Integration Universal Task
For the new Universal Task type, create a new task, and enter the task-specific details that were created in the Universal Template.
Field Descriptions for Azure Data Factory Integration Universal Task
Field | Description |
|---|---|
Azure Location | Provide the Azure location. |
Tenant ID | Provide the Azure Tenant ID or Directory ID. |
Azure Client ID and Client Secret | Provide the Azure client ID and client secret (Client ID as user name and client secret as the password) - This comes from your Azure App registration services. |
Subscription | Azure Subscription ID. |
Azure Resource Group | Azure Resource group responsible for the Data Factory. |
Data factory Operation | Select the required Azure Data Factory function for the task. |
Data Factory Name | Provide the Azure Data Factory Name. |
Pipeline Name | Pipeline name that needs to be executed in Azure Data factory. |
Pipeline Parameters | Provide the pipeline run time parameters that needs to be passed for the pipeline execution. |
Run ID | Run id for the Azure Data Factory Pipeline execution - Required in case of a restart. |
Restart Pipeline | Check if the pipeline execution needs to be restarted for a failure. |
Start From Failure | Pipeline needs to be restarted from a Failure step. |
Start Activity Name | If Start from Failure is not checked, specify if a failed pipeline needs to be restarted from a specific activity. |
Run ID for Restart | Provide the Pipeline run ID for the restart. |
Trigger Name | Trigger Name in the Azure Data Factory. |
Polling | True for ARM Polling, False for no polling, or a polling object for personal polling strategy. |
Examples for Azure Data Factory Integration Universal Tasks
Run a Pipeline
Pipeline Execution Logs
Restart a Failed Pipeline Execution
Cancel a Pipeline Run
Other Azure Data Factory Operation from UAC
Document References
This document references the following documents:
Name | Description |
|---|---|
User documentation for creating Universal Templates in the Universal Controller user interface. | |
User documentation for creating Universal Tasks in the Universal Controller user interface. |
Changelog
ut-cs-azure-datafactory-1.2.0 (2025-10-30)
Enhancements
- Added: install_if_missing function to install azure python packages if missing
- Added: Authenticate ADF via Certificate