Versions Compared

Old Version 6

changes.mady.by.user Stonebranch

Saved on

compared with

New Version Current

changes.mady.by.user Stonebranch

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Panel

Table of Contents

Disclaimer

...

  • Run a Pipeline.
  • Run a Pipeline with parameters.
  • List all Pipelines in a Workspace.
  • Monitor the started Synapse Pipeline.
  • Cancel a Pipeline Run.
  • Cancel a Pipeline Run Recursive.
  • Rerun a Pipeline from a specified activity or the beginning.
  • Service Principal-based authentication to Azure Synapse.
  • Certificate-based TLS connection 

Version Information

Template NameExtension NameVersion
CS Azure Synapseue-cs-azure-synapse1.01.12

Refer to Changelog for version history information.

...

  1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.

  2. To import the Universal Template into your Controller, follow the instructions here.

  3. When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Modifications of this integration, applied by users or customers, before or after import, might affect the support of this integration. For more information, refer to Integration Modifications.

Configure Universal Task

...

FieldInput TypeValueTypeDescription

Azure Location

Requirede.g., eastus2

Text

The Azure location.

Azure Tenant ID

Requirede.g., 44af330b-a795-47b6-82fe-64db323a499a

Text

The Azure Tenant ID or Directory ID.

Azure Client Id & Secret

Required

Credential

The Azure client ID and client secret (Client ID as username and client secret as the password) - This comes from your Azure App registration services.

CertificateRequiredNone | Certificate File | Agent DefaultChoice

Specify, if a certificate should be used for the connection

  • None: No certificate will be used for the connection
  • Certificate File: A local certificate pem file will be used. The location of the pem file needs to entered in the field Certificate File Path. 
  • Agent Default: The default certificate pem file of the agent installed python certifi module will be used. The required certificate(s) for the connection need to entered in the cacert.pem file used by the python certifi module.
    • Sample locations:  
      • Windows: C:\Program Files\Universal\UAGSrv\uext\_3pp\public\certifi\cacert.pem
      • Linux: /opt/universal/uagsrv/uext/_3pp/public/certifi/cacert.pem
Certificate File PathOptional

Path to the certification pem file

Test

Enter the file system path to the Certificate ;e.g. C:\path_to_your_certificate.pem

Synapse Operation

RequiredCreate Pipeline Run | Get Pipelines By Workspace

Choice

Select the required Azure Synapse function for the task.

Create Pipeline Run | Get Pipelines By Workspace

Workspace Name

RequiredSBDemoWorkspace

Text

The Azure Synapse Workspace Name. 

Pipeline Name

Optional

All Pipelines in the selected Workspace are listed for selection, e.g.

  • SBDemoWorkspace
  • SBDemoWorkspace2
  • ..

Dynamic Choice

Pipeline name that needs to be executed in Azure Synapse.

This field is provided as Dynamic Choice field to retrieve the available pipelines in the given workspace.

By providing the Azure Location, Tenant ID, Azure Client ID and Client Secret and Workspace Development Endpoint all Azure Synapse Pipelines in the Workspace are listed for selection.

ParametersOptionalTrue | FalseBooleanCheck to enable the Pipeline Parameters Field   

Pipeline Parameters

Optional

Parameter should be provided in JSON format, e.g.

Code Block
{
    "PARM1": "hello",
    "PARM2": "world"
}


Script

The pipeline run time parameters that needs to be passed for the pipeline execution as script from the Universal Controller script library. The format should be JSON.

Restart Pipeline

OptionalTrue | False

Boolean

Check, if the pipeline execution needs to be restarted for failure.

The field Restart Pipeline Run ID will appear.

Is RecoveryOptionalTrue | FalseBoolean

Check, if the pipeline execution needs to be restarted in recovery mode.

If recovery mode is set to True, the specified referenced pipeline run and the new run are grouped under the same groupId.

Restart Pipeline Run IDOptionale.g., 5128441a-626e-4a7c-b7c9-152da8435272TextThe Pipeline Run ID of the Pipeline to be re-started. The Pipeline Run ID can be retrieved from the Task instance Field Pipeline Run ID.

Start Activity Name

Optional

e.g., PipelineSBDemo3-2

Would perform a restart from Activity:

PipelineSBDemo3-2

Text

Specify if a failed pipeline needs to be restarted from a specific activity.

If the field is left empty, the restart is done from the beginning.

Monitor IntervalRequiredDefault Value is 60sIntegerInterval to check for the status of the pipeline.
Monitor RetryRequiredDefault Value is 99 retriesIntegerNumber of retries to check for the pipeline status. If the number is reached the task goes to success. This value is used in combination with Monitor Interval

...

These fields are shown as soon as the pipeline is started

Field

Type

Description

Introduced in Version

Pipeline Run ID

Text

Displays the Pipeline Run ID, e.g., 5128441a-626e-4a7c-b7c9-152da8435272

The field is shown as soon as the pipeline is started.

The Run ID is used in case of a pipeline restart.

1.0.0

Pipeline Status

Text

Displays the Pipeline Run Status in Azure Synapse:

  • Queued
  • InProgress
  • Succeeded
  • Failed
  • Canceling
  • Canceled

1.0.0


...

The following Task starts an Azure Synapse Pipeline, with Basic Pipeline Parameters. 

Image RemovedImage Added


Azure Synapse Credential Details:

...

When using Pipeline Parameters in the Universal Task, those Parameters need to be set as dynamic Parameters in Azure Synapse Studio.

Example: @pipeline().parameters.note

Finished Task

Once the execution of the Pipeline in Azure is finished, the Task goes to "Success" and the Pipeline Status field shows "Succeeded".

...

STDOUT and STDERR provide additional information to User. The populated content can be changed in future versions of this Universal Task without notice. Backward compatibility is not guaranteed.

Anchor

...

Integration Modifications

...

Integration Modifications

Integration Modifications

...

Anchor
changelog
changelog

Changelog

ue-cs-azure-synapse-1.1.2 (2024-04-25)

Enhancements

  • Suport for dynamic Bearer Token

ue-cs-azure-synapse-1.1.0

Enhancements

  • Suport for TLS Certificate based connections using either a local certificate file or the default Agent Python certify module certificate file

ue-cs-azure-synapse-1.0.1 (2023-08-24)

Enhancements

  • Added: Parameter Support

ue-cs-azure-synapse-1.0.0

...