Azure Blob: Manage File Transfers

Owned by Former user (Deleted)
Last updated: Dec 05, 2023 by Stonebranch
13 min read

Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/

Introduction

Storing data in the cloud becomes an integral part of most modern IT landscapes. With Universal Automation Center you can securely automate your AWS, Azure, Google and MinIO File Transfers and integrate them into your existing scheduling flows.

As security is one of the key concerns, when moving to the cloud, the provided solution supports multi-level of security:

  • Credentials for Azure Keys are stored in an encrypted form in the database

  • Support for Azure Token based Shared Access Signatures ( SAS )

  • Communication to Azure is done via the HTTPS protocol

  • A Proxy Server connection to Azure with basic authentication is supported

This Universal Task focuses on file transfer to, from and between an Azure Blob Storage.

A similar solution as for Azure Blob Storage is also available for AWS S3 and Google Cloud Storage.

Overview

The Universal Task for Azure Blob: Managed File Transfers allows to securely transfers files from, to and between Azure Blob Storage container and folders.

The Universal Task for Azure Blob: Managed File Transfers supports the following main features:

  • The following file transfer commands are supported:

    • Upload a file(s) to an Azure Blob Storage container

    • Download of file(s) from an Azure Blob Storage container

    • Transfer files between Azure Blob Storage containers

    • List objects in an Azure Blob Storage container

    • Delete object(s) in an Azure Blob Storage container

    • List Azure Blob Storage container names

    • Create an Azure Blob Storage container

    • Monitor a Blob in a Storage container

  • File Transfer can be triggered by a third-party application using the Universal Automation Center RESTfull Webservice API: REST API

  • Universal Task for Azure Blob Storage can be integrated into any existing scheduling workflow in the same way as any standard Linux or Windows Task type.

  • Security is ensured by using the HTTPS protocol with support for an optional Proxy Server.

  • Support for Azure Token based Shared Access Signatures ( SAS )

  • No Universal Agent needs to be installed on the Azure Cloud – the communication goes via HTTPS

Software Requirements

Software Requirements for Universal Agent

  • Universal Agent for Linux or Windows Version 7.0.0.0 or later are required

  • The Universal Agent needs to be installed with python option (--python yes)

Software Requirements for Universal Controller

  • Universal Controller 7.0.0.0. or later is required

Software Requirements for the Application to be Scheduled

The Universal Task has been tested for the python Azure Storage SDK: azure-storage-blob version: 12.7.1

Key Features

The solution supports the following file transfer scenarios:

Feature

Description

Upload a file(s) to a container (Copy or Move)

  • One or more files can be uploaded to a container.

  • Move or copy are supported for the upload

  • Unix filename pattern matching support; for example, wildcard support “*” to upload multiple files

  • One or more files can be Uploaded to a folder using a prefix

  • It can be decided via an the Upload Write Options to:

    • Overwrite an existing object (Replace existing Object)

    • Cancel the operations in case an object with a similar name exists (Do not overwrite existing Object)

    • Add a timestamp to the uploaded Object (Timestamp) 

Download of file(s) from a container (Copy or Move)

  • One or more files should be downloaded from a bucket.

  • Move or copy must be supported for the download

  • Unix filename pattern matching support; for example, wildcard support “*” to download multiple files

  • Download to a specific folder is supported

  • It can be decided via an the Download Write Options to:

    • Overwrite an existing file (Replace existing File)

    • Cancel the operations in case a File with a similar name exists (Do not overwrite existing File)

    • Add a timestamp to the uploaded file (Timestamp) 

    • Perform the default Windows behaviour for copying files (Default Windows behaviour)

      • If a file with a similar name exists, the file names that are similar will be edited so that the files you copied have a number appended at the end of them. For example, if you’re copying a file named image.png to a folder that already has a file named image.png in it, the copied file will be named image (1).png.

Copy object to a container

  • An object can be copied from one container to another.

  • Folders are support

Note: wildcards are not supported for this method.

List objects in a container

  • Show all Objects in a container

  • Show all Objects in a folder

  • Unix filename pattern matching support e.g. wildcard support “*” to narrow down the objects to display

Delete object(s) in a container

  • Delete one or multiple objects in a container

  • Delete one or multiple objects in folder in a container

  • Unix filename pattern matching support e.g. wildcard support “*” to narrow down the objects to be deleted

List buckets

List all container in an Azure account

Create a container

Create a new container

Monitor Blob

Monitor a Blob in a Storage container

Proxy Server Connection

A proxy server connection with or without basic authentication can be configured

Integration into 3rd Party Applications

An Azure Blob Storage file transfer can be triggered via the Universal Automation Center RESTfull Webservice API within an application.

Self-Service through Web-Client

The Azure Blob Storage Task can be fully configured, monitored and updated via the Universal Controller Web-GUI

Import Azure Blob Storage Universal Template

The Universal Template is provided as a Zip-file. This Zip-file can be imported from any local folder using the Universal Controller UI.

To load the Universal Template Zip-file perform the following steps:

  1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.

  2. Copy the Universal Template Zip-file to a local directory.

  3. In the Universal Controller UI, select Administration > Universal Templates > Import Template.

  4. Select the Universal Template Zip-file to import.

When the Universal Template Zip-file has been imported successfully, the Universal Template will appear on the list.

Configure Azure Blob Storage Universal Tasks

For the new Universal Task type, create a new task and enter the task-specific Details that were created in the Universal Template.

Field Descriptions for Azure Blob Storage Universal Task - Action

The Azure Blob Storage Task provides multiple different file transfer actions. For each action the specific fields are described.

Azure Blob Storage Container is abbreviated to Container in the following.

Create Container - Action

The Action Creates a new Container.

Field

Description

Action

create-container action

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Connection Type

[Azure Account Key | SAS Token ]

If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.

If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.

Container

Name of the Container to create

Example for Azure Blob Storage Universal Tasks - Create Container

The following Task creates the Container: stonebranchpm2

List Container - Action

The Action list all container of an Azure Account.

Field

Description

Action

list-container action

Connection Type

[Azure Account Key | SAS Token ]

If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.

If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Example for Azure Blob Storage Universal Tasks - List Containers

The following Task list all container of the selected Azure Account.

Upload File - Action

The Action is used to upload a single or multiple files from a Windows or Linux server to an container or a folder in a container.

Field

Description

Action

Upload-file action

Connection Type

[Azure Account Key | SAS Token ]

If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.

If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Sourcefile

Source file(s) to be uploaded to a bucket or specific folder in a container.

Unix filename pattern are supported to upload a selection of files.

Supported wildcards are:

  • ? matches any single character

  • [seq] matches any character in seq

  • [!seq] matches any character not in seq

  • ”*” matches everything

Operation

[copy | move]

In case of a “move” the source file(s) is/are deleted after the upload to the container.

Prefix

A folder in a container is called prefix. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name.

Upload Write Options

Upload Write Options

[Do not overwrite existing Object | Timestamp]

  • Do not overwrite existing Object: Cancel the operations in case an object with a similar name exists

  • Timestamp: Add a timestamp to the uploaded Object

Example for Azure Blob Storage Universal Tasks - Upload File

The following Task uploads from the Linux directory the files: /home/stonebranch/demo/azure/out/test[1-4]* to the the container stonebranchpm, folder incoming.

List Objects - Action

The Action is used to display objects in a Azure Container or a specific folder in an Azure Container (prefix).

Field

Description

Action

List-objects action

Connection Type

[Azure Account Key | SAS Token ]

If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.

If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Container

Container in which the objects should be listed

Prefix

A folder in a container is called prefix. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name.

If a prefix is provided only objects in the folder with the prefix name are listed in the output.

Blob

Objects matching the given Blob are listed.

Unix filename pattern are supported to list only a selection of files:
Supported wildcards are:

  • ? matches any single character

  • [seq] matches any character in seq

  • [!seq] matches any character not in seq

  • ”*” matches everything

Example:

Blob = test* : matches everything starting with test

Blob = test[1-2].txt : matches test1.txt, test2.txt

Blob = test[!1].txt: does not match test1.txt

Blob = test?.txt: matches test1.txt, test2.txt etc.

Show Details

Show details like creation timestamp in the output

Example for Azure Blob Storage Universal Tasks - List Objects

The following Task list all objects matching the criteria test* in the container stonebranchpm in the folder incoming.

Download File - Action

This Action downloads one or multiple files from an Azure Container to a Linux or Windows folder.

Field

Description

Action

Download-file action

Connection Type

[Azure Account Key | SAS Token ]

If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.

If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Target Directory

Linux or Windows Target Directory

For example, C:\tmp\ or /home/ubuntu/download

Unix filename pattern are supported to download a selection of files.

Supported wildcards are:

  • ? matches any single character

  • [seq] matches any character in seq

  • [!seq] matches any character not in seq

  • ”*” matches everything

Operation

[copy | move]

In case of a “move” the objects are deleted after they have been download from the container.

Download Write Options

Download Write Options:

[Replace existing File | Do not overwrite existing File | Timestamp | Default Windows behaviour]

  • Replace existing File : overwrite an existing file

  • Do not overwrite existing File: cancel the operations in case a File with a similar name exists

  • Timestamp: add a timestamp to the uploaded file

  • Default Windows behaviour: perform the default Windows behaviour for copying files. If a file with a similar  name exists, the file names that are similar will be edited so that the files you copied have a number appended at the end of them. For example, if you are copying a file named image.png to a folder that already has a file named image.png in it, the copied file will be named image (1).png.

Example for Azure Blob Storage Universal Tasks - Download File

The following Task downloads from the container stonebranchpm all blobs matching the criteria test[1-2]* to the Linux directory: /home/stonebranch/demo/azure/in

Delete Objects - Action

The Actions is used to delete an object in an Azure Container or folder

Field

Description

Action

Delete-objects action

Connection Type

[Azure Account Key | SAS Token ]

If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.

If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Container

Container in which the Blobs should be deleted.

Blob

Blobs to be deleted in the given container

Note: Due to security reasons, wildcard is only supported if at least one character is provided e.g. t* would delete all files starting with at “t”.

Example for Azure Blob Storage Universal Task - Delete Objects

The following Task deletes in the container stonbranchpm, folder incoming all Blobs matching the criteria test[3-4]*.

Copy Object to Container - Action

This Action is used copy an objects from one Azure Container to another Azure Container.

Field

Description

Action

Copy-object-to-bucket action

Connection Type

[Azure Account Key | SAS Token ]

If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.

If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Container

Source container

Target Container

Target container, where the object(s) will be copied to

Blob

File ( Blob ) to be copied from the source container to target container.

Upload Write Options

Upload Write Options

[Do not overwrite existing Object | Timestamp]

  • Do not overwrite existing Object: Cancel the operations in case an object with a similar name exists

  • Timestamp: Add a timestamp to the uploaded Object

Example for Azure Blob Storage Universal Tasks - Copy Object to Container

The following Task copies the Blob: test1.txt from the source container: stonebranchpm to the target container: stonebranchpm2

Delete Container - Action

This action is used to delete an Azure Container

Field

Description

Action

Delete-container action

Connection Type

[Azure Account Key | SAS Token ]

If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.

If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Container

Name of the Container to be deleted

Note: The container is delete including all objects.

Example for Azure Blob Storage Universal Tasks - Delete Container

The following Task delete the container stonebranchpm2

Upload File - Action

The Action is used to upload a single or multiple files from a Windows or Linux server to an container or a folder in a container.

Field

Description

Action

Upload-file action

Connection Type

[Azure Account Key | SAS Token ]

If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.

If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Sourcefile

Source file(s) to be uploaded to a bucket or specific folder in a container.

Unix filename pattern are supported to upload a selection of files.

Supported wildcards are:

  • ? matches any single character

  • [seq] matches any character in seq

  • [!seq] matches any character not in seq

  • ”*” matches everything

Operation

[copy | move]

In case of a “move” the source file(s) is/are deleted after the upload to the container.

Prefix

A folder in a container is called prefix. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name.

Upload Write Options

Upload Write Options

[Do not overwrite existing Object | Timestamp]

  • Do not overwrite existing Object: Cancel the operations in case an object with a similar name exists

  • Timestamp: Add a timestamp to the uploaded Object

Example for Azure Blob Storage Universal Tasks - Upload File

The following Task uploads from the Linux directory the files: /home/stonebranch/demo/azure/out/test[1-4]* to the the container stonebranchpm, folder incoming.

Monitor Blob - Action

The Action is used to monitor a Blob in an Azure Container.

Field

Description

Action

Monitor-Blob action

Connection Type

[Azure Account Key | SAS Token ]

If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.

If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Container

Container in which the objects should be monitored

Prefix

A folder in a container is called prefix. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name.

If a prefix is provided only objects in the folder with the prefix name are listed in the output.

Blob

Blob to monitor. The task goes to status success, if the Blob is found.

Unix filename pattern are supported to list only a selection of files:
Supported wildcards are:

  • ? matches any single character

  • [seq] matches any character in seq

  • [!seq] matches any character not in seq

  • ”*” matches everything

Example:

Blob = test* : matches everything starting with test

Blob = test[1-2].txt : matches test1.txt, test2.txt

Blob = test[!1].txt: does not match test1.txt

Blob = test?.txt: matches test1.txt, test2.txt etc.

Example for Azure Blob Storage Universal Tasks - Monitor Blob

The following Task monitors in the Container: stonebranchpm all Blob’s starting with the name report. If a Blob starting with the name report is found the task goes to task status success.