...
The following Task monitors in the Container: stonebranchpm all Blob’s starting with the name report. If a Blob starting with the name report is found the task goes to task status success.
| Panel | |
|---|---|
|
Disclaimer
Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/
Introduction
Storing data in the cloud becomes an integral part of most modern IT landscapes. With Universal Automation Center you can securely automate your AWS, Azure, Google and MinIO File Transfers and integrate them into your existing scheduling flows.
As security is one of the key concerns, when moving to the cloud, the provided solution supports multi-level of security:
Credentials for Azure Keys are stored in an encrypted form in the database
Support for Azure Token based Shared Access Signatures ( SAS )
Communication to Azure is done via the HTTPS protocol
A Proxy Server connection to Azure with basic authentication is supported
This Universal Task focuses on file transfer to, from and between an Azure Blob Storage.
A similar solution as for Azure Blob Storage is also available for AWS S3 and Google Cloud Storage.
Overview
The Universal Task for Azure Blob: Managed File Transfers allows to securely transfers files from, to and between Azure Blob Storage container and folders.
The Universal Task for Azure Blob: Managed File Transfers supports the following main features:
The following file transfer commands are supported:
Upload a file(s) to an Azure Blob Storage container
Download of file(s) from an Azure Blob Storage container
Transfer files between Azure Blob Storage containers
List objects in an Azure Blob Storage container
Delete object(s) in an Azure Blob Storage container
List Azure Blob Storage container names
Create an Azure Blob Storage container
Monitor a Blob in a Storage container
File Transfer can be triggered by a third-party application using the Universal Automation Center RESTfull Webservice API: REST API
Universal Task for Azure Blob Storage can be integrated into any existing scheduling workflow in the same way as any standard Linux or Windows Task type.
Security is ensured by using the HTTPS protocol with support for an optional Proxy Server.
Support for Azure Token based Shared Access Signatures ( SAS )
No Universal Agent needs to be installed on the Azure Cloud – the communication goes via HTTPS
Software Requirements
Software Requirements for Universal Agent
Universal Agent for Linux or Windows Version 7.0.0.0 or later are required
The Universal Agent needs to be installed with python option (--python yes)
Software Requirements for Universal Controller
Universal Controller 7.0.0.0. or later is required
Software Requirements for the Application to be Scheduled
The Universal Task has been tested for the python Azure Storage SDK: azure-storage-blob version: 12.7.1
Key Features
The solution supports the following file transfer scenarios:
...
Feature
...
Description
...
Upload a file(s) to a container (Copy or Move)
...
One or more files can be uploaded to a container.
Move or copy are supported for the upload
Unix filename pattern matching support; for example, wild card support “*” to upload multiple files
One or more files can be Uploaded to a folder using a prefix
It can be decided via an the Upload Write Options to:
Overwrite an existing object (Replace existing Object)
Cancel the operations in case an object with a similar name exists (Do not overwrite existing Object)
Add a timestamp to the uploaded Object (Timestamp)
...
Download of file(s) from a container (Copy or Move)
...
One or more files should be downloaded from a bucket.
Move or copy must be supported for the download
Unix filename pattern matching support; for example, wild card support “*” to download multiple files
Download to a specific folder is supported
It can be decided via an the Download Write Options to:
Overwrite an existing file (Replace existing File)
Cancel the operations in case a File with a similar name exists (Do not overwrite existing File)
Add a timestamp to the uploaded file (Timestamp)
Perform the default Windows behaviour for copying files (Default Windows behaviour)
If a file with a similar name exists, the file names that are similar will be edited so that the files you copied have a number appended at the end of them. For example, if you’re copying a file named image.png to a folder that already has a file named image.png in it, the copied file will be named image (1).png.
...
Copy object to a container
...
An object can be copied from one container to another.
Folders are support
Note: wild cards are not supported for this method.
...
List objects in a container
...
Show all Objects in a container
Show all Objects in a folder
Unix filename pattern matching support e.g. wild card support “*” to narrow down the objects to display
...
Delete object(s) in a container
...
Delete one or multiple objects in a container
Delete one or multiple objects in folder in a container
Unix filename pattern matching support e.g. wild card support “*” to narrow down the objects to be deleted
...
List buckets
...
List all container in an Azure account
...
Create a container
...
Create a new container
...
Monitor Blob
...
Monitor a Blob in a Storage container
...
Proxy Server Connection
...
A proxy server connection with or without basic authentication can be configured
...
Integration into 3rd Party Applications
...
An Azure Blob Storage file transfer can be triggered via the Universal Automation Center RESTfull Webservice API within an application.
...
Self-Service through Web-Client
...
The Azure Blob Storage Task can be fully configured, monitored and updated via the Universal Controller Web-GUI
Import Azure Blob Storage Universal Template
The Universal Template is provided as a Zip-file. This Zip-file can be imported from any local folder using the Universal Controller UI.
To load the Universal Template Zip-file perform the following steps:
- This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
Copy the Universal Template Zip-file to a local directory.
In the Universal Controller UI, select Administration > Universal Templates > Import Template.
Select the Universal Template Zip-file to import.
When the Universal Template Zip-file has been imported successfully, the Universal Template will appear on the list.
Configure Azure Blob Storage Universal Tasks
For the new Universal Task type, create a new task and enter the task-specific Details that were created in the Universal Template.
Field Descriptions for Azure Blob Storage Universal Task - Action
The Azure Blob Storage Task provides multiple different file transfer actions. For each action the specific fields are described.
Azure Blob Storage Container is abbreviated to Container in the following.
Create Container - Action
The Action Creates a new Container.
...
Field
...
Description
...
Action
...
create-container action
...
Useproxy ( default is NO )
...
[NO | YES]
If set to YES, the fields to set-up the proxy server connections are displayed:
Proxy Server IP or hostname
Proxy Server Port
Proxy Server Credentials (optional)
...
Loglevel ( default is INFO )
...
Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]
...
Connection Type
...
[Azure Account Key | SAS Token ]
If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.
If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
...
Container
...
Name of the Container to create
Example for Azure Blob Storage Universal Tasks - Create Container
The following Task creates the Container: stonebranchpm2
List Container - Action
The Action list all container of an Azure Account.
...
Field
...
Description
...
Action
...
list-container action
...
Connection Type
...
[Azure Account Key | SAS Token ]
If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.
If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
...
Useproxy ( default is NO )
...
[NO | YES]
If set to YES, the fields to set-up the proxy server connections are displayed:
Proxy Server IP or hostname
Proxy Server Port
Proxy Server Credentials (optional)
...
Loglevel ( default is INFO )
...
Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]
Example for Azure Blob Storage Universal Tasks - List Containers
The following Task list all container of the selected Azure Account.
Upload File - Action
The Action is used to upload a single or multiple files from a Windows or Linux server to an container or a folder in a container.
...
Field
...
Description
...
Action
...
Upload-file action
...
Connection Type
...
[Azure Account Key | SAS Token ]
If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.
If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
...
Useproxy ( default is NO )
...
[NO | YES]
If set to YES, the fields to set-up the proxy server connections are displayed:
Proxy Server IP or hostname
Proxy Server Port
Proxy Server Credentials (optional)
...
Loglevel ( default is INFO )
...
Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]
...
Sourcefile
...
Source file(s) to be uploaded to a bucket or specific folder in a container.
Unix filename pattern are supported to upload a selection of files.
Supported wildcards are:
? matches any single character
[seq] matches any character in seq
[!seq] matches any character not in seq
”*” matches everything
...
Operation
...
[copy | move]
In case of a “move” the source file(s) is/are deleted after the upload to the container.
...
Prefix
...
A folder in a container is called prefix. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name.
...
Upload Write Options
[Do not overwrite existing Object | Timestamp]
Do not overwrite existing Object: Cancel the operations in case an object with a similar name exists
Timestamp: Add a timestamp to the uploaded Object
Example for Azure Blob Storage Universal Tasks - Upload File
The following Task uploads from the Linux directory the files: /home/stonebranch/demo/azure/out/test[1-4]* to the the container stonebranchpm, folder incoming.
List Objects - Action
The Action is used to display objects in a Azure Container or a specific folder in an Azure Container (prefix).
...
Field
...
Description
...
Action
...
List-objects action
...
Connection Type
...
[Azure Account Key | SAS Token ]
If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.
If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
...
Useproxy ( default is NO )
...
[NO | YES]
If set to YES, the fields to set-up the proxy server connections are displayed:
Proxy Server IP or hostname
Proxy Server Port
Proxy Server Credentials (optional)
...
Loglevel ( default is INFO )
...
Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]
...
Container
...
Container in which the objects should be listed
...
Prefix
...
A folder in a container is called prefix. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name.
If a prefix is provided only objects in the folder with the prefix name are listed in the output.
...
Blob
...
Objects matching the given Blob are listed.
Unix filename pattern are supported to list only a selection of files:
Supported wildcards are:
? matches any single character
[seq] matches any character in seq
[!seq] matches any character not in seq
”*” matches everything
Example:
Blob = test* : matches everything starting with test
Blob = test[1-2].txt : matches test1.txt, test2.txt
Blob = test[!1].txt: does not match test1.txt
Blob = test?.txt: matches test1.txt, test2.txt etc.
...
Show Details
...
Show details like creation timestamp in the output
Example for Azure Blob Storage Universal Tasks - List Objects
The following Task list all objects matching the criteria test* in the container stonebranchpm in the folder incoming.
Download File - Action
This Action downloads one or multiple files from an Azure Container to a Linux or Windows folder.
...
Field
...
Description
...
Action
...
Download-file action
...
Connection Type
...
[Azure Account Key | SAS Token ]
If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.
If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
...
Useproxy ( default is NO )
...
[NO | YES]
If set to YES, the fields to set-up the proxy server connections are displayed:
Proxy Server IP or hostname
Proxy Server Port
Proxy Server Credentials (optional)
...
Loglevel ( default is INFO )
...
Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]
...
Target Directory
...
Linux or Windows Target Directory
For example, C:\tmp\ or /home/ubuntu/download
Unix filename pattern are supported to download a selection of files.
Supported wildcards are:
? matches any single character
[seq] matches any character in seq
[!seq] matches any character not in seq
”*” matches everything
...
Operation
...
[copy | move]
In case of a “move” the objects are deleted after they have been download from the container.
...
Download Write Options
...
Download Write Options:
[Replace existing File | Do not overwrite existing File | Timestamp | Default Windows behaviour]
Replace existing File : overwrite an existing file
Do not overwrite existing File: cancel the operations in case a File with a similar name exists
Timestamp: add a timestamp to the uploaded file
Default Windows behaviour: perform the default Windows behaviour for copying files. If a file with a similar name exists, the file names that are similar will be edited so that the files you copied have a number appended at the end of them. For example, if you are copying a file named image.png to a folder that already has a file named image.png in it, the copied file will be named image (1).png.
Example for Azure Blob Storage Universal Tasks - Download File
The following Task downloads from the container stonebranchpm all blobs matching the criteria test[1-2]* to the Linux directory: /home/stonebranch/demo/azure/in
Delete Objects - Action
The Actions is used to delete an object in an Azure Container or folder
...
Field
...
Description
...
Action
...
Delete-objects action
...
Connection Type
...
[Azure Account Key | SAS Token ]
If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.
If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
...
Useproxy ( default is NO )
...
[NO | YES]
If set to YES, the fields to set-up the proxy server connections are displayed:
Proxy Server IP or hostname
Proxy Server Port
Proxy Server Credentials (optional)
...
Loglevel ( default is INFO )
...
Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]
...
Container
...
Container in which the Blobs should be deleted.
...
Blob
...
Blobs to be deleted in the given container
Note: Due to security reasons wild card is only support, if at least one character is provided e.g. t* would delete all files starting with at “t”.
Example for Azure Blob Storage Universal Task - Delete Objects
The following Task deletes in the container stonbranchpm, folder incoming all Blobs matching the criteria test[3-4]*.
Copy Object to Container - Action
This Action is used copy an objects from one Azure Container to another Azure Container.
...
Field
...
Description
...
Action
...
Copy-object-to-bucket action
...
Connection Type
...
[Azure Account Key | SAS Token ]
If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.
If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
...
Useproxy ( default is NO )
...
[NO | YES]
If set to YES, the fields to set-up the proxy server connections are displayed:
Proxy Server IP or hostname
Proxy Server Port
Proxy Server Credentials (optional)
...
Loglevel ( default is INFO )
...
Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]
...
Container
...
Source container
...
Target Container
...
Target container, where the object(s) will be copied to
...
Blob
...
File ( Blob ) to be copied from the source container to target container.
...
Upload Write Options
...
Upload Write Options
[Do not overwrite existing Object | Timestamp]
Do not overwrite existing Object: Cancel the operations in case an object with a similar name exists
Timestamp: Add a timestamp to the uploaded Object
Example for Azure Blob Storage Universal Tasks - Copy Object to Container
The following Task copies the Blob: test1.txt from the source container: stonebranchpm to the target container: stonebranchpm2
Delete Container - Action
This action is used to delete an Azure Container
...
Field
...
Description
...
Action
...
Delete-container action
...
Connection Type
...
[Azure Account Key | SAS Token ]
If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.
If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
...
Useproxy ( default is NO )
...
[NO | YES]
If set to YES, the fields to set-up the proxy server connections are displayed:
Proxy Server IP or hostname
Proxy Server Port
Proxy Server Credentials (optional)
...
Loglevel ( default is INFO )
...
Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]
...
Container
...
Name of the Container to be deleted
Note: The container is delete including all objects.
Example for Azure Blob Storage Universal Tasks - Delete Container
The following Task delete the container stonebranchpm2
Upload File - Action
The Action is used to upload a single or multiple files from a Windows or Linux server to an container or a folder in a container.
...
Field
...
Description
...
Action
...
Upload-file action
...
Connection Type
...
[Azure Account Key | SAS Token ]
If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.
If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
...
Useproxy ( default is NO )
...
[NO | YES]
If set to YES, the fields to set-up the proxy server connections are displayed:
Proxy Server IP or hostname
Proxy Server Port
Proxy Server Credentials (optional)
...
Loglevel ( default is INFO )
...
Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]
...
Sourcefile
...
Source file(s) to be uploaded to a bucket or specific folder in a container.
Unix filename pattern are supported to upload a selection of files.
Supported wildcards are:
? matches any single character
[seq] matches any character in seq
[!seq] matches any character not in seq
”*” matches everything
...
Operation
...
[copy | move]
In case of a “move” the source file(s) is/are deleted after the upload to the container.
...
Prefix
...
A folder in a container is called prefix. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name.
...
Upload Write Options
...
Upload Write Options
[Do not overwrite existing Object | Timestamp]
Do not overwrite existing Object: Cancel the operations in case an object with a similar name exists
Timestamp: Add a timestamp to the uploaded Object
Example for Azure Blob Storage Universal Tasks - Upload File
The following Task uploads from the Linux directory the files: /home/stonebranch/demo/azure/out/test[1-4]* to the the container stonebranchpm, folder incoming.
Monitor Blob - Action
The Action is used to monitor a Blob in an Azure Container.
...
Field
...
Description
...
Action
...
Monitor-Blob action
...
Connection Type
...
[Azure Account Key | SAS Token ]
If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration.
If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
...
Useproxy ( default is NO )
...
[NO | YES]
If set to YES, the fields to set-up the proxy server connections are displayed:
Proxy Server IP or hostname
Proxy Server Port
Proxy Server Credentials (optional)
...
Loglevel ( default is INFO )
...
Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]
...
Container
...
Container in which the objects should be monitored
...
Prefix
...
A folder in a container is called prefix. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name.
If a prefix is provided only objects in the folder with the prefix name are listed in the output.
...
Blob
...
Blob to monitor. The task goes to status success, if the Blob is found.
Unix filename pattern are supported to list only a selection of files:
Supported wildcards are:
? matches any single character
[seq] matches any character in seq
[!seq] matches any character not in seq
”*” matches everything
Example:
Blob = test* : matches everything starting with test
Blob = test[1-2].txt : matches test1.txt, test2.txt
Blob = test[!1].txt: does not match test1.txt
Blob = test?.txt: matches test1.txt, test2.txt etc.
Example for Azure Blob Storage Universal Tasks - Monitor Blob
The following Task monitors in the Container: stonebranchpm all Blob’s starting with the name report. If a Blob starting with the name report is found the task goes to task status success.