Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/

Introduction

Storing data in the cloud becomes an integral part of most modern IT landscapes. With Universal Automation Center you can securely automate your AWS, Azure, Google and MinIO File Transfers and integrate them into your existing scheduling flows.

As security is one of the key concerns when moving to the cloud, the provided solution supports multi-level of security:

Credentials for Amazon S3 (Access Key, Secret Access key and Region) are stored in an encrypted form in the database
IAM Role-Based Access Control (RBAC) is supported
Communication to Amazon is done via the HTTPS protocol
A Proxy Server connection to Amazon S3 with basic authentication is supported
Secure access to Amazon S3 buckets using AWS bucket policies can be configured in the AWS console
Restrict sending files only to specific buckets using Amazon S3 End Points can be configured in the AWS console

This Universal Task focuses on the AMAZON AWS S3 file transfer, including support for MinIO. MinIO is an Open Source object storage server for private cloud environments based on Amazon’s S3 API. All file transfer scenarios supported for AMAZON AWS S3 are also support for MinIO. The scenarios described in this documentation are also valid for MinIO.

A similar solution as for AWS S3 is also available for Microsoft Azure Blob Storage and Google Cloud Storage.

Overview

The Universal Task for Amazon S3 allows to securely transfers files from, to and between AWS S3 cloud storage buckets and folders.

The Universal Task for Amazon S3 support the following main features:

The following file transfer commands are supported:
- Upload a file(s) to an S3 bucket
- Download of file(s) from an S3 bucket
- Transfer files between S3 buckets
- List objects in an S3 bucket
- Delete object(s) in an S3 bucket
- List S3 bucket names
- Create an S3 bucket
Monitor for a key(s) in a bucket.
File Transfer can be trigger by a third-party application using the Universal Automation Center RESTfull Webservice API: REST API.
Universal Task for AWS S3 can be integrated into any existing scheduling workflow in the same way as any standard Linux or Windows Task type.
Security is ensured by using the HTTPS protocol with support for an optional Proxy Server.
AWS IAM Role Based Access ( RBCA ) is supported.
AWS canned ACLs are supported; for example, to grant full access to the bucket owner.
No Universal Agent needs to be installed on the AWS Cloud – the communication goes via HTTPS.

Software Requirements

Software Requirements Universal Agents and Universal Controller

Universal Agent for Linux or Windows Version 7.0.0.0 or later are required.
The Universal Agent needs to be installed with python option (--python yes).

Software Requirements Universal Controller

Universal Controller 7.0.0.0. or later is required.

Software Requirements for the Application to be scheduled

The Universal Task has been tested for the AWS S3 SDK for python (boto3) V1.15.6.

Universal Task for AWS S3 Key Features

The solution supports the following file transfer scenarios:

Name	Description
Upload a file(s) to a bucket (Copy or Move)	A single or multiple files can be uploaded to a bucket. Move or copy are supported for the upload Unix filename pattern matching support e.g. wildcard support “” to upload multiple files A single or multiple files can be Uploaded to a folder using a prefix It can be decided via an the Upload Write Options: to overwrite an existing object (Replace existing Object) to cancel the operations in case an object with a similar name exists (Do not overwrite existing Object) to add a timestamp to the uploaded Object (Timestamp*)
Download of file(s) from a bucket (Copy or Move)	A single or multiple files should be downloaded from a bucket. Move or copy must be supported for the download Unix filename pattern matching support e.g. wildcard support “” to download multiple files Download to a specific folder is supported It can be decided via an the Download Write Options: to overwrite an existing file (Replace existing File) to cancel the operations in case a File with a similar name exists (Do not overwrite existing File) to add a timestamp to the uploaded file (Timestamp) to perform the default Windows behaviour for copying files (Default Windows behaviour*) If a file with a similar name exists, the file names that are similar will be edited so that the files you copied have a number appended at the end of them. For example, if you’re copying a file named image.png to a folder that already has a file named image.png in it, the copied file will be named image (1).png.
Copy object to a bucket	An object can be copied from one AWS S3 bucket to another. Folders are support
List objects in a bucket	Show all Objects in a bucket Show all Objects in a folder Unix filename pattern matching support e.g. wildcard support “*” to narrow down the objects to display
Delete object(s) in a bucket	Delete one or multiple objects in a bucket Delete one or multiple objects in folder in a bucket Unix filename pattern matching support; for example, wildcard support “*” to narrow down the objects to be deleted
List buckets	List all bucket in an AWS S3 account
Create an S3 bucket	Create a new S3 bucket
Delete an S3 bucket	Delete an S3 bucket
Monitor Key	Monitor if a certain key(s) exists in an S3 bucket. Unix filename pattern matching support; for example, wildcard support “*” to narrow down the keys to be monitored.
Proxy Server Connection	A proxy server connection with or without basic authentication can be configured
Integration into 3^rd Party Applications	An AWS S3 file transfer can be triggered via the Universal Automation Center RESTfull Webservice API within an application.
Self-Service through Web-Client	The S3 Task can be fully configured, monitored and updated via the Universal Controller Web-GUI

Import Amazon S3 Downloadable Universal Template

To use this downloadable Universal Template, you first must perform the following steps:

This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
In the Universal Controller UI, select Administration >Configuration > Universal Templates to display the current list of Universal Templates.
Click Import Template.
Select the template ZIP file and Import.

When the template has been imported successfully, the Universal Template will appear on the list. Refresh your Navigation Tree to see these tasks in the Automation Center Menu.Download the provided ZIP file.

Configure Amazon S3 Universal Tasks

For the new Universal Task type, create a new task and enter the task-specific Details that were created in the Universal Template.

Field Descriptions for Amazon S3 Universal Task - Action

The Amazon S3 Task provides multiple different file transfer actions. For each action the specific fields are described.

Create Bucket - Action

The Action Creates a new S3 bucket.

Field	Description
Action	create-bucket action
AWS_ACCESS_KEY_ID	AWS Account credentials Access Key
AWS_DEFAULT_REGION	AWS Account credentials Region
AWS_SECRET_ACCESS_KEY	AWS Account credentials Secret Access Key
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Role Based Access	[NO \| YES] If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed: Role Arn: Amazon Role, which is applied for the connection; for example, rn:aws:iam::111222333444:role/SB-AWSS3 Service Name [STS \| S3 ] , default is STS STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources.
Endpoint URL	Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system
Bucket	Name of the AWS Bucket to create

Example for AWS S3 Universal Tasks - Create Bucket

List Buckets - Action

The Action list all buckets of an AWS Account.

Field	Description
Action	list-buckets action
AWS_ACCESS_KEY_ID	AWS Account credentials Access Key
AWS_DEFAULT_REGION	AWS Account credentials Region
AWS_SECRET_ACCESS_KEY	AWS Account credentials Secret Access Key
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Role Based Access	[NO \| YES] If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed: Role Arn: Amazon Role, which is applied for the connection e.g. rn:aws:iam::111222333444:role/SB-AWSS3 Service Name [STS \| S3 ] , default is STS STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources.
Endpoint URL	Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system

Example for Amazon S3 Universal Tasks - List Buckets

Upload File - Action

The Action is used to upload a single or multiple files from a Windows or Linux server to an AWS bucket or a folder in an AWS bucket.

Field	Description
Action	Upload-file action
AWS_ACCESS_KEY_ID	AWS Account credentials Access Key
AWS_DEFAULT_REGION	AWS Account credentials Region
AWS_SECRET_ACCESS_KEY	AWS Account credentials Secret Access Key
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Role Based Access	[NO \| YES] If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed: Role Arn: Amazon Role, which is applied for the connection e.g. rn:aws:iam::111222333444:role/SB-AWSS3 Service Name [STS \| S3 ] , default is STS STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources.
ACL	bucket-owner-full-control \| private \| public-read \| public-read-write \| aws-exec-read \| authenticated-read \| bucket-owner-read \| log-delivery-write AWS canned ACLs; for details, refer to: AWS canned ACL The default ACL bucket-owner-full-control grant full access to the bucket owner.
Endpoint URL	Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage
Sourcefile	Source file(s) to be uploaded to a bucket or specific folder in a bucket. Unix filename pattern are supported to upload a selection of files. Supported wildcards are: ? matches any single character [seq] matches any character in seq [!seq] matches any character not in seq ”*” matches everything
Operation	[copy \| move] In case of a “move” the source file(s) is/are deleted after the upload to the bucket.
Prefix	A folder in a bucket is called prefix in AWS. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name.
Upload Write Options	Upload Write Options [Replace existing Object \| Do not overwrite existing Object \| Timestamp] Replace existing Object: Overwrite an existing object Do not overwrite existing Object: Cancel the operations in case an object with a similar name exists Timestamp: Add a timestamp to the uploaded Object

Example for Amazon S3 Universal Tasks - Upload File

List Objects - Action

The Action is used to display objects in a bucket or a specific bucket folder (prefix).

Field	Description
Action	List-objects action
AWS_ACCESS_KEY_ID	AWS Account credentials Access Key
AWS_DEFAULT_REGION	AWS Account credentials Region
AWS_SECRET_ACCESS_KEY	AWS Account credentials Secret Access Key
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Role Based Access	[NO \| YES] If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed: Role Arn: Amazon Role, which is applied for the connection e.g. rn:aws:iam::111222333444:role/SB-AWSS3 Service Name [STS \| S3 ] , default is STS STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources.
Endpoint URL	Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system
Bucket	Bucket name in AWS
Prefix	A folder in a bucket is called prefix in AWS. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name. If a prefix is provided only objects in the folder with the prefix name are listed in the output.
S3key	Objects matching the given S3key are listed. Unix filename pattern are supported to list only a selection of files: Supported wildcards are: ? matches any single character [seq] matches any character in seq [!seq] matches any character not in seq ”” matches everything Example: S3key = test : matches everything starting with test S3key = test[1-2].txt : matches test1.txt, test2.txt S3key = test[!1].txt: does not match test1.txt S3key = test?.txt: matches test1.txt, test2.txt etc.
Show Details	Show details like creation timestamp in the output

Example for Amazon S3 Universal Tasks - List Objects

Download File - Action

This Action downloads one or multiple files from a bucket to a Linux or Windows folder

Field	Description
Action	Download-file action
AWS_ACCESS_KEY_ID	AWS Account credentials Access Key
AWS_DEFAULT_REGION	AWS Account credentials Region
AWS_SECRET_ACCESS_KEY	AWS Account credentials Secret Access Key
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Role Based Access	[NO \| YES] If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed: Role Arn: Amazon Role, which is applied for the connection e.g. rn:aws:iam::111222333444:role/SB-AWSS3 Service Name [STS \| S3 ] , default is STS STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources.
Endpoint URL	Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system
Target Directory	Linux or Windows Target Directory For example, C:\tmp\ or /home/ubuntu/download Unix filename pattern are supported to download a selection of files. Supported wildcards are: ? matches any single character [seq] matches any character in seq [!seq] matches any character not in seq ”*” matches everything
Operation	[copy \| move] In case of a “move” the objects are deleted after they have been download from the bucket.
Download Write Options	Download Write Options: [Replace existing File \| Do not overwrite existing File \| Timestamp \| Default Windows behaviour] Replace existing File : overwrite an existing file Do not overwrite existing File: cancel the operations in case a File with a similar name exists Timestamp: add a timestamp to the uploaded file Default Windows behaviour: perform the default Windows behaviour for copying files. If a file with a similar name exists, the file names that are similar will be edited so that the files you copied have a number appended at the end of them. For example, if you’re copying a file named image.png to a folder that already has a file named image.png in it, the copied file will be named image (1).png.

Example for Amazon S3 Universal Tasks - Download File

Delete Objects - Action

The Actions is used to delete an object in a bucket or folder.

Field	Description
Action	Delete-objects action
AWS_ACCESS_KEY_ID	AWS Account credentials Access Key
AWS_DEFAULT_REGION	AWS Account credentials Region
AWS_SECRET_ACCESS_KEY	AWS Account credentials Secret Access Key
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Role Based Access	[NO \| YES] If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed: Role Arn: Amazon Role, which is applied for the connection e.g. rn:aws:iam::111222333444:role/SB-AWSS3 Service Name [STS \| S3 ] , default is STS STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources.
Endpoint URL	Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system
Bucket	AWS bucket name
S3key	Key to be deleted in AWS Note: Due to security reasons wildcard is only support, if at least one character is provided e.g. t* would delete all files starting with at “t”.

Example for Amazon S3 Universal Tasks - Delete Objects

Copy Object to Bucket - Action

This Action is used copy a single or multiple objects from one bucket to another bucket in AWS

Field	Description
Action	Copy-object-to-bucket action
AWS_ACCESS_KEY_ID	AWS Account credentials Access Key
AWS_DEFAULT_REGION	AWS Account credentials Region
AWS_SECRET_ACCESS_KEY	AWS Account credentials Secret Access Key
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Role Based Access	[NO \| YES] If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed: Role Arn: Amazon Role, which is applied for the connection e.g. rn:aws:iam::111222333444:role/SB-AWSS3 Service Name [STS \| S3 ] , default is STS STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources.
ACL	bucket-owner-full-control \| private \| public-read \| public-read-write \| aws-exec-read \| authenticated-read \| bucket-owner-read \| log-delivery-write AWS canned ACLs; for details, refer to: AWS canned ACL The default ACL bucket-owner-full-control grant full access to the bucket owner.
Endpoint URL	Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system
Bucket	AWS bucket name
Target Bucket	target Bucket, where the object(s) will be copied to
S3key	File to be copied from one bucket to another
TargetS3Key	Target file name. The TargetS3key supports pattern match. For example: Bucket: stonebranchpm Target Bucket: stonebranchpm1 S3key: dir1/report* ; Target s3key: dir2/* This copies all files from bucket stonebranchpm in the directory dir1 starting with the file name report to the directory dir2/ in the bucket stonebranchpm1. Note If the field TargetS3key is left blank, the entire source key - including the directory - will be copied to the target bucket. For example: Bucket: stonebranchpm Target Bucket: stonebranchpm1 S3key: dir1/report* ; Target s3key: This copies dir1/report* from bucket stonebranchpm to the bucket stonebranchpm1 ATTENTION Do not use a non-pattern match Target S3key when the S3key uses pattern match. For example: Bucket: stonebranchpm Target Bucket: stonebranchpm1 S3key: dir1/report* ; Target s3key: dir2/report1.txt This set-up will copy only one file, even if multiple files will match the S3Key in the source.
Upload Write Options	Upload Write Options [Replace existing Object \| Do not overwrite existing Object \| Timestamp] Replace existing Object: Overwrite an existing object Do not overwrite existing Object: Cancel the operations in case an object with a similar name exists Timestamp: Add a timestamp to the uploaded Object

Example for Amazon S3 Universal Tasks - Copy Object to Bucket

Delete Bucket - Action

This action is used to delete a bucket

Field	Description
Action	Delete-bucket action
AWS_ACCESS_KEY_ID	AWS Account credentials Access Key
AWS_DEFAULT_REGION	AWS Account credentials Region
AWS_SECRET_ACCESS_KEY	AWS Account credentials Secret Access Key
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Role Based Access	[NO \| YES] If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed: Role Arn: Amazon Role, which is applied for the connection e.g. rn:aws:iam::111222333444:role/SB-AWSS3 Service Name [STS \| S3 ] , default is STS STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources.
Endpoint URL	Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system
Bucket	Name of the bucket to be deleted

Example for Amazon S3 Universal Tasks - Delete Bucket

Monitor Key - Action

This Action is used to monitor for a key(s) in a bucket

Field	Description
Action	monitor-key action
AWS_ACCESS_KEY_ID	AWS Account credentials Access Key
AWS_DEFAULT_REGION	AWS Account credentials Region
AWS_SECRET_ACCESS_KEY	AWS Account credentials Secret Access Key
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Role Based Access	[NO \| YES] If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed: Role Arn: Amazon Role, which is applied for the connection e.g. rn:aws:iam::111222333444:role/SB-AWSS3 Service Name [STS \| S3 ] , default is STS STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources.
Endpoint URL	Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system
Bucket	Bucket name in AWS
Prefix	A folder in a bucket is called prefix in AWS. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name. If a prefix is provided only objects in the folder with the prefix name are listed in the output.
S3key	Objects matching the given S3key are listed. Unix filename pattern are supported to list only a selection of files: Supported wildcards are: ? matches any single character [seq] matches any character in seq [!seq] matches any character not in seq ”” matches everything Example: S3key = test : matches everything starting with test S3key = test[1-2].txt : matches test1.txt, test2.txt S3key = test[!1].txt: does not match test1.txt S3key = test?.txt: matches test1.txt, test2.txt etc.
Interval	[10 \| 60 \| 180] Monitoring Interval in seconds Example: An Interval of 60s means every 60s the bucket is scanned for the key(s) to monitor.

Integrations

Amazon S3: Cloud Storage Bucket File Transfer