Your use of this download is governed by Stonebranch’s Terms of Use, which are available at Stonebranch Integration Hub - Terms of Use.
Template Name | Extension Name | Extension Version |
---|---|---|
DBT Core CLI | ue-dbt-core-cli | 1.0.1 |
Refer to Changelog for version history information.
DBT (Data Build Tool) Core is an open-source too that is designed to help data engineers and analysts manage and transform data in a reliable and scalable manner. Some key features of DBT Core include version control integration, automated testing, documentation generation, and dependency management. These features enable teams to collaborate effectively, ensure the quality of their data transformations, and maintain documentation for their data pipelines. DBT Core simplifies the process of transforming data in a data warehouse environment, providing a structured and scalable approach that helps teams manage their data transformation workflows effectively.
This integration provides an interface to call the DBT Core CLI and enables users to execute DBT Core CLI tasks in Universal Controler. This is particularly useful when UC acts as a data pipeline orchestrator and DBT is a step within a data pipeline Universal Controller workflow.
Feature | Description |
---|---|
Show Environment Information | Show DBT version and DBT environment variables information as Task Instance STDOUT. Enables users to retrieve Information on the dbt core installation. |
Execute Command | Execute a given DBT command. DBT output is displayed as task instance STDOUT. |
Credentials stored encrypted and passed as environment variables | Ability to safely store and pass UC Credentials as DBT environment variables. |
Display dbt artifacts in Extension Output | Artifacts can be collected, and related information can be published within the Universal Controller for subsequent workflow tasks to use. |
This integration requires a Universal Agent and a Python runtime to execute the Universal Task.
Area | Details |
---|---|
Python Version | Requires Python of version 3.7. Tested with the Universal Agent bundled Python distribution (python version 3.7.16) |
Universal Agent | Both Windows and Linux agents are supported:
|
Universal Controller | Universal Controller Version >= 7.4.0.0 |
Network and Connectivity | Connectivity towards Warehouses that DBT uses should be established. |
DBT Module | This integration is tested against DBT core module (dbt-core) 1.5.1. It is required to be installed manually on the Agent. For more information refer to dbt-core official installation instructions. |
This action provides information about the DBT Core execution environment by listing the following information
the version of DBT
the DBT environment variables present at the execution time
the runtime directory
DBT CLI command needs to be on the execution PATH and could be set either on the environment of the user that runs the task on the Agent side or by setting Environment Variables on the task definition.
No DBT Input Parameters are required to be passed apart from the Action itself. Common input parameters such as Runtime Directory or Environment Variables might come in handy if need to be configured on the task definition.
User Scenario: Show dbt environment information. |
Output Type | Description (Successful Execution) | Example (Successful Execution) | |
---|---|---|---|
EXTENSION | The extension output follows the standard Extension Output format, providing:
|
| |
Description (Failed Execution) | Example (Failed Execution) | ||
The extension output follows the standard Extension Output format, providing:
|
| ||
STDOUT | On Extension STDOUT the following information is displayed:
|
|
When Action = “Execute Command” then the command provided as Command Line is executed. Environment variables are set for credentials purposes through “Credentials as Environment Variables” before Command Line is executed.
For DBT commands that “target path” is required, it should be set either as an environment variable (DBT_TARGET_PATH) or provided on the Command Line.
User Scenario: Execute dbt command using Credentials as DBT Environment Variables. | User Scenario: Execute dbt command using Credentials as DBT Environment Variables. The above script shows how to set DBT environment variables using UAC Functions on Resolvable Credentials. |
Output Type | Description (Successful Execution) | Example (Successful Execution) | ||
---|---|---|---|---|
EXTENSION | The extension output follows the standard Extension Output format, providing:
|
| ||
Description (Failed Execution) | Example (Failed Execution) | |||
The extension output follows the standard Extension Output format, providing:
|
| |||
STDOUT | The standard dbt command stdout appears also on the STDOUT of the Universal Task Instance. The level of information depends on the Log Level selected at Univesal Task Definition. |
|
Field | Type | Default Value | Mandatory | Description | Introduced in Version |
---|---|---|---|---|---|
Action | Choice | Show Environment Information | Yes | The action performed upon the task execution. Available options:
| 1.0.0 |
Credentials as Environment Variables | Credentials | Optional | A script that allows to set Environment Variables that are used as credentials inside DBT The format of this script should be
Available when Action = “Execute Command” | 1.0.0 | |
Artifact Content Visibility | Choice | --None-- | Optional |
Available when Action = “Execute Command” | 1.0.0 |
Command Line | Large Text | Optional | The full command line. Available when Action = “Execute Command” | 1.0.0 |
No output only fields
Cancellation from UC will just stop the execution of the Task. The Re-run of the task behaves the same way as on the initial run.
Extension Exit Code | Status | Status Description | Meaning |
---|---|---|---|
0 | SUCCESS | “Task executed successfully“ | Successful Execution, in case DBT exit code is 0. |
1 | FAIL | “Execution Failed: <error details>” | Failed Execution, in case DBT exit code is other than 0. |
20 | DATA_VALIDATION_ERROR | “Data Validation Error: <error details>.“ | Input data validation error. |
STDOUT and STDERR provide additional information to User. The populated content can be changed in future versions of this extension without notice. Backward compatibility is not guaranteed.
Depending on the log level the Universal Task is started by the controller, the DBT_LOG_LEVEL environment variable is set prior to the execution of the DBT command based on the mapping below.
Universal Task Log Level | DBT Log Level |
---|---|
Trace & Debug | Debug |
Info | info |
Warn | warn |
Error & Severe | error |
The following applies:
IF DBT_LOG_LEVEL is already set then it will not be overwritten.
IF --log-level command line option is used, by default dbt uses that value from the command line
To use the Universal Template, you first must perform the following steps.
This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
To import the Universal Template into your Controller, follow these instructions.
When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.
Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.
For a new Universal Task, create a new task, and enter the required input fields.
Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.
Python code modifications should not be done.
Template Modifications
General Section
"Name", "Extension", "Variable Prefix", "Icon" should not be changed.
Universal Template Details Section
"Template Type", "Agent Type", "Send Extension Variables", "Always Cancel on Force Finish" should not be changed.
Result Processing Defaults Section
Success and Failure Exit codes should not be changed.
Success and Failure Output processing should not be changed.
Fields Restriction Section
The setup of the template does not impose any restrictions, However with respect to "Exit Code Processing Fields" section.
Success/Failure exit codes need to be respected.
In principle, as STDERR and STDOUT outputs can change in follow-up releases of this integration, they should not be considered as a reliable source for determining success or failure of a task.
Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.
This document references the following documents:
Document Link | Description |
---|---|
User documentation for creating, working with and understanding Universal Templates and Integrations. | |
User documentation for creating Universal Tasks in the Universal Controller user interface. |
Fixed
: Include integration icon on Universal Template (#34228).
Initial Version