Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at Stonebranch Integration Hub - Terms of Use.

Version Information

Template NameExtension NameExtension Version
DBT Core CLI

ue-dbt-core-cli

1.0.

0

1

Refer to Changelog for version history information.

...

This integration provides an interface to call the DBT Core CLI and enables users to execute DBT Core CLI tasks in Universal Controler. This is particularly useful when UC acts as a data pipeline orchestrator and DBT is a step within a data pipeline Universal Controller workflow.

Key Features

Feature

Description

Show Environment Information

Show DBT version and DBT environment variables information as Task Instance STDOUT. Enables users to retrieve Information on the dbt core installation.

Execute Command

Execute a given DBT command. DBT output is displayed as task instance STDOUT.

Credentials stored encrypted and passed as environment variables

Ability to safely store and pass UC Credentials as DBT environment variables.

Display dbt artifacts in Extension Output

Artifacts can be collected, and related information can be published within the Universal Controller for subsequent workflow tasks to use. 


Requirements

This integration requires a Universal Agent and a Python runtime to execute the Universal Task.

Area

Details

Python Version

Requires Python of version 3.7.  Tested with the Universal Agent bundled Python distribution (python version 3.7.

6

16)

Universal Agent

Both Windows and Linux agents are supported:

  • Universal Agent for Windows x64 Version >= 7.4.0.0

  • Universal Agent for Linux Version >= 7.4.0.0

Universal Controller

Universal Controller Version >= 7.4.0.0

Network and Connectivity

Connectivity towards Warehouses that DBT uses should be established.

DBT Module

This integration is tested against DBT core module (dbt-core) 1.5.1. It is required to be installed manually on the Agent. For more information refer to dbt-core official installation instructions.

Supported Actions

Action: Get Environment Information

...

No DBT Input Parameters are required to be passed apart from the Action itself. Common input parameters such as Runtime Directory or Environment Variables might come in handy if need to be configured on the task definition.

Configuration example

...

Image Added
User Scenario: Show dbt environment information.



Action Output


Output Type

Description (Successful Execution)

Example (Successful Execution)

EXTENSION

The extension output follows the standard Extension Output format, providing:

  • “exit_code“, “status“, “status_description“: General info regarding the task execution.

  • “invocation”

    • “fields”: The task configuration used for this task execution.

  • “result“ fields

    • “execution_info”:

      • “command_line”: The dbt command line that is executed

      • “command_exit_code”: Is the dbt command line exit code


Code Block
{
    "exit_code": 0,
    "status": "SUCCESS",
    "status_description": "Task executed successfully.",
    "invocation": {
        "extension": "ue-dbt-core-cli",
        "version": "1.0.0",
        "fields": {
            "action": "Show Environment Information",
            "credentials": null,
            "command_line": null,
            "artifact_content_visibility": "--None--"
        }
    },
    "result": {
        "execution_info": {
            "command_line": "dbt --version",
            "command_exit_code": 0
        }
    }
}



Description (Failed Execution)

Example (Failed Execution)

The extension output follows the standard Extension Output format, providing:

  • “exit_code“, “status“, “status_description“: Those attributes are set accordingly when an error happens. A list of possible values is provided in the extension exit codes table in this document.

  • “invocation”

    • “fields”: The task configuration used for this task execution.

  • “result“ fields

    • “execution_info”:

      • “command_line”: The dbt command line that is executed and resulted in erroneous execution

      • “command_exit_code”: is the dbt command line exit code if a dbt command is being executed.


Code Block
{
    "exit_code": 1,
    "status": "FAIL",
    "status_description": "Execution Failed: Refer to STDERR or Extension Output for more details.",
    "invocation": {
        "extension": "ue-dbt-core-cli",
        "version": "1.0.0",
        "fields": {
            "action": "Show Environment Information",
            "credentials": null,
            "command_line": null,
            "artifact_content_visibility": "--None--"
        }
    },
    "result": {
        "execution_info": {
            "command_line": "dbt --version",
            "command_exit_code": 1
        }
    }
}


STDOUT

On Extension STDOUT the following information is displayed:

  • The version of DBT

  • The DBT environment variables present at the execution time

  • The runtime directory


Code Block
DBT Core and Plugins Version Information
===========================================

Core:
  - installed: 1.5.1

Plugins:
  - postgres: 1.5.1 - Update available!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation


DBT Environment Variables
===========================================

+--------------------+---------+
| DBT ENV VARIABLE   | Value   |
+====================+=========+
| DBT_STB_VAR        | stb     |
+--------------------+---------+

Runtime Directory
===========================================
C:\Program Files\Universal\UAGSrv



Action: “Execute Command”

When Action = “Execute Command” then the command provided as Command Line is executed. Environment variables are set for credentials purposes through “Credentials as Environment Variables” before Command Line is executed.

For DBT commands that “target path” is required, it should be set either as an environment variable (DBT_TARGET_PATH) or provided on the Command Line.

Configuration examples

Image RemovedImage Removed

Image Added


Image Added
User Scenario: Execute dbt command using Credentials as DBT Environment Variables.User Scenario: Execute dbt command using Credentials as DBT Environment Variables. The above script shows how to set DBT environment variables using UAC Functions on Resolvable Credentials.



Action Output


Output Type

Description (Successful Execution)

Example (Successful Execution)

EXTENSION

The extension output follows the standard Extension Output format, providing:

  • “exit_code“, “status“, “status_description“: General info regarding the task execution.

  • “invocation”

    • “fields”: The task configuration used for this task execution.

  • “result“ fields:

    • “execution_info”:

      • “command_line”: The dbt command line that is executed

      • “command_exit_code”: Is the dbt command line exit code

    • “artifacts” field: a list of artifact information produced by dbt command. for each element of the list the following attributes are available

      • “type”: the type of the artifact. Manifest and Run Results artifacts are supported.

      • “path”: The path where the artifact file is located.

      • “content”: The content of the artifact file if configured to be printed on Extension Output, else null.

Info

Note on artifacts: The artifacts that are supported are the “Manifest” and the “Run Results”. Both of them are located in the “target” directory. The target directory will be identified automatically by this integration taking into account --target-path CLI flag and DBT_TARGET_PATH environment variable, but not take into account any configuration on dbt_project.yml file, since the configuration of “target” directory path in dbt_project.yml is considered deprecated and expected not to work in dbt core version 1.6.

The precedence order for detection of the target directory is: CLI flag, Environment variable.




Code Block
{
    "exit_code": 0,
    "status": "SUCCESS",
    "status_description": "Task executed successfully.",
    "invocation": {
        "extension": "ue-dbt-core-cli",
        "version": "1.0.0",
        "fields": {
            "action": "Execute Command",
            "credentials": "/home/agent/tmp/e8917d08-0341-4239-a486-f53570237817",
            "command_line": "dbt run
--profiles-dir=/home/
.../
dbt_workspace/.dbt
--project-dir=/home/
.../dbt
dbt_workspace/my_stb_dbt_project 
--target-path=/home
/...
/dbt_workspace/my_stb_dbt_project/ue_execution",
            "artifact_content_visibility": "--None--"
        }
    },
    "result": {
        "execution_info": {
            "command_line": "dbt run --profiles-dir=/home
/...
/dbt_workspace/.dbt 
--project-dir=/home
/...
/dbt_workspace/my_stb_dbt_project 
--target-path=/home
/ue-dev/USERS/ioanna
/dbt_workspace/my_stb_dbt_project/ue_execution",
            "command_exit_code": 0,
            "artifacts": [
                {
                    "type": "manifest",
                    "path": "/home
/...
/dbt_workspace/my_stb_dbt_project/ue_execution/manifest.json",
                    "content": null
                },
                {
                    "type": "run_results",
                    "path": "/home/.../dbt_workspace/my_stb_dbt_project/ue_execution/run_results.json",
                    "content": null
                }
            ]
        }
    }
}



Description (Failed Execution)

Example (Failed Execution)


The extension output follows the standard Extension Output format, providing:

  • “exit_code“, “status“, “status_description“: Will be set according to the specific error that happened during the execution. A list of possible values is provided in the extension exit codes table in this document.

  • “invocation”

    • “fields”: Same as in successful execution

  • “result“ fields:

    • “execution_info”:

      • “command_line”: The dbt command line that is executed

      • “command_exit_code”: Is the dbt command line exit code

    • “artifacts” field: a list of artifact information produced by dbt command. for each element of the list, the following attributes are available

      • “type”: the type of the artifact. Manifest and Run Results artifacts are supported.

      • “path”: The path where the artifact file is located.

      • “content”: The content of the artifact file if it is configured to be printed on Extension Output, else null.



Code Block
{
    "exit_code": 1,
    "status": "FAIL",
    "status_description": "Execution Failed: Refer to STDERR or Extension Output for more details.",
    "invocation": {
        "extension": "ue-dbt-core-cli",
        "version": "1.0.0",
        "fields": {
            "action": "Execute Command",
            "credentials": "/home/agent/tmp/587a68dd-3220-41e8-94b2-a2d0caf69445",
            "command_line": "dbt build --profiles-dir=/home/
.../
dbt_workspace/.dbt 
--project-dir=/home
/...
/dbt_workspace/my_stb_dbt_project 
--target-path=/home
/...
/dbt_workspace/my_stb_dbt_project/ue_execution",
            "artifact_content_visibility": "Run Results on Extension Output",
        }
    },
    "result": {
        "execution_info": {
            "command_line": "dbt build --profiles-dir=/home
/...
/dbt_workspace/.dbt 
--project-dir=/home/
.../
dbt_workspace/my_stb_dbt_project 
--target-path=/home/
.../
dbt_workspace/my_stb_dbt_project/ue_execution",
            "command_exit_code": 2,
            "artifacts": []
        }
    }
}


STDOUT

The standard dbt command stdout appears also on the STDOUT of the Universal Task Instance. The level of information depends on the Log Level selected at Univesal Task Definition.


Code Block
12:14:02  Running with dbt=1.5.0
12:14:03  Found 2 models, 4 tests, 0 snapshots, 0 analyses, 307 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics, 0 groups
12:14:03
12:14:04  Concurrency: 1 threads (target='dev')
12:14:04
12:14:04  1 of 2 START sql table model dev_schema.my_first_dbt_model ..................... [RUN]
12:14:05  1 of 2 OK created sql table model dev_schema.my_first_dbt_model ................ [SELECT 2 in 0.81s]
12:14:05  2 of 2 START sql view model dev_schema.my_second_dbt_model ..................... [RUN]
12:14:06  2 of 2 OK created sql view model dev_schema.my_second_dbt_model ................ [CREATE VIEW in 0.81s]
12:14:06
12:14:06  Finished running 1 table model, 1 view model in 0 hours 0 minutes and 3.97 seconds (3.97s).
12:14:06
12:14:06  Completed successfully
12:14:06
12:14:07  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2

...


Input Fields

Field

Type

Default Value

Mandatory

Description

Introduced in Version

Action

Choice

Show Environment Information

Yes

The action performed upon the task execution.

Available options:

  • Show Environment Information

  • Execute Command

1.0.0

Credentials as Environment Variables

Credentials


Optional


A script that allows to set Environment Variables that are used as credentials inside DBT

The format of this script should be

MY_CREDENTIAL_ENV_VARIABLE_A=<UC credential function call A>
MY_CREDENTIAL_ENV_VARIABLE_B=<UC credential function call B>
...

Available when Action = “Execute Command”

1.0.0

Artifact Content Visibility

Choice

--None--

Optional

  • --None--

  • Run Results on Extension Output

Available when Action = “Execute Command”

1.0.0

Command Line

Large Text


Optional

The full command line.

Available when Action = “Execute Command”

1.0.0

Output-Only fields

No output only fields

...

Cancellation from UC will just stop the execution of the Task. The Re-run of the task behaves the same way as on the initial run.

...

Exit Codes

Extension Exit Code

Status

Status Description

 Meaning

0

SUCCESS

“Task executed successfully“

Successful Execution, in case DBT exit code is 0.

1

FAIL

“Execution Failed: <error details>”

Failed Execution, in case DBT exit code is other than 0.

20

DATA_VALIDATION_ERROR

“Data Validation Error: <error details>.“

Input data validation error.

STDOUT and STDERR


STDOUT and STDERR provide additional information to User. The populated content can be changed in future versions of this extension without notice. Backward compatibility is not guaranteed.

Depending on the log level the Universal Task is started by the controller, the DBT_LOG_LEVEL environment variable is set prior to the execution of the DBT command based on the mapping below.

Universal Task Log Level

DBT Log Level

Trace & Debug

Debug

Info

info


Warn

warn

Error & Severe

error


The following applies:

  • IF DBT_LOG_LEVEL is already set then it will not be overwritten.

  • IF --log-level command line option is used, by default dbt uses that value from the command line

...

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Universal Task

For a new Universal Task, create a new task, and enter the required input fields.

Anchor
IntegrationModifications
IntegrationModifications

Integration Modifications

...

This document references the following documents:

Document Link

Description

Universal Templates

User documentation for creating, working with and understanding Universal Templates and Integrations.

Universal Tasks

User documentation for creating Universal Tasks in the Universal Controller user interface.

Anchor
changelog
changelog

Changelog

ue-dbt-core-cli-1.0.1 (2023-10-18)

Fixes

  • Fixed:  Include integration icon on Universal Template (#34228).

ue-dbt-core-cli-1.0.0(2023-08-16)

Initial Version