Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at Stonebranch Integration Hub - Terms of Use.

Version Information

Refer to Changelog for version history information.

Overview

DBT (Data Build Tool) Core is an open-source too that is designed to help data engineers and analysts manage and transform data in a reliable and scalable manner. Some key features of DBT Core include version control integration, automated testing, documentation generation, and dependency management. These features enable teams to collaborate effectively, ensure the quality of their data transformations, and maintain documentation for their data pipelines. DBT Core simplifies the process of transforming data in a data warehouse environment, providing a structured and scalable approach that helps teams manage their data transformation workflows effectively.

This integration provides an interface to call the DBT Core CLI and enables users to execute DBT Core CLI tasks in Universal Controler. This is particularly useful when UC acts as a data pipeline orchestrator and DBT is a step within a data pipeline Universal Controller workflow.

Key Features

Requirements

This integration requires a Universal Agent and a Python runtime to execute the Universal Task.

Supported Actions

Action: Get Environment Information

This action provides information about the DBT Core execution environment by listing the following information

• the version of DBT

• the DBT environment variables present at the execution time

• the runtime directory

DBT CLI command needs to be on the execution PATH and could be set either on the environment of the user that runs the task on the Agent side or by setting Environment Variables on the task definition.

No DBT Input Parameters are required to be passed apart from the Action itself. Common input parameters such as Runtime Directory or Environment Variables might come in handy if need to be configured on the task definition.

Configuration example

User Scenario: Show dbt environment information.

Action Output

Output Type	Description (Successful Execution)	Example (Successful Execution)
EXTENSION	The extension output follows the standard Extension Output format, providing: “exit_code“, “status“, “status_description“: General info regarding the task execution. “invocation” “fields”: The task configuration used for this task execution. “result“ fields “execution_info”: “command_line”: The dbt command line that is executed “command_exit_code”: Is the dbt command line exit code	{ "exit_code": 0, "status": "SUCCESS", "status_description": "Task executed successfully.", "invocation": { "extension": "ue-dbt-core-cli", "version": "1.0.0", "fields": { "action": "Show Environment Information", "credentials": null, "command_line": null, "artifact_content_visibility": "--None--" } }, "result": { "execution_info": { "command_line": "dbt --version", "command_exit_code": 0 } } }
	Description (Failed Execution)	Example (Failed Execution)
	The extension output follows the standard Extension Output format, providing: “exit_code“, “status“, “status_description“: Those attributes are set accordingly when an error happens. A list of possible values is provided in the extension exit codes table in this document. “invocation” “fields”: The task configuration used for this task execution. “result“ fields “execution_info”: “command_line”: The dbt command line that is executed and resulted in erroneous execution “command_exit_code”: is the dbt command line exit code if a dbt command is being executed.	{     "exit_code": 1,     "status": "FAIL",     "status_description": "Execution Failed: Refer to STDERR or Extension Output for more details.",     "invocation": {         "extension": "ue-dbt-core-cli",         "version": "1.0.0",         "fields": {             "action": "Show Environment Information",             "credentials": null,             "command_line": null,             "artifact_content_visibility": "--None--"         }     },     "result": {         "execution_info": {             "command_line": "dbt --version",             "command_exit_code": 1         }     } }
STDOUT	On Extension STDOUT the following information is displayed: The version of DBT The DBT environment variables present at the execution time The runtime directory	DBT Core and Plugins Version Information =========================================== Core: - installed: 1.5.1 Plugins: - postgres: 1.5.1 - Update available! At least one plugin is out of date or incompatible with dbt-core. You can find instructions for upgrading here: https://docs.getdbt.com/docs/installation DBT Environment Variables =========================================== +--------------------+---------+ | DBT ENV VARIABLE | Value | +====================+=========+ | DBT_STB_VAR | stb | +--------------------+---------+ Runtime Directory =========================================== C:\Program Files\Universal\UAGSrv

Action: “Execute Command”

When Action = “Execute Command” then the command provided as Command Line is executed. Environment variables are set for credentials purposes through “Credentials as Environment Variables” before Command Line is executed.

For DBT commands that “target path” is required, it should be set either as an environment variable (DBT_TARGET_PATH) or provided on the Command Line.

Configuration examples

User Scenario: Execute dbt command using Credentials as DBT Environment Variables.	User Scenario: Execute dbt command using Credentials as DBT Environment Variables. The above script shows how to set DBT environment variables using UAC Functions on Resolvable Credentials.

Action Output

{
    "exit_code": 0,
    "status": "SUCCESS",
    "status_description": "Task executed successfully.",
    "invocation": {
        "extension": "ue-dbt-core-cli",
        "version": "1.0.0",
        "fields": {
            "action": "Execute Command",
            "credentials": "/home/agent/tmp/e8917d08-0341-4239-a486-f53570237817",
            "command_line": "dbt run
--profiles-dir=/home/dbt_workspace/.dbt
--project-dir=/home/dbt_workspace/my_stb_dbt_project 
--target-path=/home/dbt_workspace/my_stb_dbt_project/ue_execution",
            "artifact_content_visibility": "--None--"
        }
    },
    "result": {
        "execution_info": {
            "command_line": "dbt run --profiles-dir=/home/dbt_workspace/.dbt 
--project-dir=/home/dbt_workspace/my_stb_dbt_project 
--target-path=/home/dbt_workspace/my_stb_dbt_project/ue_execution",
            "command_exit_code": 0,
            "artifacts": [
                {
                    "type": "manifest",
                    "path": "/home/dbt_workspace/my_stb_dbt_project/ue_execution/manifest.json",
                    "content": null
                },
                {
                    "type": "run_results",
                    "path": "/home/.../dbt_workspace/my_stb_dbt_project/ue_execution/run_results.json",
                    "content": null
                }
            ]
        }
    }
}

{
    "exit_code": 1,
    "status": "FAIL",
    "status_description": "Execution Failed: Refer to STDERR or Extension Output for more details.",
    "invocation": {
        "extension": "ue-dbt-core-cli",
        "version": "1.0.0",
        "fields": {
            "action": "Execute Command",
            "credentials": "/home/agent/tmp/587a68dd-3220-41e8-94b2-a2d0caf69445",
            "command_line": "dbt build --profiles-dir=/home/dbt_workspace/.dbt 
--project-dir=/home/dbt_workspace/my_stb_dbt_project 
--target-path=/home/dbt_workspace/my_stb_dbt_project/ue_execution",
            "artifact_content_visibility": "Run Results on Extension Output",
        }
    },
    "result": {
        "execution_info": {
            "command_line": "dbt build --profiles-dir=/home/dbt_workspace/.dbt 
--project-dir=/home/dbt_workspace/my_stb_dbt_project 
--target-path=/home/dbt_workspace/my_stb_dbt_project/ue_execution",
            "command_exit_code": 2,
            "artifacts": []
        }
    }
}

12:14:02  Running with dbt=1.5.0
12:14:03  Found 2 models, 4 tests, 0 snapshots, 0 analyses, 307 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics, 0 groups
12:14:03
12:14:04  Concurrency: 1 threads (target='dev')
12:14:04
12:14:04  1 of 2 START sql table model dev_schema.my_first_dbt_model ..................... [RUN]
12:14:05  1 of 2 OK created sql table model dev_schema.my_first_dbt_model ................ [SELECT 2 in 0.81s]
12:14:05  2 of 2 START sql view model dev_schema.my_second_dbt_model ..................... [RUN]
12:14:06  2 of 2 OK created sql view model dev_schema.my_second_dbt_model ................ [CREATE VIEW in 0.81s]
12:14:06
12:14:06  Finished running 1 table model, 1 view model in 0 hours 0 minutes and 3.97 seconds (3.97s).
12:14:06
12:14:06  Completed successfully
12:14:06
12:14:07  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2

Input Fields

MY_CREDENTIAL_ENV_VARIABLE_A=<UC credential function call A>
MY_CREDENTIAL_ENV_VARIABLE_B=<UC credential function call B>
...

Output-Only fields

No output only fields

Extension Cancelation and Re-run

Cancellation from UC will just stop the execution of the Task. The Re-run of the task behaves the same way as on the initial run.

Exit Codes

STDOUT and STDERR

Depending on the log level the Universal Task is started by the controller, the DBT_LOG_LEVEL environment variable is set prior to the execution of the DBT command based on the mapping below.

The following applies:

• IF DBT_LOG_LEVEL is already set then it will not be overwritten.

• IF --log-level command line option is used, by default dbt uses that value from the command line

How To

Import Universal Template

To use the Universal Template, you first must perform the following steps.

1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.

2. To import the Universal Template into your Controller, follow these instructions.

3. When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Universal Task

For a new Universal Task, create a new task, and enter the required input fields.

Integration Modifications

Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.

• Python code modifications should not be done.

• Template Modifications

  • General Section

    • "Name", "Extension", "Variable Prefix", "Icon" should not be changed.

  • Universal Template Details Section

    • "Template Type", "Agent Type", "Send Extension Variables", "Always Cancel on Force Finish" should not be changed.

  • Result Processing Defaults Section

    • Success and Failure Exit codes should not be changed.

    • Success and Failure Output processing should not be changed.

  • Fields Restriction Section
    The setup of the template does not impose any restrictions, However with respect to "Exit Code Processing Fields" section.

    1. Success/Failure exit codes need to be respected.

    2. In principle, as STDERR and STDOUT outputs can change in follow-up releases of this integration, they should not be considered as a reliable source for determining success or failure of a task.

Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.

Document References

This document references the following documents:

Changelog

ue-dbt-core-cli-1.0.1 (2023-10-18)

Fixes

• Fixed:  Include integration icon on Universal Template (#34228).

ue-dbt-core-cli-1.0.0 (2023-08-16)

Initial Version