Skip to content

dbt integration

dbt quote

Design Partners Wanted

Continual's dbt and dbt Cloud integration is undergoing active development and evolution. To be part of our design partner program, learn more about our dbt roadmap, or give general feedback, please email support@continual.ai

Overview

Many modern data teams use dbt to model the data of their business. They may have one or more dbt models to represent entities in their business, and each of those models contains columns (or attributes, or features) of these entities.

Continual integrates with dbt by allowing dbt users to define entities, feature sets, and predictive models directly from their existing dbt models. This is done by using dbt's built-in meta config metadata mechanism.

This integration is intended to give dbt users a lightweight way to start leveraging their dbt assets in Continual without having to re-do data transformation work that is already done in dbt. This integration also provides a compatible workflow to the existing dbt workflow: users can execute continual run on a dbt project to have Continual process it and build out the required resources.

Configuration

Let's say you have a customers model, via the dbt Getting Started guide:

{{
  config(
    materialized = "table"
  )  
}}

...
final as (

    select
        customers.customer_id,
        customers.first_name,
        customers.last_name,
        customer_orders.first_order_date,
        customer_orders.most_recent_order_date,
        coalesce(customer_orders.number_of_orders, 0) as number_of_orders

    from customers

    left join customer_orders using (customer_id)

)

select * from final

This particular model has:

  • An entity id: customer_id
  • Two attributes related to the customer's name: first_name, last_name
  • Three attributes related to the customer's order history: first_order_date, most_recent_order_date, number_of_orders

We can instruct Continual about this entity and its features simply by adding some additional metadata to the dbt model config:

{{
  config(
    materialized = "table",
    meta = {
      "continual": {
        "type": "FeatureSet",
        "entity": "Customer",
        "index": "customer_id"
      }
    }
  )  
}}
...

As with all dbt model metadata, this can be added in the model file itself, or in separate schema or configuration files. For full configuration options, refer to the reference documentation.

Usage

Once Continual has been configured via dbt metadata, users can execute Continual directly on top of a dbt project by using the Continual CLI:

dbt run
continual run

Continual run functions much like dbt run, reading profile and target information directly from the dbt project:

  • --profiles-dir: The directory containing your profiles.yml file.

  • --project-dir: The dbt project directory, i.e. the directory containing your dbt_project.yml file.

  • --profile: Overrides the default profile found in dbt_project.yml.

  • --target: Overrides the default target found in profiles.yml.

Continual specific information can also be passed in to override project defaults:

  • --project: The continual project to use. Overrides the currently set project.

  • --continual-dir: The subdirectory in your dbt_project to save Continual yaml files. By default, this is set to whatever is the targets-path in the dbt_project.yml file. As a result, Continual yaml files will not be saved or versioned in git with the default dbt settings. You can modify this behavior by explicitly selecting a different directory, like continual

For a complete reference of command line options, refer to the documentation on continual run.

Workflow

Continual fits in to your existing dbt workflow:

  • You can create separate Continual projects for each of your dbt projects
  • You can define separate Continual environments to match your separate dbt profiles and targets as needed
  • continual run can be used in the same way as dbt run in your git/GitOps, CI, and CD workflows

Examples

For a full walkthrough of building a predictive model on top of dbt with Continual, check out the dbt guided example.

Back to top