git and CI/CD workflow

Continual is an operational AI platform and is designed to be GitOps-friendly. Users who routinely work in Git should feel comfortable leveraging Continual's commands to fit into their continuous integration and continuous deployment development and production workflows. Before getting started here, we recommend reading our section on Environments, which provide essential "branching" functionality that enables many of these workflows.

Using environments for dev and prod

When users wish to iterate on a project in Continual, there are a few things that are required:

1. Create a separate environment, isolated from production.

2. Copy all resources from production into the new environment.

In git, this would be embodied by the command git checkout -b <my_branch>. In continual, you can similarly execute the following command:

continual checkout <my_env>

This command create a new environment and syncs that environment with production. Users can then iterate on the new environment without affecting continual. When using checkout, Continual creates a new schema in your feature store <project_id>_<env_id>. Which will contain any resources created in the new environment. This ensures that your schemas are always distinct from those used and production and never clash.

Note

If you wish to build an environment without syncing production over, you can do so with the normal continual environments create command.

Building an environment

Now that our environment is created, we'll want to be able to push our changes into the environment. To accomplish this, we'll just use the normal continual push command:

continual push <my_dir>

This pushes the specified files into Continual and any modified files will trigger a build. By deafult this uses the current project and environment, but you can manually specify those as well via --project <project_id> --env <env_id>. You'll be able to view any new resources in the Web UI or CLI as usual, in the specified environment.

Note, you can use --train-none and --train-all to force behavior when pushing content into Continual. This is particularly useful when working in a branch if you want to force update models, even if their definitions have not changed, or if you want to prevent re-trainings while doing an update.

Viewing changes

As resources are modified and pushed into Continual, each is tracked in a change object. You can think of these as akin to commits. Changes are separated by environment and you can always view changes via:

continual changes list --project <project_id> --env <env_id>

or, by visiting the Changes section of the Web UI.

Any failed change can be rebuilt, either in the Web UI or via continual changes rerun <change_id>.

Merging an environment

The last tool we need at our disposal is some way to merge an environment back into production. In Git, we have git merge, and Continual also provides the following command:

continual merge --from <env_id>

This command now merges the changes in your environment into the production environment. Any changed or new feature-sets or models will be built. By default, merge executes in the production environment. If you wish to merge to a non-prod environment, you can do so via --to <env_id>. Note that you cannot merge across projects.

Note

You can use --plan-only with merge to see what changes will be maid during a merge without actually executing the merge.

Via the Environments section in the Web UI, you may also merge an environment into another:

Using Continual with your CI/CD

With the above commands, it should be relatively straight forward to integrate Continual into your CI/CD pipelines. For those wishing to do so, we offer the following recommendations:

1. Use a separate Git repository for each Continual project. Name the repository to be the same as the project ID.

2. If an environment is not specified when using the Continual CLI, Continual will try to use your git branch as the environment name. This is a safety mechanism to prevent unintended modifications to produtction. For Git users, this should minimize the amount of options you need to enter for various commands. If you wish to explicitly use an environment different than your branch, you can either pass in --env or set the default environment via continual config set-environment.

3. You should configure your CI/CD pipeline to build off of pull requests and execute continual push <continual-dir> into the repo@branch environment in Continual. Remember that the repo corresponds to your project_id and the branch will correspond to the environment name. Subsequent commits into the PR will push into the same environment and will only build modified resources.

4. Configure your CI/CD pipeline so that when a PR is approved, continual merge -from repo@branch is executed. This will trigger Continual to push all modifications to the production environment.

Example workflows

Development workflow

git clone <my-dbt-project>
cd <my-dbt-project>
git checkout -b <my-new-branch>
continual checkout <my-new-branch>
<modify Continual files>
continual push --env <my-new-branch> <modified-files>
git add <modified dbt files>
git commit -a
git push

Production no CI/CD

git clone <my-dbt-project>
cd <my-dbt-project>
git checkout -b <my-new-branch>
continual checkout <my-new-branch>
<modify Continual files>
continual push --env <my-new-branch> <modified-files>
git add <modified dbt files>
git commit -a
git push
continual push #pushes into prod environment

Production with CI/CD

git clone <my-dbt-project>
cd <my-dbt-project>
git checkout -b <my-new-branch>
continual checkout <my-new-branch>
<modify Continual files>
git add <modified dbt files>
git commit -a
git push
<create pull request>
<CI/CD executes `continual push` on branch env>
<review + approve pr>
<CI/CD executes `continual merge` into production>