Using Branch Deployments with GitHub Actions#

This guide is applicable to Dagster Cloud.

In this guide, we'll walk you through setting up Branch Deployments with GitHub Actions.

Using this approach to branch deployments may be a good fit if:

  • You use GitHub for version control
  • You want Dagster to fully automate Branch Deployments

If you don't use GitHub for version control or want full control over your setup, check out the Branch deployments with the dagster-cloud CLI guide.

Prerequisites#

To complete the steps in this guide, you'll need:

  • Editor or Admin permissions in Dagster Cloud
  • The ability to run a new agent in your infrastructure. This isn't required if you're using Serverless deployment.
  • The ability to configure GitHub Actions for your repository. This isn't required if you used the Dagster Cloud GitHub app to connect your repository as a code location.

Step 1: Generate a Dagster Cloud agent token#

In this step, you'll generate a token for the Dagster Cloud agent. The Dagster Cloud agent will use this to authenticate to the agent API.

  1. Sign in to your Dagster Cloud instance.
  2. Click the user menu (your icon) > Cloud settings.
  3. In the Cloud settings page, click the Configure tokens tab.
  4. Click the + Create agent token button.
  5. After the token has been created, click Reveal token.

Keep the token somewhere handy - you'll need it to complete the setup.

Step 2: Create and configure an agent#

While you can use your existing production agent, we recommend creating a dedicated branch deployment agent. This ensures that your production instance isn't negatively impacted by the workload associated with branch deployments.

AMAZON ECS AGENTS

  1. Deploy an ECS agent to serve your branch deployments. Follow the ECS agent setup guide, making sure to set the Enable Branch Deployments parameter if using the CloudFormation template. If you are running an existing agent, follow the upgrade guide to ensure your template is up-to-date. Then, turn on the Enable Branch Deployments parameter.

  2. Create a private Amazon Elastic Registry (ECR) repository. Refer to the AWS ECR documentation for instructions.

    After the repository has been created, navigate back to the list of ECR repositories.

    In the list, locate the repository and its URI:

    Highlighted repository URI in the AWS ECR console

  3. Create an IAM user. This user must:

    • Have push access to the ECR repository, and
    • Have programmatic access to AWS using an access key

    After the user is created, save the Access key ID and Secret access key values shown on the confirmation page:

    Highlighted repository URI in the AWS ECR console
DOCKER AGENTS

  1. Set up a new Docker agent. Refer to the Docker agent setup guide for instructions.

  2. After the agent is set up, modify the dagster.yaml file as follows:

    • Set the dagster_cloud_api.branch_deployments field to true
    • Remove any deployment field(s)

    For example:

    # dagster.yaml

instance_class:
  module: dagster_cloud.instance
  class: DagsterCloudAgentInstance

dagster_cloud_api:
  agent_token: <YOUR_AGENT_TOKEN>
  branch_deployments: true ## true enables branch deployments

user_code_launcher:
  module: dagster_cloud.workspace.docker
  class: DockerUserCodeLauncher
  config:
    networks:
      - dagster_cloud_agent
    server_ttl:
      enabled: true
      ttl_seconds: 7200 #2 hours
KUBERNETES AGENTS

  1. Set up a new Kubernetes agent. Refer to the Kubernetes agent setup guide for instructions.

  2. After the agent is set up, modify your Helm values file to include the following:

    dagsterCloud:
  branchDeployments: true
---
workspace:
  serverTTL:
    enabled: true
    ttlSeconds: 7200

Step 3: Add GitHub workflow files to your repository#

If you used the GitHub app to add the repository as a code location, this step isn't required. Skip to the next step.

In this step, you'll add the required GitHub workflow files to your GitHub repository. The files can be found in our Hybrid quickstart repository or Serverless quickstart repository, depending on the agent you are using.

Copy the following files to your repository (the linked files are shown in the Hybrid repo, there are equivalents in the Serverless repo).

In the next step, you'll modify these files to work with your Dagster Cloud setup.

Step 4: Configure the GitHub repository#

In this section, you'll:

  1. Add the agent registry to dagster_cloud.yaml
  2. Configure GitHub Action secrets
  3. Update GitHub Workflow files
  4. Verify Action runs

Step 4.1: Add the agent registry to dagster_cloud.yaml#

If you're using Serverless deployment, this step isn't required. Skip to the next step.

In the dagster_cloud.yaml file, replace build.registry with the registry used by the agent you created in Step 2.

For example:

# dagster_cloud.yaml

locations:
  - location_name: example_location
    code_source:
      python_file: repo.py
    build:
      directory: ./example_location
      registry: 764506304434.dkr.ecr.us-east-1.amazonaws.com/branch-deployment-agent

Step 4.2: Configure GitHub Action secrets#

  1. In the repository, click the Settings tab.
  2. In the Security section of the sidebar, click Secrets > Actions.
  3. Click New repository secret.
  4. In the Name field, enter the name of the secret. For example, DAGSTER_CLOUD_API_TOKEN
  5. In the Value field, paste the value of the secret.
  6. Click Add secret.

Repeat steps 3-6 for each of the secrets required for your registry type:

AMAZON ECR
  • DAGSTER_CLOUD_API_TOKEN - The Dagster Cloud agent token you created in Step 1
  • ORGANIZATION_ID - Your Dagster Cloud organization ID
  • AWS_ACCESS_KEY - The Access key ID of the AWS IAM user you created in Step 2
  • AWS_SECRET_ACCESS_KEY - The Secret access key of the AWS IAM user you created in Step 2
  • AWS_REGION - The AWS region where your ECR registry is located

The Actions secrets page should look like the following:

Configured GitHub Actions secrets for an AWS ECR registry
DOCKERHUB
  • DAGSTER_CLOUD_API_TOKEN - The Dagster Cloud agent token you created in Step 1
  • ORGANIZATION_ID - Your Dagster Cloud organization ID
  • DOCKERHUB_USERNAME - Your DockerHub username
  • DOCKERHUB_TOKEN -

The Actions secrets page should look like the following:

Required GitHub Actions secrets for a DockerHub registry

Step 4.3: Update GitHub Workflow files#

If you're using Serverless deployment, this step isn't required. Skip to the next step.

In this step, you'll update the GitHub Workflow files in the repository to set up Docker registry access.

In the .github/workflows/deploy.yml and .github/workflows/branch_deployments.yml files, uncomment the step associated with your registry.

For example, for an Amazon ECR registry, we'd uncomment the following portion of the workflow files:

steps:
  - name: Configure AWS credentials
    uses: aws-actions/configure-aws-credentials@v1
    with:
      aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
      aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      aws-region: ${{ secrets.AWS_REGION }}

Save and commit the files to the repository.

Step 4.4: Verify Action runs#

The last step is to verify that the Action runs successfully.

  1. In the repository, click the Actions tab.

  2. In the list of workflows, locate the latest branch deployment run. For example:

    A successful run for a Branch Deployment Action

If the run finished successfully, that's it!

Step 5: Access the branch deployment#

Now that Branch Deployments are configured, you can check out the preview in Dagster Cloud. There are two ways to do this:

From a pull request#

Every pull request in the repository contains a View in Cloud link:

View in Cloud preview link highlighted in a GitHub pull request

Clicking the link will open a branch deployment - or a preview of the changes - in Dagster Cloud.

Directly in Dagster Cloud#

You can also access branch deployments directly in Dagster Cloud from the deployment switcher:

Highlighted branch deployment in the Dagster Cloud deployment switcher