Overview

There are three non-local environments:

• Preview: This is an ephemeral PR-specific UAT environment where you can preview the change contained in a pull request actually deployed.

It is created by the Continuous Integration pipeline ( ci.yml ) when a pull request is opened and destroyed when it is merged or otherwise closed.

Therefore there are potentially many Preview environments running at any given time: one for each open pull request.

This is a replica of the Production environment in terms of infrastructure components, but the data in database is test data and not from production database.

• Staging: This is a particularly short-lived ephemeral environment.

Continuous Deployment Pipeline(cd.yml) which deploys to Production on push to main, creates Staging first: it deploys all deployable components to it (the database, API and frontend web app). It then runs tests and when all tests pass, it tears down Staging and the proceeds to deploy to Production.

Staging environment is a replica of the Production environment in the sense that it uses exactly the same infrastructure components and even the same data as the Production database via a database branch of the Neon DB production database taken a the time the Staging environment is created.

• Production

Terraform Modules and Workspaces

TODO: Summarise the module we have, then describe each environment’s workspaces (including their purpose, e.g. in Production environment, prod only creates the infrastructure but prod-api is used to deploy the API app flowmazonbackend as well) and the relationships between them (e.g. in Production, prod workspace should run before prod-api, then container for flowmazonbackend is built and pushed to the container registry i nthe GtHub Actions workflow for the environment, then prod-api is run in GitHub Actions workflow to deploy the app and the first time this happens the ACA app and environment would be created from scratch).

Also note that:

Most of the the environment variables required by flowmazonfrontend and flowmazonbackend are provided as arguments to the environment-specific Terraform workspaces. Their values are stored in HCP workspaces.

Others - probably only NEXT_PUBLIC_ variables requried by flowmazonfrontend - are required during build in GitHub Actions workflows. Their values are stores in secrets and/or environment variable stores in GitHub Environments.

GitHub Actions Workflows

• For each environment, in addition to the infrastructure pieces created by Terraform, there is also a GitHub Environment which is created by hand.

It stores secrets, environment variables and environment protection rules for that environment that are used by the deployment pipeline that deploys to that environment.

Production and Staging share a single GitHub Environment object as the same deployment pipeline (cd.yml ) deploys to both.

Observability Configuration for Grafana Cloud

All three environments - Production, Staging and the group of PR-specific Preview environments - produce telemetry. See Observability article for which Grafana Cloud account to use in each environment.

An environment specific Frontend Observability instance for each environment for use with the flowmazonfrontend Next.js app is automatically created by Terraform. So we don’t need to create it for any non-local environment.

However, we still need to configure rest of observability for each of Production and Staging environments and for all the Preview environments as a group . This gives us observability configuration values for flowmazonbackend and flowmazonfrontend. These would be provided via arguments for Terraform workspaces for an environment and/or to GitHub Actions workflow via the relevant GitHub Environment’s secrets and environment variables.

FOR EACH NON-LOCAL ENVIRONMENT (or, in case of Preview environments, for the whole group of Preview environments), configure the environment as follows:

• Create a Grafana Cloud in the same region in which flowmaonfrontend and flowmazonbackend apps for an environment would be located, as described in Grafana Cloud Setup.

This would give us an OTLP endpoint that is close to the server-side code of the apps. Consider this table of Grafana Cloud regions, the list of Azure Regions and list of Vercel regions when deciding the region in which to create the Grafana Account (this is the region in which the single LGTM Stack associated with a Grafana Account would be created).

You cannot use a Grafana Cloud endpoint for a different region from the one in which your Stack was created (no telemetry would be ingested). Therefore location of the account/stack is an important decision that you need to make upfront.

• Create a Cloud Access Policy for the environment (for ephemeral environments, a single policy to be created for a whole group of ephemeral environments such as Preview environment) by going to your stack, then Administration | User and access, then create a Cloud access policy:

  

  Name it after the environment or group of environments and select Write permission for Metrics, Logs, Traces and Profiles:

  

  Next, within this policy, click Add token to create a new access token:

  

  Copy the token that will be shown to clipboard. This would be provided the value of the OTEL_EXPORTER_OTLP_HEADERS environment variable to flowmazonfrontend (for its server side) and flowmazonbackend components. It should be set in the environment as follows:

  • As value of Terraform variable named env_OTEL_EXPORTER_OTLP_HEADERS in long-lived environments
  • As value of a secret named ENV_OTEL_EXPORTER_OTLP_HEADERS in GitHub Environment in the repo in which a GitHub Actions workflow would create ephemeral instances of the corresponding group of environments.