feat(examples/templates/gcp-devcontainer): add envbuilder provider (#14405)

This PR modifies the gcp-devcontainer example template to include
support for devcontainer caching using the envbuilder provider.

Co-authored-by: Mathias Fredriksson <mafredri@gmail.com>
Co-authored-by: Muhammad Atif Ali <atif@coder.com>
This commit is contained in:
Cian Johnston
2024-08-23 17:36:24 +01:00
committed by GitHub
parent a4d785dec5
commit 53e5746636
4 changed files with 239 additions and 91 deletions

View File

@@ -92,7 +92,7 @@
"gcp",
"devcontainer"
],
"markdown": "\n# Remote Development in a Devcontainer on Google Compute Engine\n\n![Architecture Diagram](./architecture.svg)\n\n## Prerequisites\n\n### Authentication\n\nThis template assumes that coderd is run in an environment that is authenticated\nwith Google Cloud. For example, run `gcloud auth application-default login` to\nimport credentials on the system and user running coderd. For other ways to\nauthenticate [consult the Terraform\ndocs](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/getting_started#adding-credentials).\n\nCoder requires a Google Cloud Service Account to provision workspaces. To create\na service account:\n\n1. Navigate to the [CGP\n console](https://console.cloud.google.com/projectselector/iam-admin/serviceaccounts/create),\n and select your Cloud project (if you have more than one project associated\n with your account)\n\n1. Provide a service account name (this name is used to generate the service\n account ID)\n\n1. Click **Create and continue**, and choose the following IAM roles to grant to\n the service account:\n\n - Compute Admin\n - Service Account User\n\n Click **Continue**.\n\n1. Click on the created key, and navigate to the **Keys** tab.\n\n1. Click **Add key** \u003e **Create new key**.\n\n1. Generate a **JSON private key**, which will be what you provide to Coder\n during the setup process.\n\n## Architecture\n\nThis template provisions the following resources:\n\n- GCP VM (persistent)\n- GCP Disk (persistent, mounted to root)\n\nCoder persists the root volume. The full filesystem is preserved when the workspace restarts.\n\n\u003e **Note**\n\u003e This template is designed to be a starting point! Edit the Terraform to extend the template to support your use case.\n\n## code-server\n\n`code-server` is installed via the [`code-server`](https://registry.coder.com/modules/code-server) registry module. Please check [Coder Registry](https://registry.coder.com) for a list of all modules and templates.\n"
"markdown": "\n# Remote Development in a Devcontainer on Google Compute Engine\n\n![Architecture Diagram](./architecture.svg)\n\n## Prerequisites\n\n### Authentication\n\nThis template assumes that coderd is run in an environment that is authenticated\nwith Google Cloud. For example, run `gcloud auth application-default login` to\nimport credentials on the system and user running coderd. For other ways to\nauthenticate [consult the Terraform\ndocs](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/getting_started#adding-credentials).\n\nCoder requires a Google Cloud Service Account to provision workspaces. To create\na service account:\n\n1. Navigate to the [CGP\n console](https://console.cloud.google.com/projectselector/iam-admin/serviceaccounts/create),\n and select your Cloud project (if you have more than one project associated\n with your account)\n\n1. Provide a service account name (this name is used to generate the service\n account ID)\n\n1. Click **Create and continue**, and choose the following IAM roles to grant to\n the service account:\n\n - Compute Admin\n - Service Account User\n\n Click **Continue**.\n\n1. Click on the created key, and navigate to the **Keys** tab.\n\n1. Click **Add key** \u003e **Create new key**.\n\n1. Generate a **JSON private key**, which will be what you provide to Coder\n during the setup process.\n\n## Architecture\n\nThis template provisions the following resources:\n\n- Envbuilder cached image (conditional, persistent) using [`terraform-provider-envbuilder`](https://github.com/coder/terraform-provider-envbuilder)\n- GCP VM (persistent) with a running Docker daemon\n- GCP Disk (persistent, mounted to root)\n- [Envbuilder container](https://github.com/coder/envbuilder) inside the GCP VM\n\nCoder persists the root volume. The full filesystem is preserved when the workspace restarts.\nWhen the GCP VM starts, a startup script runs that ensures a running Docker daemon, and starts\nan Envbuilder container using this Docker daemon. The Docker socket is also mounted inside the container to allow running Docker containers inside the workspace.\n\n\u003e **Note**\n\u003e This template is designed to be a starting point! Edit the Terraform to extend the template to support your use case.\n\n## Caching\n\nTo speed up your builds, you can use a container registry as a cache.\nWhen creating the template, set the parameter `cache_repo` to a valid Docker repository in the form `host.tld/path/to/repo`.\n\nSee the [Envbuilder Terraform Provider Examples](https://github.com/coder/terraform-provider-envbuilder/blob/main/examples/resources/envbuilder_cached_image/envbuilder_cached_image_resource.tf/) for a more complete example of how the provider works.\n\n\u003e [!NOTE] We recommend using a registry cache with authentication enabled.\n\u003e To allow Envbuilder to authenticate with the registry cache, specify the variable `cache_repo_docker_config_path`\n\u003e with the path to a Docker config `.json` on disk containing valid credentials for the registry.\n\n## code-server\n\n`code-server` is installed via the [`code-server`](https://registry.coder.com/modules/code-server) registry module. Please check [Coder Registry](https://registry.coder.com) for a list of all modules and templates.\n"
},
{
"id": "gcp-linux",

View File

@@ -34,12 +34,11 @@ Coder supports Devcontainers via [envbuilder](https://github.com/coder/envbuilde
This template provisions the following resources:
- Envbuilder cached image (conditional, persistent)
- Docker image (persistent)
- Envbuilder cached image (conditional, persistent) using [`terraform-provider-envbuilder`](https://github.com/coder/terraform-provider-envbuilder)
- Docker image (persistent) using [`envbuilder`](https://github.com/coder/envbuilder)
- Docker container (ephemeral)
- Docker volume (persistent on `/workspaces`)
with [`envbuilder`](https://github.com/coder/envbuilder) and [`terraform-provider-envbuilder`](https://github.com/coder/terraform-provider-envbuilder).
The Git repository is cloned inside the `/workspaces` volume if not present.
Any local changes to the Devcontainer files inside the volume will be applied when you restart the workspace.
Keep in mind that any tools or files outside of `/workspaces` or not added as part of the Devcontainer specification are not persisted.

View File

@@ -51,14 +51,29 @@ a service account:
This template provisions the following resources:
- GCP VM (persistent)
- Envbuilder cached image (conditional, persistent) using [`terraform-provider-envbuilder`](https://github.com/coder/terraform-provider-envbuilder)
- GCP VM (persistent) with a running Docker daemon
- GCP Disk (persistent, mounted to root)
- [Envbuilder container](https://github.com/coder/envbuilder) inside the GCP VM
Coder persists the root volume. The full filesystem is preserved when the workspace restarts.
When the GCP VM starts, a startup script runs that ensures a running Docker daemon, and starts
an Envbuilder container using this Docker daemon. The Docker socket is also mounted inside the container to allow running Docker containers inside the workspace.
> **Note**
> This template is designed to be a starting point! Edit the Terraform to extend the template to support your use case.
## Caching
To speed up your builds, you can use a container registry as a cache.
When creating the template, set the parameter `cache_repo` to a valid Docker repository in the form `host.tld/path/to/repo`.
See the [Envbuilder Terraform Provider Examples](https://github.com/coder/terraform-provider-envbuilder/blob/main/examples/resources/envbuilder_cached_image/envbuilder_cached_image_resource.tf/) for a more complete example of how the provider works.
> [!NOTE] We recommend using a registry cache with authentication enabled.
> To allow Envbuilder to authenticate with the registry cache, specify the variable `cache_repo_docker_config_path`
> with the path to a Docker config `.json` on disk containing valid credentials for the registry.
## code-server
`code-server` is installed via the [`code-server`](https://registry.coder.com/modules/code-server) registry module. Please check [Coder Registry](https://registry.coder.com) for a list of all modules and templates.

View File

@@ -6,16 +6,41 @@ terraform {
google = {
source = "hashicorp/google"
}
envbuilder = {
source = "coder/envbuilder"
}
}
}
provider "coder" {
provider "coder" {}
provider "google" {
zone = data.coder_parameter.zone.value
project = var.project_id
}
data "google_compute_default_service_account" "default" {}
data "coder_workspace" "me" {}
data "coder_workspace_owner" "me" {}
variable "project_id" {
description = "Which Google Compute Project should your workspace live in?"
}
variable "cache_repo" {
default = ""
description = "(Optional) Use a container registry as a cache to speed up builds. Example: host.tld/path/to/repo."
type = string
}
variable "cache_repo_docker_config_path" {
default = ""
description = "(Optional) Path to a docker config.json containing credentials to the provided cache repo, if required. This will depend on your Coder setup. Example: `/home/coder/.docker/config.json`."
sensitive = true
type = string
}
data "coder_parameter" "zone" {
name = "zone"
display_name = "Zone"
@@ -24,6 +49,7 @@ data "coder_parameter" "zone" {
icon = "/emojis/1f30e.png"
default = "us-central1-a"
mutable = false
order = 1
option {
name = "North America (Northeast)"
value = "northamerica-northeast1-a"
@@ -51,25 +77,48 @@ data "coder_parameter" "zone" {
}
}
provider "google" {
zone = data.coder_parameter.zone.value
project = var.project_id
}
data "google_compute_default_service_account" "default" {
}
data "coder_workspace" "me" {
}
data "coder_workspace_owner" "me" {}
resource "google_compute_disk" "root" {
name = "coder-${data.coder_workspace.me.id}-root"
type = "pd-ssd"
image = "debian-cloud/debian-12"
lifecycle {
ignore_changes = [name, image]
data "coder_parameter" "instance_type" {
name = "instance_type"
display_name = "Instance Type"
description = "Select an instance type for your workspace."
type = "string"
mutable = false
order = 2
default = "e2-micro"
option {
name = "e2-micro (2C, 1G)"
value = "e2-micro"
}
option {
name = "e2-small (2C, 2G)"
value = "e2-small"
}
option {
name = "e2-medium (2C, 2G)"
value = "e2-medium"
}
}
data "coder_parameter" "fallback_image" {
default = "codercom/enterprise-base:ubuntu"
description = "This image runs if the devcontainer fails to build."
display_name = "Fallback Image"
mutable = true
name = "fallback_image"
order = 3
}
data "coder_parameter" "devcontainer_builder" {
description = <<-EOF
Image that will build the devcontainer.
Find the latest version of Envbuilder here: https://ghcr.io/coder/envbuilder
Be aware that using the `:latest` tag may expose you to breaking changes.
EOF
display_name = "Devcontainer Builder"
mutable = true
name = "devcontainer_builder"
default = "ghcr.io/coder/envbuilder:latest"
order = 4
}
data "coder_parameter" "repo_url" {
@@ -80,6 +129,154 @@ data "coder_parameter" "repo_url" {
mutable = true
}
data "local_sensitive_file" "cache_repo_dockerconfigjson" {
count = var.cache_repo_docker_config_path == "" ? 0 : 1
filename = var.cache_repo_docker_config_path
}
# Be careful when modifying the below locals!
locals {
# Ensure Coder username is a valid Linux username
linux_user = lower(substr(data.coder_workspace_owner.me.name, 0, 32))
# Name the container after the workspace and owner.
container_name = "coder-${data.coder_workspace_owner.me.name}-${lower(data.coder_workspace.me.name)}"
# The devcontainer builder image is the image that will build the devcontainer.
devcontainer_builder_image = data.coder_parameter.devcontainer_builder.value
# We may need to authenticate with a registry. If so, the user will provide a path to a docker config.json.
docker_config_json_base64 = try(data.local_sensitive_file.cache_repo_dockerconfigjson[0].content_base64, "")
# The envbuilder provider requires a key-value map of environment variables. Build this here.
envbuilder_env = {
# ENVBUILDER_GIT_URL and ENVBUILDER_CACHE_REPO will be overridden by the provider
# if the cache repo is enabled.
"ENVBUILDER_GIT_URL" : data.coder_parameter.repo_url.value,
# The agent token is required for the agent to connect to the Coder platform.
"CODER_AGENT_TOKEN" : try(coder_agent.dev.0.token, ""),
# The agent URL is required for the agent to connect to the Coder platform.
"CODER_AGENT_URL" : data.coder_workspace.me.access_url,
# The agent init script is required for the agent to start up. We base64 encode it here
# to avoid quoting issues.
"ENVBUILDER_INIT_SCRIPT" : "echo ${base64encode(try(coder_agent.dev[0].init_script, ""))} | base64 -d | sh",
"ENVBUILDER_DOCKER_CONFIG_BASE64" : try(data.local_sensitive_file.cache_repo_dockerconfigjson[0].content_base64, ""),
# The fallback image is the image that will run if the devcontainer fails to build.
"ENVBUILDER_FALLBACK_IMAGE" : data.coder_parameter.fallback_image.value,
# The following are used to push the image to the cache repo, if defined.
"ENVBUILDER_CACHE_REPO" : var.cache_repo,
"ENVBUILDER_PUSH_IMAGE" : var.cache_repo == "" ? "" : "true",
# You can add other required environment variables here.
# See: https://github.com/coder/envbuilder/?tab=readme-ov-file#environment-variables
}
# If we have a cached image, use the cached image's environment variables. Otherwise, just use
# the environment variables we've defined above.
docker_env_input = try(envbuilder_cached_image.cached.0.env_map, local.envbuilder_env)
# Convert the above to the list of arguments for the Docker run command.
# The startup script will write this to a file, which the Docker run command will reference.
docker_env_list_base64 = base64encode(join("\n", [for k, v in local.docker_env_input : "${k}=${v}"]))
# The GCP VM needs a startup script to set up the environment and start the container. Defining this here.
# NOTE: make sure to test changes by uncommenting the local_file resource at the bottom of this file
# and running `terraform apply` to see the generated script. You should also run shellcheck on the script
# to ensure it is valid.
startup_script = <<-META
#!/usr/bin/env sh
set -eux
# If user does not exist, create it and set up passwordless sudo
if ! id -u "${local.linux_user}" >/dev/null 2>&1; then
useradd -m -s /bin/bash "${local.linux_user}"
echo "${local.linux_user} ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/coder-user
fi
# Check for Docker, install if not present
if ! command -v docker >/dev/null 2>&1; then
echo "Docker not found, installing..."
curl -fsSL https://get.docker.com -o get-docker.sh && sudo sh get-docker.sh >/dev/null 2>&1
sudo usermod -aG docker ${local.linux_user}
newgrp docker
else
echo "Docker is already installed."
fi
# Write the Docker config JSON to disk if it is provided.
if [ -n "${local.docker_config_json_base64}" ]; then
mkdir -p "/home/${local.linux_user}/.docker"
printf "%s" "${local.docker_config_json_base64}" | base64 -d | tee "/home/${local.linux_user}/.docker/config.json"
chown -R ${local.linux_user}:${local.linux_user} "/home/${local.linux_user}/.docker"
fi
# Write the container env to disk.
printf "%s" "${local.docker_env_list_base64}" | base64 -d | tee "/home/${local.linux_user}/env.txt"
# Start envbuilder.
docker run \
--rm \
--net=host \
-h ${lower(data.coder_workspace.me.name)} \
-v /home/${local.linux_user}/envbuilder:/workspaces \
-v /var/run/docker.sock:/var/run/docker.sock \
--env-file env.txt \
${data.coder_parameter.devcontainer_builder.value}
META
}
# Create a persistent disk to store the workspace data.
resource "google_compute_disk" "root" {
name = "coder-${data.coder_workspace.me.id}-root"
type = "pd-ssd"
image = "debian-cloud/debian-12"
lifecycle {
ignore_changes = all
}
}
# Check for the presence of a prebuilt image in the cache repo
# that we can use instead.
resource "envbuilder_cached_image" "cached" {
count = var.cache_repo == "" ? 0 : data.coder_workspace.me.start_count
builder_image = local.devcontainer_builder_image
git_url = data.coder_parameter.repo_url.value
cache_repo = var.cache_repo
extra_env = local.envbuilder_env
}
# This is useful for debugging the startup script. Left here for reference.
# resource local_file "startup_script" {
# content = local.startup_script
# filename = "${path.module}/startup_script.sh"
# }
# Create a VM where the workspace will run.
resource "google_compute_instance" "vm" {
name = "coder-${lower(data.coder_workspace_owner.me.name)}-${lower(data.coder_workspace.me.name)}-root"
machine_type = data.coder_parameter.instance_type.value
# data.coder_workspace_owner.me.name == "default" is a workaround to suppress error in the terraform plan phase while creating a new workspace.
desired_status = (data.coder_workspace_owner.me.name == "default" || data.coder_workspace.me.start_count == 1) ? "RUNNING" : "TERMINATED"
network_interface {
network = "default"
access_config {
// Ephemeral public IP
}
}
boot_disk {
auto_delete = false
source = google_compute_disk.root.name
}
service_account {
email = data.google_compute_default_service_account.default.email
scopes = ["cloud-platform"]
}
metadata = {
# The startup script runs as root with no $HOME environment set up, so instead of directly
# running the agent init script, create a user (with a homedir, default shell and sudo
# permissions) and execute the init script as that user.
startup-script = local.startup_script
}
}
# Create a Coder agent to manage the workspace.
resource "coder_agent" "dev" {
count = data.coder_workspace.me.start_count
arch = "amd64"
@@ -111,78 +308,15 @@ resource "coder_agent" "dev" {
}
}
# Install code-server via Terraform module.
module "code-server" {
count = data.coder_workspace.me.start_count
source = "https://registry.coder.com/modules/code-server"
source = "registry.coder.com/modules/code-server/coder"
version = "1.0.17"
agent_id = coder_agent.dev[0].id
}
resource "google_compute_instance" "vm" {
name = "coder-${lower(data.coder_workspace_owner.me.name)}-${lower(data.coder_workspace.me.name)}-root"
machine_type = "e2-medium"
# data.coder_workspace_owner.me.name == "default" is a workaround to suppress error in the terraform plan phase while creating a new workspace.
desired_status = (data.coder_workspace_owner.me.name == "default" || data.coder_workspace.me.start_count == 1) ? "RUNNING" : "TERMINATED"
network_interface {
network = "default"
access_config {
// Ephemeral public IP
}
}
boot_disk {
auto_delete = false
source = google_compute_disk.root.name
}
service_account {
email = data.google_compute_default_service_account.default.email
scopes = ["cloud-platform"]
}
metadata = {
# The startup script runs as root with no $HOME environment set up, so instead of directly
# running the agent init script, create a user (with a homedir, default shell and sudo
# permissions) and execute the init script as that user.
startup-script = <<-META
#!/usr/bin/env sh
set -eux
# If user does not exist, create it and set up passwordless sudo
if ! id -u "${local.linux_user}" >/dev/null 2>&1; then
useradd -m -s /bin/bash "${local.linux_user}"
echo "${local.linux_user} ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/coder-user
fi
# Check for Docker, install if not present
if ! command -v docker &> /dev/null
then
echo "Docker not found, installing..."
curl -fsSL https://get.docker.com -o get-docker.sh && sudo sh get-docker.sh 2>&1 >/dev/null
sudo usermod -aG docker ${local.linux_user}
newgrp docker
else
echo "Docker is already installed."
fi
# Start envbuilder
docker run --rm \
-h ${lower(data.coder_workspace.me.name)} \
-v /home/${local.linux_user}/envbuilder:/workspaces \
-e CODER_AGENT_TOKEN="${try(coder_agent.dev[0].token, "")}" \
-e CODER_AGENT_URL="${data.coder_workspace.me.access_url}" \
-e GIT_URL="${data.coder_parameter.repo_url.value}" \
-e INIT_SCRIPT="echo ${base64encode(try(coder_agent.dev[0].init_script, ""))} | base64 -d | sh" \
-e FALLBACK_IMAGE="codercom/enterprise-base:ubuntu" \
ghcr.io/coder/envbuilder
META
}
}
locals {
# Ensure Coder username is a valid Linux username
linux_user = lower(substr(data.coder_workspace_owner.me.name, 0, 32))
}
# Create metadata for the workspace and home disk.
resource "coder_metadata" "workspace_info" {
count = data.coder_workspace.me.start_count
resource_id = google_compute_instance.vm.id