Files
coder/docs/templates/docker-in-workspaces.md
Muhammad Atif Ali a49e6b88f9 docs: reorganize template docs (#10297)
* docs: rework our "templates" section

* wikistuff

* fix formatting

* add diagram

* reorganize some things

* docs: improve workspaces and templates doc (#9139)

* Reorg, updated/new screenshots, consistent terminology

* First pass

* Another pass

* Added integration section

* New outline for template pages, small updates

* Revised outline for templates, added tutorial

* First pass at tutorial

* Some feedback from Ben.

* Update docs/workspaces.md

Co-authored-by: Muhammad Atif Ali <matifali@live.com>

* Update docs/workspaces.md

Co-authored-by: Muhammad Atif Ali <matifali@live.com>

* Update docs/workspaces.md

Co-authored-by: Muhammad Atif Ali <matifali@live.com>

* Fixed typos

* Expanded tutorial

I have read the CLA Document and I hereby sign the CLA

* New screenshots, improved tutorial, revised anatomy

* Improved tutorial. Anatomy is now a guided tour.

* First pass at guided tour

* Updated authentication info

* Reorganized the guided tour

* Edited more template pages

* Update docs/templates/tour.md

Co-authored-by: Muhammad Atif Ali <matifali@live.com>

* Update docs/templates/tour.md

Co-authored-by: Muhammad Atif Ali <matifali@live.com>

* Update docs/templates/tour.md

Co-authored-by: Muhammad Atif Ali <matifali@live.com>

* Update docs/templates/tutorial.md

Co-authored-by: Muhammad Atif Ali <matifali@live.com>

* Update docs/templates/tour.md

Co-authored-by: Muhammad Atif Ali <matifali@live.com>

* Update docs/templates/tour.md

Co-authored-by: Muhammad Atif Ali <matifali@live.com>

* Update docs/templates/tour.md

Co-authored-by: Muhammad Atif Ali <matifali@live.com>

* Update docs/templates/tour.md

Co-authored-by: Muhammad Atif Ali <matifali@live.com>

* Update docs/templates/tour.md

Co-authored-by: Muhammad Atif Ali <matifali@live.com>

* Revised devcontainers and docker-in-workspaces

* Edited and added screenshots

* Prepared first draft, except docs/templates/open-in-coder.md

* Fix typo

* remove legacy parameters and migration guide

* Use coder templates create

* Added screenshot for workspace template variables

* Made it prettier

* Fixed minor typos and markdown problems

* edits to repairing workspaces

* fix broken links in product

* Added troubleshooting, minor corrections.

* fix terminal links

* fmt

---------

Co-authored-by: Muhammad Atif Ali <matifali@live.com>
Co-authored-by: Ben Potter <me@bpmct.net>
Co-authored-by: Atif Ali <atif@coder.com>

* make fmt

* fix merge conflict

* make fmt

* make gen

* update

* lint

* Discard changes to coderd/database/queries.sql.go

* Discard changes to cli/templates.go

* Discard changes to cli/templateversionarchive.go

* Discard changes to cli/templateversions.go

* Update docker-in-workspaces.md

* replace ```sh with ```shell

* open-in-coder

* fmt

* mention coder_metadata in icons.md

* resource_metadata

* use shell

* modules.md

* mention coder registry module

* workspace.md

* resource_metadata

* remove duplication

* address comments

* cleanup

* fmt

* fix broken links

* fix numbering

* mention module registry

* add example

* demote heading

* remove top level entry from manifest

* fmt

---------

Co-authored-by: Ben <me@bpmct.net>
Co-authored-by: Marc Paquette <22124737+marcpaq@users.noreply.github.com>
2023-10-17 14:47:12 +00:00

16 KiB

Docker in Workspaces

There are a few ways to run Docker within container-based Coder workspaces.

Method Description Limitations
Sysbox container runtime Install the sysbox runtime on your Kubernetes nodes for secure docker-in-docker and systemd-in-docker. Works with GKE, EKS, AKS. Requires compatible nodes. Limitations
Envbox A container image with all the packages necessary to run an inner sysbox container. Removes the need to setup sysbox-runc on your nodes. Works with GKE, EKS, AKS. Requires running the outer container as privileged (the inner container that acts as the workspace is locked down). Requires compatible nodes.
Rootless Podman Run podman inside Coder workspaces. Does not require a custom runtime or privileged containers. Works with GKE, EKS, AKS, RKE, OpenShift Requires smarter-device-manager for FUSE mounts. See all
Privileged docker sidecar Run docker as a privileged sidecar container. Requires a privileged container. Workspaces can break out to root on the host machine.

Sysbox container runtime

The Sysbox container runtime allows unprivileged users to run system-level applications, such as Docker, securely from the workspace containers. Sysbox requires a compatible Linux distribution to implement these security features. Sysbox can also be used to run systemd inside Coder workspaces. See Systemd in Docker.

Use Sysbox in Docker-based templates

After installing Sysbox on the Coder host, modify your template to use the sysbox-runc runtime:

resource "docker_container" "workspace" {
  # ...
  name    = "coder-${data.coder_workspace.me.owner}-${lower(data.coder_workspace.me.name)}"
  image   = "codercom/enterprise-base:ubuntu"
  env     = ["CODER_AGENT_TOKEN=${coder_agent.main.token}"]
  command = ["sh", "-c", coder_agent.main.init_script]
  # Use the Sysbox container runtime (required)
  runtime = "sysbox-runc"
}

resource "coder_agent" "main" {
  arch           = data.coder_provisioner.me.arch
  os             = "linux"
  startup_script = <<EOF
    #!/bin/sh

    # Start Docker
    sudo dockerd &

    # ...
    EOF
}

Use Sysbox in Kubernetes-based templates

After installing Sysbox on Kubernetes, modify your template to use the sysbox-runc RuntimeClass. This requires the Kubernetes Terraform provider version 2.16.0 or greater.

terraform {
  required_providers {
    coder = {
      source  = "coder/coder"
    }
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "2.16.0"
    }
  }
}

variable "workspaces_namespace" {
  default = "coder-namespace"
}

data "coder_workspace" "me" {}

resource "coder_agent" "main" {
  os   = "linux"
  arch = "amd64"
  dir  = "/home/coder"
  startup_script = <<EOF
    #!/bin/sh

    # Start Docker
    sudo dockerd &

    # ...
  EOF
}

resource "kubernetes_pod" "dev" {
  count = data.coder_workspace.me.start_count
  metadata {
    name      = "coder-${data.coder_workspace.me.owner}-${data.coder_workspace.me.name}"
    namespace = var.workspaces_namespace
    annotations = {
      "io.kubernetes.cri-o.userns-mode" = "auto:size=65536"
    }
  }

  spec {
  runtime_class_name = "sysbox-runc"
  # Use the Sysbox container runtime (required)
    security_context {
      run_as_user = 1000
      fs_group    = 1000
    }
    container {
      name = "dev"
      env {
        name  = "CODER_AGENT_TOKEN"
        value = coder_agent.main.token
      }
      image = "codercom/enterprise-base:ubuntu"
      command = ["sh", "-c", coder_agent.main.init_script]
    }
  }
}

Envbox

Envbox is an image developed and maintained by Coder that bundles the sysbox runtime. It works by starting an outer container that manages the various sysbox daemons and spawns an unprivileged inner container that acts as the user's workspace. The inner container is able to run system-level software similar to a regular virtual machine (e.g. systemd, dockerd, etc). Envbox offers the following benefits over running sysbox directly on the nodes:

  • No custom runtime installation or management on your Kubernetes nodes.
  • No limit to the number of pods that run envbox.

Some drawbacks include:

  • The outer container must be run as privileged
    • Note: the inner container is not privileged. For more information on the security of sysbox containers see sysbox's official documentation.
  • Initial workspace startup is slower than running sysbox-runc directly on the nodes. This is due to envbox having to pull the image to its own Docker cache on its initial startup. Once the image is cached in envbox, startup performance is similar.

Envbox requires the same kernel requirements as running sysbox directly on the nodes. Refer to sysbox's compatibility matrix to ensure your nodes are compliant.

To get started with envbox check out the starter template or visit the repo.

Authenticating with a Private Registry

Authenticating with a private container registry can be done by referencing the credentials via the CODER_IMAGE_PULL_SECRET environment variable. It is encouraged to populate this environment variable by using a Kubernetes secret.

Refer to your container registry documentation to understand how to best create this secret.

The following shows a minimal example using a the JSON API key from a GCP service account to pull a private image:

# Create the secret
$ kubectl create secret docker-registry <name> \
  --docker-server=us.gcr.io \
  --docker-username=_json_key \
  --docker-password="$(cat ./json-key-file.yaml)" \
  --docker-email=<service-account-email>
env {
  name = "CODER_IMAGE_PULL_SECRET"
  value_from {
    secret_key_ref {
      name = "<name>"
      key = ".dockerconfigjson"
    }
  }
}

Rootless podman

Podman is Docker alternative that is compatible with OCI containers specification. which can run rootless inside Kubernetes pods. No custom RuntimeClass is required.

Before using Podman, please review the following documentation:

  1. Enable smart-device-manager to securely expose a FUSE devices to pods.

    cat <<EOF | kubectl create -f -
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: fuse-device-plugin-daemonset
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          name: fuse-device-plugin-ds
      template:
        metadata:
          labels:
            name: fuse-device-plugin-ds
        spec:
          hostNetwork: true
          containers:
          - name: fuse-device-plugin-ctr
            image: soolaugust/fuse-device-plugin:v1.0
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop: ["ALL"]
            volumeMounts:
            - name: device-plugin
              mountPath: /var/lib/kubelet/device-plugins
          volumes:
          - name: device-plugin
            hostPath:
              path: /var/lib/kubelet/device-plugins
          imagePullSecrets:
          - name: registry-secret
    EOF
    
  2. Be sure to label your nodes to enable smarter-device-manager:

    kubectl get nodes
    kubectl label nodes --all smarter-device-manager=enabled
    

    ⚠️ Warning: If you are using a managed Kubernetes distribution (e.g. AKS, EKS, GKE), be sure to set node labels via your cloud provider. Otherwise, your nodes may drop the labels and break podman functionality.

  3. For systems running SELinux (typically Fedora-, CentOS-, and Red Hat-based systems), you might need to disable SELinux or set it to permissive mode.

  4. Import our kubernetes-with-podman example template, or make your own.

    echo "kubernetes-with-podman" | coder templates init
    cd ./kubernetes-with-podman
    coder templates create
    

    For more information around the requirements of rootless podman pods, see: How to run Podman inside of Kubernetes

Privileged sidecar container

A privileged container can be added to your templates to add docker support. This may come in handy if your nodes cannot run Sysbox.

⚠️ Warning: This is insecure. Workspaces will be able to gain root access to the host machine.

Use a privileged sidecar container in Docker-based templates

resource "coder_agent" "main" {
  os             = "linux"
  arch           = "amd64"
}

resource "docker_network" "private_network" {
  name = "network-${data.coder_workspace.me.id}"
}

resource "docker_container" "dind" {
  image      = "docker:dind"
  privileged = true
  name       = "dind-${data.coder_workspace.me.id}"
  entrypoint = ["dockerd", "-H", "tcp://0.0.0.0:2375"]
  networks_advanced {
    name = docker_network.private_network.name
  }
}

resource "docker_container" "workspace" {
  count   = data.coder_workspace.me.start_count
  image   = "codercom/enterprise-base:ubuntu"
  name    = "dev-${data.coder_workspace.me.id}"
  command = ["sh", "-c", coder_agent.main.init_script]
  env = [
    "CODER_AGENT_TOKEN=${coder_agent.main.token}",
    "DOCKER_HOST=${docker_container.dind.name}:2375"
  ]
  networks_advanced {
    name = docker_network.private_network.name
  }
}

Use a privileged sidecar container in Kubernetes-based templates

terraform {
  required_providers {
    coder = {
      source  = "coder/coder"
    }
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "2.16.0"
    }
  }
}

variable "workspaces_namespace" {
  default = "coder-namespace"
}

data "coder_workspace" "me" {}

resource "coder_agent" "main" {
  os             = "linux"
  arch           = "amd64"
}

resource "kubernetes_pod" "main" {
  count = data.coder_workspace.me.start_count
  metadata {
    name      = "coder-${data.coder_workspace.me.owner}-${data.coder_workspace.me.name}"
    namespace = var.namespace
  }
  spec {
    # Run a privileged dind (Docker in Docker) container
    container {
      name  = "docker-sidecar"
      image = "docker:dind"
      security_context {
        privileged = true
      }
      command = ["dockerd", "-H", "tcp://127.0.0.1:2375"]
    }
    container {
      name    = "dev"
      image   = "codercom/enterprise-base:ubuntu"
      command = ["sh", "-c", coder_agent.main.init_script]
      security_context {
        run_as_user = "1000"
      }
      env {
        name  = "CODER_AGENT_TOKEN"
        value = coder_agent.main.token
      }
      # Use the Docker daemon in the "docker-sidecar" container
      env {
        name  = "DOCKER_HOST"
        value = "localhost:2375"
      }
    }
  }
}

Systemd in Docker

Additionally, Sysbox can be used to give workspaces full systemd capabilities.

After installing Sysbox on Kubernetes, modify your template to use the sysbox-runc RuntimeClass. This requires the Kubernetes Terraform provider version 2.16.0 or greater.

terraform {
  required_providers {
    coder = {
      source  = "coder/coder"
    }
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "2.16.0"
    }
  }
}

variable "workspaces_namespace" {
  default = "coder-namespace"
}

data "coder_workspace" "me" {}

resource "coder_agent" "main" {
  os   = "linux"
  arch = "amd64"
  dir  = "/home/coder"
}

resource "kubernetes_pod" "dev" {
  count = data.coder_workspace.me.start_count
  metadata {
    name      = "coder-${data.coder_workspace.me.owner}-${data.coder_workspace.me.name}"
    namespace = var.workspaces_namespace
    annotations = {
      "io.kubernetes.cri-o.userns-mode" = "auto:size=65536"
    }
  }

  spec {

    # Use Sysbox container runtime (required)
    runtime_class_name = "sysbox-runc"

    # Run as root in order to start systemd (required)
    security_context {
      run_as_user = 0
      fs_group    = 0
    }

    container {
      name = "dev"
      env {
        name  = "CODER_AGENT_TOKEN"
        value = coder_agent.main.token
      }
      image = "codercom/enterprise-base:ubuntu"
      command = ["sh", "-c", <<EOF
    # Start the Coder agent as the "coder" user
    # once systemd has started up
    sudo -u coder --preserve-env=CODER_AGENT_TOKEN /bin/bash -- <<-'    EOT' &
    while [[ ! $(systemctl is-system-running) =~ ^(running|degraded) ]]
    do
      echo "Waiting for system to start... $(systemctl is-system-running)"
      sleep 2
    done
    ${coder_agent.main.init_script}
    EOT

    exec /sbin/init
    EOF
      ]
    }
  }
}