mirror of
https://github.com/coder/coder.git
synced 2025-07-12 00:14:10 +00:00
316 lines
9.9 KiB
Markdown
316 lines
9.9 KiB
Markdown
# Workspace Process Logging
|
|
|
|
The workspace process logging feature allows you to log all system-level
|
|
processes executing in the workspace.
|
|
|
|
> **Note:** This feature is only available on Linux in Kubernetes. There are
|
|
> additional requirements outlined further in this document.
|
|
|
|
Workspace process logging adds a sidecar container to workspace pods that will
|
|
log all processes started in the workspace container (e.g., commands executed in
|
|
the terminal or processes created in the background by other processes).
|
|
Processes launched inside containers or nested containers within the workspace
|
|
are also logged. You can view the output from the sidecar or send it to a
|
|
monitoring stack, such as CloudWatch, for further analysis or long-term storage.
|
|
|
|
Please note that these logs are not recorded or captured by the Coder
|
|
organization in any way, shape, or form.
|
|
|
|
> This is an [Enterprise](https://coder.com/docs/v2/latest/enterprise) feature.
|
|
> To learn more about Coder Enterprise, please
|
|
> [contact sales](https://coder.com/contact).
|
|
|
|
## How this works
|
|
|
|
Coder uses [eBPF](https://ebpf.io/) (which we chose for its minimal performance
|
|
impact) to perform in-kernel logging and filtering of all exec system calls
|
|
originating from the workspace container.
|
|
|
|
The core of this feature is also open source and can be found in the
|
|
[exectrace](https://github.com/coder/exectrace) GitHub repo. The enterprise
|
|
component (in the `enterprise/` directory of the repo) is responsible for
|
|
starting the eBPF program with the correct filtering options for the specific
|
|
workspace.
|
|
|
|
## Requirements
|
|
|
|
The host machine must be running a Linux kernel >= 5.8 with the kernel config
|
|
`CONFIG_DEBUG_INFO_BTF=y` enabled.
|
|
|
|
To check your kernel version, run:
|
|
|
|
```shell
|
|
uname -r
|
|
```
|
|
|
|
To validate the required kernel config is enabled, run either of the following
|
|
commands on your nodes directly (_not_ from a workspace terminal):
|
|
|
|
```shell
|
|
cat /proc/config.gz | gunzip | grep CONFIG_DEBUG_INFO_BTF
|
|
```
|
|
|
|
```shell
|
|
cat "/boot/config-$(uname -r)" | grep CONFIG_DEBUG_INFO_BTF
|
|
```
|
|
|
|
If these requirements are not met, workspaces will fail to start for security
|
|
reasons.
|
|
|
|
Your template must be a Kubernetes template. Workspace process logging is not
|
|
compatible with the `sysbox-runc` runtime due to technical limitations, but it
|
|
is compatible with our `envbox` template family.
|
|
|
|
## Example templates
|
|
|
|
We provide working example templates for Kubernetes, and Kubernetes with
|
|
`envbox` (for [Docker support in workspaces](./docker-in-workspaces.md)). You
|
|
can view these templates in the
|
|
[exectrace repo](https://github.com/coder/exectrace/tree/main/enterprise/templates).
|
|
|
|
## Configuring custom templates to use workspace process logging
|
|
|
|
If you have an existing Kubernetes or Kubernetes with `envbox` template that you
|
|
would like to add workspace process logging to, follow these steps:
|
|
|
|
1. Ensure the image used in your template has `curl` installed.
|
|
|
|
1. Add the following section to your template's `main.tf` file:
|
|
|
|
<!--
|
|
If you are updating this section, please also update the example templates
|
|
in the exectrace repo.
|
|
-->
|
|
|
|
```hcl
|
|
locals {
|
|
# This is the init script for the main workspace container that runs before the
|
|
# agent starts to configure workspace process logging.
|
|
exectrace_init_script = <<EOT
|
|
set -eu
|
|
pidns_inum=$(readlink /proc/self/ns/pid | sed 's/[^0-9]//g')
|
|
if [ -z "$pidns_inum" ]; then
|
|
echo "Could not determine process ID namespace inum"
|
|
exit 1
|
|
fi
|
|
|
|
# Before we start the script, does curl exist?
|
|
if ! command -v curl >/dev/null 2>&1; then
|
|
echo "curl is required to download the Coder binary"
|
|
echo "Please install curl to your image and try again"
|
|
# 127 is command not found.
|
|
exit 127
|
|
fi
|
|
|
|
echo "Sending process ID namespace inum to exectrace sidecar"
|
|
rc=0
|
|
max_retry=5
|
|
counter=0
|
|
until [ $counter -ge $max_retry ]; do
|
|
set +e
|
|
curl \
|
|
--fail \
|
|
--silent \
|
|
--connect-timeout 5 \
|
|
-X POST \
|
|
-H "Content-Type: text/plain" \
|
|
--data "$pidns_inum" \
|
|
http://127.0.0.1:56123
|
|
rc=$?
|
|
set -e
|
|
if [ $rc -eq 0 ]; then
|
|
break
|
|
fi
|
|
|
|
counter=$((counter+1))
|
|
echo "Curl failed with exit code $${rc}, attempt $${counter}/$${max_retry}; Retrying in 3 seconds..."
|
|
sleep 3
|
|
done
|
|
if [ $rc -ne 0 ]; then
|
|
echo "Failed to send process ID namespace inum to exectrace sidecar"
|
|
exit $rc
|
|
fi
|
|
|
|
EOT
|
|
}
|
|
```
|
|
|
|
1. Update the `command` of your workspace container like the following:
|
|
|
|
<!--
|
|
If you are updating this section, please also update the example templates
|
|
in the exectrace repo.
|
|
-->
|
|
|
|
```hcl
|
|
resource "kubernetes_pod" "main" {
|
|
...
|
|
spec {
|
|
...
|
|
container {
|
|
...
|
|
// NOTE: this command is changed compared to the upstream kubernetes
|
|
// template
|
|
command = [
|
|
"sh",
|
|
"-c",
|
|
"${local.exectrace_init_script}\n\n${coder_agent.main.init_script}",
|
|
]
|
|
...
|
|
}
|
|
...
|
|
}
|
|
...
|
|
}
|
|
```
|
|
|
|
> **Note:** If you are using the `envbox` template, you will need to update
|
|
> the third argument to be
|
|
> `"${local.exectrace_init_script}\n\nexec /envbox docker"` instead.
|
|
|
|
1. Add the following container to your workspace pod spec.
|
|
|
|
<!--
|
|
If you are updating this section, please also update the example templates
|
|
in the exectrace repo.
|
|
-->
|
|
|
|
```hcl
|
|
resource "kubernetes_pod" "main" {
|
|
...
|
|
spec {
|
|
...
|
|
// NOTE: this container is added compared to the upstream kubernetes
|
|
// template
|
|
container {
|
|
name = "exectrace"
|
|
image = "ghcr.io/coder/exectrace:latest"
|
|
image_pull_policy = "Always"
|
|
command = [
|
|
"/opt/exectrace",
|
|
"--init-address", "127.0.0.1:56123",
|
|
"--label", "workspace_id=${data.coder_workspace.me.id}",
|
|
"--label", "workspace_name=${data.coder_workspace.me.name}",
|
|
"--label", "user_id=${data.coder_workspace.me.owner_id}",
|
|
"--label", "username=${data.coder_workspace.me.owner}",
|
|
"--label", "user_email=${data.coder_workspace.me.owner_email}",
|
|
]
|
|
security_context {
|
|
// exectrace must be started as root so it can attach probes into the
|
|
// kernel to record process events with high throughput.
|
|
run_as_user = "0"
|
|
run_as_group = "0"
|
|
// exectrace requires a privileged container so it can control mounts
|
|
// and perform privileged syscalls against the host kernel to attach
|
|
// probes.
|
|
privileged = true
|
|
}
|
|
}
|
|
...
|
|
}
|
|
...
|
|
}
|
|
```
|
|
|
|
> **Note:** `exectrace` requires root privileges and a privileged container
|
|
> to attach probes to the kernel. This is a requirement of eBPF.
|
|
|
|
1. Add the following environment variable to your workspace pod:
|
|
|
|
<!--
|
|
If you are updating this section, please also update the example templates
|
|
in the exectrace repo.
|
|
-->
|
|
|
|
```hcl
|
|
resource "kubernetes_pod" "main" {
|
|
...
|
|
spec {
|
|
...
|
|
env {
|
|
name = "CODER_AGENT_SUBSYSTEM"
|
|
value = "exectrace"
|
|
}
|
|
...
|
|
}
|
|
...
|
|
}
|
|
```
|
|
|
|
Once you have made these changes, you can push a new version of your template
|
|
and workspace process logging will be enabled for all workspaces once they are
|
|
restarted.
|
|
|
|
## Viewing workspace process logs
|
|
|
|
To view the process logs for a specific workspace you can use `kubectl` to print
|
|
the logs:
|
|
|
|
```bash
|
|
kubectl logs pod-name --container exectrace
|
|
```
|
|
|
|
The raw logs will look something like this:
|
|
|
|
```json
|
|
{
|
|
"ts": "2022-02-28T20:29:38.038452202Z",
|
|
"level": "INFO",
|
|
"msg": "exec",
|
|
"fields": {
|
|
"labels": {
|
|
"user_email": "jessie@coder.com",
|
|
"user_id": "5e876e9a-121663f01ebd1522060d5270",
|
|
"username": "jessie",
|
|
"workspace_id": "621d2e52-a6987ef6c56210058ee2593c",
|
|
"workspace_name": "main"
|
|
},
|
|
"cmdline": "uname -a",
|
|
"event": {
|
|
"filename": "/usr/bin/uname",
|
|
"argv": ["uname", "-a"],
|
|
"truncated": false,
|
|
"pid": 920684,
|
|
"uid": 101000,
|
|
"gid": 101000,
|
|
"comm": "bash"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### View logs in AWS EKS
|
|
|
|
If you're using AWS' Elastic Kubernetes Service, you can
|
|
[configure your cluster](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-logs.html)
|
|
to send logs to CloudWatch. This allows you to view the logs for a specific user
|
|
or workspace.
|
|
|
|
To view your logs, go to the CloudWatch dashboard (which is available on the
|
|
**Log Insights** tab) and run a query similar to the following:
|
|
|
|
```text
|
|
fields @timestamp, log_processed.fields.cmdline
|
|
| sort @timestamp asc
|
|
| filter kubernetes.container_name="exectrace"
|
|
| filter log_processed.fields.labels.username="zac"
|
|
| filter log_processed.fields.labels.workspace_name="code"
|
|
```
|
|
|
|
## Usage considerations
|
|
|
|
- The sidecar attached to each workspace is a privileged container, so you may
|
|
need to review your organization's security policies before enabling this
|
|
feature. Enabling workspace process logging does _not_ grant extra privileges
|
|
to the workspace container itself, however.
|
|
- `exectrace` will log processes from nested Docker containers (including deeply
|
|
nested containers) correctly, but Coder does not distinguish between processes
|
|
started in the workspace and processes started in a child container in the
|
|
logs.
|
|
- With `envbox` workspaces, this feature will detect and log startup processes
|
|
begun in the outer container (including container initialization processes).
|
|
- Because this feature logs **all** processes in the workspace, high levels of
|
|
usage (e.g., during a `make` run) will result in an abundance of output in the
|
|
sidecar container. Depending on how your Kubernetes cluster is configured, you
|
|
may incur extra charges from your cloud provider to store the additional logs.
|