docs: add new section on managing provisioners from the dashboard (#16563)

closes #16513 


[preview](https://coder.com/docs/@16513-manage-ext-provisioners/admin/provisioners/manage-provisioner-jobs)

---------

Co-authored-by: EdwardAngert <17991901+EdwardAngert@users.noreply.github.com>

.vscode/settings.json

@@ -57,5 +57,8 @@
 	"[css][html][markdown][yaml]": {
 		"editor.defaultFormatter": "esbenp.prettier-vscode"
 	},
-	"typos.config": ".github/workflows/typos.toml"
+	"typos.config": ".github/workflows/typos.toml",
+	"[markdown]": {
+		"editor.defaultFormatter": "DavidAnson.vscode-markdownlint"
+	}
 }

coderd/provisionerdserver/acquirer_test.go

@@ -518,7 +518,7 @@ func TestAcquirer_MatchTags(t *testing.T) {
 
 	t.Run("GenTable", func(t *testing.T) {
 		t.Parallel()
-		// Generate a table that can be copy-pasted into docs/admin/provisioners.md
+		// Generate a table that can be copy-pasted into docs/admin/provisioners/index.md
 		lines := []string{
 			"\n",
 			"| Provisioner Tags | Job Tags | Same Org | Can Run Job? |",
@@ -547,7 +547,7 @@ func TestAcquirer_MatchTags(t *testing.T) {
 			s := fmt.Sprintf("| %s | %s | %s | %s |", kvs(tt.acquireJobTags), kvs(tt.provisionerJobTags), sameOrg, acquire)
 			lines = append(lines, s)
 		}
-		t.Log("You can paste this into docs/admin/provisioners.md")
+		t.Log("You can paste this into docs/admin/provisioners/index.md")
 		t.Log(strings.Join(lines, "\n"))
 	})
 }

docs/admin/infrastructure/architecture.md

@@ -42,7 +42,7 @@ _provisionerd_ is the execution context for infrastructure modifying providers.
 At the moment, the only provider is Terraform (running `terraform`).
 
 By default, the Coder server runs multiple provisioner daemons.
-[External provisioners](../provisioners.md) can be added for security or
+[External provisioners](../provisioners/index.md) can be added for security or
 scalability purposes.
 
 ### Workspaces

docs/admin/monitoring/health-check.md

@@ -294,7 +294,7 @@ be built until there is at least one provisioner daemon running.
 **Solution:**
 
 If you are using
-[External Provisioner Daemons](../provisioners.md#external-provisioners), ensure
+[External Provisioner Daemons](../provisioners/index.md#external-provisioners), ensure
 that they are able to successfully connect to Coder. Otherwise, ensure
 [`--provisioner-daemons`](../../reference/cli/server.md#--provisioner-daemons)
 is set to a value greater than 0.

docs/admin/monitoring/logs.md

@@ -24,7 +24,7 @@ Connect logs are all captured in the `coderd` logs.
 
 ## `provisionerd` Logs
 
-Logs for [external provisioners](../provisioners.md) are structured
+Logs for [external provisioners](../provisioners/index.md) are structured
 [and configured](../../reference/cli/provisioner_start.md#--log-human) similarly
 to `coderd` logs. Use these logs to troubleshoot and monitor the Terraform
 operations behind workspaces and templates.

docs/admin/provisioners/index.md

@@ -1,7 +1,7 @@
 # External provisioners
 
 By default, the Coder server runs
-[built-in provisioner daemons](../reference/cli/server.md#--provisioner-daemons),
+[built-in provisioner daemons](../../reference/cli/server.md#--provisioner-daemons),
 which execute `terraform` during workspace and template builds. However, there
 are often benefits to running external provisioner daemons:
 
@@ -11,7 +11,7 @@ are often benefits to running external provisioner daemons:
 - **Isolate APIs:** Deploy provisioners in isolated environments (on-prem, AWS,
   Azure) instead of exposing APIs (Docker, Kubernetes, VMware) to the Coder
   server. See
-  [Provider Authentication](../admin/templates/extending-templates/provider-authentication.md)
+  [Provider Authentication](../../admin/templates/extending-templates/provider-authentication.md)
   for more details.
 
 - **Isolate secrets**: Keep Coder unaware of cloud secrets, manage/rotate
@@ -19,19 +19,21 @@ are often benefits to running external provisioner daemons:
 
 - **Reduce server load**: External provisioners reduce load and build queue
   times from the Coder server. See
-  [Scaling Coder](../admin/infrastructure/index.md#scale-tests) for more
+  [Scaling Coder](../../admin/infrastructure/index.md#scale-tests) for more
   details.
 
 Each provisioner runs a single
-[concurrent workspace build](../admin/infrastructure/scale-testing.md#control-plane-provisionerd).
+[concurrent workspace build](../../admin/infrastructure/scale-testing.md#control-plane-provisionerd).
 For example, running 30 provisioner containers will allow 30 users to start
 workspaces at the same time.
 
 Provisioners are started with the
-[`coder provisioner start`](../reference/cli/provisioner_start.md) command in
+[`coder provisioner start`](../../reference/cli/provisioner_start.md) command in
 the [full Coder binary](https://github.com/coder/coder/releases). Keep reading
 to learn how to start provisioners via Docker, Kubernetes, Systemd, etc.
 
+You can use the dashboard, CLI, or API to [manage provisioners](./manage-provisioner-jobs.md).
+
 ## Authentication
 
 The provisioner daemon must authenticate with your Coder deployment.
@@ -83,7 +85,7 @@ Kubernetes/Docker/etc.
 
 A user account with the role `Template Admin` or `Owner` can start provisioners
 using their user account. This may be beneficial if you are running provisioners
-via [automation](../reference/index.md).
+via [automation](../../reference/index.md).
 
 ```sh
 coder login https://<your-coder-url>
@@ -110,7 +112,7 @@ Global pre-shared keys (PSK) make it difficult to rotate keys or isolate provisi
 
 A deployment-wide PSK can be used to authenticate any provisioner. To use a
 global PSK, set a
-[provisioner daemon pre-shared key (PSK)](../reference/cli/server.md#--provisioner-daemon-psk)
+[provisioner daemon pre-shared key (PSK)](../../reference/cli/server.md#--provisioner-daemon-psk)
 on the Coder server.
 
 Next, start the provisioner:
@@ -157,12 +159,12 @@ coder templates push on-prem-chicago \
 
 This can also be done in the UI when building a template:
 
-![template tags](../images/admin/provisioner-tags.png)
+![template tags](../../images/admin/provisioner-tags.png)
 
 Alternatively, a template can target a provisioner via
 [workspace tags](https://github.com/coder/coder/tree/main/examples/workspace-tags)
 inside the Terraform. See the
-[workspace tags documentation](../admin/templates/extending-templates/workspace-tags.md)
+[workspace tags documentation](../../admin/templates/extending-templates/workspace-tags.md)
 for more information.
 
 > [!NOTE]
@@ -237,17 +239,17 @@ This is illustrated in the below table:
 
 Provisioners can broadly be categorized by scope: `organization` or `user`. The
 scope of a provisioner can be specified with
-[`-tag=scope=<scope>`](../reference/cli/provisioner_start.md#-t---tag) when
+[`-tag=scope=<scope>`](../../reference/cli/provisioner_start.md#-t---tag) when
 starting the provisioner daemon. Only users with at least the
-[Template Admin](./users/index.md#roles) role or higher may create
+[Template Admin](../users/index.md#roles) role or higher may create
 organization-scoped provisioner daemons.
 
 There are two exceptions:
 
-- [Built-in provisioners](../reference/cli/server.md#--provisioner-daemons) are
+- [Built-in provisioners](../../reference/cli/server.md#--provisioner-daemons) are
   always organization-scoped.
 - External provisioners started using a
-  [pre-shared key (PSK)](../reference/cli/provisioner_start.md#--psk) are always
+  [pre-shared key (PSK)](../../reference/cli/provisioner_start.md#--psk) are always
   organization-scoped.
 
 ### Organization-Scoped Provisioners
@@ -371,7 +373,7 @@ docker run --rm -it \
 
 As mentioned above, the Coder server will run built-in provisioners by default.
 This can be disabled with a server-wide
-[flag or environment variable](../reference/cli/server.md#--provisioner-daemons).
+[flag or environment variable](../../reference/cli/server.md#--provisioner-daemons).
 
 ```sh
 coder server --provisioner-daemons=0
@@ -390,3 +392,7 @@ address.
 
 If you have provisioners daemons deployed as pods, it is advised to monitor them
 separately.
+
+## Next
+
+- [Manage Provisioners](./manage-provisioner-jobs.md)

docs/admin/provisioners/manage-provisioner-jobs.md

@@ -0,0 +1,80 @@
+# Manage provisioner jobs
+
+[Provisioners](./index.md) start and run provisioner jobs to create or delete workspaces.
+Each time a workspace is built, rebuilt, or destroyed, it generates a new job and assigns
+the job to an available provisioner daemon for execution.
+
+While most jobs complete smoothly, issues with templates, cloud resources, or misconfigured
+provisioners can cause jobs to fail or hang indefinitely (these are in a `Pending` state).
+
+![Provisioner jobs in the dashboard](../../images/admin/provisioners/provisioner-jobs.png)
+
+## How to find provisioner jobs
+
+Coder admins can view and manage provisioner jobs.
+
+Use the dashboard, CLI, or API:
+
+- **Dashboard**:
+
+   Select **Admin settings** > **Organizations** > **Provisioner Jobs**
+
+   Provisioners are organization-specific. If you have more than one organization, select it first.
+
+- **CLI**: `coder provisioner jobs list`
+- **API**: `/api/v2/provisioner/jobs`
+
+## Manage provisioner jobs from the dashboard
+
+View more information about and manage your provisioner jobs from the Coder dashboard.
+
+1. Under **Admin settings** select **Organizations**, then select **Provisioner jobs**.
+
+1. Select the **>** to expand each entry for more information.
+
+1. To delete a job, select the 🚫 at the end of the entry's row.
+
+   If your user doesn't have the correct permissions, this option is greyed out.
+
+## Provisioner job status
+
+Each provisioner job has a lifecycle state:
+
+| Status        | Description                                                    |
+|---------------|----------------------------------------------------------------|
+| **Pending**   | Job is queued but has not yet been picked up by a provisioner. |
+| **Running**   | A provisioner is actively working on the job.                  |
+| **Completed** | Job succeeded.                                                 |
+| **Failed**    | Provisioner encountered an error while executing the job.      |
+| **Canceled**  | Job was manually terminated by an admin.                       |
+
+## When to cancel provisioner jobs
+
+A job might need to be cancelled when:
+
+- It has been stuck in **Pending** for too long. This can be due to misconfigured tags or unavailable provisioners.
+- It is **Running** indefinitely, often caused by external system failures or buggy templates.
+- An admin wants to abort a failed attempt, fix the root cause, and retry provisioning.
+- A workspace was deleted in the UI but the underlying cloud resource wasn’t cleaned up, causing a hanging delete job.
+
+Cancelling a job does not automatically retry the operation.
+It clears the stuck state and allows the admin or user to trigger the action again if needed.
+
+## Troubleshoot provisioner jobs
+
+Provisioner jobs can fail or slow workspace creation for a number of reasons.
+Follow these steps to identify problematic jobs or daemons:
+
+1. Filter jobs by `pending` status in the dashboard, or use the CLI:
+
+   ```bash
+   coder provisioner jobs list -s pending
+   ```
+
+1. Look for daemons with multiple failed jobs and for template [tag mismatches](./index.md#provisioner-tags).
+
+1. Cancel the job through the dashboard, or use the CLI:
+
+   ```shell
+   coder provisioner jobs cancel <job-id>
+   ```

docs/admin/security/secrets.md

@@ -7,7 +7,7 @@ guide to
 
 This article explains how to use secrets in a workspace. To authenticate the
 workspace provisioner, see the
-<a href="../provisioners.md#authentication">provisioners documentation</a>.
+<a href="../provisioners/index.md#authentication">provisioners documentation</a>.
 
 ## Before you begin
 

docs/admin/setup/index.md

@@ -154,4 +154,4 @@ more information.
 ## Up Next
 
 - [Setup and manage templates](../templates/index.md)
-- [Setup external provisioners](../provisioners.md)
+- [Setup external provisioners](../provisioners/index.md)

docs/admin/templates/creating-templates.md

@@ -68,7 +68,7 @@ coder templates push
 > [!NOTE]
 > If `template push` fails, Coder is likely not authorized to deploy
 > infrastructure in the given location. Learn how to configure
-> [provisioner authentication](../provisioners.md).
+> [provisioner authentication](../provisioners/index.md).
 
 You can edit the metadata of the template such as the display name with the
 [`templates edit`](../../reference/cli/templates_edit.md) command:

docs/admin/templates/extending-templates/modules.md

@@ -120,7 +120,7 @@ template as the underlying module.
 ### Private git repository
 
 If you are importing a module from a private git repository, the Coder server or
-[provisioner](../../provisioners.md) needs git credentials. Since this token
+[provisioner](../../provisioners/index.md) needs git credentials. Since this token
 will only be used for cloning your repositories with modules, it is best to
 create a token with access limited to the repository and no extra permissions.
 In GitHub, you can generate a

docs/admin/templates/extending-templates/provider-authentication.md

@@ -46,7 +46,7 @@ There are two ways to use a remote Docker host for authentication:
 
 - Configure the Docker provider to use a
   [remote host over SSH or TCP](https://registry.terraform.io/providers/kreuzwerker/docker/latest/docs#remote-hosts).
-- Run an [external provisioner](../../provisioners.md) on the remote docker
+- Run an [external provisioner](../../provisioners/index.md) on the remote docker
   host.
 
 Other providers might also support authenticated environments. Check the

docs/admin/users/groups-roles.md

@@ -24,7 +24,7 @@ Roles determine which actions users can take within the platform.
 | Manage **ALL** Templates                                        |         |            | ✅              | ✅     |
 | View **ALL** Workspaces                                         |         |            | ✅              | ✅     |
 | Update and delete **ALL** Workspaces                            |         |            |                | ✅     |
-| Run [external provisioners](../provisioners.md)                 |         |            | ✅              | ✅     |
+| Run [external provisioners](../provisioners/index.md)           |         |            | ✅              | ✅     |
 | Execute and use **ALL** Workspaces                              |         |            |                | ✅     |
 | View all user operation [Audit Logs](../security/audit-logs.md) | ✅       |            |                | ✅     |
 
@@ -80,7 +80,7 @@ Note that these permissions only apply to the scope of an
 A malicious Template Admin could write a template that executes commands on the
 host (or `coder server` container), which potentially escalates their privileges
 or shuts down the Coder server. To avoid this, run
-[external provisioners](../provisioners.md).
+[external provisioners](../provisioners/index.md).
 
 In low-trust environments, we do not recommend giving users direct access to
 edit templates. Instead, use

docs/admin/users/organizations.md

@@ -37,7 +37,7 @@ From there, you can manage the name, icon, description, users, and groups:
 
 Any additional organizations have unique admins, users, templates, provisioners,
 groups, and workspaces. Each organization must have at least one dedicated
-[provisioner](../provisioners.md) since the built-in provisioners only apply to
+[provisioner](../provisioners/index.md) since the built-in provisioners only apply to
 the default organization.
 
 You can configure [organization/role/group sync](./idp-sync.md) from your
@@ -71,7 +71,7 @@ Next deploy a provisioner and template for this organization.
 
 ### 2. Deploy a provisioner
 
-[Provisioners](../provisioners.md) are organization-scoped and are responsible
+[Provisioners](../provisioners/index.md) are organization-scoped and are responsible
 for executing Terraform/OpenTofu to provision the infrastructure for workspaces
 and testing templates. Before creating templates, we must deploy at least one
 provisioner as the built-in provisioners are scoped to the default organization.
@@ -90,7 +90,7 @@ provisioner as the built-in provisioners are scoped to the default organization.
 
    In this example, start the provisioner using the Coder CLI on a host with
    Docker. For instructions on using other platforms like Kubernetes, see our
-   [provisioner documentation](../provisioners.md).
+   [provisioner documentation](../provisioners/index.md).
 
    ```sh
    export CODER_URL=https://<your-coder-url>

docs/manifest.json

@@ -499,9 +499,17 @@
 				{
 					"title": "External Provisioners",
 					"description": "Learn how to run external provisioners with Coder",
-					"path": "./admin/provisioners.md",
+					"path": "./admin/provisioners/index.md",
 					"icon_path": "./images/icons/key.svg",
+					"state": ["enterprise", "premium"],
+					"children": [
+						{
+							"title": "Manage Provisioner Jobs",
+							"description": "Learn how to run external provisioners with Coder",
+							"path": "./admin/provisioners/manage-provisioner-jobs.md",
 							"state": ["enterprise", "premium"]
+						}
+					]
 				},
 				{
 					"title": "External Auth",

docs/tutorials/best-practices/scale-coder.md

@@ -141,7 +141,7 @@ maintenance window to minimize disruption.
 ### Locality
 
 We recommend that you run one or more
-[provisioner daemon deployments external to Coder Server](../../admin/provisioners.md)
+[provisioner daemon deployments external to Coder Server](../../admin/provisioners/index.md)
 and disable provisioner daemons within your Coder Server.
 This allows you to scale them independently of the Coder Server:
 

docs/tutorials/best-practices/security-best-practices.md

@@ -161,7 +161,7 @@ provision:
 
 ### Authentication
 
-1. Use a [scoped key](../../admin/provisioners.md#scoped-key-recommended) to
+1. Use a [scoped key](../../admin/provisioners/index.md#scoped-key-recommended) to
    authenticate the provisioner daemons with Coder. These keys can only be used
    to authenticate provisioner daemons (not other APIs on the Coder Server).
 

docs/tutorials/best-practices/speed-up-templates.md

@@ -83,7 +83,7 @@ config option.
 You risk overloading Coder if you use too many built-in provisioners, so we
 recommend a maximum of five built-in provisioners per `coderd` replica. For more
 than five provisioners, we recommend that you move to
-[External Provisioners](../../admin/provisioners.md) and also consider
+[External Provisioners](../../admin/provisioners/index.md) and also consider
 [High Availability](../../admin/networking/high-availability.md) to run multiple
 `coderd` replicas.
 
@@ -165,4 +165,4 @@ directory.
 
 Ensure that this directory is set to a location on disk which will persist
 across restarts of Coder or
-[external provisioners](../../admin/provisioners.md), if you're using them.
+[external provisioners](../../admin/provisioners/index.md), if you're using them.