Commit Graph

78 Commits

Author SHA1 Message Date
44499315ed chore: reduce log volume on server startup (#16608)
Addresses https://github.com/coder/coder/issues/16231.

This PR reduces the volume of logs we print after server startup in
order to surface the web UI URL better.

Here are the logs after the changes a couple of seconds after starting
the server:

<img width="868" alt="Screenshot 2025-02-18 at 16 31 32"
src="https://github.com/user-attachments/assets/786dc4b8-7383-48c8-a5c3-a997c01ca915"
/>

The warning is due to running a development site-less build. It wouldn't
show in a release build.
2025-02-20 16:33:14 +01:00
9e9a5fd204 chore(coderd/coderdtest): wait for provisioner daemons to be connected (#15936)
Fixes https://github.com/coder/internal/issues/260
2024-12-20 09:49:19 +00:00
c5287910f9 feat: add workspace build timing metrics (#15771)
This PR introduces a new prometheus metrics for
`workspace_build_timing_seconds`, which specifically reports workspace
build times. To reduce cardinality, this metrics excludes
`workspace_name` and `workspace_owner` that are present on the
`workspace_builds_total` metrics.
2024-12-11 05:36:48 +00:00
09f00c08df chore: shutdown provisioner should stop waiting on client (#13118)
* chore: shutdown provisioner should stop waiting on client
* chore: add unit test that replicates failed client conn
2024-05-03 10:15:17 -05:00
895df54051 fix: separate signals for passive, active, and forced shutdown (#12358)
* fix: separate signals for passive, active, and forced shutdown

`SIGTERM`: Passive shutdown stopping provisioner daemons from accepting new
jobs but waiting for existing jobs to successfully complete.

`SIGINT` (old existing behavior): Notify provisioner daemons to cancel in-flight jobs, wait 5s for jobs to be exited, then force quit.

`SIGKILL`: Untouched from before, will force-quit.

* Revert dramatic signal changes

* Rename

* Fix shutdown behavior for provisioner daemons

* Add test for graceful shutdown
2024-03-15 13:16:36 +00:00
213b768785 feat(coderd): insert provisioner daemons (#11207)
* Adds UpdateProvisionerDaemonLastSeenAt
* Adds heartbeat to provisioner daemons
* Inserts provisioner daemons to database upon start
* Ensures TagOwner is an empty string and not nil
* Adds COALESCE() in idx_provisioner_daemons_name_owner_key
2023-12-18 16:44:52 +00:00
375c70d141 feat: integrate Acquirer for provisioner jobs (#9717)
* chore: add Acquirer to provisionerdserver pkg

Signed-off-by: Spike Curtis <spike@coder.com>

* code review improvements & fixes

Signed-off-by: Spike Curtis <spike@coder.com>

* feat: integrate Acquirer for provisioner jobs

Signed-off-by: Spike Curtis <spike@coder.com>

* Fix imports, whitespace

Signed-off-by: Spike Curtis <spike@coder.com>

* provisionerdserver always closes; remove poll interval from playwright

Signed-off-by: Spike Curtis <spike@coder.com>

* post jobs outside transactions

Signed-off-by: Spike Curtis <spike@coder.com>

* graceful shutdown in test

Signed-off-by: Spike Curtis <spike@coder.com>

* Mark AcquireJob deprecated

Signed-off-by: Spike Curtis <spike@coder.com>

* Graceful shutdown on all provisionerd tests

Signed-off-by: Spike Curtis <spike@coder.com>

* Deprecate, not remove CLI flags

Signed-off-by: Spike Curtis <spike@coder.com>

---------

Signed-off-by: Spike Curtis <spike@coder.com>
2023-09-19 10:25:57 +04:00
11b6068112 feat: add support for networked provisioners (#9593)
* Refactor provisionerd to use interface to connect to provisioners

Signed-off-by: Spike Curtis <spike@coder.com>

* feat: add support for networked provisioners

Signed-off-by: Spike Curtis <spike@coder.com>

* fix token length and linting

Signed-off-by: Spike Curtis <spike@coder.com>

---------

Signed-off-by: Spike Curtis <spike@coder.com>
2023-09-08 09:53:48 +00:00
b240799f47 refactor(cli): use codersdk for provisioner types (#9508)
This change removes one use of `coderd/database` from the slim binary
and more correctly uses codersdk instead of database or provisionerd
packages.

No size change (yet).

Ref: #9380
2023-09-04 21:42:15 +03:00
90acf998bf fix: fix null pointer on external provisioner daemons with daily_cost (#9401)
* fix: fix null pointer on external provisioner daemons with daily_cost

Signed-off-by: Spike Curtis <spike@coder.com>

* Add logging for debounce and job acquire

Signed-off-by: Spike Curtis <spike@coder.com>

* Return error instead of panic

Signed-off-by: Spike Curtis <spike@coder.com>

* remove debounce on external provisioners to fix test flakes

Signed-off-by: Spike Curtis <spike@coder.com>

---------

Signed-off-by: Spike Curtis <spike@coder.com>
2023-08-30 10:48:35 +00:00
60d5002eb6 refactor: change template archive extraction to be on provisioner (#9264)
* refactor provisionersdk protocol

Signed-off-by: Spike Curtis <spike@coder.com>

* refactor provisioners to use new protocol

Signed-off-by: Spike Curtis <spike@coder.com>

* refactor provisionerd to use new protocol

Signed-off-by: Spike Curtis <spike@coder.com>

* refactor tests & proto renames

* Fixes from self-review

Signed-off-by: Spike Curtis <spike@coder.com>

* appease fmt & link

Signed-off-by: Spike Curtis <spike@coder.com>

* code review fixes & e2e fixes

Signed-off-by: Spike Curtis <spike@coder.com>

* More fmt

Signed-off-by: Spike Curtis <spike@coder.com>

* Code review fixes

Signed-off-by: Spike Curtis <spike@coder.com>

* new gen; use uuid for session workdir

Signed-off-by: Spike Curtis <spike@coder.com>

* Revert nix-based gen CI task until dogfood is on nix

Signed-off-by: Spike Curtis <spike@coder.com>

* revert deleting dogfood Docker stuff

Signed-off-by: Spike Curtis <spike@coder.com>

* Revert "revert deleting dogfood Docker stuff"

This reverts commit 9762158167.

---------

Signed-off-by: Spike Curtis <spike@coder.com>
2023-08-25 06:10:15 +00:00
6b8102cf4c feat(cli): add daily_cost to coder ls (#9200) 2023-08-19 12:56:08 -05:00
22e781eced chore: add /v2 to import module path (#9072)
* chore: add /v2 to import module path

go mod requires semantic versioning with versions greater than 1.x

This was a mechanical update by running:
```
go install github.com/marwan-at-work/mod/cmd/mod@latest
mod upgrade
```

Migrate generated files to import /v2

* Fix gen
2023-08-18 18:55:43 +00:00
be40dc85ab chore: cleanup extraneous logging (#9156)
* The batchstats warning went out on every Ctrl+C in my development

Rule of silence:

The provisioner and connect messages messages were sent out on every startup
without a corresponding user event, making them annoying and more-so
debug messages.
2023-08-17 21:01:55 +00:00
21af020386 feat: add external provisioner daemon helm chart (#8939)
* Refactor helm to extract common templates to libcoder

Signed-off-by: Spike Curtis <spike@coder.com>

* Remove comment from libcoder Chart.yaml

Signed-off-by: Spike Curtis <spike@coder.com>

* Add provisioner helm chart

* Fix prettier, linting, docs

Signed-off-by: Spike Curtis <spike@coder.com>

* Log at INFO when provisionerd connects to coderd

Signed-off-by: Spike Curtis <spike@coder.com>

* remove unnecessary exports in helm tests

Signed-off-by: Spike Curtis <spike@coder.com>

---------

Signed-off-by: Spike Curtis <spike@coder.com>
2023-08-10 13:59:43 +04:00
4fb4c9b270 chore: add more rules to ensure logs consistency (#8104) 2023-06-21 12:00:38 +02:00
c12c9f1f4e chore(go.mod): update cdr.dev/slog (#7994)
* chore(mod): update cdr.dev/slog

* fix: change uses of []slog.Field to []any to match new API
2023-06-13 18:17:04 +00:00
0665a6c2f2 feat: add metric for provisioner daemons (#7858) 2023-06-06 16:50:11 -05:00
8bd9f9c351 feat: unified tracing between coderd<->provisionerd (#7370) 2023-05-03 23:02:35 +00:00
085330ad96 fix(provisionerd): only heartbeat when logs aren't being flushed (#7110) 2023-04-13 14:02:10 -05:00
a29fc7dd6f chore: update otel to v1.14.0 (#6963) 2023-04-03 00:31:39 -05:00
3b87316ad7 feat: propagate job error codes (#6507)
* feat: propagate job error_code

* fix

* Fix

* Fix

* Fix

* add errors to typesGenerated

* Address PR comments

* Fix
2023-03-08 16:32:00 +01:00
7f226d4f90 feat: add support for coder_git_auth data source (#6334)
* Add git auth providers schema

* Pipe git auth providers to the schema

* Add git auth providers to the API

* Add gitauth endpoint to query authenticated state

* Add endpoint to query git state

* Use BroadcastChannel to automatically authenticate with Git

* Add error validation for submitting the create workspace form

* Fix panic on template dry-run

* Add tests for the template version Git auth endpoint

* Show error if no gitauth is configured

* Add gitauth to cliui

* Fix unused method receiver

* Fix linting errors

* Fix dbauthz querier test

* Fix make gen

* Add JavaScript test for git auth

* Fix bad error message

* Fix provisionerd test race

See https://github.com/coder/coder/actions/runs/4277960646/jobs/7447232814

* Fix requested changes

* Add comment to CreateWorkspacePageView
2023-02-27 10:18:19 -06:00
473ab208af fix: replace owner_email with workspace_owner, change action to workspace_transition (#6337) 2023-02-24 04:38:58 +00:00
282507f0fb fix(provisionerd): add more detailed workspace build logging (#6331) 2023-02-24 00:03:12 +00:00
43e8ba0811 feat(api): add prometheus metric coderd_workspace_builds_total (#6314)
This PR adds the prometheus metric coderd_workspace_builds_total.
It measures the total number of workspace builds, along with a number of labels intended to be useful for an operator debugging a failed workspace build trying to discover the scope of the issue.
2023-02-23 01:28:10 +00:00
860e2829c5 fix: Prevent race between provisionerd connect and close (#6206)
* fix: Prevent race between provisionerd connect and close

* test: Add detection for provisioner creation after test completion
2023-02-14 16:37:43 +00:00
41ae01d2e9 fix: Improve closure of provisioner and agent tailnet dial (#6199) 2023-02-14 14:57:48 +00:00
f76ef98a32 chore!: Standardize prometheus time metrics to seconds (#5709)
* chore!: Standardize prometheus time metrics to seconds
* Update prometheus docs
2023-01-13 11:15:25 -06:00
6651c1632d fix: avoid terraform state concurrent access, remove global mutex (#5273) 2022-12-06 17:05:14 +00:00
ab3b3d5fca feat: add debouncing to provisionerd rpc calls (#5198) 2022-12-01 16:54:53 -06:00
883cf8afa9 chore: Add missing metrics description (#5212)
* chore: Add missing metrics description

* Update provisionerd/provisionerd.go

Co-authored-by: Mathias Fredriksson <mafredri@gmail.com>

* Fix

Co-authored-by: Mathias Fredriksson <mafredri@gmail.com>
2022-12-01 12:50:57 +01:00
1f20cab110 fix: don't use yamux for in-memory provisioner{,d} streams (#5136) 2022-11-22 12:19:32 -06:00
b6703b11c6 feat: Add external provisioner daemons (#4935)
* Start to port over provisioner daemons PR

* Move to Enterprise

* Begin adding tests for external registration

* Move provisioner daemons query to enterprise

* Move around provisioner daemons schema

* Add tags to provisioner daemons

* make gen

* Add user local provisioner daemons

* Add provisioner daemons

* Add feature for external daemons

* Add command to start a provisioner daemon

* Add provisioner tags to template push and create

* Rename migration files

* Fix tests

* Fix entitlements test

* PR comments

* Update migration

* Fix FE types
2022-11-16 16:34:06 -06:00
97dbd4dc5d Implement Quotas v3 (#5012)
* provisioner/terraform: add cost to resource_metadata

* provisionerd/runner: use Options struct

* Complete provisionerd implementation

* Add quota_allowance to groups

* Combine Quota and RBAC licenses

* Add Opts to InTx
2022-11-14 17:57:33 +00:00
95fb59696e Refactor Provisioner to distinguish Plan and Apply (#5036) 2022-11-11 16:45:58 -06:00
16384f8594 feat: Add provisioner force-cancel flag (#4947)
* feat: Add provisionerd force cancel flag

* Golden files

* Fix: typesGenerated.ts

* Use single struct for Provisioner config
2022-11-08 14:19:40 +01:00
53f2449e4f chore: Fix changes from buffer provisioner logs (#4924)
Comments from #4918 were missed because of auto-merge.
2022-11-06 23:59:01 -06:00
30281852d6 feat: Add buffering to provisioner job logs (#4918)
* feat: Add bufferring to provisioner job logs

This should improve overall build performance, and especially under load.

It removes the old `id` column on the `provisioner_job_logs` table
and replaces it with an auto-incrementing big integer to preserve order.

Funny enough, we never had to care about order before because inserts
would at minimum be 1ms different. Now they aren't, so the order needs
to be preserved.

* Fix log bufferring

* Fix frontend log streaming

* Fix JS test
2022-11-06 20:50:34 -06:00
e740aebf26 feat: add provisionerd prometheus metrics (#4909) 2022-11-04 19:03:01 -05:00
dde9a43b7e chore: fix otel dependency versions (#4813) 2022-10-31 19:16:16 +00:00
5de6f86959 feat: trace httpapi.{Read,Write} (#4134) 2022-09-21 17:07:00 -05:00
b340634aaa feat: add rbac tracing (#4093) 2022-09-16 18:32:15 +00:00
77acf0c340 feat: provisionerd tracing, add public trace ingestion (#4070) 2022-09-16 11:43:22 -05:00
f310aeb4cb Disable skipping job acquire log (#3240)
Signed-off-by: Spike Curtis <spike@coder.com>
2022-07-26 16:36:45 -07:00
22febc749a provisionerd sends failed or complete last (#2732)
* provisionerd sends failed or complete last

Signed-off-by: Spike Curtis <spike@coder.com>

* Move runner into package

Signed-off-by: Spike Curtis <spike@coder.com>

* Remove jobRunner interface

Signed-off-by: Spike Curtis <spike@coder.com>

* renames and slight reworking from code review

Signed-off-by: Spike Curtis <spike@coder.com>

* Reword comment about okToSend

Signed-off-by: Spike Curtis <spike@coder.com>
2022-07-01 09:55:46 -07:00
024ab6df57 fix: Use in-memory filesystem for echo provisioner tests (#2408)
* fix: Use in-memory filesystem for echo provisioner tests

This should reduce IO in CI to shave some time off tests!

* test: Increase timeouts to reduce flakes

It's difficult to understand what's timing out due to a lock
vs. taking a long time. This should help resolve! 🕵️
2022-06-16 15:09:22 +00:00
ccd061652b feat: Add built-in PostgreSQL for simple production setup (#2345)
* feat: Add built-in PostgreSQL for simple production setup

Fixes #2321.

* Use fork of embedded-postgres for cache path
2022-06-15 16:02:18 -05:00
45eb1b4980 feat: improve terraform template parsing errors (#2331) 2022-06-16 04:12:17 +10:00
6be8a373e0 feat: run a terraform plan before creating workspaces with the given template parameters (#1732) 2022-06-02 00:44:53 +10:00