Commit Graph

57 Commits

Author SHA1 Message Date
3b87316ad7 feat: propagate job error codes (#6507)
* feat: propagate job error_code

* fix

* Fix

* Fix

* Fix

* add errors to typesGenerated

* Address PR comments

* Fix
2023-03-08 16:32:00 +01:00
7f226d4f90 feat: add support for coder_git_auth data source (#6334)
* Add git auth providers schema

* Pipe git auth providers to the schema

* Add git auth providers to the API

* Add gitauth endpoint to query authenticated state

* Add endpoint to query git state

* Use BroadcastChannel to automatically authenticate with Git

* Add error validation for submitting the create workspace form

* Fix panic on template dry-run

* Add tests for the template version Git auth endpoint

* Show error if no gitauth is configured

* Add gitauth to cliui

* Fix unused method receiver

* Fix linting errors

* Fix dbauthz querier test

* Fix make gen

* Add JavaScript test for git auth

* Fix bad error message

* Fix provisionerd test race

See https://github.com/coder/coder/actions/runs/4277960646/jobs/7447232814

* Fix requested changes

* Add comment to CreateWorkspacePageView
2023-02-27 10:18:19 -06:00
473ab208af fix: replace owner_email with workspace_owner, change action to workspace_transition (#6337) 2023-02-24 04:38:58 +00:00
282507f0fb fix(provisionerd): add more detailed workspace build logging (#6331) 2023-02-24 00:03:12 +00:00
43e8ba0811 feat(api): add prometheus metric coderd_workspace_builds_total (#6314)
This PR adds the prometheus metric coderd_workspace_builds_total.
It measures the total number of workspace builds, along with a number of labels intended to be useful for an operator debugging a failed workspace build trying to discover the scope of the issue.
2023-02-23 01:28:10 +00:00
860e2829c5 fix: Prevent race between provisionerd connect and close (#6206)
* fix: Prevent race between provisionerd connect and close

* test: Add detection for provisioner creation after test completion
2023-02-14 16:37:43 +00:00
41ae01d2e9 fix: Improve closure of provisioner and agent tailnet dial (#6199) 2023-02-14 14:57:48 +00:00
f76ef98a32 chore!: Standardize prometheus time metrics to seconds (#5709)
* chore!: Standardize prometheus time metrics to seconds
* Update prometheus docs
2023-01-13 11:15:25 -06:00
6651c1632d fix: avoid terraform state concurrent access, remove global mutex (#5273) 2022-12-06 17:05:14 +00:00
ab3b3d5fca feat: add debouncing to provisionerd rpc calls (#5198) 2022-12-01 16:54:53 -06:00
883cf8afa9 chore: Add missing metrics description (#5212)
* chore: Add missing metrics description

* Update provisionerd/provisionerd.go

Co-authored-by: Mathias Fredriksson <mafredri@gmail.com>

* Fix

Co-authored-by: Mathias Fredriksson <mafredri@gmail.com>
2022-12-01 12:50:57 +01:00
1f20cab110 fix: don't use yamux for in-memory provisioner{,d} streams (#5136) 2022-11-22 12:19:32 -06:00
b6703b11c6 feat: Add external provisioner daemons (#4935)
* Start to port over provisioner daemons PR

* Move to Enterprise

* Begin adding tests for external registration

* Move provisioner daemons query to enterprise

* Move around provisioner daemons schema

* Add tags to provisioner daemons

* make gen

* Add user local provisioner daemons

* Add provisioner daemons

* Add feature for external daemons

* Add command to start a provisioner daemon

* Add provisioner tags to template push and create

* Rename migration files

* Fix tests

* Fix entitlements test

* PR comments

* Update migration

* Fix FE types
2022-11-16 16:34:06 -06:00
97dbd4dc5d Implement Quotas v3 (#5012)
* provisioner/terraform: add cost to resource_metadata

* provisionerd/runner: use Options struct

* Complete provisionerd implementation

* Add quota_allowance to groups

* Combine Quota and RBAC licenses

* Add Opts to InTx
2022-11-14 17:57:33 +00:00
95fb59696e Refactor Provisioner to distinguish Plan and Apply (#5036) 2022-11-11 16:45:58 -06:00
16384f8594 feat: Add provisioner force-cancel flag (#4947)
* feat: Add provisionerd force cancel flag

* Golden files

* Fix: typesGenerated.ts

* Use single struct for Provisioner config
2022-11-08 14:19:40 +01:00
53f2449e4f chore: Fix changes from buffer provisioner logs (#4924)
Comments from #4918 were missed because of auto-merge.
2022-11-06 23:59:01 -06:00
30281852d6 feat: Add buffering to provisioner job logs (#4918)
* feat: Add bufferring to provisioner job logs

This should improve overall build performance, and especially under load.

It removes the old `id` column on the `provisioner_job_logs` table
and replaces it with an auto-incrementing big integer to preserve order.

Funny enough, we never had to care about order before because inserts
would at minimum be 1ms different. Now they aren't, so the order needs
to be preserved.

* Fix log bufferring

* Fix frontend log streaming

* Fix JS test
2022-11-06 20:50:34 -06:00
e740aebf26 feat: add provisionerd prometheus metrics (#4909) 2022-11-04 19:03:01 -05:00
dde9a43b7e chore: fix otel dependency versions (#4813) 2022-10-31 19:16:16 +00:00
5de6f86959 feat: trace httpapi.{Read,Write} (#4134) 2022-09-21 17:07:00 -05:00
b340634aaa feat: add rbac tracing (#4093) 2022-09-16 18:32:15 +00:00
77acf0c340 feat: provisionerd tracing, add public trace ingestion (#4070) 2022-09-16 11:43:22 -05:00
f310aeb4cb Disable skipping job acquire log (#3240)
Signed-off-by: Spike Curtis <spike@coder.com>
2022-07-26 16:36:45 -07:00
22febc749a provisionerd sends failed or complete last (#2732)
* provisionerd sends failed or complete last

Signed-off-by: Spike Curtis <spike@coder.com>

* Move runner into package

Signed-off-by: Spike Curtis <spike@coder.com>

* Remove jobRunner interface

Signed-off-by: Spike Curtis <spike@coder.com>

* renames and slight reworking from code review

Signed-off-by: Spike Curtis <spike@coder.com>

* Reword comment about okToSend

Signed-off-by: Spike Curtis <spike@coder.com>
2022-07-01 09:55:46 -07:00
024ab6df57 fix: Use in-memory filesystem for echo provisioner tests (#2408)
* fix: Use in-memory filesystem for echo provisioner tests

This should reduce IO in CI to shave some time off tests!

* test: Increase timeouts to reduce flakes

It's difficult to understand what's timing out due to a lock
vs. taking a long time. This should help resolve! 🕵️
2022-06-16 15:09:22 +00:00
ccd061652b feat: Add built-in PostgreSQL for simple production setup (#2345)
* feat: Add built-in PostgreSQL for simple production setup

Fixes #2321.

* Use fork of embedded-postgres for cache path
2022-06-15 16:02:18 -05:00
45eb1b4980 feat: improve terraform template parsing errors (#2331) 2022-06-16 04:12:17 +10:00
6be8a373e0 feat: run a terraform plan before creating workspaces with the given template parameters (#1732) 2022-06-02 00:44:53 +10:00
e2ed581708 Add stages to all proto.Logs (#1563) 2022-05-18 17:33:29 +00:00
98ccd0eb89 feat: add README parsing to template versions (#1500) 2022-05-17 15:00:48 -05:00
914a2f477c fix: Restore terminal on interrupt when connecting (#1312)
Fixes #1292.
2022-05-05 20:11:44 -05:00
877854a2f3 fix: Display proper access URL on server start (#1172)
Fixes #1170.
2022-04-25 23:30:22 +00:00
66d45f391e test: Check if provisionerd is closed before setting run chan (#1145)
This race can be seen here:
https://github.com/coder/coder/runs/6159662393?check_suite_focus=true
2022-04-25 12:00:54 -05:00
a6ea99541e fix: Return from update loop when job completes (#1121)
Update was running forever, which stopped jobs from timing
out unless a restart occurred. This also fixes complete properly
reporting an error.
2022-04-25 03:41:03 +00:00
db7ed4d019 fix: Add resiliency to daemon connections (#1116)
Connections could fail when massive payloads were transmitted.
This fixes an upstream bug in dRPC where the connection would
end with a context canceled if a message was too large.

This adds retransmission of completion and failures too. If
Coder somehow loses connection with a provisioner daemon,
upon the next connection the state will be properly reported.
2022-04-24 20:33:19 -05:00
fb9dc4f346 feat: Improve resource preview and first-time experience (#946)
* Improve CLI documentation

* feat: Allow workspace resources to attach multiple agents

This enables a "kubernetes_pod" to attach multiple agents that
could be for multiple services. Each agent is required to have
a unique name, so SSH syntax is:

`coder ssh <workspace>.<agent>`

A resource can have zero agents too, they aren't required.

* Add tree view

* Improve table UI

* feat: Allow workspace resources to attach multiple agents

This enables a "kubernetes_pod" to attach multiple agents that
could be for multiple services. Each agent is required to have
a unique name, so SSH syntax is:

`coder ssh <workspace>.<agent>`

A resource can have zero agents too, they aren't required.

* Rename `tunnel` to `skip-tunnel`

This command was `true` by default, which causes
a confusing user experience.

* Add disclaimer about editing templates

* Add help to template create

* Improve workspace create flow

* Add end-to-end test for config-ssh

* Improve testing of config-ssh

* Fix workspace list

* Fix config ssh tests

* Update cli/configssh.go

Co-authored-by: Cian Johnston <public@cianjohnston.ie>

* Fix requested changes

* Remove socat requirement

* Fix resources not reading in TTY

Co-authored-by: Cian Johnston <public@cianjohnston.ie>
2022-04-11 18:54:30 -05:00
02ad3f14f5 chore: Rename Projects to Templates (#880)
Customer feedback indicated projects was a confusing name.
After querying the team internally, it seemed unanimous
that it is indeed a confusing name.

Here's for a lil less confusion @ashmeer7 🥂
2022-04-06 12:42:40 -05:00
b33dec9d38 feat: Add stage to build logs (#577)
* feat: Add stage to build logs

This adds a stage property to logs, and refactors the job logs
cliui.

It also adds tests to the cliui for build logs!

* Fix comments
2022-03-28 18:43:22 +00:00
26d24f4508 chore: Improve CI builds by caching Go modules (#528)
* chore: Improve CI builds by caching Go modules

* Skip running with `race` on non-Linux systems

* Fix darwin file descriptor error

* Fix log after close

* Improve PostgreSQL test speeds

* Fix parallel connections with PostgreSQL tests

* Fix CI flake

* Separate test/go into PostgreSQL
2022-03-22 17:09:04 -05:00
c451f4e685 feat: Add templates to create working release (#422)
* Add templates

* Move API structs to codersdk

* Back to green tests!

* It all works, but now with tea! 🧋

* It works!

* Add cancellation to provisionerd

* Tests pass!

* Add deletion of workspaces and projects

* Fix agent lock

* Add clog

* Fix linting errors

* Remove unused CLI tests

* Rename daemon to start

* Fix leaking command

* Fix promptui test

* Update agent connection frequency

* Skip login tests on Windows

* Increase tunnel connect timeout

* Fix templater

* Lower test requirements

* Fix embed

* Disable promptui tests for Windows

* Fix write newline

* Fix PTY write newline

* Fix CloseReader

* Fix compilation on Windows

* Fix linting error

* Remove bubbletea

* Cleanup readwriter

* Use embedded templates instead of serving over API

* Move templates to examples

* Improve workspace create flow

* Fix Windows build

* Fix tests

* Fix linting errors

* Fix untar with extracting max size

* Fix newline char
2022-03-22 13:17:50 -06:00
bf0ae8f573 feat: Refactor API routes to use UUIDs instead of friendly names (#401)
* Add client for agent

* Cleanup code

* Fix linting error

* Rename routes to be simpler

* Rename workspace history to workspace build

* Refactor HTTP middlewares to use UUIDs

* Cleanup routes

* Compiles!

* Fix files and organizations

* Fix querying

* Fix agent lock

* Cleanup database abstraction

* Add parameters

* Fix linting errors

* Fix log race

* Lock on close wait

* Fix log cleanup

* Fix e2e tests

* Fix upstream version of opencensus-go

* Update coderdtest.go

* Fix coverpkg

* Fix codecov ignore
2022-03-07 11:40:54 -06:00
9d2803e07a feat: Add graceful exits to provisionerd (#372)
* ci: Update DataDog GitHub branch to fallback to GITHUB_REF

This was detecting branches, but not our "main" branch before.
Hopefully this fixes it!

* Add basic Terraform Provider

* Rename post files to upload

* Add tests for resources

* Skip instance identity test

* Add tests for ensuring agent get's passed through properly

* Fix linting errors

* Add echo path

* Fix agent authentication

* fix: Convert all jobs to use a common resource and agent type

This enables a consistent API for project import and provisioned resources.

* Add "coder_workspace" data source

* feat: Remove magical parameters from being injected

This is a much cleaner abstraction. Explicitly declaring the user
parameters for each provisioner makes for significantly simpler
testing.

* feat: Add graceful exits to provisionerd

Terraform (or other provisioners) may need to cleanup state, or
cancel actions before exit. This adds the ability to gracefully
exit provisionerd.

* Fix cancel error check
2022-02-28 18:40:49 +00:00
e5c95552cd feat: Remove magical parameters from being injected (#371)
* ci: Update DataDog GitHub branch to fallback to GITHUB_REF

This was detecting branches, but not our "main" branch before.
Hopefully this fixes it!

* Add basic Terraform Provider

* Rename post files to upload

* Add tests for resources

* Skip instance identity test

* Add tests for ensuring agent get's passed through properly

* Fix linting errors

* Add echo path

* Fix agent authentication

* fix: Convert all jobs to use a common resource and agent type

This enables a consistent API for project import and provisioned resources.

* Add "coder_workspace" data source

* feat: Remove magical parameters from being injected

This is a much cleaner abstraction. Explicitly declaring the user
parameters for each provisioner makes for significantly simpler
testing.
2022-02-28 18:26:01 +00:00
65de96c8b4 fix: Leaking yamux session after HTTP handler is closed (#329)
* fix: Leaking yamux session after HTTP handler is closed

Closes #317. The httptest server cancels the context after the connection
is closed, but if a connection takes a long time to close, the request
would never end. This applies a context to the entire listener that cancels
on test cleanup.

After discussion with @bryphe-coder, reducing the parallel limit on
Windows is likely to reduce failures as well.

* Switch to windows-2022 to improve decompression

* Invalidate cache on matrix OS
2022-02-18 22:06:56 -06:00
154b9bce57 feat: Add "coder projects create" command (#246)
* Refactor parameter parsing to return nil values if none computed

* Refactor parameter to allow for hiding redisplay

* Refactor parameters to enable schema matching

* Refactor provisionerd to dynamically update parameter schemas

* Refactor job update for provisionerd

* Handle multiple states correctly when provisioning a project

* Add project import job resource table

* Basic creation flow works!

* Create project fully works!!!

* Only show job status if completed

* Add create workspace support

* Replace Netflix/go-expect with ActiveState

* Fix linting errors

* Use forked chzyer/readline

* Add create workspace CLI

* Add CLI test

* Move jobs to their own APIs

* Remove go-expect

* Fix requested changes

* Skip workspacecreate test on windows
2022-02-12 13:34:04 -06:00
7364933e65 refactor: Allow provisioner jobs to be disconnected from projects (#194)
* Nest jobs under an organization

* Rename project parameter to parameter schema

* Update references when computing project parameters

* Add files endpoint

* Allow one-off project import jobs

* Allow variables to be injected that are not defined by the schema

* Update API to use jobs first

* Fix CLI tests

* Fix linting

* Fix hex length for files table

* Reduce memory allocation for windows
2022-02-08 12:00:44 -06:00
795bba2af4 feat: Add dry run for provisioners (#178)
* refactor: Rename ProjectParameter to ProjectVersionParameter

This was confusing with ParameterValue before. It still is a bit,
but this should help distinguish scope.

* Add project version resources table

* Allow project parameters to optionally have user and workspace

* Add dry run for provisioners

* Add resource detection on project import
2022-02-07 19:35:18 -06:00
ed705f6af2 refactor: Generalize log ownership to allow for scratch jobs (#182)
* refactor: Generalize log ownership to allow for scratch jobs

Importing may fail when creating a project. We don't want to lose this output,
but we don't want to allow users to create a failing project.

This generalizes logs to soon enable one-off situations where a user can upload
their archive, create a project, and watch the output parse to completion.

* Improve file table schema by using hash

* Fix racey test by allowing logs before

* Add debug logging for PostgreSQL insert
2022-02-07 15:32:37 -06:00
1796dc6c2f chore: Add test helpers to improve coverage (#166)
* chore: Rename ProjectHistory to ProjectVersion

Version more accurately represents version storage. This
forks from the WorkspaceHistory name, but I think it's
easier to understand Workspace history.

* Rename files

* Standardize tests a bit more

* Remove Server struct from coderdtest

* Improve test coverage for workspace history

* Fix linting errors

* Fix coderd test leak

* Fix coderd test leak

* Improve workspace history logs

* Standardize test structure for codersdk

* Fix linting errors

* Fix WebSocket compression

* Update coderd/workspaces.go

Co-authored-by: Bryan <bryan@coder.com>

* Add test for listing project parameters

* Cache npm dependencies with setup node

* Remove windows npm cache key

Co-authored-by: Bryan <bryan@coder.com>
2022-02-05 18:24:51 -06:00