It's possible for websocket close messages to be too long, which cause
them to silently fail without a proper close message. See error below:
```
2022-03-31 17:08:34.862 [INFO] (stdlib) <close_notjs.go:72> "2022/03/31 17:08:34 websocket: failed to marshal close frame: reason string max is 123 but got \"insert provisioner daemon:Cannot encode []database.ProvisionerType into oid 19098 - []database.ProvisionerType must implement Encoder or be converted to a string\" with length 161"
```
* feat: Add stage to build logs
This adds a stage property to logs, and refactors the job logs
cliui.
It also adds tests to the cliui for build logs!
* feat: Add stage to build logs
This adds a stage property to logs, and refactors the job logs
cliui.
It also adds tests to the cliui for build logs!
* feat: Add config-ssh and tests for resiliency
* Rename "Echo" test to "ImmediateExit"
* Fix Terraform resource agent association
* Fix logs post-cancel
* Fix select on Windows
* Remove terraform init logs
* Move timer into it's own loop
* Fix race condition in provisioner jobs
* Fix requested changes
* feat: Add AWS instance identity authentication
This allows zero-trust authentication for all AWS instances.
Prior to this, AWS instances could be used by passing `CODER_TOKEN`
as an environment variable to the startup script. AWS explicitly
states that secrets should not be passed in startup scripts because
it's user-readable.
* Fix sha256 verbosity
* Fix HTTP client being exposed on auth
* feat: Add stage to build logs
This adds a stage property to logs, and refactors the job logs
cliui.
It also adds tests to the cliui for build logs!
* Fix comments
* chore: Move httpmw to /coderd directory
httpmw is specific to coderd and should be scoped under coderd
* chore: Move httpapi to /coderd directory
httpapi is specific to coderd and should be scoped under coderd
* chore: Move database to /coderd directory
database is specific to coderd and should be scoped under coderd
* chore: Update codecov & gitattributes for generated files
* chore: Update Makefile
This update exposes the workspace name and owner, and changes
authentication methods to be explicit. Implicit authentication
added unnecessary complexity and introduced inconsistency.
* feat: Add systemd service and production deployment
This modifies CI to use a dpkg produced from release to update and
run Coder on a tiny VM in GCP.
It's intentionally kept simple, because customers should
be able to get this same easy install experience.
* Update globalSetup.ts
* Update globalSetup.ts
* Update globalSetup.ts
* Update coder.yaml
* Use pinned version of Go
* chore: Improve CI builds by caching Go modules
* Skip running with `race` on non-Linux systems
* Fix darwin file descriptor error
* Fix log after close
* Improve PostgreSQL test speeds
* Fix parallel connections with PostgreSQL tests
* Fix CI flake
* Separate test/go into PostgreSQL
Fix for #348 - migrate our NextJS project to a pure webpack project w/ a single bundle
- [x] Switch from `next/link` to `react-router-dom`'s link
> This part was easy - just change the import to `import { Link } from "react-router-dom"` and `<Link href={...} />` to `<Link to={...} />`
- [x] Switch from `next/router` to `react-router-dom`'s paradigms (`useNavigation`, `useLocation`, and `useParams`)
> `router.push` can be converted to `navigate(...)` (provided by the `useNavigate` hook)
> `router.replace` can be converted `navigate(..., {replace: true})`
> Query parameters (`const { query } = useRouter`) can be converted to `const query = useParams()`)
- [x] Implement client-side routing with `react-router-dom`
> Parameterized routes in NextJS like `projects/[organization]/[project]` would look like:
> ```
> <Route path="projects">
> <Route path=":organization/:project">
> <Route index element={<ProjectPage />} />
> </Route>
> </Route>
> ```
I've hooked up a `build:analyze` command that spins up a server to show the bundle size:
<img width="1303" alt="image" src="https://user-images.githubusercontent.com/88213859/157496889-87c5fdcd-fad1-4f2e-b7b6-437aebf99641.png">
The bundle looks OK, but there are some opportunities for improvement - the heavy-weight dependencies, like React, ReactDOM, Material-UI, and lodash could be brought in via a CDN: https://stackoverflow.com/questions/50645796/how-to-import-reactjs-material-ui-using-a-cdn-through-webpacks-externals
* ci: Update DataDog GitHub branch to fallback to GITHUB_REF
This was detecting branches, but not our "main" branch before.
Hopefully this fixes it!
* Add basic Terraform Provider
* Rename post files to upload
* Add tests for resources
* Skip instance identity test
* Add tests for ensuring agent get's passed through properly
* Fix linting errors
* Add echo path
* Fix agent authentication
* fix: Convert all jobs to use a common resource and agent type
This enables a consistent API for project import and provisioned resources.
* Add "coder_workspace" data source
* feat: Remove magical parameters from being injected
This is a much cleaner abstraction. Explicitly declaring the user
parameters for each provisioner makes for significantly simpler
testing.
* feat: Add graceful exits to provisionerd
Terraform (or other provisioners) may need to cleanup state, or
cancel actions before exit. This adds the ability to gracefully
exit provisionerd.
* Fix cancel error check
* feat: Add destroy to workspace provision job
This enables the full flow of create/update/delete.
* ci: Update DataDog GitHub branch to fallback to GITHUB_REF
This was detecting branches, but not our "main" branch before.
Hopefully this fixes it!
* Add basic Terraform Provider
* Rename post files to upload
* Add tests for resources
* Skip instance identity test
* Add tests for ensuring agent get's passed through properly
* Fix linting errors
* Add echo path
* Fix agent authentication
* fix: Convert all jobs to use a common resource and agent type
This enables a consistent API for project import and provisioned resources.
* Add "coder_workspace" data source
* feat: Remove magical parameters from being injected
This is a much cleaner abstraction. Explicitly declaring the user
parameters for each provisioner makes for significantly simpler
testing.
* feat: Add graceful exits to provisionerd
Terraform (or other provisioners) may need to cleanup state, or
cancel actions before exit. This adds the ability to gracefully
exit provisionerd.
* Fix cancel error check
* ci: Update DataDog GitHub branch to fallback to GITHUB_REF
This was detecting branches, but not our "main" branch before.
Hopefully this fixes it!
* Add basic Terraform Provider
* Rename post files to upload
* Add tests for resources
* Skip instance identity test
* Add tests for ensuring agent get's passed through properly
* Fix linting errors
* Add echo path
* Fix agent authentication
* fix: Convert all jobs to use a common resource and agent type
This enables a consistent API for project import and provisioned resources.
* Add "coder_workspace" data source
* feat: Remove magical parameters from being injected
This is a much cleaner abstraction. Explicitly declaring the user
parameters for each provisioner makes for significantly simpler
testing.
* ci: Update DataDog GitHub branch to fallback to GITHUB_REF
This was detecting branches, but not our "main" branch before.
Hopefully this fixes it!
* Add basic Terraform Provider
* Rename post files to upload
* Add tests for resources
* Skip instance identity test
* Add tests for ensuring agent get's passed through properly
* Fix linting errors
* Add echo path
* Fix agent authentication
* fix: Convert all jobs to use a common resource and agent type
This enables a consistent API for project import and provisioned resources.
* feat: Add agent authentication based on instance ID
Each cloud has it's own unique instance identity signatures, which
can be used for zero-token authentication. This change adds support
for tracking by "instance_id", and automatically authenticating
with Google Cloud.
* Add test for CLI
* Fix workspace agent request name
* Fix race with adding to wait group
* Fix name of instance identity token
WebSockets hijack the HTTP connection from the server, causing
server.Close() to not wait for these connections to fully cleanup.
This adds a global wait-group to the coderd API, which ensures all
WebSocket HTTP handlers have properly exited before returning.
* feat: Add workspace agent for SSH
This adds the initial agent that supports TTY
and execution over SSH. It functions across MacOS,
Windows, and Linux.
This does not handle the coderd interaction yet,
but does setup a simple path forward.
* Fix pty tests on Windows
* Fix log race
* Lock around dial error to fix log output
* Fix context return early
* fix: Leaking yamux session after HTTP handler is closed
Closes#317. We depended on the context canceling the yamux connection,
but this isn't a sync operation. Explicitly calling close ensures the
handler waits for yamux to complete before exit.
* Lock around close return
* Force failure with log
* Fix failed handler
* Upgrade dep
* Fix defer inside loops
* Fix context cancel for HTTP requests
* Fix resize
* fix: Leaking yamux session after HTTP handler is closed
Closes#317. The httptest server cancels the context after the connection
is closed, but if a connection takes a long time to close, the request
would never end. This applies a context to the entire listener that cancels
on test cleanup.
After discussion with @bryphe-coder, reducing the parallel limit on
Windows is likely to reduce failures as well.
* Switch to windows-2022 to improve decompression
* Invalidate cache on matrix OS
Fixes#210 - this isPR implements `coder login` in the case where the default user is already created.
This change adds:
- A prompt in the case where there is not an initial user that opens the server URL + requests a session token
- This ports over some code from v1 for the `openURL` and `isWSL` functions to support opening the browser
- A `/api/v2/api-keys` endpoint that can be `POST`'d to in order to request a new api key for a user
- This route was inspired by the v1 functionality
- A `cli-auth` route + page that shows the generated api key
- Tests for the new code + storybook for the new UI
The `/cli-auth` route, like in v1, is very minimal:
<img width="624" alt="Screen Shot 2022-02-16 at 5 05 07 PM" src="https://user-images.githubusercontent.com/88213859/154384627-78ab9841-27bf-490f-9bbe-23f8173c9e97.png">
And the terminal UX looks like this:
* Initial agent
* fix: Use buffered reader in peer to fix ShortBuffer
This prevents a io.ErrShortBuffer from occurring when the byte
slice being read is smaller than the chunks sent from the opposite
pipe.
This makes sense for unordered connections, where transmission is
not guarunteed, but does not make sense for TCP-like connections.
We use a bufio.Reader when ordered to ensure data isn't lost.
* SSH server works!
* Start Windows support
* Something works
* Refactor pty package to support Windows spawn
* SSH server now works on Windows
* Fix non-Windows
* Fix Linux PTY render
* FIx linux build tests
* Remove agent and wintest
* Add test for Windows resize
* Fix linting errors
* Add Windows environment variables
* Add strings import
* Add comment for attrs
* Add goleak
* Add require import
* Refactor parameter parsing to return nil values if none computed
* Refactor parameter to allow for hiding redisplay
* Refactor parameters to enable schema matching
* Refactor provisionerd to dynamically update parameter schemas
* Refactor job update for provisionerd
* Handle multiple states correctly when provisioning a project
* Add project import job resource table
* Basic creation flow works!
* Create project fully works!!!
* Only show job status if completed
* Add create workspace support
* Replace Netflix/go-expect with ActiveState
* Fix linting errors
* Use forked chzyer/readline
* Add create workspace CLI
* Add CLI test
* Move jobs to their own APIs
* Remove go-expect
* Fix requested changes
* Skip workspacecreate test on windows
An issue came up last week... our `embed.go` strategy doesn't handle dynamic NextJS-style routes! This is a blocker, because I'm aiming to set up CD on Monday, and the v2 UI makes heavy use of dynamic routing.
As a potential solution, this implements a go pkg `nextrouter` that serves `html` files, but respecting the dynamic routing behavior of NextJS:
- Files that have square brackets - ie `[providers]` provide a single-level dynamic route
- Files that have `[[...` prefix - ie `[[...any]]` - are catch-all routes.
- Files should be preferred over folders (ie, `providers.html` is preferred over `/providers`)
- Fixes the trailing-slash bug we hit in the previous `embed` strategy
This also integrates with `slog.Logger` for tracing, and handles injecting template parameters - a feature we need in v1 and v2 to be able to inject stuff like CSRF tokens.
This implements testing by using an in-memory file-system, so that we can exercise all of these cases.
In addition, this adjust V2's `embed.go` strategy to use `nextrouter`, which simplifies that file considerably. I'm tempted to factor out the `secureheaders` logic into a separate package, too.
If this works OK, it could be used for V1 too (although that scenario is more complex due to our hybrid-routing strategy). Based on our FE variety meeting, there's always a chance we could move away from NextJS in v1 - if that's the case, this router will still work and be more tested than our previous strategy (it just won't make use of dynamic routing). So I figured this was worth doing to make sure we can make forward progress in V2.
* Nest jobs under an organization
* Rename project parameter to parameter schema
* Update references when computing project parameters
* Add files endpoint
* Allow one-off project import jobs
* Allow variables to be injected that are not defined by the schema
* Update API to use jobs first
* Fix CLI tests
* Fix linting
* Fix hex length for files table
* Reduce memory allocation for windows
* refactor: Rename ProjectParameter to ProjectVersionParameter
This was confusing with ParameterValue before. It still is a bit,
but this should help distinguish scope.
* Add project version resources table
* Allow project parameters to optionally have user and workspace
* Add dry run for provisioners
* Add resource detection on project import
* refactor: Generalize log ownership to allow for scratch jobs
Importing may fail when creating a project. We don't want to lose this output,
but we don't want to allow users to create a failing project.
This generalizes logs to soon enable one-off situations where a user can upload
their archive, create a project, and watch the output parse to completion.
* Improve file table schema by using hash
* Fix racey test by allowing logs before
* Add debug logging for PostgreSQL insert
* ci: Replace DataDog CI with custom upload script
This will reduce CI time by ~6 minutes across all of
our runners. It's a bit janky, but I believe worth
the slight maintainance burden.
* Fix test race when job would complete too early
* Fix job cancelation override
* Fix race where provisioner job is inserted before project version
* chore: Rename ProjectHistory to ProjectVersion
Version more accurately represents version storage. This
forks from the WorkspaceHistory name, but I think it's
easier to understand Workspace history.
* Rename files
* Standardize tests a bit more
* Remove Server struct from coderdtest
* Improve test coverage for workspace history
* Fix linting errors
* Fix coderd test leak
* Fix coderd test leak
* Improve workspace history logs
* Standardize test structure for codersdk
* Fix linting errors
* Fix WebSocket compression
* Update coderd/workspaces.go
Co-authored-by: Bryan <bryan@coder.com>
* Add test for listing project parameters
* Cache npm dependencies with setup node
* Remove windows npm cache key
Co-authored-by: Bryan <bryan@coder.com>
* chore: Rename ProjectHistory to ProjectVersion
Version more accurately represents version storage. This
forks from the WorkspaceHistory name, but I think it's
easier to understand Workspace history.
* Rename files
This replaces the cdr-basic provisioner type with
"echo". It reads binary data from the directory
and returns the responses in order.
This is used to test project and workspace job logic.
* feat: Add parameter querying to the API
* feat: Add streaming endpoint for workspace history
Enables a buildlog-like flow for reading job output.
* Fix empty parameter source and destination
* Add comment for usage of workspace history logs endpoint
