Files
coder/docs/admin/notifications.md

293 lines
14 KiB
Markdown
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Notifications
Notifications are sent by Coder in response to specific internal events, such as
a workspace being deleted or a user being created.
**Notifications are currently an experimental feature.**
## Enable experiment
In order to activate the notifications feature, you'll need to enable the
`notifications` experiment.
```bash
# Using the CLI flag
$ coder server --experiments=notifications
# Alternatively, using the `CODER_EXPERIMENTS` environment variable
$ CODER_EXPERIMENTS=notifications coder server
```
More information on experiments can be found
[here](https://coder.com/docs/contributing/feature-stages#experimental-features).
## Event Types
Notifications are sent in response to internal events, to alert the affected
user(s) of this event. Currently we support the following list of events:
### Workspace Events
_These notifications are sent to the workspace owner._
- Workspace Deleted
- Workspace Manual Build Failure
- Workspace Automatic Build Failure
- Workspace Automatically Updated
- Workspace Dormant
- Workspace Marked For Deletion
### User Events
_These notifications are sent to users with **owner** and **user admin** roles._
- User Account Created
- User Account Deleted
- User Account Suspended
- User Account Activated
- _(coming soon) User Password Reset_
- _(coming soon) User Email Verification_
_These notifications are sent to the user themselves._
- User Account Suspended
- User Account Activated
### Template Events
_These notifications are sent to users with **template admin** roles._
- Template Deleted
## Configuration
You can modify the notification delivery behavior using the following server
flags.
| Required | CLI | Env | Type | Description | Default |
| :------: | ----------------------------------- | --------------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------- | ------- |
| ✔️ | `--notifications-dispatch-timeout` | `CODER_NOTIFICATIONS_DISPATCH_TIMEOUT` | `duration` | How long to wait while a notification is being sent before giving up. | 1m |
| ✔️ | `--notifications-method` | `CODER_NOTIFICATIONS_METHOD` | `string` | Which delivery method to use (available options: 'smtp', 'webhook'). See [Delivery Methods](#delivery-methods) below. | smtp |
| - | `--notifications-max-send-attempts` | `CODER_NOTIFICATIONS_MAX_SEND_ATTEMPTS` | `int` | The upper limit of attempts to send a notification. | 5 |
## Delivery Methods
Notifications can currently be delivered by either SMTP or webhook. Each message
can only be delivered to one method, and this method is configured globally with
[`CODER_NOTIFICATIONS_METHOD`](https://coder.com/docs/reference/cli/server#--notifications-method)
(default: `smtp`).
Enterprise customers can configured which method to use for each of the
supported [Events](#events); see the [Preferences](#preferences) section below
for more details.
## SMTP (Email)
Use the `smtp` method to deliver notifications by email to your users. Coder
does not ship with an SMTP server, so you will need to configure Coder to use an
existing one.
**Server Settings:**
| Required | CLI | Env | Type | Description | Default |
| :------: | --------------------------------- | ------------------------------------- | ----------- | ----------------------------------------- | ------------- |
| ✔️ | `--notifications-email-from` | `CODER_NOTIFICATIONS_EMAIL_FROM` | `string` | The sender's address to use. | |
| ✔️ | `--notifications-email-smarthost` | `CODER_NOTIFICATIONS_EMAIL_SMARTHOST` | `host:port` | The SMTP relay to send messages through. | localhost:587 |
| - | `--notifications-email-hello` | `CODER_NOTIFICATIONS_EMAIL_HELLO` | `string` | The hostname identifying the SMTP server. | localhost |
**Authentication Settings:**
| Required | CLI | Env | Type | Description |
| :------: | ------------------------------------------ | ---------------------------------------------- | -------- | ------------------------------------------------------------------------- |
| - | `--notifications-email-auth-username` | `CODER_NOTIFICATIONS_EMAIL_AUTH_USERNAME` | `string` | Username to use with PLAIN/LOGIN authentication. |
| - | `--notifications-email-auth-password` | `CODER_NOTIFICATIONS_EMAIL_AUTH_PASSWORD` | `string` | Password to use with PLAIN/LOGIN authentication. |
| - | `--notifications-email-auth-password-file` | `CODER_NOTIFICATIONS_EMAIL_AUTH_PASSWORD_FILE` | `string` | File from which to load password for use with PLAIN/LOGIN authentication. |
| - | `--notifications-email-auth-identity` | `CODER_NOTIFICATIONS_EMAIL_AUTH_IDENTITY` | `string` | Identity to use with PLAIN authentication. |
**TLS Settings:**
| Required | CLI | Env | Type | Description | Default |
| :------: | ----------------------------------------- | ------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| - | `--notifications-email-force-tls` | `CODER_NOTIFICATIONS_EMAIL_FORCE_TLS` | `bool` | Force a TLS connection to the configured SMTP smarthost. If port 465 is used, TLS will be forced. See https://datatracker.ietf.org/doc/html/rfc8314#section-3.3. | false |
| - | `--notifications-email-tls-starttls` | `CODER_NOTIFICATIONS_EMAIL_TLS_STARTTLS` | `bool` | Enable STARTTLS to upgrade insecure SMTP connections using TLS. Ignored if `CODER_NOTIFICATIONS_EMAIL_FORCE_TLS` is set. | false |
| - | `--notifications-email-tls-skip-verify` | `CODER_NOTIFICATIONS_EMAIL_TLS_SKIPVERIFY` | `bool` | Skip verification of the target server's certificate (**insecure**). | false |
| - | `--notifications-email-tls-server-name` | `CODER_NOTIFICATIONS_EMAIL_TLS_SERVERNAME` | `string` | Server name to verify against the target certificate. | |
| - | `--notifications-email-tls-cert-file` | `CODER_NOTIFICATIONS_EMAIL_TLS_CERTFILE` | `string` | Certificate file to use. | |
| - | `--notifications-email-tls-cert-key-file` | `CODER_NOTIFICATIONS_EMAIL_TLS_CERTKEYFILE` | `string` | Certificate key file to use. | |
**NOTE:** you _MUST_ use `CODER_NOTIFICATIONS_EMAIL_FORCE_TLS` if your smarthost
supports TLS on a port other than `465`.
### Send emails using G-Suite
After setting the required fields above:
1. Create an [App Password](https://myaccount.google.com/apppasswords) using the
account you wish to send from
2. Set the following configuration options:
```
CODER_NOTIFICATIONS_EMAIL_SMARTHOST=smtp.gmail.com:465
CODER_NOTIFICATIONS_EMAIL_AUTH_USERNAME=<user>@<domain>
CODER_NOTIFICATIONS_EMAIL_AUTH_PASSWORD="<app password created above>"
```
See
[this help article from Google](https://support.google.com/a/answer/176600?hl=en)
for more options.
### Send emails using Outlook.com
After setting the required fields above:
1. Setup an account on Microsoft 365 or outlook.com
2. Set the following configuration options:
```
CODER_NOTIFICATIONS_EMAIL_SMARTHOST=smtp-mail.outlook.com:587
CODER_NOTIFICATIONS_EMAIL_TLS_STARTTLS=true
CODER_NOTIFICATIONS_EMAIL_AUTH_USERNAME=<user>@<domain>
CODER_NOTIFICATIONS_EMAIL_AUTH_PASSWORD="<account password>"
```
See
[this help article from Microsoft](https://support.microsoft.com/en-us/office/pop-imap-and-smtp-settings-for-outlook-com-d088b986-291d-42b8-9564-9c414e2aa040)
for more options.
## Webhook
The webhook delivery method sends an HTTP POST request to the defined endpoint.
The purpose of webhook notifications is to enable integrations with other
systems.
**Settings**:
| Required | CLI | Env | Type | Description |
| :------: | ---------------------------------- | -------------------------------------- | ----- | --------------------------------------- |
| ✔️ | `--notifications-webhook-endpoint` | `CODER_NOTIFICATIONS_WEBHOOK_ENDPOINT` | `url` | The endpoint to which to send webhooks. |
Here is an example payload for Coder's webhook notification:
```json
{
"_version": "1.0",
"msg_id": "88750cad-77d4-4663-8bc0-f46855f5019b",
"payload": {
"_version": "1.0",
"notification_name": "Workspace Deleted",
"user_id": "4ac34fcb-8155-44d5-8301-e3cd46e88b35",
"user_email": "danny@coder.com",
"user_name": "danny",
"user_username": "danny",
"actions": [
{
"label": "View workspaces",
"url": "https://et23ntkhpueak.pit-1.try.coder.app/workspaces"
},
{
"label": "View templates",
"url": "https://et23ntkhpueak.pit-1.try.coder.app/templates"
}
],
"labels": {
"initiator": "danny",
"name": "my-workspace",
"reason": "initiated by user"
}
},
"title": "Workspace \"my-workspace\" deleted",
"body": "Hi danny\n\nYour workspace my-workspace was deleted.\nThe specified reason was \"initiated by user (danny)\"."
}
```
The top-level object has these keys:
- `_version`: describes the version of this schema; follows semantic versioning
- `msg_id`: the UUID of the notification (matches the ID in the
`notification_messages` table)
- `payload`: contains the specific details of the notification; described below
- `title`: the title of the notification message (equivalent to a subject in
SMTP delivery)
- `body`: the body of the notification message (equivalent to the message body
in SMTP delivery)
The `payload` object has these keys:
- `_version`: describes the version of this inner schema; follows semantic
versioning
- `notification_name`: name of the event which triggered the notification
- `user_id`: Coder internal user identifier of the target user (UUID)
- `user_email`: email address of the target user
- `user_name`: name of the target user
- `user_username`: username of the target user
- `actions`: a list of CTAs (Call-To-Action); these are mainly relevant for SMTP
delivery in which they're shown as buttons
- `labels`: dynamic map of zero or more string key-value pairs; these vary from
event to event
## User Preferences
All users have the option to opt-out of any notifications. Go to **Account** ->
**Notifications** to turn notifications on or off. The delivery method for each
notification is indicated on the right hand side of this table.
![User Notification Preferences](../images/user-notification-preferences.png)
## Delivery Preferences (enterprise)
Administrators can configure which delivery methods are used for each different
[event type](#event-types).
![preferences](../images/admin/notification-admin-prefs.png)
You can find this page under
`https://$CODER_ACCESS_URL/deployment/notifications?tab=events`.
## Stop sending notifications
Administrators may wish to stop _all_ notifications across the deployment. We
support a killswitch in the CLI for these cases.
To pause sending notifications, execute
[`coder notifications pause`](https://coder.com/docs/reference/cli/notifications_pause).
To resume sending notifications, execute
[`coder notifications resume`](https://coder.com/docs/reference/cli/notifications_resume).
## Internals
The notification system is built to operate concurrently in a single- or
multi-replica Coder deployment, and has a built-in retry mechanism. It uses the
configured Postgres database to store notifications in a queue and facilitate
concurrency.
All messages are stored in the `notification_messages` table.
Messages older than 7 days are deleted.
### Message States
![states](../images/admin/notification-states.png)
_A notifier here refers to a Coder replica which is responsible for dispatching
the notification. All running replicas act as notifiers to process pending
messages._
- a message begins in `pending` state
- transitions to `leased` when a Coder replica acquires new messages from the
database
- new messages are checked for every `CODER_NOTIFICATIONS_FETCH_INTERVAL`
(default: 15s)
- if a message is delivered successfully, it transitions to `sent` state
- if a message encounters a non-retryable error (e.g. misconfiguration), it
transitions to `permanent_failure`
- if a message encounters a retryable error (e.g. temporary server outage), it
transitions to `temporary_failure`
- this message will be retried up to `CODER_NOTIFICATIONS_MAX_SEND_ATTEMPTS`
(default: 5)
- this message will transition back to `pending` state after
`CODER_NOTIFICATIONS_RETRY_INTERVAL` (default: 5m) and be retried
- after `CODER_NOTIFICATIONS_MAX_SEND_ATTEMPTS` is exceeded, it transitions to
`permanent_failure`
Diagnostic messages will be saved in the `notification_messages` table and will
be logged, in the case of failure.