Compare commits

..

53 Commits

Author SHA1 Message Date
Daniel Hougaard
d2a4f265de Update ExternalMigrationsTab.tsx 2025-07-24 19:58:29 +04:00
Daniel Hougaard
4af872e504 fix: ui state 2025-07-24 19:14:50 +04:00
Daniel Hougaard
716b88fa49 requested changes and docs 2025-07-24 19:09:24 +04:00
Daniel Hougaard
716f061c01 Merge branch 'heads/main' into daniel/vault-import 2025-07-24 17:29:55 +04:00
carlosmonastyrski
75622ed03e Merge pull request #3926 from Infisical/feat/remindersImprovement
feat(secret-reminders): rework secret reminders logic
2025-07-23 16:07:04 -03:00
Scott Wilson
a7041fcade Merge pull request #4199 from Infisical/search-by-tags-metadata
improvement(dashboard): add secret tag/metadata search functionality to single env view dashboard
2025-07-23 11:27:11 -07:00
Scott Wilson
0b38fc7843 Merge pull request #4181 from Infisical/org-policy-edit-page-revisions
improvements(frontend): org and project policy page ui improvements
2025-07-23 11:26:38 -07:00
Maidul Islam
e678c19874 Merge pull request #4225 from Infisical/fix/secret-scanning-delete
feat: updated invalid url
2025-07-23 13:38:45 -04:00
=
878e12ea5c feat: updated invalid url 2025-07-23 23:06:38 +05:30
x032205
485a90bde1 Merge pull request #4224 from Infisical/fix-secret-rotation-defaults
Fix secret rotation defaults
2025-07-23 12:45:39 -04:00
x032205
98b6bdad76 Fix secret rotation defaults 2025-07-23 12:44:23 -04:00
Carlos Monastyrski
f490ca22ac Small fix on new permission field actionProjectType missin on reminders service 2025-07-23 13:07:58 -03:00
Maidul Islam
2d8de9e782 update product names for project templates 2025-07-23 10:50:46 -04:00
Maidul Islam
14d4cfdbe4 Merge pull request #4222 from Infisical/fix/secret-scanning-delete
fix: resolved project deletion not working for secret scanning on missing plan
2025-07-23 10:47:18 -04:00
x032205
e8bd73c0d0 Merge pull request #4201 from Infisical/check-gateway-license-in-service
License check on fnGetGatewayClientTlsByGatewayId
2025-07-23 10:41:58 -04:00
Akhil Mohan
3406457c08 Merge pull request #4218 from dcs-soni/bug/banner-flicker
fix: redis banner appears only when it is not configured
2025-07-23 20:10:18 +05:30
=
c16764b62b fix: resolved project deletion not working for secret scanning on missing plan 2025-07-23 20:07:45 +05:30
Sid
ab56a69d59 feat: Digital Ocean App connection and App Platform secret sync (#4203)
* fix: save wip

* feat: final impl

* feat: docs

* Update backend/src/services/app-connection/digital-ocean/digital-ocean-connection-service.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* chore: remove empty conflict files

* Update backend/src/server/routes/v1/app-connection-routers/app-connection-router.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update frontend/src/components/secret-syncs/forms/schemas/digital-ocean-app-platform-sync-destination-schema.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update frontend/src/components/secret-syncs/forms/schemas/digital-ocean-app-platform-sync-destination-schema.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update frontend/src/components/secret-syncs/forms/SecretSyncDestinationFields/DigitalOceanAppPlatformSyncFields.tsx

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update backend/src/services/secret-sync/digital-ocean-app-platform/digital-ocean-app-platform-sync-schemas.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix: lint

* fix: api client

* fix: lint and types

* fix: typecheck lint

* fix: docs

* fix: docs

* fix: linting

---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-07-23 19:59:29 +05:30
Carlos Monastyrski
8520ca98c7 Merge branch 'main' into feat/remindersImprovement 2025-07-23 11:27:31 -03:00
carlosmonastyrski
95b997c100 Merge pull request #4214 from Infisical/fix/confirmCommitButtonFix
Redo confirm changes box
2025-07-23 11:08:38 -03:00
carlosmonastyrski
b433582ca6 Merge pull request #4210 from Infisical/feat/bitbucketSecretSync
Add Bitbucket Secret Sync
2025-07-23 11:07:07 -03:00
Maidul Islam
242cfe82c5 update product names 2025-07-23 09:38:08 -04:00
Maidul Islam
af4f7ec4f3 Merge pull request #4207 from Infisical/feat/split-back
feat: Move products out of projects
2025-07-23 09:27:53 -04:00
dcs-soni
9ec4419d83 fix testing vars 2025-07-23 13:04:10 +05:30
dcs-soni
7ff7e5882a fix: redis banner appears only when it is not configured 2025-07-23 12:44:05 +05:30
Scott Wilson
d4bdf04061 improvement: responsive and border color 2025-07-22 09:34:43 -07:00
Scott Wilson
4dcb3938e0 improvements: minor adjustments 2025-07-22 08:58:12 -07:00
Carlos Monastyrski
f992535812 Redo confirm changes box 2025-07-22 11:03:25 -03:00
Daniel Hougaard
464e32b0e9 Update VaultPlatformModal.tsx 2025-07-22 13:04:00 +04:00
Scott Wilson
4547b61d8f improvement: add metadata support to deep search 2025-07-21 18:18:04 -07:00
Carlos Monastyrski
047fd9371f Fix bitbucket iterationCount limit 2025-07-21 21:39:57 -03:00
Daniel Hougaard
bfd8b64871 requested changes 2025-07-22 02:15:21 +04:00
Daniel Hougaard
185cc4efba Update VaultPlatformModal copy.tsx 2025-07-22 01:50:28 +04:00
Daniel Hougaard
7150b9314d feat(external-migrations): vault migrations 2025-07-22 01:35:02 +04:00
Carlos Monastyrski
328f929a29 Addressed PR comments 2025-07-21 18:24:48 -03:00
Carlos Monastyrski
5019918516 Add secret sync app connection permission set 2025-07-21 11:44:21 -03:00
Carlos Monastyrski
ce877cd352 Addressed PR suggestions 2025-07-21 11:01:22 -03:00
Carlos Monastyrski
d44b3293b6 Add Bitbucket Secret Sync 2025-07-21 10:28:31 -03:00
x032205
4d8000e331 License check on fnGetGatewayClientTlsByGatewayId 2025-07-19 02:41:41 -04:00
Scott Wilson
90c341cf53 improvement: add secret tag/metadata search functionality to single env view dashboard 2025-07-18 18:22:11 -07:00
Scott Wilson
8df53dde3b improvements: address feedback 2025-07-17 15:27:28 -07:00
Carlos Monastyrski
394ecd24a0 Merge branch 'main' into feat/remindersImprovement 2025-07-17 17:35:41 -03:00
Daniel Hougaard
6d3acb5514 Update models.ts 2025-07-18 00:28:15 +04:00
Scott Wilson
1e08b3cdc2 chore: remove unused export 2025-07-16 15:05:10 -07:00
Scott Wilson
844f2bb72c improvements: org and project policy page ui improvements 2025-07-16 14:48:57 -07:00
Carlos Monastyrski
bd4968b60d Minor improvements on new reminders api 2025-07-14 16:48:05 -03:00
Carlos Monastyrski
6449699f03 Merge branch 'main' into feat/remindersImprovement 2025-07-14 10:19:33 -03:00
Carlos Monastyrski
0e680e366b Improve reminders router 2025-07-11 15:26:09 -03:00
Carlos Monastyrski
0af00ce82d Minor fix on add reminder table migration 2025-07-11 09:21:57 -03:00
Carlos Monastyrski
3153450dc5 Merge branch 'main' into feat/remindersImprovement 2025-07-11 08:59:21 -03:00
Carlos Monastyrski
50ba2e543c Minor improvements on new reminders logic 2025-07-11 08:02:18 -03:00
Carlos Monastyrski
e2559f10bc feat(secret-reminders): addressed PR suggestions and improvements 2025-07-04 11:58:09 -03:00
Carlos Monastyrski
0efc314f33 feat(secret-reminders): rework secret reminders logic 2025-07-04 09:47:36 -03:00
231 changed files with 5973 additions and 1355 deletions

View File

@@ -24,6 +24,7 @@ export const mockQueue = (): TQueueServiceFactory => {
events[name] = event;
},
getRepeatableJobs: async () => [],
getDelayedJobs: async () => [],
clearQueue: async () => {},
stopJobById: async () => {},
stopJobByIdPg: async () => {},

View File

@@ -93,6 +93,7 @@ import { TProjectEnvServiceFactory } from "@app/services/project-env/project-env
import { TProjectKeyServiceFactory } from "@app/services/project-key/project-key-service";
import { TProjectMembershipServiceFactory } from "@app/services/project-membership/project-membership-service";
import { TProjectRoleServiceFactory } from "@app/services/project-role/project-role-service";
import { TReminderServiceFactory } from "@app/services/reminder/reminder-types";
import { TSecretServiceFactory } from "@app/services/secret/secret-service";
import { TSecretBlindIndexServiceFactory } from "@app/services/secret-blind-index/secret-blind-index-service";
import { TSecretFolderServiceFactory } from "@app/services/secret-folder/secret-folder-service";
@@ -285,6 +286,7 @@ declare module "fastify" {
secretScanningV2: TSecretScanningV2ServiceFactory;
internalCertificateAuthority: TInternalCertificateAuthorityServiceFactory;
pkiTemplate: TPkiTemplatesServiceFactory;
reminder: TReminderServiceFactory;
};
// this is exclusive use for middlewares in which we need to inject data
// everywhere else access using service layer

View File

@@ -504,6 +504,12 @@ import {
TProjectMicrosoftTeamsConfigsInsert,
TProjectMicrosoftTeamsConfigsUpdate
} from "@app/db/schemas/project-microsoft-teams-configs";
import { TReminders, TRemindersInsert, TRemindersUpdate } from "@app/db/schemas/reminders";
import {
TRemindersRecipients,
TRemindersRecipientsInsert,
TRemindersRecipientsUpdate
} from "@app/db/schemas/reminders-recipients";
import {
TSecretReminderRecipients,
TSecretReminderRecipientsInsert,
@@ -1211,5 +1217,11 @@ declare module "knex/types/tables" {
TSecretScanningConfigsInsert,
TSecretScanningConfigsUpdate
>;
[TableName.Reminder]: KnexOriginal.CompositeTableType<TReminders, TRemindersInsert, TRemindersUpdate>;
[TableName.ReminderRecipient]: KnexOriginal.CompositeTableType<
TRemindersRecipients,
TRemindersRecipientsInsert,
TRemindersRecipientsUpdate
>;
}
}

View File

@@ -0,0 +1,43 @@
import { Knex } from "knex";
import { TableName } from "../schemas";
import { createOnUpdateTrigger, dropOnUpdateTrigger } from "../utils";
export async function up(knex: Knex): Promise<void> {
if (!(await knex.schema.hasTable(TableName.Reminder))) {
await knex.schema.createTable(TableName.Reminder, (t) => {
t.uuid("id", { primaryKey: true }).defaultTo(knex.fn.uuid());
t.uuid("secretId").nullable();
t.foreign("secretId").references("id").inTable(TableName.SecretV2).onDelete("CASCADE");
t.string("message", 1024).nullable();
t.integer("repeatDays").checkPositive().nullable();
t.timestamp("nextReminderDate").notNullable();
t.timestamps(true, true, true);
t.unique("secretId");
});
}
if (!(await knex.schema.hasTable(TableName.ReminderRecipient))) {
await knex.schema.createTable(TableName.ReminderRecipient, (t) => {
t.uuid("id", { primaryKey: true }).defaultTo(knex.fn.uuid());
t.uuid("reminderId").notNullable();
t.foreign("reminderId").references("id").inTable(TableName.Reminder).onDelete("CASCADE");
t.uuid("userId").notNullable();
t.foreign("userId").references("id").inTable(TableName.Users).onDelete("CASCADE");
t.timestamps(true, true, true);
t.index("reminderId");
t.index("userId");
t.unique(["reminderId", "userId"]);
});
}
await createOnUpdateTrigger(knex, TableName.Reminder);
await createOnUpdateTrigger(knex, TableName.ReminderRecipient);
}
export async function down(knex: Knex): Promise<void> {
await dropOnUpdateTrigger(knex, TableName.Reminder);
await dropOnUpdateTrigger(knex, TableName.ReminderRecipient);
await knex.schema.dropTableIfExists(TableName.ReminderRecipient);
await knex.schema.dropTableIfExists(TableName.Reminder);
}

View File

@@ -430,4 +430,3 @@ export async function down(knex: Knex): Promise<void> {
});
}
}

View File

@@ -160,7 +160,7 @@ export enum TableName {
SecretRotationV2SecretMapping = "secret_rotation_v2_secret_mappings",
MicrosoftTeamsIntegrations = "microsoft_teams_integrations",
ProjectMicrosoftTeamsConfigs = "project_microsoft_teams_configs",
SecretReminderRecipients = "secret_reminder_recipients",
SecretReminderRecipients = "secret_reminder_recipients", // TODO(Carlos): Remove this in the future after migrating to the new reminder recipients table
GithubOrgSyncConfig = "github_org_sync_configs",
FolderCommit = "folder_commits",
FolderCommitChanges = "folder_commit_changes",
@@ -172,7 +172,10 @@ export enum TableName {
SecretScanningResource = "secret_scanning_resources",
SecretScanningScan = "secret_scanning_scans",
SecretScanningFinding = "secret_scanning_findings",
SecretScanningConfig = "secret_scanning_configs"
SecretScanningConfig = "secret_scanning_configs",
// reminders
Reminder = "reminders",
ReminderRecipient = "reminders_recipients"
}
export type TImmutableDBKeys = "id" | "createdAt" | "updatedAt" | "commitId";

View File

@@ -0,0 +1,20 @@
// Code generated by automation script, DO NOT EDIT.
// Automated by pulling database and generating zod schema
// To update. Just run npm run generate:schema
// Written by akhilmhdh.
import { z } from "zod";
import { TImmutableDBKeys } from "./models";
export const RemindersRecipientsSchema = z.object({
id: z.string().uuid(),
reminderId: z.string().uuid(),
userId: z.string().uuid(),
createdAt: z.date(),
updatedAt: z.date()
});
export type TRemindersRecipients = z.infer<typeof RemindersRecipientsSchema>;
export type TRemindersRecipientsInsert = Omit<z.input<typeof RemindersRecipientsSchema>, TImmutableDBKeys>;
export type TRemindersRecipientsUpdate = Partial<Omit<z.input<typeof RemindersRecipientsSchema>, TImmutableDBKeys>>;

View File

@@ -0,0 +1,22 @@
// Code generated by automation script, DO NOT EDIT.
// Automated by pulling database and generating zod schema
// To update. Just run npm run generate:schema
// Written by akhilmhdh.
import { z } from "zod";
import { TImmutableDBKeys } from "./models";
export const RemindersSchema = z.object({
id: z.string().uuid(),
secretId: z.string().uuid().nullable().optional(),
message: z.string().nullable().optional(),
repeatDays: z.number().nullable().optional(),
nextReminderDate: z.date(),
createdAt: z.date(),
updatedAt: z.date()
});
export type TReminders = z.infer<typeof RemindersSchema>;
export type TRemindersInsert = Omit<z.input<typeof RemindersSchema>, TImmutableDBKeys>;
export type TRemindersUpdate = Partial<Omit<z.input<typeof RemindersSchema>, TImmutableDBKeys>>;

View File

@@ -315,10 +315,12 @@ export const registerSecretRotationEndpoints = <
querystring: z.object({
deleteSecrets: z
.enum(["true", "false"])
.optional()
.transform((value) => value === "true")
.describe(SecretRotations.DELETE(type).deleteSecrets),
revokeGeneratedCredentials: z
.enum(["true", "false"])
.optional()
.transform((value) => value === "true")
.describe(SecretRotations.DELETE(type).revokeGeneratedCredentials)
}),

View File

@@ -468,7 +468,11 @@ export enum EventType {
CREATE_PROJECT = "create-project",
UPDATE_PROJECT = "update-project",
DELETE_PROJECT = "delete-project"
DELETE_PROJECT = "delete-project",
CREATE_SECRET_REMINDER = "create-secret-reminder",
GET_SECRET_REMINDER = "get-secret-reminder",
DELETE_SECRET_REMINDER = "delete-secret-reminder"
}
export const filterableSecretEvents: EventType[] = [
@@ -3326,6 +3330,31 @@ interface SecretScanningConfigUpdateEvent {
};
}
interface SecretReminderCreateEvent {
type: EventType.CREATE_SECRET_REMINDER;
metadata: {
secretId: string;
message?: string | null;
repeatDays?: number | null;
nextReminderDate?: string | null;
recipients?: string[] | null;
};
}
interface SecretReminderGetEvent {
type: EventType.GET_SECRET_REMINDER;
metadata: {
secretId: string;
};
}
interface SecretReminderDeleteEvent {
type: EventType.DELETE_SECRET_REMINDER;
metadata: {
secretId: string;
};
}
interface SecretScanningConfigReadEvent {
type: EventType.SECRET_SCANNING_CONFIG_GET;
metadata?: Record<string, never>; // not needed, based off projectId
@@ -3689,4 +3718,7 @@ export type Event =
| OrgUpdateEvent
| ProjectCreateEvent
| ProjectUpdateEvent
| ProjectDeleteEvent;
| ProjectDeleteEvent
| SecretReminderCreateEvent
| SecretReminderGetEvent
| SecretReminderDeleteEvent;

View File

@@ -566,6 +566,14 @@ export const gatewayServiceFactory = ({
if (!gateway) throw new NotFoundError({ message: `Gateway with ID ${gatewayId} not found.` });
const orgGatewayConfig = await orgGatewayConfigDAL.findById(gateway.orgGatewayRootCaId);
const orgLicensePlan = await licenseService.getPlan(orgGatewayConfig.orgId);
if (!orgLicensePlan.gateway) {
throw new BadRequestError({
message: "Please upgrade your instance to Infisical's Enterprise plan to use gateways."
});
}
const { decryptor: orgKmsDecryptor } = await kmsService.createCipherPairWithDataKey({
type: KmsDataKey.Organization,
orgId: orgGatewayConfig.orgId

View File

@@ -167,7 +167,7 @@ export const secretRotationV2QueueServiceFactory = async ({
environment: environment.name,
projectName: project.name,
rotationUrl: encodeURI(
`${appCfg.SITE_URL}/projects/${projectId}/secret-manager/secrets/${environment.slug}`
`${appCfg.SITE_URL}/projects/secret-management/${projectId}/secrets/${environment.slug}`
)
}
});

View File

@@ -596,7 +596,7 @@ export const secretScanningV2QueueServiceFactory = async ({
numberOfSecrets: payload.numberOfSecrets,
isDiffScan: payload.isDiffScan,
url: encodeURI(
`${appCfg.SITE_URL}/projects/${projectId}/secret-scanning/findings?search=scanId:${payload.scanId}`
`${appCfg.SITE_URL}/projects/secret-scanning/${projectId}/findings?search=scanId:${payload.scanId}`
),
timestamp
}
@@ -607,7 +607,7 @@ export const secretScanningV2QueueServiceFactory = async ({
timestamp,
errorMessage: payload.errorMessage,
url: encodeURI(
`${appCfg.SITE_URL}/projects/${projectId}/secret-scanning/data-sources/${dataSource.type}/${dataSource.id}`
`${appCfg.SITE_URL}/projects/secret-scanning/${projectId}/data-sources/${dataSource.type}/${dataSource.id}`
)
}
});

View File

@@ -2290,6 +2290,9 @@ export const AppConnections = {
accessKey: "The Key used to access Supabase.",
instanceUrl: "The URL used to access Supabase."
},
DIGITAL_OCEAN_APP_PLATFORM: {
apiToken: "The API token used to authenticate with Digital Ocean App Platform."
},
OKTA: {
instanceUrl: "The URL used to access your Okta organization.",
apiToken: "The API token used to authenticate with Okta."
@@ -2506,6 +2509,11 @@ export const SecretSyncs = {
SUPABASE: {
projectId: "The ID of the Supabase project to sync secrets to.",
projectName: "The name of the Supabase project to sync secrets to."
},
BITBUCKET: {
workspaceSlug: "The Bitbucket Workspace slug to sync secrets to.",
repositorySlug: "The Bitbucket Repository slug to sync secrets to.",
environmentId: "The Bitbucket Deployment Environment uuid to sync secrets to."
}
}
};

View File

@@ -64,7 +64,9 @@ export enum QueueName {
FolderTreeCheckpoint = "folder-tree-checkpoint",
InvalidateCache = "invalidate-cache",
SecretScanningV2 = "secret-scanning-v2",
TelemetryAggregatedEvents = "telemetry-aggregated-events"
TelemetryAggregatedEvents = "telemetry-aggregated-events",
DailyReminders = "daily-reminders",
SecretReminderMigration = "secret-reminder-migration"
}
export enum QueueJobs {
@@ -104,7 +106,9 @@ export enum QueueJobs {
SecretScanningV2SendNotification = "secret-scanning-v2-notification",
CaOrderCertificateForSubscriber = "ca-order-certificate-for-subscriber",
PkiSubscriberDailyAutoRenewal = "pki-subscriber-daily-auto-renewal",
TelemetryAggregatedEvents = "telemetry-aggregated-events"
TelemetryAggregatedEvents = "telemetry-aggregated-events",
DailyReminders = "daily-reminders",
SecretReminderMigration = "secret-reminder-migration"
}
export type TQueueJobTypes = {
@@ -291,6 +295,14 @@ export type TQueueJobTypes = {
caType: CaType;
};
};
[QueueName.DailyReminders]: {
name: QueueJobs.DailyReminders;
payload: undefined;
};
[QueueName.SecretReminderMigration]: {
name: QueueJobs.SecretReminderMigration;
payload: undefined;
};
[QueueName.PkiSubscriber]: {
name: QueueJobs.PkiSubscriberDailyAutoRenewal;
payload: undefined;
@@ -390,6 +402,11 @@ export type TQueueServiceFactory = {
startOffset?: number,
endOffset?: number
) => Promise<{ key: string; name: string; id: string | null }[]>;
getDelayedJobs: (
name: QueueName,
startOffset?: number,
endOffset?: number
) => Promise<{ delay: number; timestamp: number; repeatJobKey?: string; data?: unknown }[]>;
};
export const queueServiceFactory = (
@@ -552,6 +569,13 @@ export const queueServiceFactory = (
return q.getRepeatableJobs(startOffset, endOffset);
};
const getDelayedJobs: TQueueServiceFactory["getDelayedJobs"] = (name, startOffset, endOffset) => {
const q = queueContainer[name];
if (!q) throw new Error(`Queue '${name}' not initialized`);
return q.getDelayed(startOffset, endOffset);
};
const stopRepeatableJobByJobId: TQueueServiceFactory["stopRepeatableJobByJobId"] = async (name, jobId) => {
const q = queueContainer[name];
const job = await q.getJob(jobId);
@@ -598,6 +622,7 @@ export const queueServiceFactory = (
stopJobById,
stopJobByIdPg,
getRepeatableJobs,
getDelayedJobs,
startPg,
queuePg,
schedulePg

View File

@@ -246,6 +246,10 @@ import { projectMembershipServiceFactory } from "@app/services/project-membershi
import { projectUserMembershipRoleDALFactory } from "@app/services/project-membership/project-user-membership-role-dal";
import { projectRoleDALFactory } from "@app/services/project-role/project-role-dal";
import { projectRoleServiceFactory } from "@app/services/project-role/project-role-service";
import { reminderDALFactory } from "@app/services/reminder/reminder-dal";
import { dailyReminderQueueServiceFactory } from "@app/services/reminder/reminder-queue";
import { reminderServiceFactory } from "@app/services/reminder/reminder-service";
import { reminderRecipientDALFactory } from "@app/services/reminder-recipients/reminder-recipient-dal";
import { dailyResourceCleanUpQueueServiceFactory } from "@app/services/resource-cleanup/resource-cleanup-queue";
import { resourceMetadataDALFactory } from "@app/services/resource-metadata/resource-metadata-dal";
import { secretDALFactory } from "@app/services/secret/secret-dal";
@@ -371,6 +375,9 @@ export const registerRoutes = async (
const secretVersionV2BridgeDAL = secretVersionV2BridgeDALFactory(db);
const secretVersionTagV2BridgeDAL = secretVersionV2TagBridgeDALFactory(db);
const reminderDAL = reminderDALFactory(db);
const reminderRecipientDAL = reminderRecipientDALFactory(db);
const integrationDAL = integrationDALFactory(db);
const integrationAuthDAL = integrationAuthDALFactory(db);
const webhookDAL = webhookDALFactory(db);
@@ -734,9 +741,17 @@ export const registerRoutes = async (
const projectBotService = projectBotServiceFactory({ permissionService, projectBotDAL, projectDAL });
const reminderService = reminderServiceFactory({
reminderDAL,
reminderRecipientDAL,
smtpService,
projectMembershipDAL,
permissionService,
secretV2BridgeDAL
});
const orgService = orgServiceFactory({
userAliasDAL,
queueService,
identityMetadataDAL,
secretDAL,
secretV2BridgeDAL,
@@ -762,7 +777,8 @@ export const registerRoutes = async (
orgBotDAL,
oidcConfigDAL,
loginService,
projectBotService
projectBotService,
reminderService
});
const signupService = authSignupServiceFactory({
tokenService,
@@ -1060,7 +1076,6 @@ export const registerRoutes = async (
secretImportDAL,
projectEnvDAL,
webhookDAL,
orgDAL,
auditLogService,
userDAL,
projectMembershipDAL,
@@ -1082,11 +1097,11 @@ export const registerRoutes = async (
secretApprovalRequestDAL,
projectKeyDAL,
projectUserMembershipRoleDAL,
secretReminderRecipientsDAL,
orgService,
resourceMetadataDAL,
folderCommitService,
secretSyncQueue
secretSyncQueue,
reminderService
});
const projectService = projectServiceFactory({
@@ -1095,7 +1110,6 @@ export const registerRoutes = async (
projectSshConfigDAL,
secretDAL,
secretV2BridgeDAL,
queueService,
projectQueue: projectQueueService,
projectBotService,
identityProjectDAL,
@@ -1132,7 +1146,8 @@ export const registerRoutes = async (
microsoftTeamsIntegrationDAL,
projectTemplateService,
groupProjectDAL,
smtpService
smtpService,
reminderService
});
const projectEnvService = projectEnvServiceFactory({
@@ -1231,6 +1246,7 @@ export const registerRoutes = async (
kmsService,
snapshotService,
resourceMetadataDAL,
reminderService,
keyStore
});
@@ -1284,7 +1300,8 @@ export const registerRoutes = async (
secretApprovalRequestSecretDAL,
secretV2BridgeService,
secretApprovalRequestService,
licenseService
licenseService,
reminderService
});
const secretSharingService = secretSharingServiceFactory({
@@ -1616,7 +1633,6 @@ export const registerRoutes = async (
auditLogDAL,
queueService,
secretVersionDAL,
secretDAL,
secretFolderVersionDAL: folderVersionDAL,
snapshotDAL,
identityAccessTokenDAL,
@@ -1627,6 +1643,13 @@ export const registerRoutes = async (
orgService
});
const dailyReminderQueueService = dailyReminderQueueServiceFactory({
reminderService,
queueService,
secretDAL: secretV2BridgeDAL,
secretReminderRecipientsDAL
});
const dailyExpiringPkiItemAlert = dailyExpiringPkiItemAlertQueueServiceFactory({
queueService,
pkiAlertService
@@ -1926,6 +1949,8 @@ export const registerRoutes = async (
await telemetryQueue.startTelemetryCheck();
await telemetryQueue.startAggregatedEventsJob();
await dailyResourceCleanUp.startCleanUp();
await dailyReminderQueueService.startDailyRemindersJob();
await dailyReminderQueueService.startSecretReminderMigrationJob();
await dailyExpiringPkiItemAlert.startSendingAlerts();
await pkiSubscriberQueue.startDailyAutoRenewalJob();
await kmsService.startService();
@@ -2036,7 +2061,8 @@ export const registerRoutes = async (
assumePrivileges: assumePrivilegeService,
githubOrgSync: githubOrgSyncConfigService,
folderCommit: folderCommitService,
secretScanningV2: secretScanningV2Service
secretScanningV2: secretScanningV2Service,
reminder: reminderService
});
const cronJobs: CronJob[] = [];

View File

@@ -51,6 +51,10 @@ import {
DatabricksConnectionListItemSchema,
SanitizedDatabricksConnectionSchema
} from "@app/services/app-connection/databricks";
import {
DigitalOceanConnectionListItemSchema,
SanitizedDigitalOceanConnectionSchema
} from "@app/services/app-connection/digital-ocean";
import { FlyioConnectionListItemSchema, SanitizedFlyioConnectionSchema } from "@app/services/app-connection/flyio";
import { GcpConnectionListItemSchema, SanitizedGcpConnectionSchema } from "@app/services/app-connection/gcp";
import { GitHubConnectionListItemSchema, SanitizedGitHubConnectionSchema } from "@app/services/app-connection/github";
@@ -140,6 +144,7 @@ const SanitizedAppConnectionSchema = z.union([
...SanitizedRailwayConnectionSchema.options,
...SanitizedChecklyConnectionSchema.options,
...SanitizedSupabaseConnectionSchema.options,
...SanitizedDigitalOceanConnectionSchema.options,
...SanitizedOktaConnectionSchema.options
]);
@@ -178,6 +183,7 @@ const AppConnectionOptionsSchema = z.discriminatedUnion("app", [
RailwayConnectionListItemSchema,
ChecklyConnectionListItemSchema,
SupabaseConnectionListItemSchema,
DigitalOceanConnectionListItemSchema,
OktaConnectionListItemSchema
]);

View File

@@ -85,4 +85,40 @@ export const registerBitbucketConnectionRouter = async (server: FastifyZodProvid
return { repositories };
}
});
server.route({
method: "GET",
url: `/:connectionId/environments`,
config: {
rateLimit: readLimit
},
schema: {
params: z.object({
connectionId: z.string().uuid()
}),
querystring: z.object({
workspaceSlug: z.string().min(1).max(255),
repositorySlug: z.string().min(1).max(255)
}),
response: {
200: z.object({
environments: z.object({ slug: z.string(), name: z.string(), uuid: z.string() }).array()
})
}
},
onRequest: verifyAuth([AuthMode.JWT]),
handler: async (req) => {
const {
params: { connectionId },
query: { workspaceSlug, repositorySlug }
} = req;
const environments = await server.services.appConnection.bitbucket.listEnvironments(
{ connectionId, workspaceSlug, repositorySlug },
req.permission
);
return { environments };
}
});
};

View File

@@ -0,0 +1,57 @@
import { z } from "zod";
import { readLimit } from "@app/server/config/rateLimiter";
import { verifyAuth } from "@app/server/plugins/auth/verify-auth";
import { AppConnection } from "@app/services/app-connection/app-connection-enums";
import {
CreateDigitalOceanConnectionSchema,
SanitizedDigitalOceanConnectionSchema,
UpdateDigitalOceanConnectionSchema
} from "@app/services/app-connection/digital-ocean";
import { AuthMode } from "@app/services/auth/auth-type";
import { registerAppConnectionEndpoints } from "./app-connection-endpoints";
export const registerDigitalOceanConnectionRouter = async (server: FastifyZodProvider) => {
registerAppConnectionEndpoints({
app: AppConnection.DigitalOcean,
server,
createSchema: CreateDigitalOceanConnectionSchema,
updateSchema: UpdateDigitalOceanConnectionSchema,
sanitizedResponseSchema: SanitizedDigitalOceanConnectionSchema
});
// The below endpoints are not exposed and for Infisical App use
server.route({
method: "GET",
url: `/:connectionId/apps`,
config: {
rateLimit: readLimit
},
schema: {
params: z.object({
connectionId: z.string().uuid()
}),
response: {
200: z.object({
apps: z
.object({
id: z.string(),
spec: z.object({
name: z.string()
})
})
.array()
})
}
},
onRequest: verifyAuth([AuthMode.JWT]),
handler: async (req) => {
const { connectionId } = req.params;
const apps = await server.services.appConnection.digitalOcean.listApps(connectionId, req.permission);
return { apps };
}
});
};

View File

@@ -14,6 +14,7 @@ import { registerCamundaConnectionRouter } from "./camunda-connection-router";
import { registerChecklyConnectionRouter } from "./checkly-connection-router";
import { registerCloudflareConnectionRouter } from "./cloudflare-connection-router";
import { registerDatabricksConnectionRouter } from "./databricks-connection-router";
import { registerDigitalOceanConnectionRouter } from "./digital-ocean-connection-router";
import { registerFlyioConnectionRouter } from "./flyio-connection-router";
import { registerGcpConnectionRouter } from "./gcp-connection-router";
import { registerGitHubConnectionRouter } from "./github-connection-router";
@@ -74,5 +75,6 @@ export const APP_CONNECTION_REGISTER_ROUTER_MAP: Record<AppConnection, (server:
[AppConnection.Railway]: registerRailwayConnectionRouter,
[AppConnection.Checkly]: registerChecklyConnectionRouter,
[AppConnection.Supabase]: registerSupabaseConnectionRouter,
[AppConnection.DigitalOcean]: registerDigitalOceanConnectionRouter,
[AppConnection.Okta]: registerOktaConnectionRouter
};

View File

@@ -904,7 +904,9 @@ export const registerDashboardRouter = async (server: FastifyZodProvider) => {
projectId,
path: secretPath,
search,
tagSlugs: tags
tagSlugs: tags,
includeTagsInSearch: true,
includeMetadataInSearch: true
});
if (remainingLimit > 0 && totalSecretCount > adjustedOffset) {
@@ -924,7 +926,9 @@ export const registerDashboardRouter = async (server: FastifyZodProvider) => {
search,
limit: remainingLimit,
offset: adjustedOffset,
tagSlugs: tags
tagSlugs: tags,
includeTagsInSearch: true,
includeMetadataInSearch: true
})
).secrets;
}
@@ -1097,7 +1101,8 @@ export const registerDashboardRouter = async (server: FastifyZodProvider) => {
filters: {
...sharedFilters,
tagSlugs: tags,
includeTagsInSearch: true
includeTagsInSearch: true,
includeMetadataInSearch: true
}
},
req.permission

View File

@@ -42,6 +42,7 @@ import { registerProjectEnvRouter } from "./project-env-router";
import { registerProjectKeyRouter } from "./project-key-router";
import { registerProjectMembershipRouter } from "./project-membership-router";
import { registerProjectRouter } from "./project-router";
import { SECRET_REMINDER_REGISTER_ROUTER_MAP } from "./reminder-routers";
import { registerSecretFolderRouter } from "./secret-folder-router";
import { registerSecretImportRouter } from "./secret-import-router";
import { registerSecretRequestsRouter } from "./secret-requests-router";
@@ -172,4 +173,14 @@ export const registerV1Routes = async (server: FastifyZodProvider) => {
},
{ prefix: "/secret-syncs" }
);
await server.register(
async (reminderRouter) => {
// register service specific reminder endpoints (reminders/secret)
for await (const [reminderType, router] of Object.entries(SECRET_REMINDER_REGISTER_ROUTER_MAP)) {
await reminderRouter.register(router, { prefix: `/${reminderType}` });
}
},
{ prefix: "/reminders" }
);
};

View File

@@ -0,0 +1,8 @@
import { ReminderType } from "@app/services/reminder/reminder-enums";
import { registerSecretReminderRouter } from "./secret-reminder-router";
export const SECRET_REMINDER_REGISTER_ROUTER_MAP: Record<ReminderType, (server: FastifyZodProvider) => Promise<void>> =
{
[ReminderType.SECRETS]: registerSecretReminderRouter
};

View File

@@ -0,0 +1,154 @@
import { z } from "zod";
import { RemindersSchema } from "@app/db/schemas/reminders";
import { EventType } from "@app/ee/services/audit-log/audit-log-types";
import { readLimit, writeLimit } from "@app/server/config/rateLimiter";
import { verifyAuth } from "@app/server/plugins/auth/verify-auth";
import { AuthMode } from "@app/services/auth/auth-type";
export const registerSecretReminderRouter = async (server: FastifyZodProvider) => {
server.route({
url: "/:secretId",
method: "POST",
config: {
rateLimit: writeLimit
},
schema: {
params: z.object({
secretId: z.string().uuid()
}),
body: z
.object({
message: z.string().trim().max(1024).optional(),
repeatDays: z.number().min(1).nullable().optional(),
nextReminderDate: z.string().datetime().nullable().optional(),
recipients: z.string().array().optional()
})
.refine((data) => {
return data.repeatDays || data.nextReminderDate;
}, "At least one of repeatDays or nextReminderDate is required"),
response: {
200: z.object({
message: z.string()
})
}
},
onRequest: verifyAuth([AuthMode.JWT]),
handler: async (req) => {
await server.services.reminder.createReminder({
actorId: req.permission.id,
actor: req.permission.type,
actorOrgId: req.permission.orgId,
actorAuthMethod: req.permission.authMethod,
reminder: {
secretId: req.params.secretId,
message: req.body.message,
repeatDays: req.body.repeatDays,
nextReminderDate: req.body.nextReminderDate,
recipients: req.body.recipients
}
});
await server.services.auditLog.createAuditLog({
...req.auditLogInfo,
orgId: req.permission.orgId,
event: {
type: EventType.CREATE_SECRET_REMINDER,
metadata: {
secretId: req.params.secretId,
message: req.body.message,
repeatDays: req.body.repeatDays,
nextReminderDate: req.body.nextReminderDate,
recipients: req.body.recipients
}
}
});
return { message: "Successfully created reminder" };
}
});
server.route({
url: "/:secretId",
method: "GET",
config: {
rateLimit: readLimit
},
schema: {
params: z.object({
secretId: z.string().uuid()
}),
response: {
200: z.object({
reminder: RemindersSchema.extend({
recipients: z.string().array().optional()
})
.optional()
.nullable()
})
}
},
onRequest: verifyAuth([AuthMode.JWT]),
handler: async (req) => {
const reminder = await server.services.reminder.getReminder({
actorId: req.permission.id,
actor: req.permission.type,
actorOrgId: req.permission.orgId,
actorAuthMethod: req.permission.authMethod,
secretId: req.params.secretId
});
await server.services.auditLog.createAuditLog({
...req.auditLogInfo,
orgId: req.permission.orgId,
event: {
type: EventType.GET_SECRET_REMINDER,
metadata: {
secretId: req.params.secretId
}
}
});
return { reminder };
}
});
server.route({
url: "/:secretId",
method: "DELETE",
config: {
rateLimit: writeLimit
},
schema: {
params: z.object({
secretId: z.string().uuid()
}),
response: {
200: z.object({
message: z.string()
})
}
},
onRequest: verifyAuth([AuthMode.JWT]),
handler: async (req) => {
await server.services.reminder.deleteReminder({
actorId: req.permission.id,
actor: req.permission.type,
actorOrgId: req.permission.orgId,
actorAuthMethod: req.permission.authMethod,
secretId: req.params.secretId
});
await server.services.auditLog.createAuditLog({
...req.auditLogInfo,
orgId: req.permission.orgId,
event: {
type: EventType.DELETE_SECRET_REMINDER,
metadata: {
secretId: req.params.secretId
}
}
});
return { message: "Successfully deleted reminder" };
}
});
};

View File

@@ -0,0 +1,17 @@
import {
BitbucketSyncSchema,
CreateBitbucketSyncSchema,
UpdateBitbucketSyncSchema
} from "@app/services/secret-sync/bitbucket";
import { SecretSync } from "@app/services/secret-sync/secret-sync-enums";
import { registerSyncSecretsEndpoints } from "./secret-sync-endpoints";
export const registerBitbucketSyncRouter = async (server: FastifyZodProvider) =>
registerSyncSecretsEndpoints({
destination: SecretSync.Bitbucket,
server,
responseSchema: BitbucketSyncSchema,
createSchema: CreateBitbucketSyncSchema,
updateSchema: UpdateBitbucketSyncSchema
});

View File

@@ -0,0 +1,17 @@
import {
CreateDigitalOceanAppPlatformSyncSchema,
DigitalOceanAppPlatformSyncSchema,
UpdateDigitalOceanAppPlatformSyncSchema
} from "@app/services/secret-sync/digital-ocean-app-platform";
import { SecretSync } from "@app/services/secret-sync/secret-sync-enums";
import { registerSyncSecretsEndpoints } from "./secret-sync-endpoints";
export const registerDigitalOceanAppPlatformSyncRouter = async (server: FastifyZodProvider) =>
registerSyncSecretsEndpoints({
destination: SecretSync.DigitalOceanAppPlatform,
server,
responseSchema: DigitalOceanAppPlatformSyncSchema,
createSchema: CreateDigitalOceanAppPlatformSyncSchema,
updateSchema: UpdateDigitalOceanAppPlatformSyncSchema
});

View File

@@ -7,11 +7,13 @@ import { registerAwsSecretsManagerSyncRouter } from "./aws-secrets-manager-sync-
import { registerAzureAppConfigurationSyncRouter } from "./azure-app-configuration-sync-router";
import { registerAzureDevOpsSyncRouter } from "./azure-devops-sync-router";
import { registerAzureKeyVaultSyncRouter } from "./azure-key-vault-sync-router";
import { registerBitbucketSyncRouter } from "./bitbucket-sync-router";
import { registerCamundaSyncRouter } from "./camunda-sync-router";
import { registerChecklySyncRouter } from "./checkly-sync-router";
import { registerCloudflarePagesSyncRouter } from "./cloudflare-pages-sync-router";
import { registerCloudflareWorkersSyncRouter } from "./cloudflare-workers-sync-router";
import { registerDatabricksSyncRouter } from "./databricks-sync-router";
import { registerDigitalOceanAppPlatformSyncRouter } from "./digital-ocean-app-platform-sync-router";
import { registerFlyioSyncRouter } from "./flyio-sync-router";
import { registerGcpSyncRouter } from "./gcp-sync-router";
import { registerGitHubSyncRouter } from "./github-sync-router";
@@ -57,5 +59,7 @@ export const SECRET_SYNC_REGISTER_ROUTER_MAP: Record<SecretSync, (server: Fastif
[SecretSync.Supabase]: registerSupabaseSyncRouter,
[SecretSync.Zabbix]: registerZabbixSyncRouter,
[SecretSync.Railway]: registerRailwaySyncRouter,
[SecretSync.Checkly]: registerChecklySyncRouter
[SecretSync.Checkly]: registerChecklySyncRouter,
[SecretSync.DigitalOceanAppPlatform]: registerDigitalOceanAppPlatformSyncRouter,
[SecretSync.Bitbucket]: registerBitbucketSyncRouter
};

View File

@@ -21,6 +21,7 @@ import {
} from "@app/services/secret-sync/azure-app-configuration";
import { AzureDevOpsSyncListItemSchema, AzureDevOpsSyncSchema } from "@app/services/secret-sync/azure-devops";
import { AzureKeyVaultSyncListItemSchema, AzureKeyVaultSyncSchema } from "@app/services/secret-sync/azure-key-vault";
import { BitbucketSyncListItemSchema, BitbucketSyncSchema } from "@app/services/secret-sync/bitbucket";
import { CamundaSyncListItemSchema, CamundaSyncSchema } from "@app/services/secret-sync/camunda";
import { ChecklySyncListItemSchema, ChecklySyncSchema } from "@app/services/secret-sync/checkly/checkly-sync-schemas";
import {
@@ -32,6 +33,10 @@ import {
CloudflareWorkersSyncSchema
} from "@app/services/secret-sync/cloudflare-workers/cloudflare-workers-schemas";
import { DatabricksSyncListItemSchema, DatabricksSyncSchema } from "@app/services/secret-sync/databricks";
import {
DigitalOceanAppPlatformSyncListItemSchema,
DigitalOceanAppPlatformSyncSchema
} from "@app/services/secret-sync/digital-ocean-app-platform";
import { FlyioSyncListItemSchema, FlyioSyncSchema } from "@app/services/secret-sync/flyio";
import { GcpSyncListItemSchema, GcpSyncSchema } from "@app/services/secret-sync/gcp";
import { GitHubSyncListItemSchema, GitHubSyncSchema } from "@app/services/secret-sync/github";
@@ -75,7 +80,9 @@ const SecretSyncSchema = z.discriminatedUnion("destination", [
SupabaseSyncSchema,
ZabbixSyncSchema,
RailwaySyncSchema,
ChecklySyncSchema
ChecklySyncSchema,
DigitalOceanAppPlatformSyncSchema,
BitbucketSyncSchema
]);
const SecretSyncOptionsSchema = z.discriminatedUnion("destination", [
@@ -102,11 +109,12 @@ const SecretSyncOptionsSchema = z.discriminatedUnion("destination", [
GitLabSyncListItemSchema,
CloudflarePagesSyncListItemSchema,
CloudflareWorkersSyncListItemSchema,
DigitalOceanAppPlatformSyncListItemSchema,
ZabbixSyncListItemSchema,
RailwaySyncListItemSchema,
ChecklySyncListItemSchema,
SupabaseSyncListItemSchema
SupabaseSyncListItemSchema,
BitbucketSyncListItemSchema
]);
export const registerSecretSyncRouter = async (server: FastifyZodProvider) => {

View File

@@ -1,9 +1,11 @@
import fastifyMultipart from "@fastify/multipart";
import { z } from "zod";
import { BadRequestError } from "@app/lib/errors";
import { readLimit } from "@app/server/config/rateLimiter";
import { writeLimit } from "@app/server/config/rateLimiter";
import { verifyAuth } from "@app/server/plugins/auth/verify-auth";
import { AuthMode } from "@app/services/auth/auth-type";
import { VaultMappingType } from "@app/services/external-migration/external-migration-types";
const MB25_IN_BYTES = 26214400;
@@ -15,7 +17,7 @@ export const registerExternalMigrationRouter = async (server: FastifyZodProvider
bodyLimit: MB25_IN_BYTES,
url: "/env-key",
config: {
rateLimit: readLimit
rateLimit: writeLimit
},
onRequest: verifyAuth([AuthMode.JWT, AuthMode.IDENTITY_ACCESS_TOKEN]),
handler: async (req) => {
@@ -52,4 +54,30 @@ export const registerExternalMigrationRouter = async (server: FastifyZodProvider
});
}
});
server.route({
method: "POST",
url: "/vault",
config: {
rateLimit: writeLimit
},
schema: {
body: z.object({
vaultAccessToken: z.string(),
vaultNamespace: z.string().trim().optional(),
vaultUrl: z.string(),
mappingType: z.nativeEnum(VaultMappingType)
})
},
onRequest: verifyAuth([AuthMode.JWT, AuthMode.IDENTITY_ACCESS_TOKEN]),
handler: async (req) => {
await server.services.migration.importVaultData({
actorId: req.permission.id,
actor: req.permission.type,
actorOrgId: req.permission.orgId,
actorAuthMethod: req.permission.authMethod,
...req.body
});
}
});
};

View File

@@ -11,5 +11,5 @@ export const registerV3Routes = async (server: FastifyZodProvider) => {
await server.register(registerUserRouter, { prefix: "/users" });
await server.register(registerSecretRouter, { prefix: "/secrets" });
await server.register(registerSecretBlindIndexRouter, { prefix: "/workspaces" });
await server.register(registerExternalMigrationRouter, { prefix: "/migrate" });
await server.register(registerExternalMigrationRouter, { prefix: "/external-migration" });
};

View File

@@ -33,6 +33,7 @@ export enum AppConnection {
Bitbucket = "bitbucket",
Checkly = "checkly",
Supabase = "supabase",
DigitalOcean = "digital-ocean",
Okta = "okta"
}

View File

@@ -68,6 +68,11 @@ import {
getDatabricksConnectionListItem,
validateDatabricksConnectionCredentials
} from "./databricks";
import {
DigitalOceanConnectionMethod,
getDigitalOceanConnectionListItem,
validateDigitalOceanConnectionCredentials
} from "./digital-ocean";
import { FlyioConnectionMethod, getFlyioConnectionListItem, validateFlyioConnectionCredentials } from "./flyio";
import { GcpConnectionMethod, getGcpConnectionListItem, validateGcpConnectionCredentials } from "./gcp";
import { getGitHubConnectionListItem, GitHubConnectionMethod, validateGitHubConnectionCredentials } from "./github";
@@ -157,6 +162,7 @@ export const listAppConnectionOptions = () => {
getBitbucketConnectionListItem(),
getChecklyConnectionListItem(),
getSupabaseConnectionListItem(),
getDigitalOceanConnectionListItem(),
getOktaConnectionListItem()
].sort((a, b) => a.name.localeCompare(b.name));
};
@@ -244,6 +250,7 @@ export const validateAppConnectionCredentials = async (
[AppConnection.Bitbucket]: validateBitbucketConnectionCredentials as TAppConnectionCredentialsValidator,
[AppConnection.Checkly]: validateChecklyConnectionCredentials as TAppConnectionCredentialsValidator,
[AppConnection.Supabase]: validateSupabaseConnectionCredentials as TAppConnectionCredentialsValidator,
[AppConnection.DigitalOcean]: validateDigitalOceanConnectionCredentials as TAppConnectionCredentialsValidator,
[AppConnection.Okta]: validateOktaConnectionCredentials as TAppConnectionCredentialsValidator
};
@@ -283,6 +290,7 @@ export const getAppConnectionMethodName = (method: TAppConnection["method"]) =>
case CloudflareConnectionMethod.APIToken:
case BitbucketConnectionMethod.ApiToken:
case ZabbixConnectionMethod.ApiToken:
case DigitalOceanConnectionMethod.ApiToken:
case OktaConnectionMethod.ApiToken:
return "API Token";
case PostgresConnectionMethod.UsernameAndPassword:
@@ -372,6 +380,7 @@ export const TRANSITION_CONNECTION_CREDENTIALS_TO_PLATFORM: Record<
[AppConnection.Bitbucket]: platformManagedCredentialsNotSupported,
[AppConnection.Checkly]: platformManagedCredentialsNotSupported,
[AppConnection.Supabase]: platformManagedCredentialsNotSupported,
[AppConnection.DigitalOcean]: platformManagedCredentialsNotSupported,
[AppConnection.Okta]: platformManagedCredentialsNotSupported
};

View File

@@ -35,6 +35,7 @@ export const APP_CONNECTION_NAME_MAP: Record<AppConnection, string> = {
[AppConnection.Bitbucket]: "Bitbucket",
[AppConnection.Checkly]: "Checkly",
[AppConnection.Supabase]: "Supabase",
[AppConnection.DigitalOcean]: "DigitalOcean App Platform",
[AppConnection.Okta]: "Okta"
};
@@ -73,5 +74,6 @@ export const APP_CONNECTION_PLAN_MAP: Record<AppConnection, AppConnectionPlanTyp
[AppConnection.Bitbucket]: AppConnectionPlanType.Regular,
[AppConnection.Checkly]: AppConnectionPlanType.Regular,
[AppConnection.Supabase]: AppConnectionPlanType.Regular,
[AppConnection.DigitalOcean]: AppConnectionPlanType.Regular,
[AppConnection.Okta]: AppConnectionPlanType.Regular
};

View File

@@ -61,6 +61,8 @@ import { ValidateCloudflareConnectionCredentialsSchema } from "./cloudflare/clou
import { cloudflareConnectionService } from "./cloudflare/cloudflare-connection-service";
import { ValidateDatabricksConnectionCredentialsSchema } from "./databricks";
import { databricksConnectionService } from "./databricks/databricks-connection-service";
import { ValidateDigitalOceanConnectionCredentialsSchema } from "./digital-ocean";
import { digitalOceanAppPlatformConnectionService } from "./digital-ocean/digital-ocean-connection-service";
import { ValidateFlyioConnectionCredentialsSchema } from "./flyio";
import { flyioConnectionService } from "./flyio/flyio-connection-service";
import { ValidateGcpConnectionCredentialsSchema } from "./gcp";
@@ -145,6 +147,7 @@ const VALIDATE_APP_CONNECTION_CREDENTIALS_MAP: Record<AppConnection, TValidateAp
[AppConnection.Bitbucket]: ValidateBitbucketConnectionCredentialsSchema,
[AppConnection.Checkly]: ValidateChecklyConnectionCredentialsSchema,
[AppConnection.Supabase]: ValidateSupabaseConnectionCredentialsSchema,
[AppConnection.DigitalOcean]: ValidateDigitalOceanConnectionCredentialsSchema,
[AppConnection.Okta]: ValidateOktaConnectionCredentialsSchema
};
@@ -607,6 +610,7 @@ export const appConnectionServiceFactory = ({
bitbucket: bitbucketConnectionService(connectAppConnectionById),
checkly: checklyConnectionService(connectAppConnectionById),
supabase: supabaseConnectionService(connectAppConnectionById),
digitalOcean: digitalOceanAppPlatformConnectionService(connectAppConnectionById),
okta: oktaConnectionService(connectAppConnectionById)
};
};

View File

@@ -87,6 +87,12 @@ import {
TDatabricksConnectionInput,
TValidateDatabricksConnectionCredentialsSchema
} from "./databricks";
import {
TDigitalOceanConnection,
TDigitalOceanConnectionConfig,
TDigitalOceanConnectionInput,
TValidateDigitalOceanCredentialsSchema
} from "./digital-ocean";
import {
TFlyioConnection,
TFlyioConnectionConfig,
@@ -238,6 +244,7 @@ export type TAppConnection = { id: string } & (
| TRailwayConnection
| TChecklyConnection
| TSupabaseConnection
| TDigitalOceanConnection
| TOktaConnection
);
@@ -280,6 +287,7 @@ export type TAppConnectionInput = { id: string } & (
| TRailwayConnectionInput
| TChecklyConnectionInput
| TSupabaseConnectionInput
| TDigitalOceanConnectionInput
| TOktaConnectionInput
);
@@ -330,6 +338,7 @@ export type TAppConnectionConfig =
| TRailwayConnectionConfig
| TChecklyConnectionConfig
| TSupabaseConnectionConfig
| TDigitalOceanConnectionConfig
| TOktaConnectionConfig;
export type TValidateAppConnectionCredentialsSchema =
@@ -367,6 +376,7 @@ export type TValidateAppConnectionCredentialsSchema =
| TValidateRailwayConnectionCredentialsSchema
| TValidateChecklyConnectionCredentialsSchema
| TValidateSupabaseConnectionCredentialsSchema
| TValidateDigitalOceanCredentialsSchema
| TValidateOktaConnectionCredentialsSchema;
export type TListAwsConnectionKmsKeys = {

View File

@@ -9,6 +9,7 @@ import { BitbucketConnectionMethod } from "./bitbucket-connection-enums";
import {
TBitbucketConnection,
TBitbucketConnectionConfig,
TBitbucketEnvironment,
TBitbucketRepo,
TBitbucketWorkspace
} from "./bitbucket-connection-types";
@@ -21,11 +22,15 @@ export const getBitbucketConnectionListItem = () => {
};
};
export const createAuthHeader = (email: string, apiToken: string): string => {
return `Basic ${Buffer.from(`${email}:${apiToken}`).toString("base64")}`;
};
export const getBitbucketUser = async ({ email, apiToken }: { email: string; apiToken: string }) => {
try {
const { data } = await request.get<{ username: string }>(`${IntegrationUrls.BITBUCKET_API_URL}/2.0/user`, {
headers: {
Authorization: `Basic ${Buffer.from(`${email}:${apiToken}`).toString("base64")}`,
Authorization: createAuthHeader(email, apiToken),
Accept: "application/json"
}
});
@@ -57,7 +62,7 @@ export const listBitbucketWorkspaces = async (appConnection: TBitbucketConnectio
const { email, apiToken } = appConnection.credentials;
const headers = {
Authorization: `Basic ${Buffer.from(`${email}:${apiToken}`).toString("base64")}`,
Authorization: createAuthHeader(email, apiToken),
Accept: "application/json"
};
@@ -89,7 +94,7 @@ export const listBitbucketRepositories = async (appConnection: TBitbucketConnect
const { email, apiToken } = appConnection.credentials;
const headers = {
Authorization: `Basic ${Buffer.from(`${email}:${apiToken}`).toString("base64")}`,
Authorization: createAuthHeader(email, apiToken),
Accept: "application/json"
};
@@ -115,3 +120,43 @@ export const listBitbucketRepositories = async (appConnection: TBitbucketConnect
return allRepos;
};
export const listBitbucketEnvironments = async (
appConnection: TBitbucketConnection,
workspaceSlug: string,
repositorySlug: string
) => {
const { email, apiToken } = appConnection.credentials;
const headers = {
Authorization: createAuthHeader(email, apiToken),
Accept: "application/json"
};
const environments: TBitbucketEnvironment[] = [];
let hasNextPage = true;
let environmentsUrl = `${IntegrationUrls.BITBUCKET_API_URL}/2.0/repositories/${encodeURIComponent(workspaceSlug)}/${encodeURIComponent(repositorySlug)}/environments?pagelen=100`;
let iterationCount = 0;
// Limit to 10 iterations, fetching at most 10 * 100 = 1000 environments
while (hasNextPage && iterationCount < 10) {
// eslint-disable-next-line no-await-in-loop
const { data }: { data: { values: TBitbucketEnvironment[]; next: string } } = await request.get(environmentsUrl, {
headers
});
if (data?.values.length > 0) {
environments.push(...data.values);
}
if (data.next) {
environmentsUrl = data.next;
} else {
hasNextPage = false;
}
iterationCount += 1;
}
return environments;
};

View File

@@ -1,8 +1,16 @@
import { OrgServiceActor } from "@app/lib/types";
import { AppConnection } from "../app-connection-enums";
import { listBitbucketRepositories, listBitbucketWorkspaces } from "./bitbucket-connection-fns";
import { TBitbucketConnection, TGetBitbucketRepositoriesDTO } from "./bitbucket-connection-types";
import {
listBitbucketEnvironments,
listBitbucketRepositories,
listBitbucketWorkspaces
} from "./bitbucket-connection-fns";
import {
TBitbucketConnection,
TGetBitbucketEnvironmentsDTO,
TGetBitbucketRepositoriesDTO
} from "./bitbucket-connection-types";
type TGetAppConnectionFunc = (
app: AppConnection,
@@ -26,8 +34,18 @@ export const bitbucketConnectionService = (getAppConnection: TGetAppConnectionFu
return repositories;
};
const listEnvironments = async (
{ connectionId, workspaceSlug, repositorySlug }: TGetBitbucketEnvironmentsDTO,
actor: OrgServiceActor
) => {
const appConnection = await getAppConnection(AppConnection.Bitbucket, connectionId, actor);
const environments = await listBitbucketEnvironments(appConnection, workspaceSlug, repositorySlug);
return environments;
};
return {
listWorkspaces,
listRepositories
listRepositories,
listEnvironments
};
};

View File

@@ -38,3 +38,20 @@ export type TBitbucketRepo = {
full_name: string; // workspace-slug/repo-slug
slug: string;
};
export type TGetBitbucketEnvironmentsDTO = {
connectionId: string;
workspaceSlug: string;
repositorySlug: string;
};
export type TBitbucketEnvironment = {
uuid: string;
slug: string;
name: string;
};
export type TBitbucketEnvironmentsResponse = {
values: TBitbucketEnvironment[];
next?: string;
};

View File

@@ -0,0 +1,3 @@
export enum DigitalOceanConnectionMethod {
ApiToken = "api-token"
}

View File

@@ -0,0 +1,37 @@
/* eslint-disable no-await-in-loop */
import { AxiosError } from "axios";
import { z } from "zod";
import { BadRequestError } from "@app/lib/errors";
import { AppConnection } from "@app/services/app-connection/app-connection-enums";
import { DigitalOceanConnectionMethod } from "./digital-ocean-connection-constants";
import { DigitalOceanAppPlatformPublicAPI } from "./digital-ocean-connection-public-client";
import { DigitalOceanConnectionListItemSchema } from "./digital-ocean-connection-schemas";
import { TDigitalOceanConnectionConfig } from "./digital-ocean-connection-types";
export const getDigitalOceanConnectionListItem = () => {
return {
name: "Digital Ocean" as z.infer<typeof DigitalOceanConnectionListItemSchema>["name"],
app: AppConnection.DigitalOcean as const,
methods: Object.values(DigitalOceanConnectionMethod)
};
};
export const validateDigitalOceanConnectionCredentials = async (config: TDigitalOceanConnectionConfig) => {
try {
await DigitalOceanAppPlatformPublicAPI.healthcheck(config);
} catch (error: unknown) {
if (error instanceof AxiosError) {
throw new BadRequestError({
message: `Failed to validate credentials: ${error.message || "Unknown error"}`
});
}
throw new BadRequestError({
message: "Unable to validate connection - verify credentials"
});
}
return config.credentials;
};

View File

@@ -0,0 +1,105 @@
/* eslint-disable no-await-in-loop */
/* eslint-disable class-methods-use-this */
import { AxiosInstance } from "axios";
import { createRequestClient } from "@app/lib/config/request";
import { IntegrationUrls } from "@app/services/integration-auth/integration-list";
import { DigitalOceanConnectionMethod } from "./digital-ocean-connection-constants";
import {
TDigitalOceanApp,
TDigitalOceanConnectionConfig,
TDigitalOceanVariable
} from "./digital-ocean-connection-types";
class DigitalOceanAppPlatformPublicClient {
private readonly client: AxiosInstance;
constructor() {
this.client = createRequestClient({
baseURL: `${IntegrationUrls.DIGITAL_OCEAN_API_URL}/v2`,
headers: {
"Content-Type": "application/json",
Accept: "application/json"
}
});
}
async healthcheck(connection: TDigitalOceanConnectionConfig) {
switch (connection.method) {
case DigitalOceanConnectionMethod.ApiToken:
await this.getApps(connection);
break;
default:
throw new Error(`Unsupported connection method`);
}
}
async getApps(connection: TDigitalOceanConnectionConfig) {
const response = await this.client.get<{ apps: TDigitalOceanApp[] }>(`/apps`, {
headers: {
Authorization: `Bearer ${connection.credentials.apiToken}`
}
});
return response.data.apps;
}
async getApp(connection: TDigitalOceanConnectionConfig, appId: string) {
const response = await this.client.get<{ app: TDigitalOceanApp }>(`/apps/${appId}`, {
headers: {
Authorization: `Bearer ${connection.credentials.apiToken}`
}
});
return response.data.app;
}
async getVariables(connection: TDigitalOceanConnectionConfig, appId: string): Promise<TDigitalOceanVariable[]> {
const app = await this.getApp(connection, appId);
return app.spec.envs || [];
}
async putVariables(connection: TDigitalOceanConnectionConfig, appId: string, ...input: TDigitalOceanVariable[]) {
const response = await this.getApp(connection, appId);
return this.client.put(
`/apps/${appId}`,
{
spec: {
...response.spec,
envs: input
}
},
{
headers: {
Authorization: `Bearer ${connection.credentials.apiToken}`
}
}
);
}
async deleteVariables(connection: TDigitalOceanConnectionConfig, appId: string, ...input: TDigitalOceanVariable[]) {
const response = await this.getApp(connection, appId);
const existing = response.spec.envs || [];
const variables = existing.filter((v) => input.find((i) => i.key === v.key));
return this.client.put(
`/apps/${appId}`,
{
spec: {
...response.spec,
envs: variables
}
},
{
headers: {
Authorization: `Bearer ${connection.credentials.apiToken}`
}
}
);
}
}
export const DigitalOceanAppPlatformPublicAPI = new DigitalOceanAppPlatformPublicClient();

View File

@@ -0,0 +1,67 @@
import z from "zod";
import { AppConnections } from "@app/lib/api-docs";
import { AppConnection } from "@app/services/app-connection/app-connection-enums";
import {
BaseAppConnectionSchema,
GenericCreateAppConnectionFieldsSchema,
GenericUpdateAppConnectionFieldsSchema
} from "@app/services/app-connection/app-connection-schemas";
import { DigitalOceanConnectionMethod } from "./digital-ocean-connection-constants";
export const DigitalOceanConnectionMethodSchema = z
.nativeEnum(DigitalOceanConnectionMethod)
.describe(AppConnections.CREATE(AppConnection.DigitalOcean).method);
export const DigitalOceanConnectionAccessTokenCredentialsSchema = z.object({
apiToken: z
.string()
.trim()
.min(1, "API Token required")
.max(255)
.describe(AppConnections.CREDENTIALS.DIGITAL_OCEAN_APP_PLATFORM.apiToken)
});
const BaseDigitalOceanConnectionSchema = BaseAppConnectionSchema.extend({
app: z.literal(AppConnection.DigitalOcean)
});
export const DigitalOceanConnectionSchema = BaseDigitalOceanConnectionSchema.extend({
method: DigitalOceanConnectionMethodSchema,
credentials: DigitalOceanConnectionAccessTokenCredentialsSchema
});
export const SanitizedDigitalOceanConnectionSchema = z.discriminatedUnion("method", [
BaseDigitalOceanConnectionSchema.extend({
method: DigitalOceanConnectionMethodSchema,
credentials: DigitalOceanConnectionAccessTokenCredentialsSchema.pick({})
})
]);
export const ValidateDigitalOceanConnectionCredentialsSchema = z.discriminatedUnion("method", [
z.object({
method: DigitalOceanConnectionMethodSchema,
credentials: DigitalOceanConnectionAccessTokenCredentialsSchema.describe(
AppConnections.CREATE(AppConnection.DigitalOcean).credentials
)
})
]);
export const CreateDigitalOceanConnectionSchema = ValidateDigitalOceanConnectionCredentialsSchema.and(
GenericCreateAppConnectionFieldsSchema(AppConnection.DigitalOcean)
);
export const UpdateDigitalOceanConnectionSchema = z
.object({
credentials: DigitalOceanConnectionAccessTokenCredentialsSchema.optional().describe(
AppConnections.UPDATE(AppConnection.DigitalOcean).credentials
)
})
.and(GenericUpdateAppConnectionFieldsSchema(AppConnection.DigitalOcean));
export const DigitalOceanConnectionListItemSchema = z.object({
name: z.literal("Digital Ocean"),
app: z.literal(AppConnection.DigitalOcean),
methods: z.nativeEnum(DigitalOceanConnectionMethod).array()
});

View File

@@ -0,0 +1,29 @@
import { logger } from "@app/lib/logger";
import { OrgServiceActor } from "@app/lib/types";
import { AppConnection } from "../app-connection-enums";
import { DigitalOceanAppPlatformPublicAPI } from "./digital-ocean-connection-public-client";
import { TDigitalOceanConnection } from "./digital-ocean-connection-types";
type TGetAppConnectionFunc = (
app: AppConnection,
connectionId: string,
actor: OrgServiceActor
) => Promise<TDigitalOceanConnection>;
export const digitalOceanAppPlatformConnectionService = (getAppConnection: TGetAppConnectionFunc) => {
const listApps = async (connectionId: string, actor: OrgServiceActor) => {
const connection = await getAppConnection(AppConnection.DigitalOcean, connectionId, actor);
try {
const apps = await DigitalOceanAppPlatformPublicAPI.getApps(connection);
return apps;
} catch (error) {
logger.error(error, "Failed to list apps on Digital Ocean");
return [];
}
};
return {
listApps
};
};

View File

@@ -0,0 +1,42 @@
import z from "zod";
import { DiscriminativePick } from "@app/lib/types";
import { AppConnection } from "../app-connection-enums";
import {
CreateDigitalOceanConnectionSchema,
DigitalOceanConnectionSchema,
ValidateDigitalOceanConnectionCredentialsSchema
} from "./digital-ocean-connection-schemas";
export type TDigitalOceanConnection = z.infer<typeof DigitalOceanConnectionSchema>;
export type TDigitalOceanConnectionInput = z.infer<typeof CreateDigitalOceanConnectionSchema> & {
app: AppConnection.DigitalOcean;
};
export type TValidateDigitalOceanCredentialsSchema = typeof ValidateDigitalOceanConnectionCredentialsSchema;
export type TDigitalOceanConnectionConfig = DiscriminativePick<
TDigitalOceanConnection,
"method" | "app" | "credentials"
> & {
orgId: string;
};
export type TDigitalOceanVariable = {
key: string;
value: string;
type: "SECRET" | "GENERAL";
};
export type TDigitalOceanApp = {
id: string;
spec: {
name: string;
services: Array<{
name: string;
}>;
envs?: TDigitalOceanVariable[];
};
};

View File

@@ -0,0 +1,4 @@
export * from "./digital-ocean-connection-constants";
export * from "./digital-ocean-connection-fns";
export * from "./digital-ocean-connection-schemas";
export * from "./digital-ocean-connection-types";

View File

@@ -1,32 +1,26 @@
import slugify from "@sindresorhus/slugify";
import sjcl from "sjcl";
import tweetnacl from "tweetnacl";
import tweetnaclUtil from "tweetnacl-util";
import { SecretType, TSecretFolders } from "@app/db/schemas";
import { crypto } from "@app/lib/crypto/cryptography";
import { BadRequestError, NotFoundError } from "@app/lib/errors";
import { chunkArray } from "@app/lib/fn";
import { BadRequestError } from "@app/lib/errors";
import { logger } from "@app/lib/logger";
import { alphaNumericNanoId } from "@app/lib/nanoid";
import { CommitType, TFolderCommitServiceFactory } from "../folder-commit/folder-commit-service";
import { TKmsServiceFactory } from "../kms/kms-service";
import { KmsDataKey } from "../kms/kms-types";
import { TProjectDALFactory } from "../project/project-dal";
import { TProjectServiceFactory } from "../project/project-service";
import { TProjectEnvDALFactory } from "../project-env/project-env-dal";
import { TProjectEnvServiceFactory } from "../project-env/project-env-service";
import { TResourceMetadataDALFactory } from "../resource-metadata/resource-metadata-dal";
import { TSecretFolderDALFactory } from "../secret-folder/secret-folder-dal";
import { TSecretFolderVersionDALFactory } from "../secret-folder/secret-folder-version-dal";
import { TSecretTagDALFactory } from "../secret-tag/secret-tag-dal";
import { TSecretV2BridgeDALFactory } from "../secret-v2-bridge/secret-v2-bridge-dal";
import { fnSecretBulkInsert, getAllSecretReferences } from "../secret-v2-bridge/secret-v2-bridge-fns";
import type { TSecretV2BridgeServiceFactory } from "../secret-v2-bridge/secret-v2-bridge-service";
import { TSecretVersionV2DALFactory } from "../secret-v2-bridge/secret-version-dal";
import { TSecretVersionV2TagDALFactory } from "../secret-v2-bridge/secret-version-tag-dal";
import { InfisicalImportData, TEnvKeyExportJSON, TImportInfisicalDataCreate } from "./external-migration-types";
import { TFolderCommitServiceFactory } from "../../folder-commit/folder-commit-service";
import { TKmsServiceFactory } from "../../kms/kms-service";
import { TProjectDALFactory } from "../../project/project-dal";
import { TProjectServiceFactory } from "../../project/project-service";
import { TProjectEnvDALFactory } from "../../project-env/project-env-dal";
import { TProjectEnvServiceFactory } from "../../project-env/project-env-service";
import { TResourceMetadataDALFactory } from "../../resource-metadata/resource-metadata-dal";
import { TSecretFolderDALFactory } from "../../secret-folder/secret-folder-dal";
import { TSecretFolderVersionDALFactory } from "../../secret-folder/secret-folder-version-dal";
import { TSecretTagDALFactory } from "../../secret-tag/secret-tag-dal";
import { TSecretV2BridgeDALFactory } from "../../secret-v2-bridge/secret-v2-bridge-dal";
import type { TSecretV2BridgeServiceFactory } from "../../secret-v2-bridge/secret-v2-bridge-service";
import { TSecretVersionV2DALFactory } from "../../secret-v2-bridge/secret-version-dal";
import { TSecretVersionV2TagDALFactory } from "../../secret-v2-bridge/secret-version-tag-dal";
import { InfisicalImportData, TEnvKeyExportJSON, TImportInfisicalDataCreate } from "../external-migration-types";
export type TImportDataIntoInfisicalDTO = {
projectDAL: Pick<TProjectDALFactory, "transaction">;
@@ -499,326 +493,3 @@ export const parseEnvKeyDataFn = async (decryptedJson: string): Promise<Infisica
return infisicalImportData;
};
export const importDataIntoInfisicalFn = async ({
projectService,
projectEnvDAL,
projectDAL,
secretDAL,
kmsService,
secretVersionDAL,
secretTagDAL,
secretVersionTagDAL,
folderDAL,
resourceMetadataDAL,
folderVersionDAL,
folderCommitService,
input: { data, actor, actorId, actorOrgId, actorAuthMethod }
}: TImportDataIntoInfisicalDTO) => {
// Import data to infisical
if (!data || !data.projects) {
throw new BadRequestError({ message: "No projects found in data" });
}
const originalToNewProjectId = new Map<string, string>();
const originalToNewEnvironmentId = new Map<
string,
{ envId: string; envSlug: string; rootFolderId: string; projectId: string }
>();
const originalToNewFolderId = new Map<
string,
{
folderId: string;
projectId: string;
}
>();
const projectsNotImported: string[] = [];
await projectDAL.transaction(async (tx) => {
for await (const project of data.projects) {
const newProject = await projectService
.createProject({
actor,
actorId,
actorOrgId,
actorAuthMethod,
workspaceName: project.name,
createDefaultEnvs: false,
tx
})
.catch((e) => {
logger.error(e, `Failed to import to project [name:${project.name}]`);
throw new BadRequestError({ message: `Failed to import to project [name:${project.name}]` });
});
originalToNewProjectId.set(project.id, newProject.id);
}
// Import environments
if (data.environments) {
for await (const environment of data.environments) {
const projectId = originalToNewProjectId.get(environment.projectId);
const slug = slugify(`${environment.name}-${alphaNumericNanoId(4)}`);
if (!projectId) {
projectsNotImported.push(environment.projectId);
// eslint-disable-next-line no-continue
continue;
}
const existingEnv = await projectEnvDAL.findOne({ projectId, slug }, tx);
if (existingEnv) {
throw new BadRequestError({
message: `Environment with slug '${slug}' already exist`,
name: "CreateEnvironment"
});
}
const lastPos = await projectEnvDAL.findLastEnvPosition(projectId, tx);
const doc = await projectEnvDAL.create({ slug, name: environment.name, projectId, position: lastPos + 1 }, tx);
const folder = await folderDAL.create({ name: "root", parentId: null, envId: doc.id, version: 1 }, tx);
originalToNewEnvironmentId.set(environment.id, {
envSlug: doc.slug,
envId: doc.id,
rootFolderId: folder.id,
projectId
});
}
}
if (data.folders) {
for await (const folder of data.folders) {
const parentEnv = originalToNewEnvironmentId.get(folder.parentFolderId as string);
if (!parentEnv) {
// eslint-disable-next-line no-continue
continue;
}
const newFolder = await folderDAL.create(
{
name: folder.name,
envId: parentEnv.envId,
parentId: parentEnv.rootFolderId
},
tx
);
const newFolderVersion = await folderVersionDAL.create(
{
name: newFolder.name,
envId: newFolder.envId,
version: newFolder.version,
folderId: newFolder.id
},
tx
);
await folderCommitService.createCommit(
{
actor: {
type: actor,
metadata: {
id: actorId
}
},
message: "Changed by external migration",
folderId: parentEnv.rootFolderId,
changes: [
{
type: CommitType.ADD,
folderVersionId: newFolderVersion.id
}
]
},
tx
);
originalToNewFolderId.set(folder.id, {
folderId: newFolder.id,
projectId: parentEnv.projectId
});
}
}
// Useful for debugging:
// console.log("data.secrets", data.secrets);
// console.log("data.folders", data.folders);
// console.log("data.environment", data.environments);
if (data.secrets && data.secrets.length > 0) {
const mappedToEnvironmentId = new Map<
string,
{
secretKey: string;
secretValue: string;
folderId?: string;
isFromBlock?: boolean;
}[]
>();
for (const secret of data.secrets) {
const targetId = secret.folderId || secret.environmentId;
// Skip if we can't find either an environment or folder mapping for this secret
if (!originalToNewEnvironmentId.get(secret.environmentId) && !originalToNewFolderId.get(targetId)) {
logger.info({ secret }, "[importDataIntoInfisicalFn]: Could not find environment or folder for secret");
// eslint-disable-next-line no-continue
continue;
}
if (!mappedToEnvironmentId.has(targetId)) {
mappedToEnvironmentId.set(targetId, []);
}
const alreadyHasSecret = mappedToEnvironmentId
.get(targetId)!
.find((el) => el.secretKey === secret.name && el.folderId === secret.folderId);
if (alreadyHasSecret && alreadyHasSecret.isFromBlock) {
// remove the existing secret if any
mappedToEnvironmentId
.get(targetId)!
.splice(mappedToEnvironmentId.get(targetId)!.indexOf(alreadyHasSecret), 1);
}
mappedToEnvironmentId.get(targetId)!.push({
secretKey: secret.name,
secretValue: secret.value || "",
folderId: secret.folderId,
isFromBlock: secret.appBlockOrderIndex !== undefined
});
}
// for each of the mappedEnvironmentId
for await (const [targetId, secrets] of mappedToEnvironmentId) {
logger.info("[importDataIntoInfisicalFn]: Processing secrets for targetId", targetId);
let selectedFolder: TSecretFolders | undefined;
let selectedProjectId: string | undefined;
// Case 1: Secret belongs to a folder / branch / branch of a block
const foundFolder = originalToNewFolderId.get(targetId);
if (foundFolder) {
logger.info("[importDataIntoInfisicalFn]: Processing secrets for folder");
selectedFolder = await folderDAL.findById(foundFolder.folderId, tx);
selectedProjectId = foundFolder.projectId;
} else {
logger.info("[importDataIntoInfisicalFn]: Processing secrets for normal environment");
const environment = data.environments.find((env) => env.id === targetId);
if (!environment) {
logger.info(
{
targetId
},
"[importDataIntoInfisicalFn]: Could not find environment for secret"
);
// eslint-disable-next-line no-continue
continue;
}
const projectId = originalToNewProjectId.get(environment.projectId)!;
if (!projectId) {
throw new BadRequestError({ message: `Failed to import secret, project not found` });
}
const env = originalToNewEnvironmentId.get(targetId);
if (!env) {
logger.info(
{
targetId
},
"[importDataIntoInfisicalFn]: Could not find environment for secret"
);
// eslint-disable-next-line no-continue
continue;
}
const folder = await folderDAL.findBySecretPath(projectId, env.envSlug, "/", tx);
if (!folder) {
throw new NotFoundError({
message: `Folder not found for the given environment slug (${env.envSlug}) & secret path (/)`,
name: "Create secret"
});
}
selectedFolder = folder;
selectedProjectId = projectId;
}
if (!selectedFolder) {
throw new NotFoundError({
message: `Folder not found for the given environment slug & secret path`,
name: "CreateSecret"
});
}
if (!selectedProjectId) {
throw new NotFoundError({
message: `Project not found for the given environment slug & secret path`,
name: "CreateSecret"
});
}
const { encryptor: secretManagerEncrypt } = await kmsService.createCipherPairWithDataKey(
{
type: KmsDataKey.SecretManager,
projectId: selectedProjectId
},
tx
);
const secretBatches = chunkArray(secrets, 2500);
for await (const secretBatch of secretBatches) {
const secretsByKeys = await secretDAL.findBySecretKeys(
selectedFolder.id,
secretBatch.map((el) => ({
key: el.secretKey,
type: SecretType.Shared
})),
tx
);
if (secretsByKeys.length) {
throw new BadRequestError({
message: `Secret already exist: ${secretsByKeys.map((el) => el.key).join(",")}`
});
}
await fnSecretBulkInsert({
inputSecrets: secretBatch.map((el) => {
const references = getAllSecretReferences(el.secretValue).nestedReferences;
return {
version: 1,
encryptedValue: el.secretValue
? secretManagerEncrypt({ plainText: Buffer.from(el.secretValue) }).cipherTextBlob
: undefined,
key: el.secretKey,
references,
type: SecretType.Shared
};
}),
folderId: selectedFolder.id,
orgId: actorOrgId,
resourceMetadataDAL,
secretDAL,
secretVersionDAL,
secretTagDAL,
secretVersionTagDAL,
folderCommitService,
actor: {
type: actor,
actorId
},
tx
});
}
}
}
});
return { projectsNotImported };
};

View File

@@ -0,0 +1,352 @@
import slugify from "@sindresorhus/slugify";
import { SecretType, TSecretFolders } from "@app/db/schemas";
import { BadRequestError, NotFoundError } from "@app/lib/errors";
import { chunkArray } from "@app/lib/fn";
import { logger } from "@app/lib/logger";
import { alphaNumericNanoId } from "@app/lib/nanoid";
import { CommitType } from "@app/services/folder-commit/folder-commit-service";
import { KmsDataKey } from "@app/services/kms/kms-types";
import { fnSecretBulkInsert, getAllSecretReferences } from "@app/services/secret-v2-bridge/secret-v2-bridge-fns";
import { TImportDataIntoInfisicalDTO } from "./envkey";
export const importDataIntoInfisicalFn = async ({
projectService,
projectEnvDAL,
projectDAL,
secretDAL,
kmsService,
secretVersionDAL,
secretTagDAL,
secretVersionTagDAL,
folderDAL,
resourceMetadataDAL,
folderVersionDAL,
folderCommitService,
input: { data, actor, actorId, actorOrgId, actorAuthMethod }
}: TImportDataIntoInfisicalDTO) => {
// Import data to infisical
if (!data || !data.projects) {
throw new BadRequestError({ message: "No projects found in data" });
}
const originalToNewProjectId = new Map<string, string>();
const originalToNewEnvironmentId = new Map<
string,
{ envId: string; envSlug: string; rootFolderId?: string; projectId: string }
>();
const originalToNewFolderId = new Map<
string,
{
envId: string;
envSlug: string;
folderId: string;
projectId: string;
}
>();
const projectsNotImported: string[] = [];
await projectDAL.transaction(async (tx) => {
for await (const project of data.projects) {
const newProject = await projectService
.createProject({
actor,
actorId,
actorOrgId,
actorAuthMethod,
workspaceName: project.name,
createDefaultEnvs: false,
tx
})
.catch((e) => {
logger.error(e, `Failed to import to project [name:${project.name}]`);
throw new BadRequestError({ message: `Failed to import to project [name:${project.name}]` });
});
originalToNewProjectId.set(project.id, newProject.id);
}
// Import environments
if (data.environments) {
for await (const environment of data.environments) {
const projectId = originalToNewProjectId.get(environment.projectId);
const slug = slugify(`${environment.name}-${alphaNumericNanoId(4)}`);
if (!projectId) {
projectsNotImported.push(environment.projectId);
// eslint-disable-next-line no-continue
continue;
}
const existingEnv = await projectEnvDAL.findOne({ projectId, slug }, tx);
if (existingEnv) {
throw new BadRequestError({
message: `Environment with slug '${slug}' already exist`,
name: "CreateEnvironment"
});
}
const lastPos = await projectEnvDAL.findLastEnvPosition(projectId, tx);
const doc = await projectEnvDAL.create({ slug, name: environment.name, projectId, position: lastPos + 1 }, tx);
const folder = await folderDAL.create({ name: "root", parentId: null, envId: doc.id, version: 1 }, tx);
originalToNewEnvironmentId.set(environment.id, {
envSlug: doc.slug,
envId: doc.id,
rootFolderId: folder.id,
projectId
});
}
}
if (data.folders) {
for await (const folder of data.folders) {
const parentEnv = originalToNewEnvironmentId.get(folder.parentFolderId as string);
const parentFolder = originalToNewFolderId.get(folder.parentFolderId as string);
let newFolder: TSecretFolders;
if (parentEnv?.rootFolderId) {
newFolder = await folderDAL.create(
{
name: folder.name,
envId: parentEnv.envId,
parentId: parentEnv.rootFolderId
},
tx
);
} else if (parentFolder) {
newFolder = await folderDAL.create(
{
name: folder.name,
envId: parentFolder.envId,
parentId: parentFolder.folderId
},
tx
);
} else {
logger.info({ folder }, "No parent environment found for folder");
// eslint-disable-next-line no-continue
continue;
}
const newFolderVersion = await folderVersionDAL.create(
{
name: newFolder.name,
envId: newFolder.envId,
version: newFolder.version,
folderId: newFolder.id
},
tx
);
await folderCommitService.createCommit(
{
actor: {
type: actor,
metadata: {
id: actorId
}
},
message: "Changed by external migration",
folderId: parentEnv?.rootFolderId || parentFolder?.folderId || "",
changes: [
{
type: CommitType.ADD,
folderVersionId: newFolderVersion.id
}
]
},
tx
);
originalToNewFolderId.set(folder.id, {
folderId: newFolder.id,
envId: parentEnv?.envId || parentFolder?.envId || "",
envSlug: parentEnv?.envSlug || parentFolder?.envSlug || "",
projectId: parentEnv?.projectId || parentFolder?.projectId || ""
});
}
}
// Useful for debugging:
// console.log("data.secrets", data.secrets);
// console.log("data.folders", data.folders);
// console.log("data.environment", data.environments);
if (data.secrets && data.secrets.length > 0) {
const mappedToEnvironmentId = new Map<
string,
{
secretKey: string;
secretValue: string;
folderId?: string;
isFromBlock?: boolean;
}[]
>();
for (const secret of data.secrets) {
const targetId = secret.folderId || secret.environmentId;
// Skip if we can't find either an environment or folder mapping for this secret
if (!originalToNewEnvironmentId.get(secret.environmentId) && !originalToNewFolderId.get(targetId)) {
logger.info({ secret }, "[importDataIntoInfisicalFn]: Could not find environment or folder for secret");
// eslint-disable-next-line no-continue
continue;
}
if (!mappedToEnvironmentId.has(targetId)) {
mappedToEnvironmentId.set(targetId, []);
}
const alreadyHasSecret = mappedToEnvironmentId
.get(targetId)!
.find((el) => el.secretKey === secret.name && el.folderId === secret.folderId);
if (alreadyHasSecret && alreadyHasSecret.isFromBlock) {
// remove the existing secret if any
mappedToEnvironmentId
.get(targetId)!
.splice(mappedToEnvironmentId.get(targetId)!.indexOf(alreadyHasSecret), 1);
}
mappedToEnvironmentId.get(targetId)!.push({
secretKey: secret.name,
secretValue: secret.value || "",
folderId: secret.folderId,
isFromBlock: secret.appBlockOrderIndex !== undefined
});
}
// for each of the mappedEnvironmentId
for await (const [targetId, secrets] of mappedToEnvironmentId) {
logger.info("[importDataIntoInfisicalFn]: Processing secrets for targetId", targetId);
let selectedFolder: TSecretFolders | undefined;
let selectedProjectId: string | undefined;
// Case 1: Secret belongs to a folder / branch / branch of a block
const foundFolder = originalToNewFolderId.get(targetId);
if (foundFolder) {
logger.info("[importDataIntoInfisicalFn]: Processing secrets for folder");
selectedFolder = await folderDAL.findById(foundFolder.folderId, tx);
selectedProjectId = foundFolder.projectId;
} else {
logger.info("[importDataIntoInfisicalFn]: Processing secrets for normal environment");
const environment = data.environments.find((env) => env.id === targetId);
if (!environment) {
logger.info(
{
targetId
},
"[importDataIntoInfisicalFn]: Could not find environment for secret"
);
// eslint-disable-next-line no-continue
continue;
}
const projectId = originalToNewProjectId.get(environment.projectId)!;
if (!projectId) {
throw new BadRequestError({ message: `Failed to import secret, project not found` });
}
const env = originalToNewEnvironmentId.get(targetId);
if (!env) {
logger.info(
{
targetId
},
"[importDataIntoInfisicalFn]: Could not find environment for secret"
);
// eslint-disable-next-line no-continue
continue;
}
const folder = await folderDAL.findBySecretPath(projectId, env.envSlug, "/", tx);
if (!folder) {
throw new NotFoundError({
message: `Folder not found for the given environment slug (${env.envSlug}) & secret path (/)`,
name: "Create secret"
});
}
selectedFolder = folder;
selectedProjectId = projectId;
}
if (!selectedFolder) {
throw new NotFoundError({
message: `Folder not found for the given environment slug & secret path`,
name: "CreateSecret"
});
}
if (!selectedProjectId) {
throw new NotFoundError({
message: `Project not found for the given environment slug & secret path`,
name: "CreateSecret"
});
}
const { encryptor: secretManagerEncrypt } = await kmsService.createCipherPairWithDataKey(
{
type: KmsDataKey.SecretManager,
projectId: selectedProjectId
},
tx
);
const secretBatches = chunkArray(secrets, 2500);
for await (const secretBatch of secretBatches) {
const secretsByKeys = await secretDAL.findBySecretKeys(
selectedFolder.id,
secretBatch.map((el) => ({
key: el.secretKey,
type: SecretType.Shared
})),
tx
);
if (secretsByKeys.length) {
throw new BadRequestError({
message: `Secret already exist: ${secretsByKeys.map((el) => el.key).join(",")}`
});
}
await fnSecretBulkInsert({
inputSecrets: secretBatch.map((el) => {
const references = getAllSecretReferences(el.secretValue).nestedReferences;
return {
version: 1,
encryptedValue: el.secretValue
? secretManagerEncrypt({ plainText: Buffer.from(el.secretValue) }).cipherTextBlob
: undefined,
key: el.secretKey,
references,
type: SecretType.Shared
};
}),
folderId: selectedFolder.id,
orgId: actorOrgId,
resourceMetadataDAL,
secretDAL,
secretVersionDAL,
secretTagDAL,
secretVersionTagDAL,
folderCommitService,
actor: {
type: actor,
actorId
},
tx
});
}
}
}
});
return { projectsNotImported };
};

View File

@@ -0,0 +1,3 @@
export * from "./envkey";
export * from "./import";
export * from "./vault";

View File

@@ -0,0 +1,341 @@
import axios, { AxiosInstance } from "axios";
import { v4 as uuidv4 } from "uuid";
import { BadRequestError } from "@app/lib/errors";
import { logger } from "@app/lib/logger";
import { blockLocalAndPrivateIpAddresses } from "@app/lib/validator";
import { InfisicalImportData, VaultMappingType } from "../external-migration-types";
type VaultData = {
namespace: string;
mount: string;
path: string;
secretData: Record<string, string>;
};
const vaultFactory = () => {
const getMounts = async (request: AxiosInstance) => {
const response = await request
.get<
Record<
string,
{
accessor: string;
options: {
version?: string;
} | null;
type: string;
}
>
>("/v1/sys/mounts")
.catch((err) => {
if (axios.isAxiosError(err)) {
logger.error(err.response?.data, "External migration: Failed to get Vault mounts");
}
throw err;
});
return response.data;
};
const getPaths = async (
request: AxiosInstance,
{ mountPath, secretPath = "" }: { mountPath: string; secretPath?: string }
) => {
try {
// For KV v2: /v1/{mount}/metadata/{path}?list=true
const path = secretPath ? `${mountPath}/metadata/${secretPath}` : `${mountPath}/metadata`;
const response = await request.get<{
data: {
keys: string[];
};
}>(`/v1/${path}?list=true`);
return response.data.data.keys;
} catch (err) {
if (axios.isAxiosError(err)) {
logger.error(err.response?.data, "External migration: Failed to get Vault paths");
if (err.response?.status === 404) {
return null;
}
}
throw err;
}
};
const getSecrets = async (
request: AxiosInstance,
{ mountPath, secretPath }: { mountPath: string; secretPath: string }
) => {
// For KV v2: /v1/{mount}/data/{path}
const response = await request
.get<{
data: {
data: Record<string, string>; // KV v2 has nested data structure
metadata: {
created_time: string;
deletion_time: string;
destroyed: boolean;
version: number;
};
};
}>(`/v1/${mountPath}/data/${secretPath}`)
.catch((err) => {
if (axios.isAxiosError(err)) {
logger.error(err.response?.data, "External migration: Failed to get Vault secret");
}
throw err;
});
return response.data.data.data;
};
// helper function to check if a mount is KV v2 (will be useful if we add support for Vault KV v1)
// const isKvV2Mount = (mountInfo: { type: string; options?: { version?: string } | null }) => {
// return mountInfo.type === "kv" && mountInfo.options?.version === "2";
// };
const recursivelyGetAllPaths = async (
request: AxiosInstance,
mountPath: string,
currentPath: string = ""
): Promise<string[]> => {
const paths = await getPaths(request, { mountPath, secretPath: currentPath });
if (paths === null || paths.length === 0) {
return [];
}
const allSecrets: string[] = [];
for await (const path of paths) {
const cleanPath = path.endsWith("/") ? path.slice(0, -1) : path;
const fullItemPath = currentPath ? `${currentPath}/${cleanPath}` : cleanPath;
if (path.endsWith("/")) {
// it's a folder so we recurse into it
const subSecrets = await recursivelyGetAllPaths(request, mountPath, fullItemPath);
allSecrets.push(...subSecrets);
} else {
// it's a secret so we add it to our results
allSecrets.push(`${mountPath}/${fullItemPath}`);
}
}
return allSecrets;
};
async function collectVaultData({
baseUrl,
namespace,
accessToken
}: {
baseUrl: string;
namespace?: string;
accessToken: string;
}): Promise<VaultData[]> {
const request = axios.create({
baseURL: baseUrl,
headers: {
"X-Vault-Token": accessToken,
...(namespace ? { "X-Vault-Namespace": namespace } : {})
}
});
const allData: VaultData[] = [];
// Get all mounts in this namespace
const mounts = await getMounts(request);
for (const mount of Object.keys(mounts)) {
if (!mount.endsWith("/")) {
delete mounts[mount];
}
}
for await (const [mountPath, mountInfo] of Object.entries(mounts)) {
// skip non-KV mounts
if (!mountInfo.type.startsWith("kv")) {
// eslint-disable-next-line no-continue
continue;
}
// get all paths in this mount
const paths = await recursivelyGetAllPaths(request, `${mountPath.replace(/\/$/, "")}`);
const cleanMountPath = mountPath.replace(/\/$/, "");
for await (const secretPath of paths) {
// get the actual secret data
const secretData = await getSecrets(request, {
mountPath: cleanMountPath,
secretPath: secretPath.replace(`${cleanMountPath}/`, "")
});
allData.push({
namespace: namespace || "",
mount: mountPath.replace(/\/$/, ""),
path: secretPath.replace(`${cleanMountPath}/`, ""),
secretData
});
}
}
return allData;
}
return {
collectVaultData,
getMounts,
getPaths,
getSecrets,
recursivelyGetAllPaths
};
};
export const transformToInfisicalFormatNamespaceToProjects = (
vaultData: VaultData[],
mappingType: VaultMappingType
): InfisicalImportData => {
const projects: Array<{ name: string; id: string }> = [];
const environments: Array<{ name: string; id: string; projectId: string; envParentId?: string }> = [];
const folders: Array<{ id: string; name: string; environmentId: string; parentFolderId?: string }> = [];
const secrets: Array<{ id: string; name: string; environmentId: string; value: string; folderId?: string }> = [];
// track created entities to avoid duplicates
const projectMap = new Map<string, string>(); // namespace -> projectId
const environmentMap = new Map<string, string>(); // namespace:mount -> environmentId
const folderMap = new Map<string, string>(); // namespace:mount:folderPath -> folderId
let environmentId: string = "";
for (const data of vaultData) {
const { namespace, mount, path, secretData } = data;
if (mappingType === VaultMappingType.Namespace) {
// create project (namespace)
if (!projectMap.has(namespace)) {
const projectId = uuidv4();
projectMap.set(namespace, projectId);
projects.push({
name: namespace,
id: projectId
});
}
const projectId = projectMap.get(namespace)!;
// create environment (mount)
const envKey = `${namespace}:${mount}`;
if (!environmentMap.has(envKey)) {
environmentId = uuidv4();
environmentMap.set(envKey, environmentId);
environments.push({
name: mount,
id: environmentId,
projectId
});
}
environmentId = environmentMap.get(envKey)!;
} else if (mappingType === VaultMappingType.KeyVault) {
if (!projectMap.has(mount)) {
const projectId = uuidv4();
projectMap.set(mount, projectId);
projects.push({
name: mount,
id: projectId
});
}
const projectId = projectMap.get(mount)!;
// create single "Production" environment per project, because we have no good way of determining environments from vault
if (!environmentMap.has(mount)) {
environmentId = uuidv4();
environmentMap.set(mount, environmentId);
environments.push({
name: "Production",
id: environmentId,
projectId
});
}
environmentId = environmentMap.get(mount)!;
}
// create folder structure
let currentFolderId: string | undefined;
let currentPath = "";
if (path.includes("/")) {
const pathParts = path.split("/").filter(Boolean);
const folderParts = pathParts;
// create nested folder structure for the entire path
for (const folderName of folderParts) {
currentPath = currentPath ? `${currentPath}/${folderName}` : folderName;
const folderKey = `${namespace}:${mount}:${currentPath}`;
if (!folderMap.has(folderKey)) {
const folderId = uuidv4();
folderMap.set(folderKey, folderId);
folders.push({
id: folderId,
name: folderName,
environmentId,
parentFolderId: currentFolderId || environmentId
});
currentFolderId = folderId;
} else {
currentFolderId = folderMap.get(folderKey)!;
}
}
}
for (const [key, value] of Object.entries(secretData)) {
secrets.push({
id: uuidv4(),
name: key,
environmentId,
value: String(value),
folderId: currentFolderId
});
}
}
return {
projects,
environments,
folders,
secrets
};
};
export const importVaultDataFn = async ({
vaultAccessToken,
vaultNamespace,
vaultUrl,
mappingType
}: {
vaultAccessToken: string;
vaultNamespace?: string;
vaultUrl: string;
mappingType: VaultMappingType;
}) => {
await blockLocalAndPrivateIpAddresses(vaultUrl);
if (mappingType === VaultMappingType.Namespace && !vaultNamespace) {
throw new BadRequestError({
message: "Vault namespace is required when project mapping type is set to namespace."
});
}
const vaultApi = vaultFactory();
const vaultData = await vaultApi.collectVaultData({
accessToken: vaultAccessToken,
baseUrl: vaultUrl,
namespace: vaultNamespace
});
const infisicalData = transformToInfisicalFormatNamespaceToProjects(vaultData, mappingType);
return infisicalData;
};

View File

@@ -19,7 +19,7 @@ import { TSecretVersionV2DALFactory } from "../secret-v2-bridge/secret-version-d
import { TSecretVersionV2TagDALFactory } from "../secret-v2-bridge/secret-version-tag-dal";
import { SmtpTemplates, TSmtpService } from "../smtp/smtp-service";
import { importDataIntoInfisicalFn } from "./external-migration-fns";
import { ExternalPlatforms, TImportInfisicalDataCreate } from "./external-migration-types";
import { ExternalPlatforms, ImportType, TImportInfisicalDataCreate } from "./external-migration-types";
export type TExternalMigrationQueueFactoryDep = {
smtpService: TSmtpService;
@@ -67,6 +67,7 @@ export const externalMigrationQueueFactory = ({
const startImport = async (dto: {
actorEmail: string;
data: {
importType: ImportType;
iv: string;
tag: string;
ciphertext: string;

View File

@@ -4,9 +4,9 @@ import { crypto } from "@app/lib/crypto/cryptography";
import { BadRequestError, ForbiddenRequestError } from "@app/lib/errors";
import { TUserDALFactory } from "../user/user-dal";
import { decryptEnvKeyDataFn, parseEnvKeyDataFn } from "./external-migration-fns";
import { decryptEnvKeyDataFn, importVaultDataFn, parseEnvKeyDataFn } from "./external-migration-fns";
import { TExternalMigrationQueueFactory } from "./external-migration-queue";
import { TImportEnvKeyDataCreate } from "./external-migration-types";
import { ImportType, TImportEnvKeyDataDTO, TImportVaultDataDTO } from "./external-migration-types";
type TExternalMigrationServiceFactoryDep = {
permissionService: TPermissionServiceFactory;
@@ -28,7 +28,7 @@ export const externalMigrationServiceFactory = ({
actorId,
actorOrgId,
actorAuthMethod
}: TImportEnvKeyDataCreate) => {
}: TImportEnvKeyDataDTO) => {
if (crypto.isFipsModeEnabled()) {
throw new BadRequestError({ message: "EnvKey migration is not supported when running in FIPS mode." });
}
@@ -60,11 +60,65 @@ export const externalMigrationServiceFactory = ({
await externalMigrationQueue.startImport({
actorEmail: user.email!,
data: encrypted
data: {
importType: ImportType.EnvKey,
...encrypted
}
});
};
const importVaultData = async ({
vaultAccessToken,
vaultNamespace,
mappingType,
vaultUrl,
actor,
actorId,
actorOrgId,
actorAuthMethod
}: TImportVaultDataDTO) => {
const { membership } = await permissionService.getOrgPermission(
actor,
actorId,
actorOrgId,
actorAuthMethod,
actorOrgId
);
if (membership.role !== OrgMembershipRole.Admin) {
throw new ForbiddenRequestError({ message: "Only admins can import data" });
}
const user = await userDAL.findById(actorId);
const vaultData = await importVaultDataFn({
vaultAccessToken,
vaultNamespace,
vaultUrl,
mappingType
});
const stringifiedJson = JSON.stringify({
data: vaultData,
actor,
actorId,
actorOrgId,
actorAuthMethod
});
const encrypted = crypto.encryption().symmetric().encryptWithRootEncryptionKey(stringifiedJson);
await externalMigrationQueue.startImport({
actorEmail: user.email!,
data: {
importType: ImportType.Vault,
...encrypted
}
});
};
return {
importEnvKeyData
importEnvKeyData,
importVaultData
};
};

View File

@@ -1,5 +1,17 @@
import { TOrgPermission } from "@app/lib/types";
import { ActorAuthMethod, ActorType } from "../auth/auth-type";
export enum ImportType {
EnvKey = "envkey",
Vault = "vault"
}
export enum VaultMappingType {
Namespace = "namespace",
KeyVault = "key-vault"
}
export type InfisicalImportData = {
projects: Array<{ name: string; id: string }>;
environments: Array<{ name: string; id: string; projectId: string; envParentId?: string }>;
@@ -14,14 +26,17 @@ export type InfisicalImportData = {
}>;
};
export type TImportEnvKeyDataCreate = {
export type TImportEnvKeyDataDTO = {
decryptionKey: string;
encryptedJson: { nonce: string; data: string };
actor: ActorType;
actorId: string;
actorOrgId: string;
actorAuthMethod: ActorAuthMethod;
};
} & Omit<TOrgPermission, "orgId">;
export type TImportVaultDataDTO = {
vaultAccessToken: string;
vaultNamespace?: string;
mappingType: VaultMappingType;
vaultUrl: string;
} & Omit<TOrgPermission, "orgId">;
export type TImportInfisicalDataCreate = {
data: InfisicalImportData;

View File

@@ -402,7 +402,7 @@ export const buildTeamsPayload = (notification: TNotification) => {
{
type: "Action.OpenUrl",
title: "View request in Infisical",
url: `${appCfg.SITE_URL}/projects/${payload.projectId}/secret-manager/approval?requestId=${payload.requestId}`
url: `${appCfg.SITE_URL}/projects/secret-management/${payload.projectId}/approval?requestId=${payload.requestId}`
}
]
};

View File

@@ -47,7 +47,7 @@ import { groupBy } from "@app/lib/fn";
import { logger } from "@app/lib/logger";
import { alphaNumericNanoId } from "@app/lib/nanoid";
import { isDisposableEmail } from "@app/lib/validator";
import { QueueName, TQueueServiceFactory } from "@app/queue";
import { QueueName } from "@app/queue";
import { getDefaultOrgMembershipRoleForUpdateOrg } from "@app/services/org/org-role-fns";
import { TOrgMembershipDALFactory } from "@app/services/org-membership/org-membership-dal";
import { TUserAliasDALFactory } from "@app/services/user-alias/user-alias-dal";
@@ -65,6 +65,7 @@ import { TProjectKeyDALFactory } from "../project-key/project-key-dal";
import { TProjectMembershipDALFactory } from "../project-membership/project-membership-dal";
import { TProjectUserMembershipRoleDALFactory } from "../project-membership/project-user-membership-role-dal";
import { TProjectRoleDALFactory } from "../project-role/project-role-dal";
import { TReminderServiceFactory } from "../reminder/reminder-types";
import { TSecretDALFactory } from "../secret/secret-dal";
import { fnDeleteProjectSecretReminders } from "../secret/secret-fns";
import { TSecretFolderDALFactory } from "../secret-folder/secret-folder-dal";
@@ -132,8 +133,8 @@ type TOrgServiceFactoryDep = {
projectBotDAL: Pick<TProjectBotDALFactory, "findOne" | "updateById">;
projectUserMembershipRoleDAL: Pick<TProjectUserMembershipRoleDALFactory, "insertMany" | "create">;
projectBotService: Pick<TProjectBotServiceFactory, "getBotKey">;
queueService: Pick<TQueueServiceFactory, "stopRepeatableJob">;
loginService: Pick<TAuthLoginFactory, "generateUserTokens">;
reminderService: Pick<TReminderServiceFactory, "deleteReminderBySecretId">;
};
export type TOrgServiceFactory = ReturnType<typeof orgServiceFactory>;
@@ -165,8 +166,8 @@ export const orgServiceFactory = ({
projectUserMembershipRoleDAL,
identityMetadataDAL,
projectBotService,
queueService,
loginService
loginService,
reminderService
}: TOrgServiceFactoryDep) => {
/*
* Get organization details by the organization id
@@ -609,7 +610,7 @@ export const orgServiceFactory = ({
await fnDeleteProjectSecretReminders(project.id, {
secretDAL,
secretV2BridgeDAL,
queueService,
reminderService,
projectBotService,
folderDAL
});

View File

@@ -39,7 +39,6 @@ import { BadRequestError, ForbiddenRequestError, NotFoundError } from "@app/lib/
import { groupBy } from "@app/lib/fn";
import { alphaNumericNanoId } from "@app/lib/nanoid";
import { TProjectPermission } from "@app/lib/types";
import { TQueueServiceFactory } from "@app/queue";
import { TPkiSubscriberDALFactory } from "@app/services/pki-subscriber/pki-subscriber-dal";
import { ActorAuthMethod, ActorType } from "../auth/auth-type";
@@ -67,6 +66,7 @@ import { TProjectMembershipDALFactory } from "../project-membership/project-memb
import { TProjectUserMembershipRoleDALFactory } from "../project-membership/project-user-membership-role-dal";
import { TProjectRoleDALFactory } from "../project-role/project-role-dal";
import { getPredefinedRoles } from "../project-role/project-role-fns";
import { TReminderServiceFactory } from "../reminder/reminder-types";
import { TSecretDALFactory } from "../secret/secret-dal";
import { fnDeleteProjectSecretReminders } from "../secret/secret-fns";
import { ROOT_FOLDER_NAME, TSecretFolderDALFactory } from "../secret-folder/secret-folder-dal";
@@ -169,7 +169,6 @@ type TProjectServiceFactoryDep = {
permissionService: TPermissionServiceFactory;
orgService: Pick<TOrgServiceFactory, "addGhostUser">;
licenseService: Pick<TLicenseServiceFactory, "getPlan" | "invalidateGetPlan">;
queueService: Pick<TQueueServiceFactory, "stopRepeatableJob">;
smtpService: Pick<TSmtpService, "sendMail">;
orgDAL: Pick<TOrgDALFactory, "findOne">;
keyStore: Pick<TKeyStoreFactory, "deleteItem">;
@@ -186,6 +185,7 @@ type TProjectServiceFactoryDep = {
| "createCipherPairWithDataKey"
>;
projectTemplateService: TProjectTemplateServiceFactory;
reminderService: Pick<TReminderServiceFactory, "deleteReminderBySecretId">;
};
export type TProjectServiceFactory = ReturnType<typeof projectServiceFactory>;
@@ -198,7 +198,6 @@ export const projectServiceFactory = ({
projectQueue,
projectKeyDAL,
permissionService,
queueService,
projectBotService,
orgDAL,
userDAL,
@@ -233,7 +232,8 @@ export const projectServiceFactory = ({
microsoftTeamsIntegrationDAL,
projectTemplateService,
groupProjectDAL,
smtpService
smtpService,
reminderService
}: TProjectServiceFactoryDep) => {
/*
* Create workspace. Make user the admin
@@ -574,7 +574,7 @@ export const projectServiceFactory = ({
await fnDeleteProjectSecretReminders(project.id, {
secretDAL,
secretV2BridgeDAL,
queueService,
reminderService,
projectBotService,
folderDAL
});
@@ -1948,6 +1948,13 @@ export const projectServiceFactory = ({
const userDetails = await userDAL.findById(permission.id);
const appCfg = getConfig();
let projectTypeUrl = project.type;
if (project.type === ProjectType.SecretManager) {
projectTypeUrl = "secret-management";
} else if (project.type === ProjectType.CertificateManager) {
projectTypeUrl = "cert-management";
}
await smtpService.sendMail({
template: SmtpTemplates.ProjectAccessRequest,
recipients: filteredProjectMembers,
@@ -1958,7 +1965,7 @@ export const projectServiceFactory = ({
projectName: project?.name,
orgName: org?.name,
note: comment,
callback_url: `${appCfg.SITE_URL}/${project.type}/${project.id}/access-management?selectedTab=members&requesterEmail=${userDetails.email}`
callback_url: `${appCfg.SITE_URL}/projects/${projectTypeUrl}/${project.id}/access-management?selectedTab=members&requesterEmail=${userDetails.email}`
}
});
};

View File

@@ -0,0 +1,11 @@
import { TDbClient } from "@app/db";
import { TableName } from "@app/db/schemas";
import { ormify } from "@app/lib/knex";
export type TReminderRecipientDALFactory = ReturnType<typeof reminderRecipientDALFactory>;
export const reminderRecipientDALFactory = (db: TDbClient) => {
const reminderRecipientOrm = ormify(db, TableName.ReminderRecipient);
return { ...reminderRecipientOrm };
};

View File

@@ -0,0 +1,133 @@
import { Knex } from "knex";
import { TDbClient } from "@app/db";
import {
TableName,
TOrganizations,
TProjectEnvironments,
TProjects,
TSecretFolders,
TSecretsV2,
TUsers
} from "@app/db/schemas";
import { RemindersSchema } from "@app/db/schemas/reminders";
import { ormify, selectAllTableCols, sqlNestRelationships } from "@app/lib/knex";
export type TReminderDALFactory = ReturnType<typeof reminderDALFactory>;
export const reminderDALFactory = (db: TDbClient) => {
const reminderOrm = ormify(db, TableName.Reminder);
const getTodayDateRange = () => {
const today = new Date();
const year = today.getUTCFullYear();
const month = today.getUTCMonth();
const date = today.getUTCDate();
// Start of day: 00:00:00.000 UTC
const startOfDay = new Date(Date.UTC(year, month, date, 0, 0, 0, 0));
// End of day: 23:59:59.999 UTC
const endOfDay = new Date(Date.UTC(year, month, date, 23, 59, 59, 999));
return {
startOfDay,
endOfDay
};
};
const findSecretDailyReminders = async (tx?: Knex) => {
const { startOfDay, endOfDay } = getTodayDateRange();
const rawReminders = await (tx || db)(TableName.Reminder)
.whereBetween("nextReminderDate", [startOfDay, endOfDay])
.leftJoin(TableName.ReminderRecipient, `${TableName.Reminder}.id`, `${TableName.ReminderRecipient}.reminderId`)
.leftJoin<TUsers>(TableName.Users, `${TableName.ReminderRecipient}.userId`, `${TableName.Users}.id`)
.leftJoin<TSecretsV2>(TableName.SecretV2, `${TableName.Reminder}.secretId`, `${TableName.SecretV2}.id`)
.leftJoin<TSecretFolders>(
TableName.SecretFolder,
`${TableName.SecretV2}.folderId`,
`${TableName.SecretFolder}.id`
)
.leftJoin<TProjectEnvironments>(
TableName.Environment,
`${TableName.SecretFolder}.envId`,
`${TableName.Environment}.id`
)
.leftJoin<TProjects>(TableName.Project, `${TableName.Environment}.projectId`, `${TableName.Project}.id`)
.leftJoin<TOrganizations>(TableName.Organization, `${TableName.Project}.orgId`, `${TableName.Organization}.id`)
.select(selectAllTableCols(TableName.Reminder))
.select(db.ref("email").withSchema(TableName.Users))
.select(db.ref("name").withSchema(TableName.Project).as("projectName"))
.select(db.ref("id").withSchema(TableName.Project).as("projectId"))
.select(db.ref("name").withSchema(TableName.Organization).as("organizationName"));
const reminders = sqlNestRelationships({
data: rawReminders,
key: "id",
parentMapper: (el) => ({
_id: el.id,
...RemindersSchema.parse(el),
projectName: el.projectName,
projectId: el.projectId,
organizationName: el.organizationName
}),
childrenMapper: [
{
key: "email",
label: "recipients" as const,
mapper: ({ email }) => ({
email
})
}
]
});
return reminders;
};
const findUpcomingReminders = async (daysAhead: number = 7, tx?: Knex) => {
const { startOfDay } = getTodayDateRange();
const futureDate = new Date(startOfDay);
futureDate.setDate(futureDate.getDate() + daysAhead);
const reminders = await (tx || db)(TableName.Reminder)
.where("nextReminderDate", ">=", startOfDay)
.where("nextReminderDate", "<=", futureDate)
.orderBy("nextReminderDate", "asc")
.leftJoin(TableName.ReminderRecipient, `${TableName.Reminder}.id`, `${TableName.ReminderRecipient}.reminderId`)
.select(selectAllTableCols(TableName.Reminder))
.select(db.ref("userId").withSchema(TableName.ReminderRecipient));
return reminders;
};
const findSecretReminder = async (secretId: string, tx?: Knex) => {
const rawReminders = await (tx || db)(TableName.Reminder)
.where(`${TableName.Reminder}.secretId`, secretId)
.leftJoin(TableName.ReminderRecipient, `${TableName.Reminder}.id`, `${TableName.ReminderRecipient}.reminderId`)
.select(selectAllTableCols(TableName.Reminder))
.select(db.ref("userId").withSchema(TableName.ReminderRecipient));
const reminders = sqlNestRelationships({
data: rawReminders,
key: "id",
parentMapper: (el) => ({
_id: el.id,
...RemindersSchema.parse(el)
}),
childrenMapper: [
{
key: "userId",
label: "recipients" as const,
mapper: ({ userId }) => userId
}
]
});
return reminders[0] || null;
};
return {
...reminderOrm,
findSecretDailyReminders,
findUpcomingReminders,
findSecretReminder
};
};

View File

@@ -0,0 +1,3 @@
export enum ReminderType {
SECRETS = "secrets"
}

View File

@@ -0,0 +1,196 @@
/* eslint-disable no-await-in-loop */
import RE2 from "re2";
import { logger } from "@app/lib/logger";
import { QueueJobs, QueueName, TQueueServiceFactory } from "@app/queue";
import { TSecretReminderRecipientsDALFactory } from "../secret-reminder-recipients/secret-reminder-recipients-dal";
import { TSecretV2BridgeDALFactory } from "../secret-v2-bridge/secret-v2-bridge-dal";
import { TReminderServiceFactory } from "./reminder-types";
type TDailyReminderQueueServiceFactoryDep = {
reminderService: TReminderServiceFactory;
queueService: TQueueServiceFactory;
secretDAL: Pick<TSecretV2BridgeDALFactory, "transaction" | "findSecretsWithReminderRecipients">;
secretReminderRecipientsDAL: Pick<TSecretReminderRecipientsDALFactory, "delete">;
};
export type TDailyReminderQueueServiceFactory = ReturnType<typeof dailyReminderQueueServiceFactory>;
const uuidRegex = new RE2(/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/i);
export const dailyReminderQueueServiceFactory = ({
reminderService,
queueService,
secretDAL,
secretReminderRecipientsDAL
}: TDailyReminderQueueServiceFactoryDep) => {
queueService.start(QueueName.DailyReminders, async () => {
logger.info(`${QueueName.DailyReminders}: queue task started`);
await reminderService.sendDailyReminders();
logger.info(`${QueueName.DailyReminders}: queue task completed`);
});
queueService.start(QueueName.SecretReminderMigration, async () => {
const REMINDER_PRUNE_BATCH_SIZE = 5_000;
const MAX_RETRY_ON_FAILURE = 3;
let numberOfRetryOnFailure = 0;
let deletedReminderCount = 0;
logger.info(`${QueueName.SecretReminderMigration}: queue task started`);
try {
const repeatableJobs = await queueService.getRepeatableJobs(QueueName.SecretReminder);
const delayedJobs = await queueService.getDelayedJobs(QueueName.SecretReminder);
logger.info(`${QueueName.SecretReminderMigration}: found ${repeatableJobs.length} secret reminder jobs`);
const reminderJobs = repeatableJobs
.map((job) => ({ secretId: job.id?.replace("reminder-", "") as string, jobKey: job.key }))
.filter(Boolean);
const reminderDelayedJobs = delayedJobs.reduce((map, job) => {
const match = uuidRegex.exec(job.repeatJobKey || "");
if (match) {
map.set(match[0], {
timestamp: job.timestamp,
delay: job.delay,
data: job.data
});
}
return map;
}, new Map<string, { timestamp: number; delay: number; data: unknown }>());
if (reminderJobs.length === 0) {
logger.info(`${QueueName.SecretReminderMigration}: no reminder jobs found`);
return;
}
for (let offset = 0; offset < reminderJobs.length; offset += REMINDER_PRUNE_BATCH_SIZE) {
try {
const batch = reminderJobs.slice(offset, offset + REMINDER_PRUNE_BATCH_SIZE);
const batchIds = batch.map((job) => job.secretId);
// Find existing secrets with pagination
// eslint-disable-next-line no-await-in-loop
const secrets = await secretDAL.findSecretsWithReminderRecipients(batchIds, REMINDER_PRUNE_BATCH_SIZE);
const secretsWithReminder = secrets.filter((secret) => secret.reminderRepeatDays);
const foundSecretIds = new Set(secretsWithReminder.map((secret) => secret.id));
// Find IDs that don't exist in either table
const secretIdsNotFound = batchIds.filter((secretId) => !foundSecretIds.has(secretId));
// Delete reminders for non-existent secrets
for (const secretId of secretIdsNotFound) {
const jobKey = reminderJobs.find((r) => r.secretId === secretId)?.jobKey;
if (jobKey) {
// eslint-disable-next-line no-await-in-loop
await queueService.stopRepeatableJobByKey(QueueName.SecretReminder, jobKey);
deletedReminderCount += 1;
}
}
for (const secretId of foundSecretIds) {
const jobKey = reminderJobs.find((r) => r.secretId === secretId)?.jobKey;
if (jobKey) {
await queueService.stopRepeatableJobByKey(QueueName.SecretReminder, jobKey);
deletedReminderCount += 1;
}
}
await secretDAL.transaction(async (tx) => {
await reminderService.batchCreateReminders(
secretsWithReminder.map((secret) => {
const delayedJob = reminderDelayedJobs.get(secret.id);
const projectId = (delayedJob?.data as { projectId?: string })?.projectId;
const nextDate = delayedJob ? new Date(delayedJob.timestamp + delayedJob.delay) : undefined;
return {
secretId: secret.id,
message: secret.reminderNote,
repeatDays: secret.reminderRepeatDays,
nextReminderDate: nextDate,
recipients: secret.recipients || [],
projectId
};
}),
tx
);
await secretReminderRecipientsDAL.delete({ $in: { secretId: secretsWithReminder.map((s) => s.id) } }, tx);
});
numberOfRetryOnFailure = 0;
} catch (error) {
numberOfRetryOnFailure += 1;
logger.error(error, `Failed to process batch at offset ${offset}`);
if (numberOfRetryOnFailure >= MAX_RETRY_ON_FAILURE) {
break;
}
// Retry the current batch
offset -= REMINDER_PRUNE_BATCH_SIZE;
// eslint-disable-next-line no-promise-executor-return, @typescript-eslint/no-loop-func, no-await-in-loop
await new Promise((resolve) => setTimeout(resolve, 500 * numberOfRetryOnFailure));
}
// Small delay between batches
// eslint-disable-next-line no-promise-executor-return, @typescript-eslint/no-loop-func, no-await-in-loop
await new Promise((resolve) => setTimeout(resolve, 10));
}
} catch (error) {
logger.error(error, "Failed to complete secret reminder pruning");
} finally {
logger.info(
`${QueueName.SecretReminderMigration}: secret reminders completed. Deleted ${deletedReminderCount} reminders`
);
}
});
// we do a repeat cron job in utc timezone at 12 Midnight each day
const startDailyRemindersJob = async () => {
// clear previous job
await queueService.stopRepeatableJob(
QueueName.DailyReminders,
QueueJobs.DailyReminders,
{ pattern: "0 0 * * *", utc: true },
QueueName.DailyReminders // just a job id
);
await queueService.queue(QueueName.DailyReminders, QueueJobs.DailyReminders, undefined, {
delay: 5000,
jobId: QueueName.DailyReminders,
repeat: { pattern: "0 0 * * *", utc: true }
});
};
// TODO: remove once all the old reminders in queues are migrated
const startSecretReminderMigrationJob = async () => {
// clear previous job
await queueService.stopRepeatableJob(
QueueName.SecretReminderMigration,
QueueJobs.SecretReminderMigration,
{ pattern: "0 */1 * * *", utc: true },
QueueName.SecretReminderMigration // just a job id
);
await queueService.queue(QueueName.SecretReminderMigration, QueueJobs.SecretReminderMigration, undefined, {
delay: 5000,
jobId: QueueName.SecretReminderMigration,
repeat: { pattern: "0 */1 * * *", utc: true }
});
};
queueService.listen(QueueName.DailyReminders, "failed", (_, err) => {
logger.error(err, `${QueueName.DailyReminders}: daily reminder processing failed`);
});
queueService.listen(QueueName.SecretReminderMigration, "failed", (_, err) => {
logger.error(err, `${QueueName.SecretReminderMigration}: secret reminder migration failed`);
});
return {
startDailyRemindersJob,
startSecretReminderMigrationJob
};
};

View File

@@ -0,0 +1,359 @@
/* eslint-disable no-await-in-loop */
import { ForbiddenError } from "@casl/ability";
import { Knex } from "knex";
import { ActionProjectType, TableName } from "@app/db/schemas";
import { TPermissionServiceFactory } from "@app/ee/services/permission/permission-service-types";
import { ProjectPermissionSecretActions, ProjectPermissionSub } from "@app/ee/services/permission/project-permission";
import { BadRequestError } from "@app/lib/errors";
import { logger } from "@app/lib/logger";
import { ActorAuthMethod, ActorType } from "../auth/auth-type";
import { TProjectMembershipDALFactory } from "../project-membership/project-membership-dal";
import { TReminderRecipientDALFactory } from "../reminder-recipients/reminder-recipient-dal";
import { TSecretV2BridgeDALFactory } from "../secret-v2-bridge/secret-v2-bridge-dal";
import { SmtpTemplates, TSmtpService } from "../smtp/smtp-service";
import { TReminderDALFactory } from "./reminder-dal";
import { TBatchCreateReminderDTO, TCreateReminderDTO, TReminderServiceFactory } from "./reminder-types";
type TReminderServiceFactoryDep = {
reminderDAL: TReminderDALFactory;
reminderRecipientDAL: TReminderRecipientDALFactory;
smtpService: TSmtpService;
projectMembershipDAL: Pick<TProjectMembershipDALFactory, "findAllProjectMembers">;
permissionService: Pick<TPermissionServiceFactory, "getProjectPermission">;
secretV2BridgeDAL: Pick<TSecretV2BridgeDALFactory, "invalidateSecretCacheByProjectId" | "findOneWithTags">;
};
export const reminderServiceFactory = ({
reminderDAL,
reminderRecipientDAL,
smtpService,
projectMembershipDAL,
permissionService,
secretV2BridgeDAL
}: TReminderServiceFactoryDep): TReminderServiceFactory => {
const $addDays = (days: number, fromDate: Date = new Date()): Date => {
const result = new Date(fromDate);
result.setDate(result.getDate() + days);
return result;
};
const $manageReminderRecipients = async (reminderId: string, newRecipients?: string[] | null): Promise<void> => {
if (!newRecipients || newRecipients.length === 0) {
// If no recipients provided, remove all existing recipients
await reminderRecipientDAL.deleteById(reminderId);
return;
}
// Remove duplicates from input
const uniqueRecipients = [...new Set(newRecipients)];
// Get existing recipients
const existingRecipients = await reminderRecipientDAL.find({ reminderId });
const existingUserIds = new Set(existingRecipients.map((r) => r.userId));
const newUserIds = new Set(uniqueRecipients);
// Find recipients to add and remove
const recipientsToAdd = uniqueRecipients.filter((userId) => !existingUserIds.has(userId));
const recipientsToRemove = existingRecipients.filter((r) => !newUserIds.has(r.userId));
// Perform database operations
if (recipientsToRemove.length > 0) {
await reminderRecipientDAL.delete({ $in: { id: recipientsToRemove.map((r) => r.id) } });
}
if (recipientsToAdd.length > 0) {
await reminderRecipientDAL.insertMany(
recipientsToAdd.map((userId) => ({
reminderId,
userId
}))
);
}
};
const createReminderInternal: TReminderServiceFactory["createReminderInternal"] = async ({
secretId,
message,
repeatDays,
nextReminderDate: nextReminderDateInput,
recipients,
projectId
}: {
secretId?: string;
message?: string | null;
repeatDays?: number | null;
nextReminderDate?: string | null;
recipients?: string[] | null;
projectId: string;
}) => {
if (!secretId) {
throw new BadRequestError({ message: "secretId is required" });
}
let nextReminderDate;
if (nextReminderDateInput) {
nextReminderDate = new Date(nextReminderDateInput);
}
if (repeatDays && repeatDays > 0) {
nextReminderDate = $addDays(repeatDays);
}
if (!nextReminderDate) {
throw new BadRequestError({ message: "repeatDays must be a positive number" });
}
const existingReminder = await reminderDAL.findOne({ secretId });
let reminderId: string;
if (existingReminder) {
// Update existing reminder
await reminderDAL.updateById(existingReminder.id, {
message,
repeatDays,
nextReminderDate
});
reminderId = existingReminder.id;
} else {
// Create new reminder
const newReminder = await reminderDAL.create({
secretId,
message,
repeatDays,
nextReminderDate
});
reminderId = newReminder.id;
}
// Manage recipients (add/update/delete as needed)
await $manageReminderRecipients(reminderId, recipients);
await secretV2BridgeDAL.invalidateSecretCacheByProjectId(projectId);
return { id: reminderId, created: !existingReminder };
};
const createReminder: TReminderServiceFactory["createReminder"] = async ({
actor,
actorId,
actorOrgId,
actorAuthMethod,
reminder
}: TCreateReminderDTO) => {
const secret = await secretV2BridgeDAL.findOneWithTags({ [`${TableName.SecretV2}.id` as "id"]: reminder.secretId });
if (!secret) {
throw new BadRequestError({ message: `Secret ${reminder.secretId} not found` });
}
const { permission } = await permissionService.getProjectPermission({
actor,
actorId,
projectId: secret.projectId,
actorAuthMethod,
actorOrgId,
actionProjectType: ActionProjectType.SecretManager
});
ForbiddenError.from(permission).throwUnlessCan(ProjectPermissionSecretActions.Edit, ProjectPermissionSub.Secrets);
const response = await createReminderInternal({
...reminder,
projectId: secret.projectId
});
return response;
};
const getReminder: TReminderServiceFactory["getReminder"] = async ({
secretId,
actor,
actorId,
actorOrgId,
actorAuthMethod
}: {
secretId: string;
actor: ActorType;
actorId: string;
actorOrgId: string;
actorAuthMethod: ActorAuthMethod;
}) => {
const secret = await secretV2BridgeDAL.findOneWithTags({ [`${TableName.SecretV2}.id` as "id"]: secretId });
if (!secret) {
throw new BadRequestError({ message: `Secret ${secretId} not found` });
}
const { permission } = await permissionService.getProjectPermission({
actor,
actorId,
projectId: secret.projectId,
actorAuthMethod,
actorOrgId,
actionProjectType: ActionProjectType.SecretManager
});
ForbiddenError.from(permission).throwUnlessCan(
ProjectPermissionSecretActions.DescribeSecret,
ProjectPermissionSub.Secrets
);
const reminder = await reminderDAL.findSecretReminder(secretId);
return reminder;
};
const sendDailyReminders: TReminderServiceFactory["sendDailyReminders"] = async () => {
const remindersToSend = await reminderDAL.findSecretDailyReminders();
for (const reminder of remindersToSend) {
try {
await reminderDAL.transaction(async (tx) => {
const recipients: string[] = reminder.recipients
.map((r) => r.email)
.filter((email): email is string => Boolean(email));
if (recipients.length === 0) {
const members = await projectMembershipDAL.findAllProjectMembers(reminder.projectId);
recipients.push(...members.map((m) => m.user.email).filter((email): email is string => Boolean(email)));
}
await smtpService.sendMail({
template: SmtpTemplates.SecretReminder,
subjectLine: "Infisical secret reminder",
recipients,
substitutions: {
reminderNote: reminder.message || "",
projectName: reminder.projectName || "",
organizationName: reminder.organizationName || ""
}
});
if (reminder.repeatDays) {
await reminderDAL.updateById(reminder.id, { nextReminderDate: $addDays(reminder.repeatDays) }, tx);
} else {
await reminderDAL.deleteById(reminder.id, tx);
}
});
} catch (error) {
logger.error(
error,
`Failed to send reminder to recipients ${reminder.recipients.map((r) => r.email).join(", ")}`
);
}
}
};
const deleteReminder: TReminderServiceFactory["deleteReminder"] = async ({
actor,
actorId,
actorOrgId,
actorAuthMethod,
secretId
}: {
actor: ActorType;
actorId: string;
actorOrgId: string;
actorAuthMethod: ActorAuthMethod;
secretId: string;
}) => {
const secret = await secretV2BridgeDAL.findOneWithTags({ [`${TableName.SecretV2}.id` as "id"]: secretId });
if (!secret) {
throw new BadRequestError({ message: `Secret ${secretId} not found` });
}
const { permission } = await permissionService.getProjectPermission({
actor,
actorId,
projectId: secret.projectId,
actorAuthMethod,
actorOrgId,
actionProjectType: ActionProjectType.SecretManager
});
ForbiddenError.from(permission).throwUnlessCan(ProjectPermissionSecretActions.Edit, ProjectPermissionSub.Secrets);
await reminderDAL.delete({ secretId });
await secretV2BridgeDAL.invalidateSecretCacheByProjectId(secret.projectId);
};
const deleteReminderBySecretId: TReminderServiceFactory["deleteReminderBySecretId"] = async (
secretId: string,
projectId: string,
tx?: Knex
) => {
await reminderDAL.delete({ secretId }, tx);
await secretV2BridgeDAL.invalidateSecretCacheByProjectId(projectId);
};
const batchCreateReminders: TReminderServiceFactory["batchCreateReminders"] = async (
remindersData: TBatchCreateReminderDTO,
tx?: Knex
) => {
if (!remindersData || remindersData.length === 0) {
return { created: 0, reminderIds: [] };
}
const processedReminders = remindersData.map(
({ secretId, message, repeatDays, nextReminderDate: nextReminderDateInput, recipients, projectId }) => {
let nextReminderDate;
if (nextReminderDateInput) {
nextReminderDate = new Date(nextReminderDateInput);
}
if (repeatDays && repeatDays > 0 && !nextReminderDate) {
nextReminderDate = $addDays(repeatDays);
}
if (!nextReminderDate) {
throw new BadRequestError({
message: `repeatDays must be a positive number for secretId: ${secretId}`
});
}
return {
secretId,
message,
repeatDays,
nextReminderDate,
recipients: recipients ? [...new Set(recipients)] : [],
projectId
};
}
);
const newReminders = await reminderDAL.insertMany(
processedReminders.map(({ secretId, message, repeatDays, nextReminderDate, projectId }) => ({
secretId,
message,
repeatDays,
nextReminderDate,
projectId
})),
tx
);
const allRecipientInserts: Array<{ reminderId: string; userId: string }> = [];
newReminders.forEach((reminder, index) => {
const { recipients } = processedReminders[index];
if (recipients && recipients.length > 0) {
recipients.forEach((userId) => {
allRecipientInserts.push({
reminderId: reminder.id,
userId
});
});
}
});
if (allRecipientInserts.length > 0) {
await reminderRecipientDAL.insertMany(allRecipientInserts, tx);
}
const projectIds = new Set(processedReminders.map((r) => r.projectId).filter((id): id is string => Boolean(id)));
for (const projectId of projectIds) {
await secretV2BridgeDAL.invalidateSecretCacheByProjectId(projectId);
}
return {
created: newReminders.length,
reminderIds: newReminders.map((r) => r.id)
};
};
return {
createReminder,
getReminder,
sendDailyReminders,
deleteReminder,
deleteReminderBySecretId,
batchCreateReminders,
createReminderInternal
};
};

View File

@@ -0,0 +1,102 @@
import { Knex } from "knex";
import { ActorAuthMethod, ActorType } from "../auth/auth-type";
export type TReminder = {
id: string;
secretId?: string | null;
message?: string | null;
repeatDays?: number | null;
nextReminderDate: Date;
createdAt: Date;
updatedAt: Date;
};
export type TCreateReminderDTO = {
actor: ActorType;
actorId: string;
actorOrgId: string;
actorAuthMethod: ActorAuthMethod;
reminder: {
secretId?: string;
message?: string | null;
repeatDays?: number | null;
nextReminderDate?: string | null;
recipients?: string[] | null;
};
};
export type TBatchCreateReminderDTO = {
secretId: string;
message?: string | null;
repeatDays?: number | null;
nextReminderDate?: string | Date | null;
recipients?: string[] | null;
projectId?: string;
}[];
export interface TReminderServiceFactory {
createReminder: ({ actor, actorId, actorOrgId, actorAuthMethod, reminder }: TCreateReminderDTO) => Promise<{
id: string;
created: boolean;
}>;
getReminder: ({
secretId,
actor,
actorId,
actorOrgId,
actorAuthMethod
}: {
secretId: string;
actor: ActorType;
actorId: string;
actorOrgId: string;
actorAuthMethod: ActorAuthMethod;
}) => Promise<(TReminder & { recipients: string[] }) | null>;
sendDailyReminders: () => Promise<void>;
deleteReminder: ({
actor,
actorId,
actorOrgId,
actorAuthMethod,
secretId
}: {
actor: ActorType;
actorId: string;
actorOrgId: string;
actorAuthMethod: ActorAuthMethod;
secretId: string;
}) => Promise<void>;
deleteReminderBySecretId: (secretId: string, projectId: string, tx?: Knex) => Promise<void>;
batchCreateReminders: (
remindersData: TBatchCreateReminderDTO,
tx?: Knex
) => Promise<{
created: number;
reminderIds: string[];
}>;
createReminderInternal: ({
secretId,
message,
repeatDays,
nextReminderDate,
recipients,
projectId
}: {
secretId?: string;
message?: string | null;
repeatDays?: number | null;
nextReminderDate?: string | null;
recipients?: string[] | null;
projectId: string;
}) => Promise<{
id: string;
created: boolean;
}>;
}

View File

@@ -6,7 +6,6 @@ import { QueueJobs, QueueName, TQueueServiceFactory } from "@app/queue";
import { TIdentityAccessTokenDALFactory } from "../identity-access-token/identity-access-token-dal";
import { TIdentityUaClientSecretDALFactory } from "../identity-ua/identity-ua-client-secret-dal";
import { TOrgServiceFactory } from "../org/org-service";
import { TSecretDALFactory } from "../secret/secret-dal";
import { TSecretVersionDALFactory } from "../secret/secret-version-dal";
import { TSecretFolderVersionDALFactory } from "../secret-folder/secret-folder-version-dal";
import { TSecretSharingDALFactory } from "../secret-sharing/secret-sharing-dal";
@@ -19,7 +18,6 @@ type TDailyResourceCleanUpQueueServiceFactoryDep = {
identityUniversalAuthClientSecretDAL: Pick<TIdentityUaClientSecretDALFactory, "removeExpiredClientSecrets">;
secretVersionDAL: Pick<TSecretVersionDALFactory, "pruneExcessVersions">;
secretVersionV2DAL: Pick<TSecretVersionV2DALFactory, "pruneExcessVersions">;
secretDAL: Pick<TSecretDALFactory, "pruneSecretReminders">;
secretFolderVersionDAL: Pick<TSecretFolderVersionDALFactory, "pruneExcessVersions">;
snapshotDAL: Pick<TSnapshotDALFactory, "pruneExcessSnapshots">;
secretSharingDAL: Pick<TSecretSharingDALFactory, "pruneExpiredSharedSecrets" | "pruneExpiredSecretRequests">;
@@ -36,7 +34,6 @@ export const dailyResourceCleanUpQueueServiceFactory = ({
snapshotDAL,
secretVersionDAL,
secretFolderVersionDAL,
secretDAL,
identityAccessTokenDAL,
secretSharingDAL,
secretVersionV2DAL,
@@ -46,7 +43,6 @@ export const dailyResourceCleanUpQueueServiceFactory = ({
}: TDailyResourceCleanUpQueueServiceFactoryDep) => {
queueService.start(QueueName.DailyResourceCleanUp, async () => {
logger.info(`${QueueName.DailyResourceCleanUp}: queue task started`);
await secretDAL.pruneSecretReminders(queueService);
await identityAccessTokenDAL.removeExpiredTokens();
await identityUniversalAuthClientSecretDAL.removeExpiredClientSecrets();
await secretSharingDAL.pruneExpiredSharedSecrets();

View File

@@ -0,0 +1,10 @@
import { AppConnection } from "@app/services/app-connection/app-connection-enums";
import { SecretSync } from "@app/services/secret-sync/secret-sync-enums";
import { TSecretSyncListItem } from "@app/services/secret-sync/secret-sync-types";
export const BITBUCKET_SYNC_LIST_OPTION: TSecretSyncListItem = {
name: "Bitbucket",
destination: SecretSync.Bitbucket,
connection: AppConnection.Bitbucket,
canImportSecrets: false
};

View File

@@ -0,0 +1,222 @@
import { request } from "@app/lib/config/request";
import { createAuthHeader } from "@app/services/app-connection/bitbucket";
import { IntegrationUrls } from "@app/services/integration-auth/integration-list";
import {
TBitbucketListVariables,
TBitbucketSyncWithCredentials,
TBitbucketVariable,
TDeleteBitbucketVariable,
TPutBitbucketVariable
} from "@app/services/secret-sync/bitbucket/bitbucket-sync-types";
import { SecretSyncError } from "@app/services/secret-sync/secret-sync-errors";
import { matchesSchema } from "@app/services/secret-sync/secret-sync-fns";
import { TSecretMap } from "@app/services/secret-sync/secret-sync-types";
import { SECRET_SYNC_NAME_MAP } from "../secret-sync-maps";
const buildVariablesUrl = (workspace: string, repository: string, environment?: string, uuid?: string): string => {
const baseUrl = `${IntegrationUrls.BITBUCKET_API_URL}/2.0/repositories/${encodeURIComponent(workspace)}/${encodeURIComponent(repository)}`;
if (environment) {
return `${baseUrl}/deployments_config/environments/${environment}/variables/${uuid || ""}`;
}
return `${baseUrl}/pipelines_config/variables/${uuid || ""}`;
};
const listVariables = async ({
workspaceSlug,
repositorySlug,
environmentId,
authHeader
}: TBitbucketListVariables): Promise<TBitbucketVariable[]> => {
const url = buildVariablesUrl(workspaceSlug, repositorySlug, environmentId);
const { data } = await request.get<{ values: TBitbucketVariable[] }>(url, {
headers: {
Authorization: authHeader,
Accept: "application/json"
}
});
return data.values;
};
const upsertVariable = async ({
workspaceSlug,
repositorySlug,
environmentId,
key,
value,
existingVariables,
authHeader
}: {
workspaceSlug: string;
repositorySlug: string;
environmentId?: string;
key: string;
value: string;
existingVariables: TBitbucketVariable[];
authHeader: string;
}) => {
const existingVariable = existingVariables.find((variable) => variable.key === key);
const requestData = { key, value, secured: true };
const headers = {
Authorization: authHeader,
"Content-Type": "application/json"
};
if (existingVariable) {
const url = buildVariablesUrl(workspaceSlug, repositorySlug, environmentId, existingVariable.uuid);
return request.put(url, requestData, { headers });
}
const url = buildVariablesUrl(workspaceSlug, repositorySlug, environmentId);
return request.post(url, requestData, { headers });
};
const putVariables = async ({
workspaceSlug,
repositorySlug,
environmentId,
secretMap,
authHeader
}: TPutBitbucketVariable & { secretMap: TSecretMap; authHeader: string }) => {
const existingVariables = await listVariables({
workspaceSlug,
repositorySlug,
environmentId,
authHeader
});
const promises = Object.entries(secretMap).map(([key, { value }]) =>
upsertVariable({
workspaceSlug,
repositorySlug,
environmentId,
key,
value,
existingVariables,
authHeader
})
);
return Promise.all(promises);
};
const deleteVariables = async ({
workspaceSlug,
repositorySlug,
environmentId,
keys,
authHeader
}: TDeleteBitbucketVariable) => {
const existingVariables = await listVariables({
workspaceSlug,
repositorySlug,
environmentId,
authHeader
});
const variablesToDelete = existingVariables.filter((variable) => keys.includes(variable.key));
const promises = variablesToDelete.map((variable) => {
const url = buildVariablesUrl(workspaceSlug, repositorySlug, environmentId, variable.uuid);
return request.delete(url, {
headers: { Authorization: authHeader }
});
});
return Promise.all(promises);
};
export const BitbucketSyncFns = {
syncSecrets: async (secretSync: TBitbucketSyncWithCredentials, secretMap: TSecretMap) => {
const {
connection,
environment,
destinationConfig: { workspaceSlug, repositorySlug, environmentId }
} = secretSync;
const { email, apiToken } = connection.credentials;
const authHeader = createAuthHeader(email, apiToken);
try {
await putVariables({
workspaceSlug,
repositorySlug,
environmentId,
secretMap,
authHeader
});
} catch (error) {
throw new SecretSyncError({ error });
}
if (secretSync.syncOptions.disableSecretDeletion) return;
try {
const existingVariables = await listVariables({
workspaceSlug,
repositorySlug,
environmentId,
authHeader
});
const keysToDelete = existingVariables
.map((variable) => variable.key)
.filter(
(secret) =>
matchesSchema(secret, environment?.slug || "", secretSync.syncOptions.keySchema) && !(secret in secretMap)
);
if (keysToDelete.length > 0) {
await deleteVariables({
workspaceSlug,
repositorySlug,
environmentId,
keys: keysToDelete,
authHeader
});
}
} catch (error) {
throw new SecretSyncError({ error });
}
},
removeSecrets: async (secretSync: TBitbucketSyncWithCredentials, secretMap: TSecretMap) => {
const {
connection,
destinationConfig: { workspaceSlug, repositorySlug, environmentId }
} = secretSync;
const { email, apiToken } = connection.credentials;
const authHeader = createAuthHeader(email, apiToken);
try {
const existingVariables = await listVariables({
workspaceSlug,
repositorySlug,
environmentId,
authHeader
});
const keysToRemove = existingVariables.map((variable) => variable.key).filter((secret) => secret in secretMap);
if (keysToRemove.length > 0) {
await deleteVariables({
workspaceSlug,
repositorySlug,
environmentId,
keys: keysToRemove,
authHeader
});
}
} catch (error) {
throw new SecretSyncError({ error });
}
},
getSecrets: async (secretSync: TBitbucketSyncWithCredentials): Promise<TSecretMap> => {
throw new Error(`${SECRET_SYNC_NAME_MAP[secretSync.destination]} does not support importing secrets.`);
}
};

View File

@@ -0,0 +1,45 @@
import { z } from "zod";
import { SecretSyncs } from "@app/lib/api-docs";
import { AppConnection } from "@app/services/app-connection/app-connection-enums";
import { SecretSync } from "@app/services/secret-sync/secret-sync-enums";
import {
BaseSecretSyncSchema,
GenericCreateSecretSyncFieldsSchema,
GenericUpdateSecretSyncFieldsSchema
} from "@app/services/secret-sync/secret-sync-schemas";
import { TSyncOptionsConfig } from "@app/services/secret-sync/secret-sync-types";
const BitbucketSyncDestinationConfigSchema = z.object({
repositorySlug: z.string().describe(SecretSyncs.DESTINATION_CONFIG.BITBUCKET.repositorySlug),
environmentId: z.string().optional().describe(SecretSyncs.DESTINATION_CONFIG.BITBUCKET.environmentId),
workspaceSlug: z.string().describe(SecretSyncs.DESTINATION_CONFIG.BITBUCKET.workspaceSlug)
});
const BitbucketSyncOptionsConfig: TSyncOptionsConfig = { canImportSecrets: false };
export const BitbucketSyncSchema = BaseSecretSyncSchema(SecretSync.Bitbucket, BitbucketSyncOptionsConfig).extend({
destination: z.literal(SecretSync.Bitbucket),
destinationConfig: BitbucketSyncDestinationConfigSchema
});
export const CreateBitbucketSyncSchema = GenericCreateSecretSyncFieldsSchema(
SecretSync.Bitbucket,
BitbucketSyncOptionsConfig
).extend({
destinationConfig: BitbucketSyncDestinationConfigSchema
});
export const UpdateBitbucketSyncSchema = GenericUpdateSecretSyncFieldsSchema(
SecretSync.Bitbucket,
BitbucketSyncOptionsConfig
).extend({
destinationConfig: BitbucketSyncDestinationConfigSchema.optional()
});
export const BitbucketSyncListItemSchema = z.object({
name: z.literal("Bitbucket"),
connection: z.literal(AppConnection.Bitbucket),
destination: z.literal(SecretSync.Bitbucket),
canImportSecrets: z.literal(false)
});

View File

@@ -0,0 +1,50 @@
import { z } from "zod";
import { TBitbucketConnection } from "@app/services/app-connection/bitbucket";
import { BitbucketSyncListItemSchema, BitbucketSyncSchema, CreateBitbucketSyncSchema } from "./bitbucket-sync-schemas";
export type TBitbucketSync = z.infer<typeof BitbucketSyncSchema>;
export type TBitbucketSyncInput = z.infer<typeof CreateBitbucketSyncSchema>;
export type TBitbucketSyncListItem = z.infer<typeof BitbucketSyncListItemSchema>;
export type TBitbucketSyncWithCredentials = TBitbucketSync & {
connection: TBitbucketConnection;
};
export type TBitbucketVariable = {
key: string;
value?: string;
// Secure variables values are not returned by the API neither are they shown in Bitbucket UI
secured: boolean;
uuid: string;
type: string;
};
export type TBitbucketListVariables = {
workspaceSlug: string;
repositorySlug: string;
environmentId?: string;
authHeader: string;
};
export type TPutBitbucketVariable = {
authHeader: string;
workspaceSlug: string;
repositorySlug: string;
environmentId?: string;
};
export type TDeleteBitbucketVariable = {
authHeader: string;
workspaceSlug: string;
repositorySlug: string;
environmentId?: string;
keys: string[];
};
export type TBitbucketConnectionCredentials = {
authHeader: string;
};

View File

@@ -0,0 +1,4 @@
export * from "./bitbucket-sync-constants";
export * from "./bitbucket-sync-fns";
export * from "./bitbucket-sync-schemas";
export * from "./bitbucket-sync-types";

View File

@@ -0,0 +1,10 @@
import { AppConnection } from "@app/services/app-connection/app-connection-enums";
import { SecretSync } from "@app/services/secret-sync/secret-sync-enums";
import { TSecretSyncListItem } from "@app/services/secret-sync/secret-sync-types";
export const DIGITAL_OCEAN_APP_PLATFORM_SYNC_LIST_OPTION: TSecretSyncListItem = {
name: "Digital Ocean App Platform" as const,
destination: SecretSync.DigitalOceanAppPlatform,
connection: AppConnection.DigitalOcean,
canImportSecrets: false
};

View File

@@ -0,0 +1,83 @@
/* eslint-disable no-continue */
/* eslint-disable @typescript-eslint/no-unsafe-member-access */
/* eslint-disable @typescript-eslint/no-unsafe-assignment */
import { TDigitalOceanVariable } from "@app/services/app-connection/digital-ocean";
import { DigitalOceanAppPlatformPublicAPI } from "@app/services/app-connection/digital-ocean/digital-ocean-connection-public-client";
import { matchesSchema } from "@app/services/secret-sync/secret-sync-fns";
import { SecretSyncError } from "../secret-sync-errors";
import { SECRET_SYNC_NAME_MAP } from "../secret-sync-maps";
import { TSecretMap } from "../secret-sync-types";
import { TDigitalOceanAppPlatformSyncWithCredentials } from "./digital-ocean-app-platform-sync-types";
export const DigitalOceanAppPlatformSyncFns = {
async getSecrets(secretSync: TDigitalOceanAppPlatformSyncWithCredentials) {
throw new Error(`${SECRET_SYNC_NAME_MAP[secretSync.destination]} does not support importing secrets.`);
},
async syncSecrets(secretSync: TDigitalOceanAppPlatformSyncWithCredentials, secretMap: TSecretMap) {
const {
environment,
syncOptions: { disableSecretDeletion, keySchema }
} = secretSync;
const config = secretSync.destinationConfig;
const existing = await DigitalOceanAppPlatformPublicAPI.getVariables(secretSync.connection, config.appId);
const variables: Record<string, TDigitalOceanVariable> = Object.fromEntries(existing.map((v) => [v.key, v]));
for (const [key, value] of Object.entries(secretMap)) {
variables[key] = {
key,
value: value.value,
type: "SECRET"
} as TDigitalOceanVariable;
}
if (!disableSecretDeletion) {
for (const v of existing) {
if (!matchesSchema(v.key, environment?.slug || "", keySchema)) continue;
if (!(v.key in secretMap)) {
delete variables[v.key];
}
}
}
try {
const vars = Object.values(variables);
await DigitalOceanAppPlatformPublicAPI.putVariables(secretSync.connection, config.appId, ...vars);
} catch (error) {
throw new SecretSyncError({
error
});
}
},
async removeSecrets(secretSync: TDigitalOceanAppPlatformSyncWithCredentials, secretMap: TSecretMap) {
const config = secretSync.destinationConfig;
try {
const existingSecrets = await DigitalOceanAppPlatformPublicAPI.getVariables(secretSync.connection, config.appId);
const vars = Object.entries(existingSecrets)
.map(([key, v]) => {
if (!(key in secretMap)) return;
return {
key,
value: v.value,
type: "SECRET"
} as TDigitalOceanVariable;
})
.filter(Boolean) as TDigitalOceanVariable[];
await DigitalOceanAppPlatformPublicAPI.deleteVariables(secretSync.connection, config.appId, ...vars);
} catch (error) {
throw new SecretSyncError({
error
});
}
}
};

View File

@@ -0,0 +1,46 @@
import { z } from "zod";
import { AppConnection } from "@app/services/app-connection/app-connection-enums";
import { SecretSync } from "@app/services/secret-sync/secret-sync-enums";
import {
BaseSecretSyncSchema,
GenericCreateSecretSyncFieldsSchema,
GenericUpdateSecretSyncFieldsSchema
} from "@app/services/secret-sync/secret-sync-schemas";
import { TSyncOptionsConfig } from "@app/services/secret-sync/secret-sync-types";
const DigitalOceanAppPlatformSyncDestinationConfigSchema = z.object({
appId: z.string().min(1, "Account ID is required").max(255, "Account ID must be less than 255 characters"),
appName: z.string().min(1, "Account Name is required").max(255, "Account Name must be less than 255 characters")
});
const DigitalOceanAppPlatformSyncOptionsConfig: TSyncOptionsConfig = { canImportSecrets: false };
export const DigitalOceanAppPlatformSyncSchema = BaseSecretSyncSchema(
SecretSync.DigitalOceanAppPlatform,
DigitalOceanAppPlatformSyncOptionsConfig
).extend({
destination: z.literal(SecretSync.DigitalOceanAppPlatform),
destinationConfig: DigitalOceanAppPlatformSyncDestinationConfigSchema
});
export const CreateDigitalOceanAppPlatformSyncSchema = GenericCreateSecretSyncFieldsSchema(
SecretSync.DigitalOceanAppPlatform,
DigitalOceanAppPlatformSyncOptionsConfig
).extend({
destinationConfig: DigitalOceanAppPlatformSyncDestinationConfigSchema
});
export const UpdateDigitalOceanAppPlatformSyncSchema = GenericUpdateSecretSyncFieldsSchema(
SecretSync.DigitalOceanAppPlatform,
DigitalOceanAppPlatformSyncOptionsConfig
).extend({
destinationConfig: DigitalOceanAppPlatformSyncDestinationConfigSchema.optional()
});
export const DigitalOceanAppPlatformSyncListItemSchema = z.object({
name: z.literal("Digital Ocean App Platform"),
connection: z.literal(AppConnection.DigitalOcean),
destination: z.literal(SecretSync.DigitalOceanAppPlatform),
canImportSecrets: z.literal(false)
});

View File

@@ -0,0 +1,23 @@
import z from "zod";
import { TDigitalOceanConnection, TDigitalOceanVariable } from "@app/services/app-connection/digital-ocean";
import {
CreateDigitalOceanAppPlatformSyncSchema,
DigitalOceanAppPlatformSyncListItemSchema,
DigitalOceanAppPlatformSyncSchema
} from "./digital-ocean-app-platform-sync-schemas";
export type TDigitalOceanAppPlatformSyncListItem = z.infer<typeof DigitalOceanAppPlatformSyncListItemSchema>;
export type TDigitalOceanAppPlatformSync = z.infer<typeof DigitalOceanAppPlatformSyncSchema>;
export type TDigitalOceanAppPlatformSyncInput = z.infer<typeof CreateDigitalOceanAppPlatformSyncSchema>;
export type TDigitalOceanAppPlatformSyncWithCredentials = TDigitalOceanAppPlatformSync & {
connection: TDigitalOceanConnection;
};
export type TDigitalOceanAppPlatformSecret = TDigitalOceanVariable & {
type: "SECRET";
};

View File

@@ -0,0 +1,4 @@
export * from "./digital-ocean-app-platform-sync-constants";
export * from "./digital-ocean-app-platform-sync-fns";
export * from "./digital-ocean-app-platform-sync-schemas";
export * from "./digital-ocean-app-platform-sync-types";

View File

@@ -25,7 +25,9 @@ export enum SecretSync {
Supabase = "supabase",
Zabbix = "zabbix",
Railway = "railway",
Checkly = "checkly"
Checkly = "checkly",
DigitalOceanAppPlatform = "digital-ocean-app-platform",
Bitbucket = "bitbucket"
}
export enum SecretSyncInitialSyncBehavior {

View File

@@ -28,12 +28,17 @@ import { ONEPASS_SYNC_LIST_OPTION, OnePassSyncFns } from "./1password";
import { AZURE_APP_CONFIGURATION_SYNC_LIST_OPTION, azureAppConfigurationSyncFactory } from "./azure-app-configuration";
import { AZURE_DEVOPS_SYNC_LIST_OPTION, azureDevOpsSyncFactory } from "./azure-devops";
import { AZURE_KEY_VAULT_SYNC_LIST_OPTION, azureKeyVaultSyncFactory } from "./azure-key-vault";
import { BITBUCKET_SYNC_LIST_OPTION, BitbucketSyncFns } from "./bitbucket";
import { CAMUNDA_SYNC_LIST_OPTION, camundaSyncFactory } from "./camunda";
import { CHECKLY_SYNC_LIST_OPTION } from "./checkly/checkly-sync-constants";
import { ChecklySyncFns } from "./checkly/checkly-sync-fns";
import { CLOUDFLARE_PAGES_SYNC_LIST_OPTION } from "./cloudflare-pages/cloudflare-pages-constants";
import { CloudflarePagesSyncFns } from "./cloudflare-pages/cloudflare-pages-fns";
import { CLOUDFLARE_WORKERS_SYNC_LIST_OPTION, CloudflareWorkersSyncFns } from "./cloudflare-workers";
import {
DIGITAL_OCEAN_APP_PLATFORM_SYNC_LIST_OPTION,
DigitalOceanAppPlatformSyncFns
} from "./digital-ocean-app-platform";
import { FLYIO_SYNC_LIST_OPTION, FlyioSyncFns } from "./flyio";
import { GCP_SYNC_LIST_OPTION } from "./gcp";
import { GcpSyncFns } from "./gcp/gcp-sync-fns";
@@ -80,7 +85,9 @@ const SECRET_SYNC_LIST_OPTIONS: Record<SecretSync, TSecretSyncListItem> = {
[SecretSync.Supabase]: SUPABASE_SYNC_LIST_OPTION,
[SecretSync.Zabbix]: ZABBIX_SYNC_LIST_OPTION,
[SecretSync.Railway]: RAILWAY_SYNC_LIST_OPTION,
[SecretSync.Checkly]: CHECKLY_SYNC_LIST_OPTION
[SecretSync.Checkly]: CHECKLY_SYNC_LIST_OPTION,
[SecretSync.DigitalOceanAppPlatform]: DIGITAL_OCEAN_APP_PLATFORM_SYNC_LIST_OPTION,
[SecretSync.Bitbucket]: BITBUCKET_SYNC_LIST_OPTION
};
export const listSecretSyncOptions = () => {
@@ -258,6 +265,10 @@ export const SecretSyncFns = {
return ChecklySyncFns.syncSecrets(secretSync, schemaSecretMap);
case SecretSync.Supabase:
return SupabaseSyncFns.syncSecrets(secretSync, schemaSecretMap);
case SecretSync.DigitalOceanAppPlatform:
return DigitalOceanAppPlatformSyncFns.syncSecrets(secretSync, schemaSecretMap);
case SecretSync.Bitbucket:
return BitbucketSyncFns.syncSecrets(secretSync, schemaSecretMap);
default:
throw new Error(
`Unhandled sync destination for sync secrets fns: ${(secretSync as TSecretSyncWithCredentials).destination}`
@@ -365,6 +376,12 @@ export const SecretSyncFns = {
case SecretSync.Supabase:
secretMap = await SupabaseSyncFns.getSecrets(secretSync);
break;
case SecretSync.DigitalOceanAppPlatform:
secretMap = await DigitalOceanAppPlatformSyncFns.getSecrets(secretSync);
break;
case SecretSync.Bitbucket:
secretMap = await BitbucketSyncFns.getSecrets(secretSync);
break;
default:
throw new Error(
`Unhandled sync destination for get secrets fns: ${(secretSync as TSecretSyncWithCredentials).destination}`
@@ -452,6 +469,10 @@ export const SecretSyncFns = {
return ChecklySyncFns.removeSecrets(secretSync, schemaSecretMap);
case SecretSync.Supabase:
return SupabaseSyncFns.removeSecrets(secretSync, schemaSecretMap);
case SecretSync.DigitalOceanAppPlatform:
return DigitalOceanAppPlatformSyncFns.removeSecrets(secretSync, schemaSecretMap);
case SecretSync.Bitbucket:
return BitbucketSyncFns.removeSecrets(secretSync, schemaSecretMap);
default:
throw new Error(
`Unhandled sync destination for remove secrets fns: ${(secretSync as TSecretSyncWithCredentials).destination}`

View File

@@ -28,7 +28,9 @@ export const SECRET_SYNC_NAME_MAP: Record<SecretSync, string> = {
[SecretSync.Supabase]: "Supabase",
[SecretSync.Zabbix]: "Zabbix",
[SecretSync.Railway]: "Railway",
[SecretSync.Checkly]: "Checkly"
[SecretSync.Checkly]: "Checkly",
[SecretSync.DigitalOceanAppPlatform]: "Digital Ocean App Platform",
[SecretSync.Bitbucket]: "Bitbucket"
};
export const SECRET_SYNC_CONNECTION_MAP: Record<SecretSync, AppConnection> = {
@@ -58,7 +60,9 @@ export const SECRET_SYNC_CONNECTION_MAP: Record<SecretSync, AppConnection> = {
[SecretSync.Supabase]: AppConnection.Supabase,
[SecretSync.Zabbix]: AppConnection.Zabbix,
[SecretSync.Railway]: AppConnection.Railway,
[SecretSync.Checkly]: AppConnection.Checkly
[SecretSync.Checkly]: AppConnection.Checkly,
[SecretSync.DigitalOceanAppPlatform]: AppConnection.DigitalOcean,
[SecretSync.Bitbucket]: AppConnection.Bitbucket
};
export const SECRET_SYNC_PLAN_MAP: Record<SecretSync, SecretSyncPlanType> = {
@@ -88,5 +92,7 @@ export const SECRET_SYNC_PLAN_MAP: Record<SecretSync, SecretSyncPlanType> = {
[SecretSync.Supabase]: SecretSyncPlanType.Regular,
[SecretSync.Zabbix]: SecretSyncPlanType.Regular,
[SecretSync.Railway]: SecretSyncPlanType.Regular,
[SecretSync.Checkly]: SecretSyncPlanType.Regular
[SecretSync.Checkly]: SecretSyncPlanType.Regular,
[SecretSync.DigitalOceanAppPlatform]: SecretSyncPlanType.Regular,
[SecretSync.Bitbucket]: SecretSyncPlanType.Regular
};

View File

@@ -869,7 +869,7 @@ export const secretSyncQueueFactory = ({
secretPath: folder?.path,
environment: environment?.name,
projectName: project.name,
syncUrl: `${appCfg.SITE_URL}/projects/${projectId}/secret-manager/integrations/secret-syncs/${destination}/${secretSync.id}`
syncUrl: `${appCfg.SITE_URL}/projects/secret-management/${projectId}/integrations/secret-syncs/${destination}/${secretSync.id}`
}
});
};

View File

@@ -72,6 +72,12 @@ import {
TAzureKeyVaultSyncListItem,
TAzureKeyVaultSyncWithCredentials
} from "./azure-key-vault";
import {
TBitbucketSync,
TBitbucketSyncInput,
TBitbucketSyncListItem,
TBitbucketSyncWithCredentials
} from "./bitbucket/bitbucket-sync-types";
import {
TChecklySync,
TChecklySyncInput,
@@ -90,6 +96,11 @@ import {
TCloudflareWorkersSyncListItem,
TCloudflareWorkersSyncWithCredentials
} from "./cloudflare-workers";
import {
TDigitalOceanAppPlatformSyncInput,
TDigitalOceanAppPlatformSyncListItem,
TDigitalOceanAppPlatformSyncWithCredentials
} from "./digital-ocean-app-platform/digital-ocean-app-platform-sync-types";
import { TFlyioSync, TFlyioSyncInput, TFlyioSyncListItem, TFlyioSyncWithCredentials } from "./flyio/flyio-sync-types";
import { TGcpSync, TGcpSyncInput, TGcpSyncListItem, TGcpSyncWithCredentials } from "./gcp";
import { TGitLabSync, TGitLabSyncInput, TGitLabSyncListItem, TGitLabSyncWithCredentials } from "./gitlab";
@@ -166,7 +177,8 @@ export type TSecretSync =
| TZabbixSync
| TRailwaySync
| TChecklySync
| TSupabaseSync;
| TSupabaseSync
| TBitbucketSync;
export type TSecretSyncWithCredentials =
| TAwsParameterStoreSyncWithCredentials
@@ -195,7 +207,9 @@ export type TSecretSyncWithCredentials =
| TZabbixSyncWithCredentials
| TRailwaySyncWithCredentials
| TChecklySyncWithCredentials
| TSupabaseSyncWithCredentials;
| TSupabaseSyncWithCredentials
| TDigitalOceanAppPlatformSyncWithCredentials
| TBitbucketSyncWithCredentials;
export type TSecretSyncInput =
| TAwsParameterStoreSyncInput
@@ -224,7 +238,9 @@ export type TSecretSyncInput =
| TZabbixSyncInput
| TRailwaySyncInput
| TChecklySyncInput
| TSupabaseSyncInput;
| TSupabaseSyncInput
| TDigitalOceanAppPlatformSyncInput
| TBitbucketSyncInput;
export type TSecretSyncListItem =
| TAwsParameterStoreSyncListItem
@@ -253,7 +269,9 @@ export type TSecretSyncListItem =
| TZabbixSyncListItem
| TRailwaySyncListItem
| TChecklySyncListItem
| TSupabaseSyncListItem;
| TSupabaseSyncListItem
| TDigitalOceanAppPlatformSyncListItem
| TBitbucketSyncListItem;
export type TSyncOptionsConfig = {
canImportSecrets: boolean;

View File

@@ -415,6 +415,8 @@ export const secretV2BridgeDALFactory = ({ db, keyStore }: TSecretV2DalArg) => {
filters?: {
search?: string;
tagSlugs?: string[];
includeTagsInSearch?: boolean;
includeMetadataInSearch?: boolean;
}
) => {
try {
@@ -433,17 +435,27 @@ export const secretV2BridgeDALFactory = ({ db, keyStore }: TSecretV2DalArg) => {
.whereIn("folderId", folderIds)
.where((bd) => {
if (filters?.search) {
void bd.whereILike("key", `%${filters?.search}%`);
void bd.whereILike(`${TableName.SecretV2}.key`, `%${filters?.search}%`);
if (filters?.includeTagsInSearch) {
void bd.orWhereILike(`${TableName.SecretTag}.slug`, `%${filters?.search}%`);
}
if (filters?.includeMetadataInSearch) {
void bd
.orWhereILike(`${TableName.ResourceMetadata}.key`, `%${filters?.search}%`)
.orWhereILike(`${TableName.ResourceMetadata}.value`, `%${filters?.search}%`);
}
}
})
.where((bd) => {
void bd.whereNull("userId").orWhere({ userId: userId || null });
void bd
.whereNull(`${TableName.SecretV2}.userId`)
.orWhere({ [`${TableName.SecretV2}.userId` as "userId"]: userId || null });
})
.countDistinct("key");
.countDistinct(`${TableName.SecretV2}.key`);
// only need to join tags if filtering by tag slugs
const slugs = filters?.tagSlugs?.filter(Boolean);
if (slugs && slugs.length > 0) {
if ((slugs && slugs.length > 0) || filters?.includeTagsInSearch) {
void query
.leftJoin(
TableName.SecretV2JnTag,
@@ -454,18 +466,31 @@ export const secretV2BridgeDALFactory = ({ db, keyStore }: TSecretV2DalArg) => {
TableName.SecretTag,
`${TableName.SecretV2JnTag}.${TableName.SecretTag}Id`,
`${TableName.SecretTag}.id`
)
.whereIn("slug", slugs);
);
if (slugs?.length) {
void query.whereIn("slug", slugs);
}
}
if (filters?.includeMetadataInSearch) {
void query.leftJoin(
TableName.ResourceMetadata,
`${TableName.SecretV2}.id`,
`${TableName.ResourceMetadata}.secretId`
);
}
const secrets = await query;
// @ts-expect-error not inferred by knex
return Number(secrets[0]?.count ?? 0);
} catch (error) {
throw new DatabaseError({ error, name: "get folder secret count" });
}
};
// This method currently uses too many joins which is not performant, in case we need to add more filters we should consider refactoring this method
const findByFolderIds = async (dto: {
folderIds: string[];
userId?: string;
@@ -485,12 +510,14 @@ export const secretV2BridgeDALFactory = ({ db, keyStore }: TSecretV2DalArg) => {
.whereIn(`${TableName.SecretV2}.folderId`, folderIds)
.where((bd) => {
if (filters?.search) {
void bd.whereILike(`${TableName.SecretV2}.key`, `%${filters?.search}%`);
if (filters?.includeTagsInSearch) {
void bd.orWhereILike(`${TableName.SecretTag}.slug`, `%${filters?.search}%`);
}
if (filters?.includeMetadataInSearch) {
void bd
.whereILike(`${TableName.SecretV2}.key`, `%${filters?.search}%`)
.orWhereILike(`${TableName.SecretTag}.slug`, `%${filters?.search}%`);
} else {
void bd.whereILike(`${TableName.SecretV2}.key`, `%${filters?.search}%`);
.orWhereILike(`${TableName.ResourceMetadata}.key`, `%${filters?.search}%`)
.orWhereILike(`${TableName.ResourceMetadata}.value`, `%${filters?.search}%`);
}
}
@@ -513,18 +540,15 @@ export const secretV2BridgeDALFactory = ({ db, keyStore }: TSecretV2DalArg) => {
`${TableName.SecretV2JnTag}.${TableName.SecretTag}Id`,
`${TableName.SecretTag}.id`
)
.leftJoin(
TableName.SecretReminderRecipients,
`${TableName.SecretV2}.id`,
`${TableName.SecretReminderRecipients}.secretId`
)
.leftJoin(TableName.Users, `${TableName.SecretReminderRecipients}.userId`, `${TableName.Users}.id`)
.leftJoin(TableName.ResourceMetadata, `${TableName.SecretV2}.id`, `${TableName.ResourceMetadata}.secretId`)
.leftJoin(
TableName.SecretRotationV2SecretMapping,
`${TableName.SecretV2}.id`,
`${TableName.SecretRotationV2SecretMapping}.secretId`
)
.leftJoin(TableName.Reminder, `${TableName.SecretV2}.id`, `${TableName.Reminder}.secretId`)
.leftJoin(TableName.ReminderRecipient, `${TableName.Reminder}.id`, `${TableName.ReminderRecipient}.reminderId`)
.leftJoin(TableName.Users, `${TableName.ReminderRecipient}.userId`, `${TableName.Users}.id`)
.where((qb) => {
if (filters?.metadataFilter && filters.metadataFilter.length > 0) {
filters.metadataFilter.forEach((meta) => {
@@ -547,7 +571,11 @@ export const secretV2BridgeDALFactory = ({ db, keyStore }: TSecretV2DalArg) => {
}) as rank`
)
)
.select(db.ref("id").withSchema(TableName.SecretReminderRecipients).as("reminderRecipientId"))
.select(db.ref("id").withSchema(TableName.Reminder).as("reminderId"))
.select(db.ref("message").withSchema(TableName.Reminder).as("reminderNote"))
.select(db.ref("repeatDays").withSchema(TableName.Reminder).as("reminderRepeatDays"))
.select(db.ref("nextReminderDate").withSchema(TableName.Reminder).as("nextReminderDate"))
.select(db.ref("id").withSchema(TableName.ReminderRecipient).as("reminderRecipientId"))
.select(db.ref("username").withSchema(TableName.Users).as("reminderRecipientUsername"))
.select(db.ref("email").withSchema(TableName.Users).as("reminderRecipientEmail"))
.select(db.ref("id").withSchema(TableName.Users).as("reminderRecipientUserId"))
@@ -809,6 +837,44 @@ export const secretV2BridgeDALFactory = ({ db, keyStore }: TSecretV2DalArg) => {
}
};
const findSecretsWithReminderRecipients = async (ids: string[], limit: number, tx?: Knex) => {
try {
// Create a subquery to get limited secret IDs
const limitedSecretIds = (tx || db)(TableName.SecretV2)
.whereIn(`${TableName.SecretV2}.id`, ids)
.limit(limit)
.select("id");
// Join with all recipients for the limited secrets
const docs = await (tx || db)(TableName.SecretV2)
.whereIn(`${TableName.SecretV2}.id`, limitedSecretIds)
.leftJoin(TableName.Reminder, `${TableName.SecretV2}.id`, `${TableName.Reminder}.secretId`)
.leftJoin(TableName.ReminderRecipient, `${TableName.Reminder}.id`, `${TableName.ReminderRecipient}.reminderId`)
.select(selectAllTableCols(TableName.SecretV2))
.select(db.ref("userId").withSchema(TableName.ReminderRecipient).as("reminderRecipientUserId"));
const data = sqlNestRelationships({
data: docs,
key: "id",
parentMapper: (el) => ({
_id: el.id,
...SecretsV2Schema.parse(el)
}),
childrenMapper: [
{
key: "reminderRecipientUserId",
label: "recipients" as const,
mapper: ({ reminderRecipientUserId }) => reminderRecipientUserId
}
]
});
return data;
} catch (error) {
throw new DatabaseError({ error, name: "FindSecretsWithReminderRecipients" });
}
};
return {
...secretOrm,
update,
@@ -826,6 +892,7 @@ export const secretV2BridgeDALFactory = ({ db, keyStore }: TSecretV2DalArg) => {
countByFolderIds,
findOne,
find,
invalidateSecretCacheByProjectId
invalidateSecretCacheByProjectId,
findSecretsWithReminderRecipients
};
};

View File

@@ -231,18 +231,7 @@ export const fnSecretBulkUpdate = async ({
const sanitizedInputSecrets = inputSecrets.map(
({
filter,
data: {
skipMultilineEncoding,
type,
key,
encryptedValue,
userId,
encryptedComment,
metadata,
secretMetadata,
reminderNote,
reminderRepeatDays
}
data: { skipMultilineEncoding, type, key, encryptedValue, userId, encryptedComment, metadata, secretMetadata }
}) => ({
filter: { ...filter, folderId },
data: {
@@ -252,9 +241,7 @@ export const fnSecretBulkUpdate = async ({
userId,
encryptedComment,
metadata: JSON.stringify(metadata || secretMetadata || []),
reminderNote,
encryptedValue,
reminderRepeatDays
encryptedValue
}
})
);
@@ -270,9 +257,7 @@ export const fnSecretBulkUpdate = async ({
encryptedComment,
version,
metadata,
reminderNote,
encryptedValue,
reminderRepeatDays,
id: secretId
}) => ({
skipMultilineEncoding,
@@ -282,9 +267,7 @@ export const fnSecretBulkUpdate = async ({
encryptedComment,
version,
metadata: metadata ? JSON.stringify(metadata) : [],
reminderNote,
encryptedValue,
reminderRepeatDays,
folderId,
secretId,
userActorId,
@@ -407,6 +390,7 @@ export const fnSecretBulkDelete = async ({
secretQueueService,
folderCommitService,
secretVersionDAL,
projectId,
commitChanges
}: TFnSecretBulkDelete) => {
const deletedSecrets = await secretDAL.deleteMany(
@@ -419,11 +403,14 @@ export const fnSecretBulkDelete = async ({
tx
);
await Promise.allSettled(
await Promise.all(
deletedSecrets
.filter(({ reminderRepeatDays }) => Boolean(reminderRepeatDays))
.map(({ id, reminderRepeatDays }) =>
secretQueueService.removeSecretReminder({ secretId: id, repeatDays: reminderRepeatDays as number }, tx)
secretQueueService.removeSecretReminder(
{ secretId: id, repeatDays: reminderRepeatDays as number, projectId },
tx
)
)
);

View File

@@ -39,6 +39,7 @@ import { TCommitResourceChangeDTO, TFolderCommitServiceFactory } from "../folder
import { TKmsServiceFactory } from "../kms/kms-service";
import { KmsDataKey } from "../kms/kms-types";
import { TProjectEnvDALFactory } from "../project-env/project-env-dal";
import { TReminderServiceFactory } from "../reminder/reminder-types";
import { TResourceMetadataDALFactory } from "../resource-metadata/resource-metadata-dal";
import { TSecretQueueFactory } from "../secret/secret-queue";
import { TGetASecretByIdDTO } from "../secret/secret-types";
@@ -115,6 +116,7 @@ type TSecretV2BridgeServiceFactoryDep = {
snapshotService: Pick<TSecretSnapshotServiceFactory, "performSnapshot">;
resourceMetadataDAL: Pick<TResourceMetadataDALFactory, "insertMany" | "delete">;
keyStore: Pick<TKeyStoreFactory, "getItem" | "setExpiry" | "setItemWithExpiry" | "deleteItem">;
reminderService: Pick<TReminderServiceFactory, "createReminder" | "getReminder">;
};
export type TSecretV2BridgeServiceFactory = ReturnType<typeof secretV2BridgeServiceFactory>;
@@ -139,7 +141,8 @@ export const secretV2BridgeServiceFactory = ({
secretApprovalRequestSecretDAL,
kmsService,
resourceMetadataDAL,
keyStore
keyStore,
reminderService
}: TSecretV2BridgeServiceFactoryDep) => {
const $validateSecretReferences = async (
projectId: string,
@@ -311,7 +314,6 @@ export const secretV2BridgeServiceFactory = ({
{
version: 1,
type,
reminderRepeatDays: inputSecretData.secretReminderRepeatDays,
encryptedComment: setKnexStringValue(
inputSecretData.secretComment,
(value) => secretManagerEncryptor({ plainText: Buffer.from(value) }).cipherTextBlob
@@ -319,7 +321,6 @@ export const secretV2BridgeServiceFactory = ({
encryptedValue: inputSecretData.secretValue
? secretManagerEncryptor({ plainText: Buffer.from(inputSecretData.secretValue) }).cipherTextBlob
: undefined,
reminderNote: inputSecretData.secretReminderNote,
skipMultilineEncoding: inputSecretData.skipMultilineEncoding,
key: secretName,
userId: inputSecret.type === SecretType.Personal ? actorId : null,
@@ -345,6 +346,20 @@ export const secretV2BridgeServiceFactory = ({
return createdSecret;
});
if (inputSecret.secretReminderRepeatDays) {
await reminderService.createReminder({
actor,
actorId,
actorOrgId,
actorAuthMethod,
reminder: {
secretId: secret.id,
message: inputSecret.secretReminderNote,
repeatDays: inputSecret.secretReminderRepeatDays
}
});
}
await secretDAL.invalidateSecretCacheByProjectId(projectId);
if (inputSecret.type === SecretType.Shared) {
await snapshotService.performSnapshot(folderId);
@@ -521,12 +536,10 @@ export const secretV2BridgeServiceFactory = ({
{
filter: { id: secretId },
data: {
reminderRepeatDays: inputSecret.secretReminderRepeatDays,
encryptedComment: setKnexStringValue(
inputSecret.secretComment,
(value) => secretManagerEncryptor({ plainText: Buffer.from(value) }).cipherTextBlob
),
reminderNote: inputSecret.secretReminderNote,
skipMultilineEncoding: inputSecret.skipMultilineEncoding,
key: inputSecret.newSecretName || secretName,
tags: inputSecret.tagIds,
@@ -547,19 +560,20 @@ export const secretV2BridgeServiceFactory = ({
tx
})
);
await secretQueueService.handleSecretReminder({
newSecret: {
id: updatedSecret[0].id,
...inputSecret
},
oldSecret: {
id: secret.id,
secretReminderNote: secret.reminderNote,
secretReminderRepeatDays: secret.reminderRepeatDays,
secretReminderRecipients: secret.secretReminderRecipients?.map((el) => el.user.id)
},
projectId
});
if (inputSecret.secretReminderRepeatDays) {
await reminderService.createReminder({
actor,
actorId,
actorOrgId,
actorAuthMethod,
reminder: {
secretId: secret.id,
message: inputSecret.secretReminderNote,
repeatDays: inputSecret.secretReminderRepeatDays,
recipients: inputSecret.secretReminderRecipients
}
});
}
await secretDAL.invalidateSecretCacheByProjectId(projectId);
if (inputSecret.type === SecretType.Shared) {
@@ -1930,12 +1944,10 @@ export const secretV2BridgeServiceFactory = ({
return {
filter: { id: originalSecret.id, type: SecretType.Shared },
data: {
reminderRepeatDays: el.secretReminderRepeatDays,
encryptedComment: setKnexStringValue(
el.secretComment,
(value) => secretManagerEncryptor({ plainText: Buffer.from(value) }).cipherTextBlob
),
reminderNote: el.secretReminderNote,
skipMultilineEncoding: el.skipMultilineEncoding,
key: el.newSecretName || el.secretKey,
tags: el.tagIds,
@@ -2601,9 +2613,7 @@ export const secretV2BridgeServiceFactory = ({
key: doc.key,
encryptedComment: doc.encryptedComment,
skipMultilineEncoding: doc.skipMultilineEncoding,
reminderNote: doc.reminderNote,
secretMetadata: doc.secretMetadata,
reminderRepeatDays: doc.reminderRepeatDays,
...(doc.encryptedValue
? {
encryptedValue: doc.encryptedValue,
@@ -2997,6 +3007,11 @@ export const secretV2BridgeServiceFactory = ({
});
};
const findSecretIdsByFolderIdAndKeys = async ({ folderId, keys }: { folderId: string; keys: string[] }) => {
const secrets = await secretDAL.find({ folderId, $in: { [`${TableName.SecretV2}.key` as "key"]: keys } });
return secrets.map((el) => ({ id: el.id, key: el.key }));
};
return {
createSecret,
deleteSecret,
@@ -3016,6 +3031,7 @@ export const secretV2BridgeServiceFactory = ({
getSecretsByFolderMappings,
getSecretById,
getAccessibleSecrets,
getSecretVersionsByIds
getSecretVersionsByIds,
findSecretIdsByFolderIdAndKeys
};
};

View File

@@ -249,6 +249,7 @@ export type TCreateSecretReminderDTO = {
export type TRemoveSecretReminderDTO = {
secretId: string;
repeatDays: number;
projectId: string;
};
export type TBackFillSecretReferencesDTO = TProjectPermission;
@@ -358,6 +359,7 @@ export type TFindSecretsByFolderIdsFilter = {
tagSlugs?: string[];
metadataFilter?: { key?: string; value?: string }[];
includeTagsInSearch?: boolean;
includeMetadataInSearch?: boolean;
keys?: string[];
};

View File

@@ -5,8 +5,6 @@ import { TDbClient } from "@app/db";
import { SecretsSchema, SecretType, TableName, TSecrets, TSecretsUpdate } from "@app/db/schemas";
import { BadRequestError, DatabaseError, NotFoundError } from "@app/lib/errors";
import { ormify, selectAllTableCols, sqlNestRelationships } from "@app/lib/knex";
import { logger } from "@app/lib/logger";
import { QueueName, TQueueServiceFactory } from "@app/queue";
export type TSecretDALFactory = ReturnType<typeof secretDALFactory>;
@@ -383,94 +381,6 @@ export const secretDALFactory = (db: TDbClient) => {
}
};
const pruneSecretReminders = async (queueService: TQueueServiceFactory) => {
const REMINDER_PRUNE_BATCH_SIZE = 5_000;
const MAX_RETRY_ON_FAILURE = 3;
let numberOfRetryOnFailure = 0;
let deletedReminderCount = 0;
logger.info(`${QueueName.DailyResourceCleanUp}: secret reminders started`);
try {
const repeatableJobs = await queueService.getRepeatableJobs(QueueName.SecretReminder);
const reminderJobs = repeatableJobs
.map((job) => ({ secretId: job.id?.replace("reminder-", "") as string, jobKey: job.key }))
.filter(Boolean);
if (reminderJobs.length === 0) {
logger.info(`${QueueName.DailyResourceCleanUp}: no reminder jobs found`);
return;
}
for (let offset = 0; offset < reminderJobs.length; offset += REMINDER_PRUNE_BATCH_SIZE) {
try {
const batchIds = reminderJobs.slice(offset, offset + REMINDER_PRUNE_BATCH_SIZE).map((r) => r.secretId);
const payload = {
$in: {
id: batchIds
}
};
const opts = {
limit: REMINDER_PRUNE_BATCH_SIZE
};
// Find existing secrets with pagination
// eslint-disable-next-line no-await-in-loop
const [secrets, secretsV2] = await Promise.all([
ormify(db, TableName.Secret).find(payload, opts),
ormify(db, TableName.SecretV2).find(payload, opts)
]);
const foundSecretIds = new Set([
...secrets.map((secret) => secret.id),
...secretsV2.map((secret) => secret.id)
]);
// Find IDs that don't exist in either table
const secretIdsNotFound = batchIds.filter((secretId) => !foundSecretIds.has(secretId));
// Delete reminders for non-existent secrets
for (const secretId of secretIdsNotFound) {
const jobKey = reminderJobs.find((r) => r.secretId === secretId)?.jobKey;
if (jobKey) {
// eslint-disable-next-line no-await-in-loop
await queueService.stopRepeatableJobByKey(QueueName.SecretReminder, jobKey);
deletedReminderCount += 1;
}
}
numberOfRetryOnFailure = 0;
} catch (error) {
numberOfRetryOnFailure += 1;
logger.error(error, `Failed to process batch at offset ${offset}`);
if (numberOfRetryOnFailure >= MAX_RETRY_ON_FAILURE) {
break;
}
// Retry the current batch
offset -= REMINDER_PRUNE_BATCH_SIZE;
// eslint-disable-next-line no-promise-executor-return, @typescript-eslint/no-loop-func, no-await-in-loop
await new Promise((resolve) => setTimeout(resolve, 500 * numberOfRetryOnFailure));
}
// Small delay between batches
// eslint-disable-next-line no-promise-executor-return, @typescript-eslint/no-loop-func, no-await-in-loop
await new Promise((resolve) => setTimeout(resolve, 10));
}
} catch (error) {
logger.error(error, "Failed to complete secret reminder pruning");
} finally {
logger.info(
`${QueueName.DailyResourceCleanUp}: secret reminders completed. Deleted ${deletedReminderCount} reminders`
);
}
};
return {
...secretOrm,
update,
@@ -485,7 +395,6 @@ export const secretDALFactory = (db: TDbClient) => {
upsertSecretReferences,
findReferencedSecretReferences,
findAllProjectSecretValues,
pruneSecretReminders,
findManySecretsWithTags
};
};

View File

@@ -18,11 +18,9 @@ import { ProjectPermissionSecretActions } from "@app/ee/services/permission/proj
import { getConfig } from "@app/lib/config/env";
import { buildSecretBlindIndexFromName } from "@app/lib/crypto";
import { crypto, SymmetricKeySize } from "@app/lib/crypto/cryptography";
import { daysToMillisecond, secondsToMillis } from "@app/lib/dates";
import { BadRequestError, NotFoundError } from "@app/lib/errors";
import { groupBy, unique } from "@app/lib/fn";
import { logger } from "@app/lib/logger";
import { QueueJobs, QueueName, TQueueServiceFactory } from "@app/queue";
import {
fnSecretBulkInsert as fnSecretV2BridgeBulkInsert,
fnSecretBulkUpdate as fnSecretV2BridgeBulkUpdate,
@@ -34,6 +32,7 @@ import { KmsDataKey } from "../kms/kms-types";
import { getBotKeyFnFactory } from "../project-bot/project-bot-fns";
import { TProjectBotServiceFactory } from "../project-bot/project-bot-service";
import { TProjectEnvDALFactory } from "../project-env/project-env-dal";
import { TReminderServiceFactory } from "../reminder/reminder-types";
import { TSecretFolderDALFactory } from "../secret-folder/secret-folder-dal";
import { TSecretV2BridgeDALFactory } from "../secret-v2-bridge/secret-v2-bridge-dal";
import { TSecretDALFactory } from "./secret-dal";
@@ -745,7 +744,8 @@ export const fnSecretBulkDelete = async ({
tx,
actorId,
secretDAL,
secretQueueService
secretQueueService,
projectId
}: TFnSecretBulkDelete) => {
const deletedSecrets = await secretDAL.deleteMany(
inputSecrets.map(({ type, secretBlindIndex }) => ({
@@ -761,7 +761,10 @@ export const fnSecretBulkDelete = async ({
deletedSecrets
.filter(({ secretReminderRepeatDays }) => Boolean(secretReminderRepeatDays))
.map(({ id, secretReminderRepeatDays }) =>
secretQueueService.removeSecretReminder({ secretId: id, repeatDays: secretReminderRepeatDays as number }, tx)
secretQueueService.removeSecretReminder(
{ secretId: id, repeatDays: secretReminderRepeatDays as number, projectId },
tx
)
)
);
@@ -1228,14 +1231,14 @@ export const decryptSecretWithBot = (
type TFnDeleteProjectSecretReminders = {
secretDAL: Pick<TSecretDALFactory, "find">;
secretV2BridgeDAL: Pick<TSecretV2BridgeDALFactory, "find">;
queueService: Pick<TQueueServiceFactory, "stopRepeatableJob">;
reminderService: Pick<TReminderServiceFactory, "deleteReminderBySecretId">;
projectBotService: Pick<TProjectBotServiceFactory, "getBotKey">;
folderDAL: Pick<TSecretFolderDALFactory, "findByProjectId">;
};
export const fnDeleteProjectSecretReminders = async (
projectId: string,
{ secretDAL, secretV2BridgeDAL, queueService, projectBotService, folderDAL }: TFnDeleteProjectSecretReminders
{ secretDAL, secretV2BridgeDAL, reminderService, projectBotService, folderDAL }: TFnDeleteProjectSecretReminders
) => {
const projectFolders = await folderDAL.findByProjectId(projectId);
const { shouldUseSecretV2Bridge } = await projectBotService.getBotKey(projectId, false);
@@ -1250,23 +1253,13 @@ export const fnDeleteProjectSecretReminders = async (
$notNull: ["secretReminderRepeatDays"]
});
const appCfg = getConfig();
for await (const secret of projectSecrets) {
const repeatDays = shouldUseSecretV2Bridge
? (secret as { reminderRepeatDays: number }).reminderRepeatDays
: (secret as { secretReminderRepeatDays: number }).secretReminderRepeatDays;
// We're using the queue service directly to get around conflicting imports.
if (repeatDays) {
await queueService.stopRepeatableJob(
QueueName.SecretReminder,
QueueJobs.SecretReminder,
{
// on prod it this will be in days, in development this will be second
every: appCfg.NODE_ENV === "development" ? secondsToMillis(repeatDays) : daysToMillisecond(repeatDays)
},
`reminder-${secret.id}`
);
await reminderService.deleteReminderBySecretId(secret.id, projectId);
}
}
};

View File

@@ -19,7 +19,6 @@ import { TSnapshotSecretV2DALFactory } from "@app/ee/services/secret-snapshot/sn
import { KeyStorePrefixes, KeyStoreTtls, TKeyStoreFactory } from "@app/keystore/keystore";
import { getConfig } from "@app/lib/config/env";
import { crypto, SymmetricKeySize } from "@app/lib/crypto/cryptography";
import { daysToMillisecond, secondsToMillis } from "@app/lib/dates";
import { BadRequestError, NotFoundError } from "@app/lib/errors";
import { getTimeDifferenceInSeconds, groupBy, isSamePath, unique } from "@app/lib/fn";
import { logger } from "@app/lib/logger";
@@ -40,7 +39,6 @@ import { TIntegrationAuthServiceFactory } from "../integration-auth/integration-
import { syncIntegrationSecrets } from "../integration-auth/integration-sync-secret";
import { TKmsServiceFactory } from "../kms/kms-service";
import { KmsDataKey } from "../kms/kms-types";
import { TOrgDALFactory } from "../org/org-dal";
import { TOrgServiceFactory } from "../org/org-service";
import { TProjectDALFactory } from "../project/project-dal";
import { createProjectKey } from "../project/project-fns";
@@ -49,12 +47,12 @@ import { TProjectEnvDALFactory } from "../project-env/project-env-dal";
import { TProjectKeyDALFactory } from "../project-key/project-key-dal";
import { TProjectMembershipDALFactory } from "../project-membership/project-membership-dal";
import { TProjectUserMembershipRoleDALFactory } from "../project-membership/project-user-membership-role-dal";
import { TReminderServiceFactory } from "../reminder/reminder-types";
import { TResourceMetadataDALFactory } from "../resource-metadata/resource-metadata-dal";
import { ResourceMetadataDTO } from "../resource-metadata/resource-metadata-schema";
import { TSecretFolderDALFactory } from "../secret-folder/secret-folder-dal";
import { TSecretImportDALFactory } from "../secret-import/secret-import-dal";
import { fnSecretsV2FromImports } from "../secret-import/secret-import-fns";
import { TSecretReminderRecipientsDALFactory } from "../secret-reminder-recipients/secret-reminder-recipients-dal";
import { TSecretV2BridgeDALFactory } from "../secret-v2-bridge/secret-v2-bridge-dal";
import { expandSecretReferencesFactory, getAllSecretReferences } from "../secret-v2-bridge/secret-v2-bridge-fns";
import { TSecretVersionV2DALFactory } from "../secret-v2-bridge/secret-version-dal";
@@ -92,7 +90,6 @@ type TSecretQueueFactoryDep = {
projectKeyDAL: Pick<TProjectKeyDALFactory, "create">;
projectMembershipDAL: Pick<TProjectMembershipDALFactory, "findAllProjectMembers" | "create">;
smtpService: TSmtpService;
orgDAL: Pick<TOrgDALFactory, "findOrgByProjectId">;
secretVersionDAL: TSecretVersionDALFactory;
secretBlindIndexDAL: TSecretBlindIndexDALFactory;
secretTagDAL: TSecretTagDALFactory;
@@ -112,11 +109,8 @@ type TSecretQueueFactoryDep = {
projectUserMembershipRoleDAL: Pick<TProjectUserMembershipRoleDALFactory, "create">;
resourceMetadataDAL: Pick<TResourceMetadataDALFactory, "insertMany" | "delete">;
folderCommitService: Pick<TFolderCommitServiceFactory, "createCommit">;
secretReminderRecipientsDAL: Pick<
TSecretReminderRecipientsDALFactory,
"delete" | "findUsersBySecretId" | "insertMany" | "transaction"
>;
secretSyncQueue: Pick<TSecretSyncQueueFactory, "queueSecretSyncsSyncSecretsByPath">;
reminderService: Pick<TReminderServiceFactory, "createReminderInternal" | "deleteReminderBySecretId">;
};
export type TGetSecrets = {
@@ -154,7 +148,6 @@ export const secretQueueFactory = ({
userDAL,
webhookDAL,
projectEnvDAL,
orgDAL,
smtpService,
projectDAL,
projectBotDAL,
@@ -177,9 +170,9 @@ export const secretQueueFactory = ({
projectUserMembershipRoleDAL,
projectKeyDAL,
resourceMetadataDAL,
secretReminderRecipientsDAL,
secretSyncQueue,
folderCommitService
folderCommitService,
reminderService
}: TSecretQueueFactoryDep) => {
const integrationMeter = opentelemetry.metrics.getMeter("Integrations");
const errorHistogram = integrationMeter.createHistogram("integration_secret_sync_errors", {
@@ -189,19 +182,8 @@ export const secretQueueFactory = ({
const removeSecretReminder = async ({ deleteRecipients = true, ...dto }: TRemoveSecretReminderDTO, tx?: Knex) => {
if (deleteRecipients) {
await secretReminderRecipientsDAL.delete({ secretId: dto.secretId }, tx);
await reminderService.deleteReminderBySecretId(dto.secretId, dto.projectId, tx);
}
const appCfg = getConfig();
await queueService.stopRepeatableJob(
QueueName.SecretReminder,
QueueJobs.SecretReminder,
{
// on prod it this will be in days, in development this will be second
every: appCfg.NODE_ENV === "development" ? secondsToMillis(dto.repeatDays) : daysToMillisecond(dto.repeatDays)
},
`reminder-${dto.secretId}`
);
};
const $generateActor = async (actorId?: string, isManual?: boolean): Promise<Actor> => {
@@ -241,11 +223,9 @@ export const secretQueueFactory = ({
oldSecret,
newSecret,
projectId,
deleteRecipients = true
secretReminderRecipients
}: TCreateSecretReminderDTO) => {
try {
const appCfg = getConfig();
if (oldSecret.id !== newSecret.id) {
throw new BadRequestError({
name: "SecretReminderIdMismatch",
@@ -260,38 +240,13 @@ export const secretQueueFactory = ({
});
}
// If the secret already has a reminder, we should remove the existing one first.
if (oldSecret.secretReminderRepeatDays) {
await removeSecretReminder({
repeatDays: oldSecret.secretReminderRepeatDays,
secretId: oldSecret.id,
deleteRecipients
});
}
await queueService.queue(
QueueName.SecretReminder,
QueueJobs.SecretReminder,
{
note: newSecret.secretReminderNote,
projectId,
repeatDays: newSecret.secretReminderRepeatDays,
secretId: newSecret.id
},
{
jobId: `reminder-${newSecret.id}`,
repeat: {
// on prod it this will be in days, in development this will be second
every:
appCfg.NODE_ENV === "development"
? secondsToMillis(newSecret.secretReminderRepeatDays)
: daysToMillisecond(newSecret.secretReminderRepeatDays),
immediately: true
},
removeOnComplete: true,
removeOnFail: true
}
);
await reminderService.createReminderInternal({
secretId: newSecret.id,
message: newSecret.secretReminderNote,
repeatDays: newSecret.secretReminderRepeatDays,
recipients: secretReminderRecipients,
projectId
});
} catch (err) {
logger.error(err, "Failed to create secret reminder.");
throw new BadRequestError({
@@ -304,55 +259,30 @@ export const secretQueueFactory = ({
const handleSecretReminder = async ({ newSecret, oldSecret, projectId }: THandleReminderDTO) => {
const { secretReminderRepeatDays, secretReminderNote, secretReminderRecipients } = newSecret;
const recipientsUpdated =
secretReminderRecipients?.some(
(newId) => !oldSecret.secretReminderRecipients?.find((oldId) => newId === oldId)
) || secretReminderRecipients?.length !== oldSecret.secretReminderRecipients?.length;
await secretReminderRecipientsDAL.transaction(async (tx) => {
if (newSecret.type !== SecretType.Personal && secretReminderRepeatDays !== undefined) {
if (
(secretReminderRepeatDays && oldSecret.secretReminderRepeatDays !== secretReminderRepeatDays) ||
(secretReminderNote && oldSecret.secretReminderNote !== secretReminderNote)
) {
await addSecretReminder({
oldSecret,
newSecret,
projectId,
deleteRecipients: false
});
} else if (
secretReminderRepeatDays === null &&
secretReminderNote === null &&
oldSecret.secretReminderRepeatDays
) {
await removeSecretReminder({
secretId: oldSecret.id,
repeatDays: oldSecret.secretReminderRepeatDays
});
}
if (newSecret.type !== SecretType.Personal && secretReminderRepeatDays !== undefined) {
if (
(secretReminderRepeatDays && oldSecret.secretReminderRepeatDays !== secretReminderRepeatDays) ||
(secretReminderNote && oldSecret.secretReminderNote !== secretReminderNote)
) {
await addSecretReminder({
oldSecret,
newSecret,
projectId,
secretReminderRecipients: secretReminderRecipients ?? [],
deleteRecipients: false
});
} else if (
secretReminderRepeatDays === null &&
secretReminderNote === null &&
oldSecret.secretReminderRepeatDays
) {
await removeSecretReminder({
secretId: oldSecret.id,
repeatDays: oldSecret.secretReminderRepeatDays,
projectId
});
}
if (recipientsUpdated) {
// if no recipients, delete all existing recipients
if (!secretReminderRecipients?.length) {
const existingRecipients = await secretReminderRecipientsDAL.findUsersBySecretId(newSecret.id, tx);
if (existingRecipients) {
await secretReminderRecipientsDAL.delete({ secretId: newSecret.id }, tx);
}
} else {
await secretReminderRecipientsDAL.delete({ secretId: newSecret.id }, tx);
await secretReminderRecipientsDAL.insertMany(
secretReminderRecipients.map((r) => ({
secretId: newSecret.id,
userId: r,
projectId
})),
tx
);
}
}
});
}
};
const createManySecretsRawFn = createManySecretsRawFnFactory({
projectDAL,
@@ -747,7 +677,7 @@ export const secretQueueFactory = ({
environment: jobPayload.environmentName,
count: jobPayload.count,
projectName: project.name,
integrationUrl: `${appCfg.SITE_URL}/projects/${project.id}/secret-manager/integrations?selectedTab=native-integrations`
integrationUrl: `${appCfg.SITE_URL}/projects/secret-management/${project.id}/integrations?selectedTab=native-integrations`
}
});
}
@@ -1118,62 +1048,9 @@ export const secretQueueFactory = ({
}
});
// TODO(Carlos): remove this queue (needed for queue initialization and perform the migration)
queueService.start(QueueName.SecretReminder, async ({ data }) => {
logger.info(`secretReminderQueue.process: [secretDocument=${data.secretId}]`);
const { projectId } = data;
const organization = await orgDAL.findOrgByProjectId(projectId);
const project = await projectDAL.findById(projectId);
const secret = await secretV2BridgeDAL.findById(data.secretId);
const [folder] = await folderDAL.findSecretPathByFolderIds(project.id, [secret.folderId]);
const recipients = await secretReminderRecipientsDAL.findUsersBySecretId(data.secretId);
if (!organization) {
logger.info(`secretReminderQueue.process: [secretDocument=${data.secretId}] no organization found`);
return;
}
if (!project) {
logger.info(`secretReminderQueue.process: [secretDocument=${data.secretId}] no project found`);
return;
}
const projectMembers = await projectMembershipDAL.findAllProjectMembers(projectId);
if (!projectMembers || !projectMembers.length) {
logger.info(`secretReminderQueue.process: [secretDocument=${data.secretId}] no project members found`);
return;
}
const selectedRecipients = recipients?.length
? recipients.map((r) => r.email as string)
: projectMembers.map((m) => m.user.email as string);
await smtpService.sendMail({
template: SmtpTemplates.SecretReminder,
subjectLine: "Infisical secret reminder",
recipients: selectedRecipients,
substitutions: {
reminderNote: data.note, // May not be present.
projectName: project.name,
organizationName: organization.name
}
});
await queueService.queue(QueueName.SecretWebhook, QueueJobs.SecWebhook, {
type: WebhookEvents.SecretReminderExpired,
payload: {
projectName: project.name,
projectId: project.id,
secretPath: folder?.path,
environment: folder?.environmentSlug || "",
reminderNote: data.note,
secretName: secret?.key,
secretId: data.secretId
}
});
logger.info(`(deprecated) secretReminderQueue.process: [secretDocument=${data.secretId}]`);
});
const startSecretV2Migration = async (projectId: string) => {

View File

@@ -46,6 +46,7 @@ import { ChangeType } from "../folder-commit/folder-commit-service";
import { TProjectDALFactory } from "../project/project-dal";
import { TProjectBotServiceFactory } from "../project-bot/project-bot-service";
import { TProjectEnvDALFactory } from "../project-env/project-env-dal";
import { TReminderServiceFactory } from "../reminder/reminder-types";
import { TSecretBlindIndexDALFactory } from "../secret-blind-index/secret-blind-index-dal";
import { TSecretFolderDALFactory } from "../secret-folder/secret-folder-dal";
import { TSecretImportDALFactory } from "../secret-import/secret-import-dal";
@@ -129,6 +130,7 @@ type TSecretServiceFactoryDep = {
"insertMany" | "insertApprovalSecretTags"
>;
licenseService: Pick<TLicenseServiceFactory, "getPlan">;
reminderService: Pick<TReminderServiceFactory, "createReminder">;
};
export type TSecretServiceFactory = ReturnType<typeof secretServiceFactory>;
@@ -151,7 +153,8 @@ export const secretServiceFactory = ({
secretApprovalRequestSecretDAL,
secretV2BridgeService,
secretApprovalRequestService,
licenseService
licenseService,
reminderService
}: TSecretServiceFactoryDep) => {
const getSecretReference = async (projectId: string) => {
// if bot key missing means e2e still exist
@@ -551,7 +554,8 @@ export const secretServiceFactory = ({
await secretQueueService.removeSecretReminder(
{
repeatDays: secret.secretReminderRepeatDays,
secretId: secret.id
secretId: secret.id,
projectId
},
tx
);
@@ -1082,7 +1086,8 @@ export const secretServiceFactory = ({
await secretQueueService.removeSecretReminder(
{
repeatDays: secret.secretReminderRepeatDays,
secretId: secret.id
secretId: secret.id,
projectId
},
tx
);
@@ -1137,6 +1142,8 @@ export const secretServiceFactory = ({
| "environment"
| "tagSlugs"
| "search"
| "includeTagsInSearch"
| "includeMetadataInSearch"
>) => {
const { shouldUseSecretV2Bridge } = await projectBotService.getBotKey(projectId);
@@ -1670,8 +1677,6 @@ export const secretServiceFactory = ({
secretComment,
secretValue,
tagIds,
reminderNote: secretReminderNote,
reminderRepeatDays: secretReminderRepeatDays,
secretMetadata
}
]
@@ -1853,9 +1858,6 @@ export const secretServiceFactory = ({
secretComment,
secretValue,
tagIds,
reminderNote: secretReminderNote,
reminderRepeatDays: secretReminderRepeatDays,
secretReminderRecipients,
secretMetadata
}
]
@@ -1864,9 +1866,6 @@ export const secretServiceFactory = ({
return { type: SecretProtectionType.Approval as const, approval };
}
const secret = await secretV2BridgeService.updateSecret({
secretReminderRepeatDays,
secretReminderNote,
secretReminderRecipients,
skipMultilineEncoding,
tagIds,
secretComment,
@@ -1884,6 +1883,21 @@ export const secretServiceFactory = ({
secretValue,
secretMetadata
});
if (secretReminderRepeatDays) {
await reminderService.createReminder({
actor,
actorId,
actorOrgId,
actorAuthMethod,
reminder: {
secretId: secret.id,
message: secretReminderNote,
repeatDays: secretReminderRepeatDays,
recipients: secretReminderRecipients
}
});
}
return { type: SecretProtectionType.Direct as const, secret };
}
@@ -2316,6 +2330,29 @@ export const secretServiceFactory = ({
secrets: inputSecrets,
mode
});
await Promise.all(
inputSecrets
.filter((el) => el.secretReminderRepeatDays)
.map(async (secret) => {
await reminderService.createReminder({
actor,
actorId,
actorOrgId,
actorAuthMethod,
reminder: {
secretId: secrets.find(
(el) =>
(el.secretKey === secret.secretKey || el.secretKey === secret.newSecretName) &&
el.secretPath === (secret.secretPath || secretPath)
)?.id,
message: secret.secretReminderNote,
repeatDays: secret.secretReminderRepeatDays
}
});
})
);
return { type: SecretProtectionType.Direct as const, secrets };
}

View File

@@ -212,6 +212,8 @@ export type TGetSecretsRawDTO = {
limit?: number;
search?: string;
keys?: string[];
includeTagsInSearch?: boolean;
includeMetadataInSearch?: boolean;
} & TProjectPermission;
export type TGetSecretAccessListDTO = {
@@ -310,6 +312,7 @@ export type TUpdateManySecretRawDTO = Omit<TProjectPermission, "projectId"> & {
secretMetadata?: ResourceMetadataDTO;
secretReminderRepeatDays?: number | null;
secretReminderNote?: string | null;
secretPath?: string;
}[];
};
@@ -410,6 +413,7 @@ export type TCreateSecretReminderDTO = {
oldSecret: TPartialSecret;
newSecret: TPartialSecret;
projectId: string;
secretReminderRecipients: string[];
deleteRecipients?: boolean;
};
@@ -417,6 +421,7 @@ export type TCreateSecretReminderDTO = {
export type TRemoveSecretReminderDTO = {
secretId: string;
repeatDays: number;
projectId: string;
deleteRecipients?: boolean;
};

View File

@@ -214,7 +214,7 @@ export const serviceTokenServiceFactory = ({
substitutions: {
tokenName: token.name,
projectName: token.projectName,
url: `${appCfg.SITE_URL}/projects/${token.projectId}/secret-manager/access-management?selectedTab=service-tokens`
url: `${appCfg.SITE_URL}/projects/secret-management/${token.projectId}/access-management?selectedTab=service-tokens`
}
});
await serviceTokenDAL.update({ id: token.id }, { expiryNotificationSent: true });

View File

@@ -0,0 +1,4 @@
---
title: "Available"
openapi: "GET /api/v1/app-connections/digital-ocean/available"
---

View File

@@ -0,0 +1,8 @@
---
title: "Create"
openapi: "POST /api/v1/app-connections/digital-ocean"
---
<Note>
Check out the configuration docs for [Digital Ocean Connections](/integrations/app-connections/digital-ocean) to learn how to obtain the required credentials.
</Note>

View File

@@ -0,0 +1,4 @@
---
title: "Delete"
openapi: "DELETE /api/v1/app-connections/digital-ocean/{connectionId}"
---

View File

@@ -0,0 +1,4 @@
---
title: "Get by ID"
openapi: "GET /api/v1/app-connections/digital-ocean/{connectionId}"
---

View File

@@ -0,0 +1,4 @@
---
title: "Get by Name"
openapi: "GET /api/v1/app-connections/digital-ocean/connection-name/{connectionName}"
---

View File

@@ -0,0 +1,4 @@
---
title: "List"
openapi: "GET /api/v1/app-connections/digital-ocean"
---

View File

@@ -0,0 +1,8 @@
---
title: "Update"
openapi: "PATCH /api/v1/app-connections/digital-ocean/{connectionId}"
---
<Note>
Check out the configuration docs for [Digital Ocean Connections](/integrations/app-connections/digital-ocean) to learn how to obtain the required credentials.
</Note>

View File

@@ -0,0 +1,4 @@
---
title: "Create"
openapi: "POST /api/v1/secret-syncs/bitbucket"
---

View File

@@ -0,0 +1,4 @@
---
title: "Delete"
openapi: "DELETE /api/v1/secret-syncs/bitbucket/{syncId}"
---

View File

@@ -0,0 +1,4 @@
---
title: "Get by ID"
openapi: "GET /api/v1/secret-syncs/bitbucket/{syncId}"
---

View File

@@ -0,0 +1,4 @@
---
title: "Get by Name"
openapi: "GET /api/v1/secret-syncs/bitbucket/sync-name/{syncName}"
---

View File

@@ -0,0 +1,4 @@
---
title: "Import Secrets"
openapi: "POST /api/v1/secret-syncs/bitbucket/{syncId}/import-secrets"
---

View File

@@ -0,0 +1,4 @@
---
title: "List"
openapi: "GET /api/v1/secret-syncs/bitbucket"
---

Some files were not shown because too many files have changed in this diff Show More