Merge pull request #591 from Infisical/cli-telemetry

Cli telemetry

README.md

@@ -89,6 +89,30 @@ git clone https://github.com/Infisical/infisical && cd infisical && copy .env.ex
 
 Create an account at `http://localhost:80`
 
+### Scan and prevent secret leaks 
+On top managing secrets with Infisical, you can also scan for over 140+ secret types in your files, directories and git repositories. 
+
+To scan your full git history, run:
+
+```
+infisical scan --verbose
+```
+
+To scan your uncommitted git changes, run:
+
+```
+infisical scan git-changes --verbose
+```
+
+You can also scan your uncommited but staged changes by running the command below. This command can also be used as a pre-commit hook to prevent secret leak.
+
+```
+infisical scan git-changes --staged --verbose
+```
+
+Lean about Infisical's code scanning feature [here](https://infisical.com/docs/cli/scanning-overview) 
+
+
 ## Open-source vs. paid
 
 This repo available under the [MIT expat license](https://github.com/Infisical/infisical/blob/main/LICENSE), with the exception of the `ee` directory which will contain premium enterprise features requiring a Infisical license in the future. 

docs/cli/commands/scan-git-changes.mdx

@@ -0,0 +1,104 @@
+---
+title: "scan git-changes"
+description: "Scan for secrets in your uncommitted code"
+---
+
+```bash
+infisical scan git-changes
+
+# Display the full secret findings
+infisical scan git-changes --verbose
+```
+
+## Description
+Scanning for secrets before you commit your changes is great way to prevent leaks. Infisical makes this easy with the sub command `git-changes`.
+
+The `git-changes` scans for uncommitted changes in a Git repository, and is especially designed for use on developer machines, aligning with the ['shift left'](https://cloud.google.com/architecture/devops/devops-tech-shifting-left-on-security) security approach. 
+When `git-changes` is run on a Git repository, Infisical parses the output from a `git diff` command. 
+
+To scan changes in commits that have been staged via `git add`, you can add the `--staged` flag to the sub command. This flag is particularly useful when using Infisical CLI as a pre-commit tool.
+
+### Flags
+<Accordion title="--staged">
+  **Description**
+
+  detect secrets in a --staged state
+
+  Default value: `false`
+</Accordion>
+
+<Accordion title="--log-opts">
+  **Description**
+
+  git log options
+</Accordion>
+
+<Accordion title="--baseline-path">
+  Short hand: `-b`
+
+  **Description**
+
+  path to baseline with issues that can be ignored
+</Accordion>
+
+<Accordion title="--config">
+  Short hand: `-c`
+
+  **Description**
+
+  config file path
+
+  order of precedence:
+  1. --config flag
+  2. env var INFISICAL_SCAN_CONFIG
+  3. (--source/-s)/.infisical-scan.toml
+  If none of the three options are used, then Infisical will use the default config
+</Accordion>
+
+<Accordion title="--exit-code">
+  **Description**
+
+  exit code when leaks have been encountered (default 1)
+</Accordion>
+
+<Accordion title="--max-target-megabytes">
+  **Description**
+
+  files larger than this will be skipped
+</Accordion>
+
+<Accordion title="--no-color">
+  **Description**
+
+  turn off color for verbose output
+</Accordion>
+
+<Accordion title="--redact">
+  **Description**
+
+  redact secrets from logs and stdout
+</Accordion>
+
+<Accordion title="--report-format">
+  **Description**
+
+  output format (json, csv, sarif) (default "json")
+</Accordion>
+
+<Accordion title="--report-path">
+  **Description**
+
+  report file
+</Accordion>
+
+<Accordion title="--source">
+  **Description**
+
+  path to source (default ".")
+</Accordion>
+
+<Accordion title="--verbose">
+  **Description**
+
+  show verbose output from scan
+</Accordion>

docs/cli/commands/scan.mdx

@@ -0,0 +1,126 @@
+---
+title: "scan"
+description: "Scan git history, directories, and files for secrets"
+---
+
+```bash 
+infisical scan
+
+# Display the full secret findings
+infisical scan --verbose
+```
+
+## Description
+The `infisical scan` command serves to scan repositories, directories, and files. It's compatible with both individual developer machines and Continuous Integration (CI) environments.
+
+When you run `infisical scan` on a Git repository, Infisical will parses the output of a `git log -p` command. This command generates [patches](https://stackoverflow.com/questions/8279602/what-is-a-patch-in-git-version-control) that Infisical uses to identify secrets in your code. 
+You can configure the range of commits that `git log` will cover using the `--log-opts` flag. 
+Any options you can use with `git log -p` are valid for `--log-opts`. 
+
+For instance, to instruct Infisical to scan a specific range of commits, use the following command: `infisical scan --log-opts="--all commitA..commitB"`. For more details, refer to the [Git log documentation](https://git-scm.com/docs/git-log).
+
+To scan individual files and directories, use the `--no-git` flag. 
+
+### Flags 
+<Accordion title="--log-opts">
+  **Description**
+
+  git log options
+</Accordion>
+
+<Accordion title="--no-git">
+  **Description**
+
+  treat git repo as a regular directory and scan those files, --log-opts has no effect on the scan when --no-git is set
+
+  Default value: `false`
+</Accordion>
+
+<Accordion title="--pipe">
+  Short hand: `-b`
+
+  **Description**
+
+  scan input from stdin, ex: `cat some_file | infisical scan --pipe`
+
+  Default value: `false`
+</Accordion>
+
+<Accordion title="--follow-symlinks">
+  Short hand: `-b`
+
+  **Description**
+  scan files that are symlinks to other files
+
+  Default value: `false`
+</Accordion>
+
+<Accordion title="--baseline-path">
+  Short hand: `-b`
+
+  **Description**
+
+  path to baseline with issues that can be ignored
+</Accordion>
+
+<Accordion title="--config">
+  Short hand: `-c`
+
+  **Description**
+
+  config file path
+
+  order of precedence:
+  1. --config flag
+  2. env var INFISICAL_SCAN_CONFIG
+  3. (--source/-s)/.infisical-scan.toml
+  If none of the three options are used, then Infisical will use the default config
+</Accordion>
+
+<Accordion title="--exit-code">
+  **Description**
+
+  exit code when leaks have been encountered (default 1)
+</Accordion>
+
+<Accordion title="--max-target-megabytes">
+  **Description**
+
+  files larger than this will be skipped
+</Accordion>
+
+<Accordion title="--no-color">
+  **Description**
+
+  turn off color for verbose output
+</Accordion>
+
+<Accordion title="--redact">
+  **Description**
+
+  redact secrets from logs and stdout
+</Accordion>
+
+<Accordion title="--report-format">
+  **Description**
+
+  output format (json, csv, sarif) (default "json")
+</Accordion>
+
+<Accordion title="--report-path">
+  **Description**
+
+  report file
+</Accordion>
+
+<Accordion title="--source">
+  **Description**
+
+  path to source (default ".")
+</Accordion>
+
+<Accordion title="--verbose">
+  **Description**
+
+  show verbose output from scan
+</Accordion>

docs/cli/scanning-overview.mdx

@@ -0,0 +1,220 @@
+---
+title: 'Secret scanning'
+description: "Scan and prevent secret leaks in your code base"
+---
+
+Building upon its core functionality of fetching and injecting secrets into your applications, Infisical now takes a significant step forward in bolstering your code security. 
+We've enhanced our CLI tool to include a powerful scanning feature, capable of identifying more than 140 different types of secret leaks in your codebase. 
+In addition to scanning for past leaks, this new addition also actively aids in preventing future leaks. 
+
+# Scanning 
+<Tabs>
+  <Tab title="Scanning files, directories and or git history">
+	```bash
+	infisical scan
+
+	# Display the full secret findings
+	infisical scan --verbose
+	```
+
+	The `infisical scan` command serves to scan repositories, directories, and files. It's compatible with both individual developer machines and Continuous Integration (CI) environments.
+
+	When you run `infisical scan` on a Git repository, Infisical will parses the output of a `git log -p` command. This command generates [patches](https://stackoverflow.com/questions/8279602/what-is-a-patch-in-git-version-control) that Infisical uses to identify secrets in your code. 
+	You can configure the range of commits that `git log` will cover using the `--log-opts` flag. 
+	Any options you can use with `git log -p` are valid for `--log-opts`. 
+
+	For instance, to instruct Infisical to scan a specific range of commits, use the following command: `infisical scan --log-opts="--all commitA..commitB"`. For more details, refer to the [Git log documentation](https://git-scm.com/docs/git-log).
+
+	To scan individual files and directories, use the `--no-git` flag. 
+
+	**View [full details for this command](./commands/scan)**
+  </Tab>
+  <Tab title="Scanning uncommitted files ">
+	```bash 
+	infisical scan git-changes
+
+	# Display the full secret findings
+	infisical git-changes --verbose
+	```
+
+	Scanning for secrets before you commit your changes is great way to prevent leaks. Infisical makes this easy with the sub command `git-changes`.
+
+	The `git-changes` scans for uncommitted changes in a Git repository, and is especially designed for use on developer machines, aligning with the ['shift left'](https://cloud.google.com/architecture/devops/devops-tech-shifting-left-on-security) security approach. 
+	When `git-changes` is run on a Git repository, Infisical parses the output from a `git diff` command. 
+
+	To scan changes in commits that have been staged via `git add`, you can add the `--staged` flag to the sub command. This flag is particularly useful when using Infisical CLI as a pre-commit tool.
+
+	**View [full details for this command](./commands/scan-git-changes)**
+	​
+	<Info>
+		`git-changes` command is only for Git repositories; using it on files or directories will result in an error.
+	</Info>
+  </Tab>
+</Tabs>
+
+#
+#
+# Creating a baseline
+
+When scanning large repositories or repositories with a long history, it can be helpful to use a baseline.
+
+A baseline allows Infisical to ignore any old findings that are already present in the baseline findings. You can create a infisical scan report by running `infisical scan` with the `--report-path` flag.
+
+To create a Infisical scan report and save it in a file called leaks-report.json, use the following command:
+
+```
+infisical scan --report-path leaks-report.json
+```
+
+Once a baseline is created, you can apply it when running the `infisical scan` command again. Use the following command:
+
+```
+infisical scan --baseline-path leaks-report.json --report-path findings.json
+```
+
+After running the `scan` command with the `--baseline-path` flag, the report output in findings.json will only contain new issues.
+
+#
+#
+# Configuration file
+
+To customize the scan, such as specifying your own rules or establishing exceptions for certain files or paths that should not be flagged as risks, you can define these specifications in the configuration file.
+
+<Accordion title="Example custom configuration file">
+```toml infisical-scan.toml
+# Title for the configuration file
+title = "Some title"
+
+
+# This configuration is the foundation that can be expanded. If there are any overlapping rules 
+# between this base and the expanded configuration, the rules in this base will take priority.
+# Another aspect of extending configurations is the ability to link multiple files, up to a depth of 2.
+# "Allowlist" arrays get appended and may have repeated elements.
+# "useDefault" and "path" cannot be used simultaneously. Please choose one.
+[extend]
+# useDefault will extend the base configuration with the default config:
+# https://raw.githubusercontent.com/Infisical/infisical/main/cli/config/infisical-scan.toml
+useDefault = true
+# or you can supply a path to a configuration. Path is relative to where infisical cli
+# was invoked, not the location of the base config.
+path = "common_config.toml"
+
+# An array of tables that contain information that define instructions
+# on how to detect secrets
+[[rules]]
+
+# Unique identifier for this rule
+id = "some-identifier-for-rule"
+
+# Short human readable description of the rule.
+description = "awesome rule 1"
+
+# Golang regular expression used to detect secrets. Note Golang's regex engine
+# does not support lookaheads.
+regex = '''one-go-style-regex-for-this-rule'''
+
+# Golang regular expression used to match paths. This can be used as a standalone rule or it can be used
+# in conjunction with a valid `regex` entry.
+path = '''a-file-path-regex'''
+
+# Array of strings used for metadata and reporting purposes.
+tags = ["tag","another tag"]
+
+# A regex match may have many groups, this allows you to specify the group that should be used as (which group the secret is contained in)
+# its entropy checked if `entropy` is set.
+secretGroup = 3
+
+# Float representing the minimum shannon entropy a regex group must have to be considered a secret.
+# Shannon entropy measures how random a data is. Since secrets are usually composed of many random characters, they typically have high entropy 
+entropy = 3.5
+
+# Keywords are used for pre-regex check filtering.
+# If rule has keywords but the text fragment being scanned doesn't have at least one of it's keywords, it will be skipped for processing further. 
+# Ideally these values should either be part of the idenitifer or unique strings specific to the rule's regex
+# (introduced in v8.6.0)
+keywords = [
+	"auth",
+	"password",
+	"token",
+]
+
+# You can include an allowlist table for a single rule to reduce false positives or ignore commits
+# with known/rotated secrets
+[rules.allowlist]
+description = "ignore commit A"
+commits = [ "commit-A", "commit-B"]
+paths = [
+	'''go\.mod''',
+	'''go\.sum'''
+]
+# note: (rule) regexTarget defaults to check the _Secret_ in the finding.
+# if regexTarget is not specified then _Secret_ will be used.
+# Acceptable values for regexTarget are "match" and "line"
+regexTarget = "match"
+regexes = [
+	'''process''',
+	'''getenv''',
+]
+# note: stopwords targets the extracted secret, not the entire regex match
+# if the extracted secret is found in the stopwords list, the finding will be skipped (i.e not included in report) 
+stopwords = [
+	'''client''',
+	'''endpoint''',
+]
+
+
+# This is a global allowlist which has a higher order of precedence than rule-specific allowlists.
+# If a commit listed in the `commits` field below is encountered then that commit will be skipped and no
+# secrets will be detected for said commit. The same logic applies for regexes and paths.
+[allowlist]
+description = "global allow list"
+commits = [ "commit-A", "commit-B", "commit-C"]
+paths = [
+	'''gitleaks\.toml''',
+	'''(.*?)(jpg|gif|doc)'''
+]
+
+# note: (global) regexTarget defaults to check the _Secret_ in the finding.
+# if regexTarget is not specified then _Secret_ will be used.
+# Acceptable values for regexTarget are "match" and "line"
+regexTarget = "match"
+
+regexes = [
+	'''219-09-9999''',
+	'''078-05-1120''',
+	'''(9[0-9]{2}|666)-\d{2}-\d{4}''',
+]
+# note: stopwords targets the extracted secret, not the entire regex match
+# if the extracted secret is found in the stopwords list, the finding will be skipped (i.e not included in report) 
+stopwords = [
+	'''client''',
+	'''endpoint''',
+]
+```
+</Accordion>
+
+#
+#
+# Ignoring Known Secrets
+If you're intentionally committing a test secret that `infisical scan` might flag, you can instruct Infisical to overlook that secret with the methods listed below.
+
+### infisical-scan:ignore 
+
+To ignore a secret contained in line of code, simply add `infisical-scan:ignore ` at the end of the line as comment in the given programming.
+
+```js example.js
+function helloWorld() {
+    console.log("8dyfuiRyq=vVc3RRr_edRk-fK__JItpZ"); // infisical-scan:ignore 
+}
+```
+
+### .infisicalignore
+An alternative method to exclude specific findings involves creating a .infisicalignore file at your repository's root. 
+You can then add the fingerprints of the findings you wish to exclude. The Infisical scan report provides a unique Fingerprint for each secret found.
+By incorporating these Fingerprints into the .infisicalignore file, Infisical will skip the corresponding secret findings in subsequent scans.
+
+```.ignore .infisicalignore
+bea0ff6e05a4de73a5db625d4ae181a015b50855:frontend/components/utilities/attemptLogin.js:stripe-access-token:147
+bea0ff6e05a4de73a5db625d4ae181a015b50855:backend/src/json/integrations.json:generic-api-key:5
+1961b92340e5d2613acae528b886c842427ce5d0:frontend/components/utilities/attemptLogin.js:stripe-access-token:148
+```

docs/documentation/getting-started/introduction.mdx

@@ -79,6 +79,13 @@ Start syncing environment variables with [Infisical Cloud](https://app.infisical
   >
     Explore integrations for Next.js, Express, Django, and more
   </Card>
+  <Card
+    href="/cli/scanning-overview"
+    title="Secret scanning"
+    color="#0285c7"
+  >
+    Scan and prevent 140+ secret type leaks in your codebase 
+  </Card>
   <Card
     href="https://calendly.com/team-infisical/infisical-demo"
     title="Contact Us"

docs/mint.json

@@ -140,7 +140,7 @@
         "cli/overview",
         "cli/usage",
         {
-          "group": "Commands",
+          "group": "Core commands",
           "pages": [
             "cli/commands/login",
             "cli/commands/init",
@@ -149,9 +149,17 @@
             "cli/commands/export",
             "cli/commands/vault",
             "cli/commands/user",
-            "cli/commands/reset"
+            "cli/commands/reset",
+            {
+              "group": "infisical scan",
+              "pages": [
+                "cli/commands/scan",
+                "cli/commands/scan-git-changes"
+              ]
+            }
           ]
         },
+        "cli/scanning-overview",
         "cli/project-config",
         "cli/faq"
       ]