Thread by @bnchandrapal

How we monitor secrets committed in our self hosted @gitlab instance in real time ?

(Twitter thread which summarises multiple experiments)

#ProductSecurity #gitlab #security

Use Pre-commit / Pre-receive / Post-receive git hooks ?

Pre-commit : scan for secrets before commit. Prevents committing secrets by devs

Problem: requires access to dev laptops (privacy issue?). Hard to manage regexes in their laptop. Harder in a company hiring lots of devs

Pre-receive : scan for secrets before commits are saved in Gitlab. If secrets found reject. Easy to manage server controls.

Problem: If a person commits secret to code, server rejects push. The person requires (advanced) git skills to remove secret from git history

Post-receive : scan for secrets after the commits are saved in Gitlab. Commits are in Gitlab even if secrets are present.

Problem: The secret is already committed to the code. Anyone having access to Gitlab repo can see.

After discussing with team members, decided to passively listen to commit data of devs (post-receive hook).

If secret found, create a Gitlab confidential issue. Make sure the secret is revoked and secure best practices applied to the repo and deployment.

Evaluating tools:

Found multiple secret scanning tools. Picked two of them - trufflehog and gitleaks.

Trufflehog :

- code in python
- fast for few commits, slow for many commits
- customizable regex
- easy CLI commands
- good documentation

Gitleaks :

- code in golang
- comparatively slow for smaller commits, but very fast in large repos
- customizable regex
- supports whitelisting of issues
- lacking documentation
- lots of options in CLI commands
- allows scan of single commit, but downloads the entire repo

Both tools are good. If you ask me which one I would pick, its Gitleaks.

Gitleaks won the first fight.

Task at hand:
Scan all the commits in real time to find secrets and create Gitlab confidential issues

Gitlab has "System hooks".

docs.gitlab.com/ee/system_hook…

Gitlab can be configured to notify us every time a git push is made (all the push events).

Wrote a script which initiates the tool whenever a push event is made.

Results with Gitleaks:
- has capability to scan only few commits. But downloads the entire repo. (We have a few repos which are 1GB+)
- very very memory intense. Takes up many cores and is still greedy

Results with Trufflehog:
- downloads the entire repo
- slower than gitleaks
- less memory intense

Few commits made to huge 1GB repos, gitleaks downloads the same repo again and again from scratch. So gitleaks takes more disk and memory.

Its a tie for Gitleaks and Trufflehog.

Found a hack where Gitlab can return the commit diff using the commit hash for a specific repo.

Configured gitleaks and trufflehog to scan the diff.

Trufflehog defeated Gitleaks with ease. Trufflehog takes lesser time than gitleaks to scan almost the same secrets.

Then created 2 cloud functions:

1. runs trufflehog
2. creates Gitlab confidential issues if secret found

So this is how the final product looks:

Gitlab system hooks --> cloud fn running trufflehog --> cloud fn to create Gitlab issue --> security team follow up to revoke

This is how we have implemented realtime secret scanning on our self hosted Gitlab server.

If you have followed the thread till here, please follow @fahrishb and @sanjogpanda. This thread is just a summary of our 1 month experiments.

In the final cloud function, we took the most essential parts of trufflehog. Now the cloud function that scans the code looks way different than trufflehog.