Detecting and Removing Secrets from Git History with Betterleaks and git-filter-repo

Developers often accidentally commit sensitive data such as API keys, passwords, database dumps, or private uploads to Git repositories. Even after removal in a later commit, these secrets remain accessible in the commit history and can be recovered by anyone with repository access. The recommended approach begins with scanning the entire history using specialized tools to identify leaked credentials across all branches and past commits. Once identified, the secrets must first be rotated or revoked before any history rewriting occurs. Tools like Betterleaks provide detection with keyword filtering, entropy analysis, and Base64 decoding, while git-filter-repo enables precise removal of files and replacement of secret strings throughout the repository timeline. The process requires careful backups, coordination with teams, and force-pushing rewritten history, followed by fresh clones for all contributors and CI/CD systems. Even after cleanup, organizations must assume that old secrets may persist in forks, backups, or caches and therefore treat rotation as mandatory.

HabrTranslated from Russian

Read at source

Have you ever wondered exactly what has entered your Git repository over time? Accidental commits of debug functions, incorrect comments, or copied files can usually be fixed with a follow-up commit. However, when sensitive data such as third-party service keys, integration credentials, database dumps, or entire /private/uploads directories are involved, simply committing a deletion does not remove the information from history.

Secrets committed to any repository, even a private one, remain accessible through git log or by checking out older commits. Anyone who can read the repository, clone it, fork it, or access connected CI/CD pipelines gains exposure. Contracts with clients frequently prohibit storing credentials or sensitive data in repositories for this reason, emphasizing the need for traceable password issuance and revocation processes.

The first step is to scan the repository thoroughly. The recommended tool is Betterleaks, which replaced earlier utilities such as gitleaks. It applies rule-based detection for patterns resembling AWS keys or GitHub tokens, uses BPE tokenization to reduce false positives, and automatically decodes Base64, hex, and URL-encoded content. Installation is available via Homebrew, Docker, or source build.

Running betterleaks git . --log-opts="--all" -v examines every commit, patch, and change across all branches. Results include details such as the rule ID, matched secret, author, commit message, and file path. Reports can be exported in JSON format for further processing. Manual review of findings is essential to distinguish real secrets from false positives.

After confirmation, git-filter-repo is used to rewrite history. Secrets extracted from the report are converted into replacement rules, and files matching dangerous extensions such as .sql, .env, or .bak are listed for removal. Before execution, secrets must be moved to external configuration files such as .env.local or settings.local.php so the application continues to function after cleanup.

The procedure is destructive and irreversible. A mirror backup must be created, all pull requests closed, and a clean clone prepared. The command git filter-repo --invert-paths --paths-from-file paths-to-remove.txt --replace-text secrets-to-replace.txt performs both file deletion and secret replacement in one pass. After verification with a new scan, aggressive garbage collection and a forced mirror push complete the process.

Every contributor and CI/CD pipeline must then perform a fresh clone rather than a pull or rebase, because commit hashes have changed. Even after these steps, old objects may survive in forks, backups, or caches, making secret rotation mandatory.

Meridian digestThe week's most consequential security stories, distilled into one briefing.See the digest

More in Data Breaches & Leaks

Yandex Restores Access to 96% of Yandex Disk Data After Data Center Attacks
k0t Operators Breach Corporate Firewalls and VPNs Exposing Over 70,000 Credentials
OpenCode Reports Possible Exposure of Third-Party API Keys After Legacy System Breach