Our journey begins with a simple idea - to query all possible AWS keys leaked on Github. Thankfully, @Google gives $300 free credit on BigQuery, so not a problem. We start by creating a regex for AWS Keys.
people.eecs.berkeley.edu/~rohanpadhye/f…
people.eecs.berkeley.edu/~rohanpadhye/f…
We create a BigQuery SQL query inspired by @LiveOverflow's bitcoin private key searching video. Next, we run this Query. The github dataset is about 2.7 TB and it costs $5 for each TB. So each query costs approx $10. The query takes very small time. 
And bam! In a fraction of seconds, we get 9922 AWS Access Keys. These are all from various projects, some may work and some may not. After removing the keys that are invalid, we are left with 7953 keys. Not bad I'd say. We are not disclosing any valid keys for security reasons.
We move a step furthur and create regexes for some more services. We make regexes for Bitly keys, Slack webhooks, Google API Keys, Slack API keys, @SendGrid keys, @MailChimp Tokens, Redis urls, etc. I am really excited at this point.
Query takes some time but it finishes really fast. The results are Amazing. We plot a graph too :D.
This research just shows a fraction of what's out there. This practically cost us nothing, and the results were truly mind blowing. I agree, this is not a targeted attack, but consider the havoc a malicious entity can cause with this type of data.
What can @github do?
Github can adopt filters during Push time that prevent adding code that matches against known filters. Like, when the user pushes the code, it should run that against known filters. If it does not matches, then only should it succed.
Github can adopt filters during Push time that prevent adding code that matches against known filters. Like, when the user pushes the code, it should run that against known filters. If it does not matches, then only should it succed.
What can developers do?
They can use several tools that are available like git-secrets, gitrob, truffleHog etc. Pre-commit hooks are also an interesting choice. github.com/awslabs/git-se… github.com/anshumanbh/git…
All these can prevent such accidental commits.
They can use several tools that are available like git-secrets, gitrob, truffleHog etc. Pre-commit hooks are also an interesting choice. github.com/awslabs/git-se… github.com/anshumanbh/git…
All these can prevent such accidental commits.
Finally, thank you for reading this article. The Open Source hosting platforms have caused an increase in leaks. What we need is a central platform that is free and contains filters to assist in preventing Git Leaks. If anyone does that, it would be best!
This research was inspired from @infosec_au and @nnwakelam's research on Commonspeak2 and Commonspeak. Full credit goes to them!
