Thread by @_jtmelton

Thanksgiving in the US is upon us.

With that in mind, _easily_ my favorite new open-source tool of this year is semgrep. (github.com/returntocorp/s… and semgrep.dev).

Thread: A few thoughts on why ...

1/

Background: I'm in a fairly small group of people who have actually built a static analysis product. I believe in the technology. I also know the problems.

I think Semgrep made 2 fundamentally clever decisions:

2/

1. They didn't try to solve the whole problem.

How do you want to solve the classically difficult data flow "hard problems"? Do you want to solve XSS by finding all the vulns and fixing each individually? Or ... do you want to use a modern framework and CSP?

3/

Do you want to solve sqli by fixing each, or enforcing input validation and using bind params? The former needs big static analysis. The latter can be done with a much simpler/faster tool.

4/

The folks at r2c (and facebook previously) certainly have the ability to ability to solve the deep problems, but they took the right modern customer-focused pragmatic approach IMO.

5/

2. They flipped the traditional model of "find all bugs" to "define and enforce invariants"

It's difficult to overstate the importance of this intuition practically. Static analysis products are difficult in large part due to rule management.

6/

The core tech is ~straightforward, but supporting every language and framework is a never-ending uphill battle. It's also assymetric. The product company has to support _every_ lang/fwk for every product to provide an out-of-the-box experience.

7/

By moving to invariant enforcement, you as the customer or even application get to choose what langs/fwks you want to use, and then ensure that no-one uses anything but those.

8/

My typical example here is if there are 10 sql ORM frameworks in your language that are popular, you're only using 1. The static analysis vendor needs to support and model safe/unsafe apis in all of these libraries.

9/

You need to support 1, and then just ban all the others. This is focused on the "paved path" idea I talk about often.

note: @mmadou was prescient in this space, and had this idea early on.

10/

Further, Semgrep (IMO) has finally solved the developer UX. Working in semgrep.dev/editor just _feels_ right. Copy-paste your code and start writing your rule (with some ~auto-complete capabilities). This is objectively better than anything else I've used.

11/

Also, this is a tool for many uses. I use it for security, but given it's invariant enforcement, the paved path is not just for security - it's for all of dev.

12/

This helps us have a centralized tool to speak the same language as devs. It also means devs don't have to understand arcane custom rule formats.

13/

Finally, I love/build/support open-source, and they are sharing this tool with the community. The team is also really friendly and responsive. I have high hopes.

14/

I remember having a conversation almost ten years ago now with @DinisCruz where we (passionately) discussed having an open-source static analyzer. I feel like we finally have one, and I like it.

15/

TL;DR Like all tools, semgrep's not perfect, but it's genuinely really great. This is as close as I get to fanboy.

16/16 (/fin)

/cc @clintgibler @dlukeomalley @0xine

/cc @yoann_padioleau thanks for your hard work!