Thread by @LeaKissner

Next up at #usesec19: Yueqi Chen will speak about "Toward the Detection of Inconsistencies in Public Security Vulnerability Reports"

Challenges Faced by Security Operations Engineer:
1. Keep an eye on new vulns that affect their systems
2. Patch vulnerable software as soon as possible

So you check NVD, CVE, other databases like Exploit Database, Security Focus, Red Hat bugnzilla...

One day you see a new CVE for MS Outlook (which your company uses)! Oh no! But the version you're using is not included. Huh. So you check NVD, which says your version *is* vulnerable. Which should you trust?

So these researchers decided to measure the inconsistency of vulnerability reports from 1999-2018. That's a lot of vulns! So they built VIEM, an automatic took to extract vulnerable software name&version.

Then measurement

When trying to measure, traditional NLP doesn't work, because people aren't consistent when naming software. For example, some reports might split up parts of the name, use nicknames, format names differently, lots of types of vulnerabilities.

They tried. The recall was <40%

They used a neat modeling technique that is totally in their paper and I'm not going to type (and they're not going to fully explain in this talk) to get a nice ML model that can deal with both the structured and unstructured report.

tl;dr they got accuracy .9764

Metrics:
1. Match software name: # of same words > # of different words "Internet Explorer" and "Microsoft Internet Explorer"
2. Measure version consistency: strict match vs loose match "1.1" and ".1 to 1.2"

Inconsistency exists among all vuln report websites

With loose matching ~.95. With strict matching at most .8

No category is immune from inconsistency.

[There are lots of graphs and I decline to act like the curses library over livetweet ;)]

Sometimes CVE claims more versions are vulnerable. Sometimes NVD.

The rates are quite high, somtimes because of typoes, sometimes not keeping up with external sources (not updating reports when new vulnerable versions are added). Seems to be getting better in last 3 years

What's the impact? Real-world case study.

64 versions are confirmed vulnerable... but there were even more vulnerable that weren't reported at all. 😱

Bad vulnerability reports can lead security folks not to realize they're vulnerable and mitigate, or waste time thinking they're vulnerable when they're not

We can improve this by standardize vuln reporting procedure... or even automate!

Q: How many of these inconsistencies are going to cause problems for humans rather than just the ML algorithm?

A: We did not do this sort of evaluation. First we'd need to measure the ground truth and may do that in future work.

Q: As data is copied around, inconsistencies arise. Do you get the sense that there's a real source of truth to look at?

A: No, because developers report vulnerabilities to different places.