Thread by @rmoff

For a conference about data, you'd rightly expect that we use data when evaluating sessions and building the program for #KafkaSummit and #Current22. It starts with the program committee (confluent.io/en-gb/blog/int…) reviewing all the submissions with this: en.wikipedia.org/wiki/Elo_ratin…

The output from the session reviews is a single score for each talk, which then forms the basis for the first pass of building the program. Some talks are obviously great … whilst others are obviously not

This is just the beginning of the process. If we built a program on abstract score alone it probably wouldn't be a very balanced program. There are many more factors to take into account.

One bit of data that I thought would be interesting to compare was the speaker ratings for the previous #KafkaSummit with the abstract ratings for the same sessions. How correlated is the abstract rating with the resulting talk delivery?

First up, a huge caveat. Speaker rating data is definitely sketchy at best. For #KafkaSummit it's collected through an app (that not everyone will have installed), not everyone leaves a rating, probably people who feel most strongly will take the time to leave a rating…

…and that's before you take into account the fact that a single number can't convey the full gamut of opinions a person may have (the same goes for abstract scores, BTW). Perhaps you couldn't hear the speaker and rate them down because of it (even though that's the AV's fault)

Maybe the slides were crap but delivery great, or the delivery great but the content poor. Or maybe you had a sore head from the party the night before, or it's nearly lunchtime and you're impatient for the session to finish.

All these reasons and more contribute to the speaker score being a pretty crude measure. But a measure it is nonetheless, so let's take a look at it.

For #KafkaSummit London the very best-rated sessions (top 10%) were all good picks based on the abstract score too

So does a top-rated abstract mean that you're going to get an excellent talk? Well, no, no it doesn't. Even excusing a few outliers and data burps, it's pretty clear that a great abstract is no guarantee of a great talk.

What about if we invert this? Are there bad abstracts that end up being great talks? Well, the data here is already biased for what are hopefully going to be good talks (because why would you build a conference program from abstracts that were crap?).

Of the six abstracts with review scores below the median, three tanked (speaker score in bottom quartile or even bottom 10%) – but one beat the median speaker score and two were in the top quartile!

What conclusions are there to draw from this? Firstly, the abstract isn't *everything*. But does that mean you can put in a crap abstract and expect to be accepted because it might turn out to be a 💎diamond in the rough? NO! 🙊

Per the above data, the *really bad* abstracts (bottom quartile) just don't get accepted. Period🛑

Make sure you put your best work into a good abstract because it gives you the best fighting chance. This blog gives you some advice: rmoff.net/2020/01/16/how…

If we don't pick abstracts based on score alone, then what else factors into that? The screenshot earlier in the thread gives you some clues. For example, is the subject relevant to the audience at the conference? Is there a good representation of different technologies?

Make sure you come along to #Current22 to see what you make of the program that we've got for you —tickets are on sale now: 2022.currentevent.io

(oh, and do all the speakers and future program committee a favour and *always* leave session ratings for any conference you're at if you can 😁)