Thread by @BrianNosek

Parents: Taste-testing is a great way to engage kids on science and how methodology can address potential biases.

Here's an example w/@LastCrumbCookie we did recently. This is after a few years of lots of simpler taste tests.

Most of our prior taste tests were amenable to some blinding such as comparing fast food chicken and fries and testing generic vs. brand-name.

Haven wanted to do @LastCrumbCookie and compare their 12 varieties. Blinding not possible & 12 is a lot to test! Challenging for design.

Haven and Joni also wanted to do a pre/post comparison to see if our expectations were good predictors of what we would actually like.

Finally, they wanted to evaluate the cookies holistically and needed a measurement strategy that captured important variation between cookies.

After some debate, we decided:

* Pre-rate early and then remove access to ratings to reduce expectancy effects.

* We can also reduce expectancy by making predictions & ratings different judgments. Pre: predict final rating; Judgments: eval 4 specific features & average results.

* 12 cookies is too many for 1 day, so 3 cookies per day for 4 days.

* Stratified random sampling for each day: 1 cookie from pre-rated top 1/3, 1 from pre-rated middle 1/3; and 1 from pre-rated bottom 1/3.

* Random order for each 3 each day.

* Predictions were made as rank-order from 1 to 12 knowing the cookie name and ingredients.

* Judgments were 1 to 10 ratings on four dimensions: texture, taste, originality, presentation.

* Overall judgment = taste*0.67 + texture*0.11 + originality*0.11 + presentation*0.11

* Overall rating = average of four raters (Dad, Mom, Haven, Joni)

* Convert overall rating to a ranking to compare with predictions

Data was recorded in a Google spreadsheet w/ only planned comparison of pre/post ranking & all else exploratory analysis.

docs.google.com/spreadsheets/d…

Planned comparison of predictions versus judgments combined across raters was a correlation of -0.06! But mom 0.45, dad 0.37, and Joni 0.32 had positive correlations and Haven -0.22 had a negative correlation.

Outperforming expectations:

Netflix and Crunch (Cinnamon Toast Crunch)
Not Today Mr. Muffin Man (Blueberry Muffin)

Underperforming expectations:

The James Dean (Oreo Milkshake)
Macadamnia (Salted Caramel Macadamia)

Others pictured examining means, variation, correlations.

With years of these activities, the kids have learned so much about research design, hypothesizing, blinding, bias, visualization, variability, correlation, measurement, and uncertainty.

This example illustrates how much this activity has matured.

If you want to try it with your kids, start really simple like we did:

First: Rate on an agreed scale, write down scores, compare with each other

Second: Add comparison, eg, different brands of same ice cream

Third: Add blinding

Fourth: Add visualization of data

Fifth: Add variation in measurement

Sixth: Discuss other biases and how to reduce them with good design

There are many other approaches. The point is start super simple and just add "naturally" as the kids offer ideas.

Next, we might move out of spreadsheet for analysis and move into analytic and visualization tools. Maybe we will even try preregistration on OSF.

Check out the data and you can imagine the conversation as we tried to make sense of it together.

Try it! docs.google.com/spreadsheets/d…

For our kids, the interest has become intrinsic, so much so that they set-up tests on their own for themselves, their friends, and the pets.

Science is fun and is for everyone.