0/n Thank all of you who participated in 'The demon game'. I am taking a screenshot because when knowing the whys it loses all value (there is no more asymmetry of information). These 182 responses are 'The sample'.
1/n You may have already known about this thought experiment you just run on, mainly because there are many different variants of it in the literature. This is the one that I have seen lately:
2/n This example is good because the results are clear-cut to show 2 typical sources of error. Poor experimental setups are the bain of our existence and there are myriad ways they can go wrong.
3/n These are pretty common and easy to make mistakes for anyone that is not on the professional side of understanding how to extract information from people. There is nothing to be ashamed of if you make them. There is one important invariant.
4/n The first and probably the most obvious for anyone is: "Framing". The 'evil demon' immediately triggers your innate mechanism of self-preservation, therefore pushes you to the safest option. You can see how different the results are when the daemon is *fair*.
5/n One thing I can say is: this comment made my day. That is the best example you could get of how 'Framing' works.
6/n Now, this is a one-off, I cannot run it again. This begs the question of what would be the results if there is no demon and/or Russian Roulette at all.
7/n As a real-life example, it is pretty easy to botch UX interview when the interviewer has no experience in such tests (being there, done that). For example, Bernoulli experiments (pass/no-pass) are extremely sensitive to framing. Even a single word can make a huge difference.
8/n A second common error is interviewee imperfect information or in this case, concealed information. In the example, the interviewer knows pretty well what the risk is, but doesn't provide you any contrasting information to assess the true risk. bandolier.org.uk/booth/Risk/dyi…
9/n This mechanic is useful when you want to assess if there is a sequence of actions dependence between decisions. In UX, if possible, you would usually run the same set of tasks in different orders to ensure you can control sequence dependence.
10/n Why did I do the changed example? I just provided the other side to provide proof that as I expected, it is pretty sensitive.
11/n It is pretty known also that assessing risk is something 'we' humans are very bad at. By introducing this information asymmetry (or in this case, concealing the true risk of living) we can inadvertently bias the results.
12/n And that is why even 'simple' thought experiments should be designed in such a way to avoid the controller/designer/interviewer biasing the sample. Hope this crash course on experimental design pitfalls helps you in the future.

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Federico Andres Lois

Federico Andres Lois Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @federicolois

7 Nov
1/ The first rule of Lockdown Club is: You do not talk about deaths per million. The second rule of Lockdown Club is: You do not talk about deaths per million.
2/ Third rule of Lockdown Club: someone yells Sweden or herd immunity, you point out the other Nordics. Fourth rule: only two metrics to a discussion, cases and cases.
3/ Fifth rule: one lockdown per season, fellas. Sixth rule: no deaths, no herd. Seventh rule: lockdowns will go on as long as they have to.
Read 4 tweets
17 Oct
Controversial opinion: those that say its not possible to shield the vulnerable, also won't be able to prove if there is a difference (or lack of it) between the trajectory of the virus at Madrid and Stockholm. Who do you think has let it rip?
1/ There were many "Eureka" moments while working on our paper, but probably the most important of all happened pretty early. Non-linear models are highly sensitive to:
2/ We decided early on to eliminate as many parameters as possible. Location parameters are simple to fix, they are location parameters. Viral parameters also, you can go and say R0=3.3 and you made a choice. How many parameters are left if you do that?
Read 32 tweets
13 Oct
1/ Our preprint with @LDjaparidze is online at @medrxivpreprint
"SARS-CoV-2 waves in Europe: A 2-stratum SEIRS model solution"
2/ We extended the SEIRS model to support stratified isolation levels for healthy <60 and vulnerable individuals.
3/ We forced the model to predict daily deaths curves and the reported age serology ratio for key metropolitan areas in Europe. The immunity level estimations obtained were: Madrid 43%; Catalonia 24%; Brussels 73%; and Stockholm 65%.
Read 11 tweets
1 Oct
I have my badge of honor. At this rate, the fat tail event is there won't be more twitter to block. :D Image
For context, this is why he got mad with me.
And I love the depth of the rebuttal. Image
Read 4 tweets
25 Sep
[1/n] Preparing everything to respond to the question: "Under our isolation epidemic model. Is it possible to correct government policy mismanagement starting at the end of July in 90 days for Madrid, Catalunya and a few other cities?" What do you think? Answers in an hour or so.
[2/n] For those that are new to this thread, you can prepare and hone your skills in modeling with the Harmless Virus Game:
[3/n] And the more difficult but also important for this new thread "The Vaccine Gamble" game:
Read 20 tweets
24 Sep
The arrival of a vaccine in the context of COVID can be modeled using game theory as a gamble over the expectation of the final death toll. Most countries will have negative payoff after August 2020. Change my mind. cc @LDjaparidze
Let's make it more interesting... Do you agree?
Context is king. For a gamble to exist we need to define clearly the parameters to observe the likely expected result. Let's start with vaccine efficacy (VE). What do you think is the range most manufacturers are looking for?
Read 24 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!