Thread by @federicolois

0/n Thank all of you who participated in 'The demon game'. I am taking a screenshot because when knowing the whys it loses all value (there is no more asymmetry of information). These 182 responses are 'The sample'.

1/n You may have already known about this thought experiment you just run on, mainly because there are many different variants of it in the literature. This is the one that I have seen lately:

2/n This example is good because the results are clear-cut to show 2 typical sources of error. Poor experimental setups are the bain of our existence and there are myriad ways they can go wrong.

3/n These are pretty common and easy to make mistakes for anyone that is not on the professional side of understanding how to extract information from people. There is nothing to be ashamed of if you make them. There is one important invariant.

4/n The first and probably the most obvious for anyone is: "Framing". The 'evil demon' immediately triggers your innate mechanism of self-preservation, therefore pushes you to the safest option. You can see how different the results are when the daemon is *fair*.

5/n One thing I can say is: this comment made my day. That is the best example you could get of how 'Framing' works.

6/n Now, this is a one-off, I cannot run it again. This begs the question of what would be the results if there is no demon and/or Russian Roulette at all.

7/n As a real-life example, it is pretty easy to botch UX interview when the interviewer has no experience in such tests (being there, done that). For example, Bernoulli experiments (pass/no-pass) are extremely sensitive to framing. Even a single word can make a huge difference.

8/n A second common error is interviewee imperfect information or in this case, concealed information. In the example, the interviewer knows pretty well what the risk is, but doesn't provide you any contrasting information to assess the true risk. bandolier.org.uk/booth/Risk/dyi…

9/n This mechanic is useful when you want to assess if there is a sequence of actions dependence between decisions. In UX, if possible, you would usually run the same set of tasks in different orders to ensure you can control sequence dependence.

10/n Why did I do the changed example? I just provided the other side to provide proof that as I expected, it is pretty sensitive.

11/n It is pretty known also that assessing risk is something 'we' humans are very bad at. By introducing this information asymmetry (or in this case, concealing the true risk of living) we can inadvertently bias the results.

12/n And that is why even 'simple' thought experiments should be designed in such a way to avoid the controller/designer/interviewer biasing the sample. Hope this crash course on experimental design pitfalls helps you in the future.