Thread by @benwilbrink

Logic and the achievement test item, what about them?

Logic is demanding discipline. Medieval scholars were excellent logicians, rumor has it. Yet there is a lot of logic in everyday reasoning also, not rigorous at all. Expect pseudo-logic to play havoc with test items.

Logic is not the same as reasoning. Logic is a formal discipline, not touching on the real world.
- All animals can fly.
- I am an animal.
- Therefore I can fly.
In logic it is possible to posit a false premise, ‘all animals fly’, and yet have a perfectly valid syllogism.

Test items implicitly of explicitly asking the student to assume a certain state of affairs to be true are ambiguous: how is the student to know she has to reason according to the rules of logic, or not?

In principle, all context questions in math are of this type!

'All’ in ‘all context questions’ is a universal quantifier. Universal quantifiers in achievement tests are ambiguous. What is meant by ‘all’: is it a logical ‘no exceptions assumed’, or is it a colloquial ‘exceptions are rare’ (really an existential quantifier ‘there exist’)?

There exist students who might be better informed than test item writers (think), e.g. because they have read extra books on the subject. Those students might mistake a simple premise, meant to true by the item writer, for a false one, because they know there exist exceptions.

Let’s look at an example, taken from the dissertation by Anton Boonen cps.nl/l/library/down…. Boonen, by the way, declared himself to be a believer in Realistic Math Education.
Can you spot the logical inconsistency?

Some literature.

Hertwig, Benz & Krauss 2008: The conjunction fallacy and the many meanings of ‘and’. pdf:
sci-hub.tw/10.1016/j.cogn…

Tversky and Kahneman (1983) introduced the conjunction fallacy. Ever heard of the Linda problem? See also, you should have that book: ‘Thinking fast and slow’ by David Kahneman. The conjunction fallacy spurred a lot of research.
A recent article: link.springer.com/article/10.100… open

A chapter on deductive reasoning:
Leighton & Sternberg 2012: Reasoning and Problem Solving. Handbook of Psychology, 2nd.
sci-hub.tw/10.1002/978111…
Reasoning is not ‘just reasoning’, it is a complex phenomenon. Do you still want to test students on their reasoning abilities?

The Book Stacking Problem.
A Dutch Science Quiz in 2005 had the following item:
- Build a tower of square paving stones leaning as far as possible to one side.
- The stones must be placed on top of one another, not next to others.
- How far will the tower lean over maximally?

MC:
1. exactly two stones
2. approximately one and a half stone
3. infinitely far.

(info on the math: Eric W. Weisstein (2005?). ‘Book Stacking problem’ mathworld.wolfram.com/BookStackingPr…)

The eds. of the quiz were warned for the clumsy wording of this item., yet they went along with it.

Something sneaky/funny about questioning students. The questioners already know what the answers should be. The students know that the questioners know what they should answer. It is a game that goes on and on, for many, many years.
(Lost the source of this observation)

To be continued. I don’t know yet where the question ‘logic and assessment?’ will lead to. Still to go, a.o.,: Belnap & Steel ‘The logic of questions and answers’; Hintikka ‘Socratic epistemology’; Lepore ‘Meaning and argument’; Finocchiaro ‘Arguments about arguments’.

There is an interesting open access journal ‘Informal Logic. Reasoning and Argumentation in Theory and Practice’ informallogic.ca/index.php/info…
e.g. Sara Greco 2018: The Analysis of Implicit Premises within Children’s Argumentative Inferences. Quote:

On syllogistic reasoning.

“- All animals love water.
- Cats do not like water.
- Cats are not animals.” p. 172 in:

Keith E. Stanovich (1999). ‘Who is rational? Studies of individual differences in reasoning’ crcpress.com/Who-Is-Rationa…
Reviewed: pdfs.semanticscholar.org/66a6/ed5452c27…

The syllogism is valid, its conclusion contradicts world knowledge. “The belief effect in syllogistic reasoning occurs when people endorse conclusions based on their believability or truth in the world rather than on logical validity and it has been much investigated .... ” See

Evans et al., 1983: On the conflict between logic and belief in syllogistic reasoning. Memory & Cognition, 11, 295-306. core.ac.uk/download/pdf/8…
The title is apt: students typically are struggling with new course content contradicting prior belief. E.g. in physics. #belief_change

Markovits & Nantel, 1989: The belief-bias effect in the production and evaluation of logical conclusions. Memory & Cognition, 17, 11-17. researchgate.net/publication/20…
“.... indicates a significant effect of subjects' beliefs on logical reasoning tasks”

Newstead et al. 1992: The source of belief-bias effect in syllogistic reasoning. Cognition, 45, 257-284. Abstract: sciencedirect.com/science/articl… paywalled.
There is a lot more recent research on the belief-bias effect, however. Google it.

And the last one:
Oakhill, Johnson-Laird & Garnhem, 1989: Believability and syllogistic reasoning. Cognition, 31, 117-140. Abstract: sciencedirect.com/science/articl… paywalled, no alternative available.

The general idea: reasoning is not a simple concept. Testing students' reasoning abilities is not simple either. The research literature teaches us that understanding text by analysis and inference is a complicated phenomenon, imo it belongs to problem solving (Allen Newell).

From the literature on reasoning, Leighton & Sternberg 2003 is an informative book: The nature of reasoning. Info: cambridge.org/us/academic/su…
Reasoning is related ‘critical thinking’. Be aware of the lack of research showing both to be GENERIC skills. They are content-SPECIFIC.

By the way. Google Scholar has pdf of the Oakhill Johnson-Laird Garnham article ‘Believability and syllogistic reasoning’ mentalmodels.princeton.edu/papers/1989bel…

One of the syllogisms used in the research (p. 137):

- All of the Frenchmen are wine drinkers
- Some of the wine drinkers are gourmets

What does or does not follow?

(this is an ambiguity that might easily be present in exam items)

A famous one: You’ve won a quiz, and are entitled to a chance to win a car. Just pick the right door out of 3 doors. You’ve made a choice; before opening that door, the quizmaster opens another door, behind it: no car. You are offered the chance to change. Do you change?

Examples given are toy problems. Even then many people have trouble to keep real world concerns apart from formal reasoning. That spells disaster for complex problems. Logicians use truth tables to track complexities: logicinaction.org/docs/ch8.pdf
8.2.4 Tableaus vs natural reasoning

Complex problems are not for novices to solve: they are for domain experts. (e.g. research literature on chess expertise). An expert might use truth table techniques to check solutions, but more often than not the expert will not even be aware of the many steps involved!

A serious attempt to use logic for checking on ambiguities in questioning was made by Belnap & Steel 1976: The logic of questions and answers. (Of importance in questioning by computer software / courseware) For some annotations see benwilbrink.nl/literature/bel…

unroll @threadreaderapp

The Oakley ao syllogism above is easily solved by substitution of Italians for Gourmets:

- All of the Frenchmen are wine drinkers
- Some of the wine drinkers are Italians

The 3-doors problem is known as the Monty Hall problem: always change doors! math.ucsd.edu/~crypto/Monty/…

Substituting Italians for gourmets shows the 2nd premise to be really independent of the 1st one.

Typically teachers think of questions to ask of pupils. Mixing up formal and common sense reasoning in those questions (eg contextual math) is a serious disserve to students.

In the gourmets-syllogism it is forbidden to bring in new premises, eg ‘some Frenchmen are gourmets’, even if one knows the premise to be true.
In education that restriction is extremely unnatural. In reading the main activity is bringing one’s knowledge to bear on the text.

Students, especially the better informed ones, might bring in their own extra premises, and answer questions ‘wrongly’. I suppose this is even more serious a problem with MC questions.
There is (rare) research showing better students to be let down in educational assessment.

The Monty Hall problem is also illuminating for problems in assessment that are seldom recognized.
Professors of statistics were prepared to vehemently defend the wrong action (do not change).
Indeed, the common sense idea is: the car is behind the two doors still closed, 50%!

Is there a way to ‘objectively’ solve the dispute? There is: simulate the Monty Hall problem a number of times, at the kitchen table. The chance to win is 67% after changing, 33% not changing!
A book specifically on the Monty Hall problem: Rosenhouse global.oup.com/academic/produ…

Dutch blogs on the Mont Hall problem by @ionicasmeets global.oup.com/academic/produ…
If you like this kind of problem, other ones treated by Maya Bar-Hillel and Ruma Falk (1982). ‘Some teasers concerning conditional probabilities’ Cognition clear.rice.edu/comp280/10spri…

Okay, for a few weeks I have not been able to work any further on this thread. Let’s get going again. Important work has been done by Jacqueline Leighton, on the importance of background knowledge in answering logic questions. Ah, read: text comprehension questions! Wow.

unroll @threadreaderapp