Thread by @dasingleton

Dear Fellow Profs.

We need to talk. Your lectures are clear and eloquent, you use the latest classroom innovations, and you care about your students.

But your exams suck. This undermines everything else you do.

(tl,dr: collect and use data, test achievement)

One sees endless cutesy questions that are essentially pure “IQ” tests, essay questions that measure bullshitting ability, word questions that test mind reading, multiple choice questions that assess testing strategy. (Is dimensional analysis really all you wanted to test?)

Then there are the questions that the students who know the most are less likely to get. And there is the leveling of exams that pushes the highest grades to students having just the right (meaning the prof’s) level of pedantry and neuroticism.

Students see this and it is demoralizing. Instead of the exams promoting learning, they often teach the opposite, that success is not a matter of studying more but of factors beyond the students’ current control, including simple luck. So why study?

Before you start chanting your ready-made justifications for whatever it is you do now, I have a question: What is your data? If you cannot justify what you do empirically, I do not want to hear how clever your are.

We are all taught our subject matter, but we are not taught how to make up exams, except often by bad examples. There is a whole literature on testing, but here is the most important part: Collect and analyze data on the exams you give.

This is simple. All you have to do to start is record student grades in a spreadsheet problem by problem instead of just recording the total. It is well worth the time, and if you can get someone else to do it for you, that is even better.

Now, order the total exam scores and look at the distribution of points problem by problem. How well do they correlate? Look particularly for low or negative correlations. Something went wrong. If this never happens to you, you are better than I am. But I don’t believe you.

Look for questions that almost everyone got right. Aside from just being “filler”, you may see some misses from the top of the class. Did you mean to punish overthinking?

Look for questions that almost everyone got wrong. Why? If you are convinced that the problem was good, and you want students to get it, what do you have to do differently next time?

For problems having partial credit, look at the standard deviation. If it’s low, the problem has a lower effect on grades than its point total would suggest. Is that what you wanted? This can mislead students into spending too much time. This happens lots with “explain” questions

If the standard deviation is high, the problem has a higher effect on grades than its point total suggests. This happens lots with simple questions requiring a choice, where students think short answer means brief thought. It can help to warn them.

Now look at the totals and ask what particular problems were most important toward getting an A grade. What did the exam really measure, achievement or IQ or pedantry? Grading an exam means grading the exam, and yourself, not just the students.

Subjects vary and class circumstances differ, so there are no one-size-fits-all recommendations, but we should collect tips based on data. Here are some of mine.

Don’t make parts B, C, D, and E depend on part A because part A will be missed for reasons that you cannot anticipate. You will get a bad correlation vs total exam score and a high standard deviation because good students can miss part A.

Break out each part of a question in an obvious way. I guarantee that students will forget to answer parts of a conglomerate question. Having students learn to follow all directions is perhaps important, but teach it when the stakes are low.

Don’t add superfluous information. Universally, longer versions of the same question lead to lower scores, more determined by “IQ” than achievement. Tell them the extra stuff in class later.

If the knowledge you want students to have is graphical, ask the question graphically. Questions written with just words are easy to write down, but it is common for nomenclature to be the sticking point for a question that was meant to test something else.

When you keep the data, it carries over to improving your class year after year. Did the online problems help vs conventional homework? As much as I hated to admit it, they did, because that is my data said.

Oh, and I just love the innovations that you are all doing. They really deserve an award. Tell me about your data.

There is something else about exams that beginning faculty should know. That is, on the rare occasions that teaching comes under serious scrutiny, I have seen exams become the focus. The reason is that other methods for evaluating teaching are qualitative or troubled

(e.g. student evaluations) or very hard to document (e.g. actual student learning), but bad exam questions are fully documented and easily recognized.