Thread by @Roger_Beaty

1: Introducing AuDrA: An Automated Drawing Assessment platform for evaluating creativity!

AuDrA is a neural network trained to judge the creativity of drawings like a human.

The AuDrA model and the 13,000+ drawings used to train/test it are open access🧵
psyarxiv.com/t63dm/

2: Creative thinking tasks, like drawing sketches, are often used to assess creativity. A common method is to have human raters judge each response. Yet this method is time-consuming and subjective, slowing research and limiting access for those without resources.

3: Researchers have developed methods to automate verbal creativity assessment to good effect. For example, semantic distance predicts human ratings of creativity for short stories (r ~= .7). However, tools for scoring visual creativity are limited.
link.springer.com/article/10.375…

4: Drawing on machine learning models from computer vision, we trained a modified ResNet—a deep convolutional neural network. The model was trained on a large collection of line drawings and human creativity ratings on the drawing task of Barbot (2018):
frontiersin.org/articles/10.33…

5: Across 4 datasets, nearly 60 raters, and over 13,000 drawings, AuDrA scores strongly correlated with human creativity ratings for new drawings on the same drawing task.

6: AuDrA generalized to predict the creativity of sketches from entirely independent datasets of human ratings that were not used in training the model.

7: As expected, AuDrA's predictive accuracy decreased when it was presented with drawings based on object “starting images”—a task it was never trained on—yet it still outperformed elaboration (“ink on the page”; r = .4).

8: Importantly, AuDrA showed evidence of learning visual features of drawings, beyond simply predicting their degree of elaboration. In other words, AuDrA learned something about what humans find creative about drawings, not just “more elaborate is better."

9: Here’s an overview of the dataset characteristics, along with the number of drawings/raters, the correlation between AuDrA’s predictions and human ratings (r), and average interrater consistency (ICC).

10: This is a first step with a single drawing task. We hope to expand AuDrA to handle other drawing tasks soon.

On that note, check out the recent work of @becmarrone and @dhcropley on automated drawing assessment, recently published in @APADivision10
psycnet.apa.org/doi/10.1037/ac…

11: Hats off to @jdpttrsn, the lead author and developer of AuDrA. And thanks to Baptiste Barbot for sharing the drawings, @JamesLloydCox for data wrangling, and our 50+ raters who scored the 13k drawings!

12: AuDrA is freely available online, along with a tutorial for how to use it. A web application that administers the drawing task and automatically generates creativity scores is coming soon! osf.io/kqn9v/