Abstract:Abstract Creativity research requires assessing the quality of ideas and products. In practice, conducting creativity research often involves asking several human raters to judge participants’ responses to creativity tasks, such as judging the novelty of ideas from the alternate uses task (AUT). Although such subjective scoring methods have proved useful, they have two inherent limitations—labor cost (raters typically code thousands of responses) and subjectivity (raters vary on their perceptions and preferences)—raising classic psychometric threats to reliability and validity. We sought to address the limitations of subjective scoring by capitalizing on recent developments in automated scoring of verbal creativity via semantic distance, a computational method that uses natural language processing to quantify the semantic relatedness of texts. In five studies, we compare the top performing semantic models (e.g., GloVe, continuous bag of words) previously shown to have the highest correspondence to human relatedness judgements. We assessed these semantic models in relation to human creativity ratings from a canonical verbal creativity task (AUT; Studies 1–3) and novelty/creativity ratings from two word association tasks (Studies 4–5). We find that a latent semantic distance factor—comprised of the common variance from five semantic models—reliably and strongly predicts human creativity and novelty ratings across a range of creativity tasks. We also replicate an established experimental effect in the creativity literature (i.e., the serial order effect) and show that semantic distance correlates with other creativity measures, demonstrating convergent validity. We provide an open platform to efficiently compute semantic distance, including tutorials and documentation ( https://osf.io/gz4fc/ ).

Disentangling Quantity and Quality in the Assessment of Creative Productions

Success in creative careers depends little on product quality

The Importance of Separating Appropriateness into Impact and Feasibility for the Psychology of Creativity

Dealing with the Criterion Problem by Measuring the Quality and Quantity of Creative Activity and Accomplishment

Aiming at Creativity and Ending up with a Range from Low‐Hanging Fruits to Foolishness: A Reflective Model of Creativity

A quantitative analysis of measures of quality in science

Automating creativity assessment with SemDis: An open platform for computing semantic distance

On the relationship between creative potential and creative achievement: Challenges and future directions

Quantifying Novelty and Influence, and the Patterns of Paradigm Shifts

The Patient is Thriving! Current Issues, Recent Advances, and Future Directions in Creativity Assessment

Is intelligence necessary and sufficient for creativity? An analysis of convergent and divergent thinking

Creativity assessment in neuroscience research.

The Dynamics of Creative Ideation: Introducing a New Assessment Paradigm

Measuring consumer perception of product creativity: Impact on satisfaction and purchasability

Towards Metrics for Evaluating Creativity in Visualisation Design

Insights on Metrics' Correlation of Creativity Assessment for Museum Cultural and Creative Product Design.

No strong support for a Dunning–Kruger effect in creativity: analyses of self-assessment in absolute and relative terms

A Fan Effect for Creative Thought: Semantic Richness Facilitates Idea Quantity but Constrains Idea Quality

Characteristics of creative individuals: An umbrella review protocol

Two limitations of our knowledge of quality

Quality-Quantity Tradeoffs in Consumption