Structuring, Aggregating, and Evaluating Crowdsourced Design Critique

Kurt Luther, Jari-lee Tolentino, Wei Wu, Amy Pavel, Maneesh Agrawala, Brian Bailey, Bjoern Hartmann, Steven Dow


Feedback is an important component of the design process, but gaining access to high-quality critique outside a classroom or firm is challenging. We present CrowdCrit, a web-based system that allows designers to receive design critiques from non-expert crowd workers. We evaluated CrowdCrit in three studies focusing on the designer’s experience and benefits of the critiques. In the first study, we compared crowd and expert critiques and found evidence that aggregated crowd critique approaches expert critique. In a second study, we found that designers who got crowd feedback perceived that it improved their design process. The third study showed that designers were enthusiastic about crowd critiques and used them to change their designs. We conclude with implications for the design of crowd feedback services.

CrowdCrit allows designers to submit preliminary designs to be critiqued by crowds and clients. The system then aggregates and visualizes the critiques for designers.

Research Paper

PDF (10.4M)

Structuring, Aggregating, and Evaluating Crowdsourced Design Critique
CSCW 2015, March 2015, To Appear.