CRC-G Evaluation Framework
A taxonomy-driven method for evaluating card-based ideation toolkits
Overview
This Master’s thesis explored how Card-based Ideation Toolkits—commonly used in design and creativity processes—can be systematically evaluated and compared.
The project culminated in the development of a Taxonomy-Driven Evaluation Framework that connects toolkit features with four key design dimensions: Creativity, Reflection, Collaboration, and Guidance (CRC-G).
The following sections only breifly outline the research. To read and understand the research in more detail, you can download the paper via the button below.
UX Researcher & Designer
Role
Individual with supervisory input from academic advisors
Team
8 Months
Timeline
Master’s thesis, Design research project
Context
Figma, Excel, Python (analysis), Miro, Google Forms and Optimal Sort
Tools
I. Background & Motivation
While card toolkits are widely used in design practice to spark creativity and structure ideation, existing evaluations often focus narrowly on creativity outcomes. Other essential aspects—like collaboration, reflection, and guidance—remain underexplored. The project aimed to address this gap by creating a comprehensive framework that captures how these dimensions interact, helping users select or design more balanced and effective toolkits.
II. Research Methodology
The research utilised a Cyclical Design Science Approach and divided the research into 3 main phases,
-
I reviewed and structured 23 ideation based card toolkits and through thematic clustering — grouping recurring design patterns across all 15 toolkits — I identified 12 dimensions that card toolkits use to support CRC-G.
-
I then ran a mixed-method study: a 15-participant Survey to prioritise the 12 dimensions and a Card-Sorting using a >60% consensus threshold to sort the Subcategories into Must-have, Nice-to-have, or Not-needed using Figma and Optimal Sort. Quantitative analysis (Statistical Analysis & Friedman Test) revealed that Collaboration Support and Cognitive Load Management Dimensions were the most valued, while qualitative feedback emphasized the need for flexibility and structured guidance.
-
And at last, built the Evaluation Framework based on the results from the Survey and Card Sorting and conducted a small evaluation study (n=3) where practitioners applied the Evaluation Framework which I then assesed through System Usability Scale (SUS) and Semi-Structured interviews.
Taxonomy of different Card Toolkits
The 12 Key Dimensions found out from the Taxonomy.
Subcategories within each Dimension which are the different types of format within each Dimension.
III. The Evaluation Framework
The findings were translated into a rubric-based evaluation framework covering 5 dimensions: Collaboration Support, Guidance Level, Toolkit Structure, Customization Level, and Interaction Style.
How it works
Each dimension contains subcategory indicators. Evaluators score each indicator on a 0–2 scale:
-
0 = Absent or unusable
-
1 = Partially present / limited
-
2 = Strongly present and robust
Each indicator carries a weight based on the card sorting consensus:
-
Must-have = 1.0
-
Nice-to-have = 0.66
-
Not-needed = 0.33
Weighted scores roll up to dimension scores and an overall alignment percentage, interpreted against three performance bands:
-
80–100% = Strong alignment — adopt with minimal refinement
-
50–79% = Moderate alignment — functional but with clear gaps
-
Below 50% = Weak alignment — substantial redesign needed
Indicator are the subcategories within each Dimensions. E.g Suggested Group use, Shared Asynchronous etc are different forms of Collaboration.
Example of the Evaluation Framework
The Descriptor is the explanation for what the score of 0, 1 or 2 means for each Indicator.
The score for each dimension is the product of each indicators Weightage and the Score given to it by the user.
IV. Usability Testing
Evaluation Process
Once the framework was built, it was tested through a small-scale usability evaluation (n = 3) where practitioners used it to assess real-world card toolkits. Each participant rated usability via the System Usability Scale (SUS) and participated in short interviews post study.
Results placed the framework’s usability between 75–80 (Good–Excellent), validating its structure but revealing cognitive effort during scoring.
Evaluation Insights
-
Users appreciated the clear structure and consistency of the rubric.
-
Scoring across dimensions revealed useful differentiation between toolkits, particularly in how they supported collaboration and reflection.
-
Some participants felt the weighting and scoring process was too analytical, suggesting a simpler version for quick assessments.
These findings confirmed that the CRC-G Framework worked well for structured evaluation but needed streamlining for efficiency.
V. Reflection
The CRC-G project successfully answered its research question by creating and validating a structured method to evaluate card-based ideation toolkits. It showed that creativity-oriented tools can be assessed systematically without limiting their flexibility.
Key Learnings:
-
Balancing rigour and usability is crucial for engagement.
-
Clear visual feedback and concise scoring improve consistency and clarity.
Limitations & Future Work:
A small evaluation sample limited generalisation, and some users found the process cognitively demanding. Future work should focus on simplifying the framework, developing a digital version, and expanding evaluation criteria to include engagement and accessibility.