Research Behind   Chips and Code

Comparative study results showing usability improvements from moving an existing desktop learning tool to the web.

Chips and CodeRepositoryThesis
Participants

Participants

In total, 24 participants, divided equally into Group A and Group B, participated in a comparative study.

Both groups were shown the same short introduction to Boolean gates. Group A used a version with the embedded Hardware IDE. Group B used a version with the existing desktop learning tool. Below is the summary of participant profiles showing differences in the make-up of each group.

Efficiency

Participants were observed to determine the time spent on tasks and the number of times they got confused.

The collected data included the time it took to get productive with the learning tool, the time it took to perform individual tasks, and the number of times the participant got confused and either asked for help or was offered help. Presented is the mean of the collected data points.

Questionnaire

At the end of the session, participants judged the subjective usability using Standard Usability Questionnaire (SUS).

SUS scores showed a difference of 7 points in favour of Chips and Code. However, both tools scored very highly, and due to the small difference and sample size, the probability Chips and Code can consistently score better is only 77.9% (below the industry standard of 90-95%).

Assignment Bias

Considering group differences, correlations were explored to hint at possible assignment bias.

While there was an effort to distribute participants equally between the two groups, some differences still remained. Correlations between profile attributes that differed the most can be observed visually on the following plots. The relatively strong correlation (

r(11)
ranging from 
-.27
to 
-.69
, 
p < 0.1
) in the case of education and occupation suggests the shown performance data could be skewed in favor of Group B - the existing tool.

About

Chips and Code – reimagined Nand2Tetris tools powered by the modern web.

Nand2Tetris is a very popular course that introduces a wide range of computer principles by building a virtual computer from the group up. However, it is accompanied by an older Java desktop application inaccessible from a large range of devices (like tablets, Chromebooks, smartphones... - one-fourth of people from developing countries rely on smartphones). Chips and Code aims to reimplement the same functionality using the modern web with an improved user experience.

This webpage captures just a small part of the full research conducted as a master's thesis at the University of Galway. For context and more information, feel free to check the full thesis document or take a peek at the code.

Open ThesisCheck code on Github