Wednesday, April 7, 2010

User testing with students

After musing a lot on interface design and evaluation, I've finally ended up with a thesis subject. Though slightly complex, it goes something along the lines of the following:

A lot of research takes place in Human Media Interaction, the Computer Science department and the university as a whole. But a lot of this research (at least as far as psychology and CS are considered) is done on students.

Why? Because students are a handy group to use. They're generally found near universities, have a large amount of spare time, are interested in research and are willing to take part in experiments.
In fact, they're such a handy group of students that about 50% of HMI research is done with students exclusively. And psychology has skirted up to the insane percentage of 90%.
This is -in my opinion- bad research practice. Students are a very specific subset of the population and in my research I will try to prove that there is a distinct difference in using students and non-students in HCI experiments.

Students groups differ from a theoretical "average population sample" (if there ever was such a thing) on the following features:

Age
Students are a very specific group of people age-wise, with ages ranging from 17-19 (enrollment age in the Netherlands) when starting their studies, with most finishing their bachelor and master studies at age 22 and 25 respectively. The majority of students in the Netherlands is both older than 17 and younger than 27.



Education
A no-brainer, bachelor and master students at universities are receiving the highest level of education available, discounting the scant few post-doc students that are around.

Sex
Though students in general are pretty much evenly distributed in male-female ratio, there are some interesting outliers, especially at the more technical studies in the Netherlands, which are unpopular with women for some reason. A high prevalence of men (Or a shortage of women, as they themselves call it) as CS students means that most research involving CS students will mainly (or only!) have men as test subjects.

Technological Aptitude
Technological aptitude is a measure of the ease and comfort the subjects of research feel when presented with a technological artefact, problem or interface. Most CS students are very adept and experienced at using computers. Moreso than other students and very much more so than the 'average population'.


Conclusion
I believe that CS students are a bad group to use for experimentation if this is the only group used. Good research requires good data and researchers doing user testing would do well to do so over a group varying over the 4 features named above.