TouchStone Evaluation

[ Introduction | Results | Appendix A | Appendix B | Appendix C | Appendix D ]

This report summarises the data obtained from two cohorts (2006/07 entry) of GEM students after the completion of a 1 hour online exam (14/03/2007) using TouchStone. Out of the total group of 90 students 79 questionnaires (paper-based) were returned (87.7%). The specification of the client machines is listed in Appendix A, specification of the server in Appendix B, specification of the exam paper in Appendix C and timings from the first cohort reported in Appendix D.

1.  Did you have any difficulties understanding how to use the software?

	Respondents	%
No	77	97.5%
A Few	1	1.3%
Many	0	0%

Note: One respondent did not complete this particular question so the percentages do not total 100.

2.  Did you encounter any software crashes or error messages?

	Respondents	%
No	67	84.8%
A Few	11	13.9%
Many	1	1.3%

3.  What do you think to the speed of the system?

	Respondents	%
Fast	35	44.3%
Adequate	44	55.7%
Slow	0	0%

4.  How would you rate the user interface?

	Respondents	%
Good	48	60.8%
Adequate	29	36.7%
Poor	0	0%

Note: Two respondents did not complete this particular question so the percentages do not total 100.

5.  Which medium do you prefer for assessments?

	Respondents	%
TouchStone	42	53.2%
Don't Mind	30	38.0%
Paper	7	8.9%

Comments in favour of online:

• Easy to use (x8)
• Quick results (x4)
• Avoids repetitive strain injury from writing too much. (x4)
• Easily change answers (x3)
• “Cleaner” (x2)
• “Access - familiarity - clarity”
• “more eco-friendly”
• “I like the ability to 'grey out' answers you know to be incorrect.”

Comments in favour of paper:

• “Computers are uncomfortable to use and looking at the screen for a long time makes me dizzy”
• “Easier to check back, to have an overview of how you've done so far and to work things out (by writing them down)”
• “I prefer to be able to write on a paper so I can come back to questions more easily”
• “Always harder to navigate exam on screen, harder to read screen than paper.”

6.  Any other comments:

• “System itself is fine, but spelling mistakes, repeat questions seem to be common.” (x2)
• “Problems with scroll button on mouse (dangerous)”
• Make it clearer to users that clicking on MCQ stems greys them out.
• “Why is the password different from the Novell login?”
• “Not sure what bottom fire escape links are for on each page.”
• “Computer screens should be checked before defects before the exam. Mine was defective causing me to have to move.”

Pentium 4 PCs (1.9GHz), 512Mb RAM
Windows 98
Refresh rate: 85Hz
Font Size: 100%
Font Smoothing: On
Restricted access: On (no website other than TouchStone, no applications apart from IE, no networked or removable drives)

Hardware
Sun Fire V40z
4x 2.2GHz AMD Opteron CPUs
8Gb RAM
Software
TouchStone 2.7
Debian Linux
Apache 2.0.48
MySQL 4.1.15
PHP 5.1.2

1 Hour duration
51 questions
10 screens
Bidirectional navigation

© 2006 The University of Nottingham All rights reserved