Use of online examination - Mark Bowen and Kay Millward

The School of Veterinary Medicine and Science had a clean slate from which to formulate an assessment strategy. While our external examiners liked and wanted essay and short answer based questions, based on their background and previous experiences, we argued that with objective testing we could assess a much larger range of the curriculum, be objective, assess deeper learning and be innovative in our questioning style. The School was given access to a database of over 3000 'single-best answer' style questions that had been compiled by four existing English vet schools with funding The Higher Education Funding Council (The OCTAVE project). This gave us a head start in creating a bank of questions for assessing the course; however concerns remained about the ability to assess deeper learning and about ensuring the quality of the questions.

We considered various options for assessment delivery, we felt that WebCT was too restrictive and when the learning team adopted QuestionMark Perception (QMP) we decided to at least try this system. In terms of authoring of questions, it can be very simple or if you need very complicated. Questions can even be written in a word template and imported into the database by a local administrator. Management of the question database remains more troublesome and it can be difficult to find questions that you have previously written or to know how often a particular question is used.

The problem remains with assessing understanding and deeper learning using objective testing. We have tried to do this using a series of different question styles. Assertion reason questions, extended matched questions and graphical (hotspot questions). We believe these have worked well in assessing what we want and despite initial concerns received very positive feedback from our external examiner. However, we have pushed the system beyond what it is designed to do, and generating useful question performance reports, for QA purposes, from our different question styles has been difficult.

The biggest limitation with QMP for us was the inability to deliver graphical (hotspot) based questions where the region of interest was anything other than a rectangle. This is not ideal when using this question style to assess anatomical knowledge; the body rarely comes in even boxes! The learning team offered to create Flash objects to overcome this problem, however with that we would not obtain the information we wanted for QA; where had the student marked the incorrect responses and without us all learning how to use Flash it would mean the writer and the programmer were on different sites - a recipe for disaster when you have no QA procedure. We have overcome this with a custom solution using the existing but unused power of QMP that can generate questions with up to 5 polygonal regions and generate useful reports.

Our early experiences with QMP in formative assessments were less than ideal, with many teething problems. However, since the server was upgraded in the Spring we have had virtually no problems and in the summer we ran 1200 hours of summative assessments in 3 days, with only one minor problem.

Space was a particular concern, running 100 assessments concurrently meant using every available space in the computer rooms of Sutton Bonington. We were unable to run the assessments in batches because of the need to complete the exams over a short period of time and did not want to transport students to University Park for their exams. Therefore we had plastic screens made that essentially turned each computer desk into a small booth to prevent straying eyes. As eAssessment becomes more popular space is going to become a challenge for the whole university.

Using objective testing has certainly not saved us any time, writing, checking and then standard setting the questions created many hours of work, and will continue to do so until our question database is large enough. However at least summer exam time causes academics virtually no effort and thanks to all the hard work of our school exams team (who had to work very hard) we hardly noticed exam time at all!