QCA website

WCT website

Evaluation Evidence:
Peer Review:
Qualifications and Curriculum Authority

As part of the independent external evidence on the quality of MARS work, here are reviews written for the Qualifications and Curriculum Authority (QCA).

Background

MARS work for the Qualifications and Curriculum Authority includes World Class Tests, particularly in 'Problem Solving in Mathematics, Science and Technology'. (Sept 1999-2000 about £200K.)

Following a high-level UK government decision, as part of its 'Excellence in Cities' program, to develop tests for high-achieving 9- and 13-year olds in 'Mathematics' and in 'Problem Solving in Mathematics, Science and Technology', MARS was selected for the latter (and to contribute in 'Mathematics'). Though many of the practicalities of implementation have yet to be worked out, the wish for a large computer-based element remains central.

1999 - 2001 Assessment Development: In this interesting project, MARS is committed to design by early 2001, about 5 hours of paper-based assessment for each age group, together with a similar amount of computer-based assessment. Progress has been swifter with the former than the latter, where problems not yet solved anywhere are being explored. Small-scale classroom studies and two large-scale trials (about 200 students per task), have provided valuable information on the response of high-achieving students to challenging tasks.

QCA Contacts

Martin Ripley RipleyM@qca.org.uk, Jeremy Tafler TaflerJ@qca.org.uk

QCA PEER REVIEW OF MARS

Background

MARS are currently working with QCA to develop World Class Tests in mathematics and problem solving for 9 and 13 year old children. This project is funded by the DFEE and is a part of the UK government's strategy to improve the performance of the country's most able children. The project commenced in 1999 and is due to deliver the first live test administration in hools in England in November 2001.

QCA Proposal Review Process & MARS

In 1999 QCA evaluated a number of options for generating World Class Tests. The option selected was to advertise for external agencies to work with us in three areas: mathematics test development; problem solving test development and computer programming of interactive, on-screen test questions.

We then managed the following steps:

1. Adverts placed in national press
2. Invitation to Tender provided to callers
3. Information Meeting held
4. Shortlisting meeting held
5. Interviews held
6. Interview Panel recommendation made to Chief Executive
7. Project Approval obtained from CE, Chairman and Head of Finance
8. Successfully tenderers contacted.

Although QCA rarely provides tenderers with indications of budget, in the instance of World Class Tests we decided that it would be justified to provide an indication of the budget available. This was provided at the Information Meeting.

QCA's Invitation to Tender specified outcomes (ie tests) as the main requirement, and it invited prospective partners to propose creative methods and ideas covering test design, test content and test development procedures.

QCA made clear that it was offering one-year exploratory contracts to be followed by longer-term test development contracts (these latter contracts are currently being tendered).

The shortlisting process at QCA involves a panel of raters independently scoring each tender submission against standard criteria headings. These include: experience, educational and testing background, project management proposals, quality of the proposed methodology and ideas, value for money. Having scored independently, panelists meet to exchange views, agree common scores and derive a shortlist. Tenderers are usually scored again after interview, using the same criteria.

In 1999 20 organisations expressed interest in the QCA World Class Tests tender exercise. As a result of the tender and selection exercise three separate teams were appointed:

In the scoring and interview exercises, the MARS proposal scored very highly in terms of the quality of the proposal, the ideas (creative and intellectual) for the approach to World Class Tests, as well as in terms of relevant past experience. The proposal did not score quite so highly on project management arrangements or on test development methodology (in particular in terms of trialling and protesting arrangements). However, even in these latter areas, the proposal was deemed to meet (or better) all of QCA's minimum requirements.

Working with MARS

The World Class Tests project is all about the development of new forms of testing using modes of testing that are highly innovative and in areas of the curriculum (problem solving and giftedness) which have not previously been addressed by a national body like QCA.

In working with MARS we have found (positively):

• supreme commitment to the educational quality of the endeavour;
• excellent creativity in the design of test questions;
• contributions to all aspects of the project - computer screen design, data capture, marking algorithms - not just problem solving;
• a strong team consisting of diverse but complementary strengths.

We are quite clear that the World Class Tests currently reflect the unique contributions of the MARS team not just in the quality of the items but also in many subtle and underpinning features.

In working on the project with MARS we have, with them, learned a considerable amount about:

• managing a complex project with (in this case) three agencies;
• organising the project and in particular, planning effective trialling and protesting, particularly in the context of using an unfamiliar medium;
• obtaining a sufficiently wide range of diverse test questions;
• predicting work effort necessary to meet volume deadlines.

Signed: QCA
Date 11.1.2001

	Copyright 2001 MARS Page updated 22 June 2001