Evaluation Evidence:
Double-blind Impact Studies

As part of the independent external evidence on the quality of MARS work, .

IR and MARS have designed a rigorous, indeed austere, approach to the evaluation of system impact, loosely based on a mirror image of "The Turing Test". Its essential principle is that neither the observer-interviewers nor those they meet know that a project is being evaluated - MARS is not identified. Rather they focus on discovering from those involved the perceived challenges, initiatives, improvements and problems within the broad field of mathematics education. Those interviewed are representative of the various groups of key players in the system.

The core question is:

Are MARS contributions spontaneously remarked on, and, if so, in what ways?

One pilot example is reported on here, comparing the comments from 3 school systems within Northern California's Silicon Valley Mathematics Assessment Collaborative (MAC) with those from 'control' districts which had no links with MARS. This work is ongoing, so further reports will follow.

IR EVALUATION OF THE MARS PROJECT
DISTRICT COMPARISON-TURING TEST REPORT

Brief Summary December 2000

In the fall of 2000, Inverness Research - a small educational consulting firm in Northern California with an established reputation in evaluating math and science reform initiatives- conducted an evaluation of the MARS project. One portion of their work used a creative methodology that stems from the thinking of the British mathematician, Alan Turing and the acclaimed "Turing Test" which he devised to determine how well computers model human intelligence. Similar to that test, the approach Inverness Research used in evaluating MARS centers on the notion of "distinguishability." They posed the following research question: to what extent are the districts involved with MARS distinguishable from other, otherwise equivalent, districts that do not have a relationship with MARS? To answer the question in a rigorous manner, they set up a "double-blind" study in which both researchers and subjects (in this case school districts) were not privy to the specific purposes of the research. Instead, two skilled researchers were given the assignment of documenting the capacity of each district to initiate and sustain a process of district-wide K-8 mathematics education reform.

Comparison of the researcher's ratings show a clear pattern of increased capacities in the two MARS districts studied. Particularly in the area of assessment the MARS districts were clearly distinguishable from the non-MARS district. Other evidence from interviews clearly shows that MARS is very visibly providing districts with a service that has contributed to the progress of reform in each district and is an important part of the district's overall ability to continue the improvement of mathematics teaching throughout the district.

Inverness Research Associates have prepared a number of additional reports on this research. There is an executive summary [missing] and a full report [missing] which may be downloaded as an Adobe Acrobat PDF files

They also conducted a cross-site comparison of three districts. To read this download Adobe Acrobat PDF files: report,[missing] district A, district B, district C, form showing protocol.