Centre for Research in Mathematics Education (CRME) Update

By Centre Convenor, Professor Geoff Wake

Our research team in the Centre for Research in Mathematics Education (CRME) has been grappling with the research that seeks to better understand teaching and learning of GCSE resit students during the current Covid-19 pandemic. During the pandemic this presents major challenges as schools and colleges are working in a time of much turbulence and change. However, our research methodology, that of working through a randomised controlled trial (RCT), is perhaps better understood now by our colleagues in colleges with whom we are working and probably the population at large. It seems that talk of RCTs, to a greater or lesser extent, has entered mainstream conversation recently as we try to make sense of the level of protection different vaccines for Covid-19 afford. Term such as “efficacy”, “control groups” and “sample sizes” have, it seems, all entered mainstream vocabulary. In this blog I provide some insight into how we are researching using RCT methodologies in educational research and how these have parallels with those used in the trials of drugs and vaccines designed to provide improved medical outcomes.

In education research RCTs have become mainstream, in England particularly through the work of the Education Endowment Foundation. The Foundation, established in 2011 by The Sutton Trust, with a £125m founding grant from the Department for Education, has funded over 150 RCTs involving more than half of schools in England which in turn have involved over one million young people. Our CRME research team has been involved in a number of these trials both as designers of interventions that aim to improve teaching and learning of mathematics and as evaluators of others.

Comparison of this research in education with the now well-known approach to trials of vaccines provides some interesting insights into the complexities and challenges of trying to find out what works in educational settings. Let me illustrate this with reference to our latest research that aims to investigate teaching for mastery in Further Education.

First a little background detail of what exactly we mean by the term teaching for mastery and the context of the research.

The term mastery has become something of a buzzword in maths teaching in England in recent years. This stems from recent moves to learn from high performing jurisdictions, such as Shanghai (China) and Singapore, where mastery approaches have been adopted and which government ministers have been keen that we adopt and adapt here in England in a bid to improve students’ learning. You can learn about details of much of this work at the National Centre for Excellence in Mathematics Teaching’s website.

Our own research is part of the DfE funded Centres for Excellence in Mathematics (CfEM) programme’s activity that aims to support teaching and learning maths in the FE College sector. As part of the Education and Training Foundation’s partnership, our research team at Nottingham is investigating an approach to teaching for mastery. This is informed by five key principles that are set out in a handbook that informs our work with teachers across the college sector in England. These were developed from our research briefing that considered what we might learn from research into mastery approaches together with consultation with teachers about what they considered would work with these particular students. Our research-informed approach has drawn on initial ideas about mastery from the educational psychologist Bloom (1971) through to recent evaluation of the approaches and their adaptation in primary and secondary schools by Boylan and colleagues (2019).

A massive challenge for all concerned is that the students we are working with have already had 11 years, or more, of learning maths only to find that they were unable to achieve a high grade at GCSE. How then, in a year, can we support them to make sufficient progress to master the mathematics that they have previously found difficult? There lies the challenge: one that teachers of maths in colleges have grappled with for many years (see, for example Higton et al (2017).

Now to return to the RCT design of our teaching for mastery intervention and how this contrasts with a drug/vaccine trial. Fundamental to the design is that we aim to improve the learning of students and this is to be done by teachers and their use of a new approach in their teaching. This approach is ‘administered’ to the students by teachers who are working together to learn and understand the new teaching methods based on the five key principles. Hence, the newly developed teaching for mastery approach is analogous to the vaccine.

What about the ‘control group’? We contrast the students in the intervention group, all those students taught by the ‘teaching for mastery’ teachers, with a group of students who are taught by teachers going about their ‘business as usual’. Of course, this contrasts somewhat starkly with the medical trials where everyone involved gets a vaccination of one type or another, either the newly designed vaccine or a placebo. In education RCTs everyone knows from the outset the group that they are randomly allocated to. There is no placebo.

It is important to acknowledge that instead of each participant in the intervention group receiving a carefully controlled and uniformly manufactured vaccine the experiences of the students whose outcomes will be compared all have very different teaching/learning experiences. Although measures are put in place to support teachers with understanding and working with the new approaches, in this particular case, by providing seven exemplar ‘teaching for mastery’ lessons and a professional development programme, the complexity of classrooms, teachers and learners all lead to very different intervention experiences. So much so, that when considering the outcomes of an RCT in education it is always important to read the evaluation report with great care, paying particular attention to how the classroom-based intervention was implemented. For example, how many of the professional development sessions did the teacher attend? How many lessons did students attend? How strictly did the teachers stick to the intended teaching? And so on. In this way we consider dosage and fidelity to intentions.

A challenge for us as researchers is to ensure that sufficient teachers and students are involved in the trial so that outcomes of the research are statistically significant. In medical trials, such as in the case of the covid-19 vaccine, we are often looking for clear binary indicators such as illness versus no-illness. In our education trials we are looking for improvements in learning: in the particular case here, we are looking to see if students in the intervention group score more highly than those in the ‘business as usual’ group when they resit GCSE exams. To be able to detect this with some certainty we need to ensure that the trials involve a sufficient number of teachers and their students with both being spread across a sufficiently large number of colleges.

Recruiting teachers and students to the trials is the first challenge for the research team, and of course everyone joining in the trials wants to be in the group receiving the intervention given that it is likely to work to at least some extent. This is probable given that it is informed by what is known from research and evidence that is available at the time. And of course, it soon becomes clear to the teachers who is to get the treatment and who isn’t as soon as we allocate those recruited to the trials to their different groups.

Following the excitement and activity, and indeed worries, of the trial with teachers and students working through the GCSE resit year we arrive at a period of intense activity for the research team as we first of all collect the data from surveys and ultimately GCSE scores which can be matched to prior attainment from national databases. When we eventually work with the data it is completely anonymised and blinded so that our statistician doesn’t know which group of students is which until the final stage of unblinding when all becomes clear. Here there are clear parallels with the vaccine trials process.

The potential last step is taken if the trial proves to be successful – roll out to the wider population. This is where we are right now, at the time of writing, with Covid-19 vaccines: we are in the middle of what is in effect a massive effectiveness trial. What happens when we vaccinate the entire population? Of course, we won’t know the answer to this for some time: but initial data of vaccine outcomes, suggest that things are looking very positive. Let’s assume that our ‘teaching for mastery’ efficacy trial leads to positive results. What then? Maybe we suggest that the approach be adopted widely. What will then be the outcome? Will there be fidelity to the carefully controlled model used in disseminating the approaches of the trial? Will it be effective?

We hope that we will be in a position to ask such questions further down the line. At this point, however, we are recruiting to the trial. If you are in a position to take part – and we would very much like to have you onboard - or would just like to find out more you can do so  on the Teaching for Mastery in FE research project website.

Reference

• Boylan et. al. (2019). Longitudinal evaluation of the Mathematics Teacher Exchange: China-England – Final Report. London: DfE
• Bloom B. (1971). Mastery learning: Theory and Practice. New York: Holt, Rinehart & Winston
• Higton et. al. (2017). Effective practice in the delivery and teaching of English and Mathematics to 16-18 year olds London: DfE