RLO Transcript: RLO title
This is a transcript of the Reusable Learning Object (RLO) entitled "Presenting and interpreting meta-analyses", online at:

http://www.nottingham.ac.uk/nursing/sonet/rlos/ebp/meta-analysis2/ 
Introduction
What is a Meta-Analysis?
In research, studies are conducted to try to answer questions about the clinical effectiveness of a treatment or intervention. These studies will have asked whether, for example, does treatment A provide significant benefits over treatment B in a particular group of people? We can use a meta-analysis to combine the results together to get on overall impression about what the results from all of the studies are saying and to get a single pooled result of how much the treatment is beneficial. Meta-analysis does not simply involve averaging the results of the individual studies, but requires a statistical method which combines the results whilst taking into account the sizes of the studies.
Effect Measures
What information do I need to perform a meta-analysis?
A meta-analysis uses the quantitative results from the individual studies, pools them together using a statistical method, and produces a new single quantitative result. This new quantitative result is a known as the ‘pooled result’ and is presented as a weighted average treatment result and a measure of variation.  

The quantitative results from the individual studies and the pooled weighted average treatment result calculated from a meta-analysis are commonly called ‘effect measures’. There is a range of effect measures used within research, and the type of effect measure one uses depends of the type of outcome measure of interest. For example, if our outcome of interest was death, there are only two options available, where the participant is either dead or alive. This outcome measure is known as a binary outcome. For an outcome measure like this, the commonly used effect measure would be the Odds Ratio. The Odds Ratio let us describe the likely harm that a treatment or intervention might cause.

The most commonly used effect measure in meta-analysis is the Odds Ratio. Other common effect measures used within research, and in particular in meta-analyses, are the Risk Ratio, Risk Difference, Mean Difference and Standardised Mean Difference. The Risk Ratio and Risk Difference are used for binary outcomes where there are only two options available. Where as the Mean Difference and Standardised Mean Difference are used for outcomes which are continuous. Continuous outcomes would include measures such as blood pressure or weight, where there are a defined range of possible values.

In addition to the meta-analysis producing a pooled treatment result, a measure of variation is also produced. This measure of variation lets us describe how precise the pooled treatment result is. Common measures of variation include the standard deviation, variance and standard error. These measures of variation are commonly used to calculate 95% confidence intervals for the pooled treatment result.   

Forest Plots
How do I present the results of a meta-analysis?
After performing the meta-analysis, the results could be presented as text in the review. However, more commonly the results are displayed using a figure, called a ‘forest plot’. 

Here is an example of a forest plot from a systematic review of randomised controlled trials which assessed the effect of giving an inexpensive course of corticosteroids to women who were about to give birth prematurely. Each of the circles with horizontal lines represents an individual study. Therefore there are seven studies in this review. The diamond at the bottom of the plot represents the pooled quantitative result from the meta-analysis. The pooled result from this meta-analysis indicates strongly that giving pregnant women who are about to give birth early a simple inexpensive course of corticosteroids significantly reduces the risk of babies dying from complications of immaturity. 

At the top of the plot is the title of the review, and a label to tell you what the treatment comparison and the outcome of interest are. For this review, the treatment comparison is corticosteroids versus placebo, and the outcome of interest is early neonatal deaths. At the bottom of the plot there is a line which tells you the scale for the treatment effect you are measuring. In this example the outcome is the proportion of babies that died, and the effect measure is the Odds Ratio. Towards the left of the scale, the scale is less than 1 which means that the treatment has made the risk of neonatal death less likely. Towards the right of the scale, the scale is more than 1 which means that the treatment has made the risk of neonatal death more likely. The vertical line in the middle of the plot is where the treatment and the placebo have had the same effect on the risk of neonatal death. In other words, there is no difference in risk between the two treatment groups, so that the risk of neonatal death in the corticosteroid group is the same as the risk in the placebo group.

Here are the results of the first randomised controlled trial conducted in 1972. This is the label for the trial; usually given as the first author’s name, or the acronym of the study. The quantitative results from this study are presented for each treatment group. The lower case n refers to the number of babies that died in the corticosteroid group, and the uppercase N refers to the total number of babies that were recruited into the corticosteroid group. For the Liggins and Howie study, there were 36 deaths in the corticosteroid group and a total of 352 babies in the corticosteroid group. The data for the placebo group is then entered using the same format. The weight is the percentage weight given to this study in the pooled meta-analysis. The general rule is that the bigger the study in terms of sample size, the larger the percentage of weight given to that study. The figure in the middle of the plot shows pictorially the results for the individual studies, here the treatment effect for the individual study is represented by a square. The size of the square is proportional to the percentage weight of the study in the meta-analysis. So the larger the square the more weight the study has in the meta-analysis. The horizontal line represents the certainty of the treatment effect, where the shorter the line the more certain we are about the treatment effect. The horizontal line is called the confidence interval. Usually 95% confidence intervals are presented on forest plots. If the horizontal line touches or crosses the vertical line, it means that the trial found no significant difference between the two treatments. The position the circle to the left of the vertical line shows that this trial found the corticosteroid treatment was beneficial as compared to placebo.

Looking at the plot as a whole, at the bottom of the forest plot, a diamond is shown. The middle of the diamond is the pooled meta-analysis result for the treatment effect, and the width of the diamond is the certainty of the result, again usually presented as a 95% confidence interval. If the confidence interval crosses the vertical line, then there is no significant difference in the effectiveness of the corticosteroid treatment group and the placebo group. However, in this example, we can see that the effect is significant and a course of corticosteroids is beneficial in reducing the risk of neonatal deaths, since the horizontal line does not cross the vertical line, and the diamond is to the left of the plot

Heterogeneity
What is heterogeneity?
Looking at the forest plot, one can see that the results for the individual studies are all different. This difference is called ‘heterogeneity’. Heterogeneity may be due to differences in the study protocols, such as differences in doses of treatments, or study populations. However, heterogeneity may exist purely by chance too. Heterogeneity is an important factor to consider in all meta-analyses, since too much heterogeneity  can invalidate the pooled results.  

Heterogeneity is usually assessed either visually using a forest plot or more formally using statistical tests, such as the Cochran’s Q test. However, more recently a statistical test called the 'I2 test' has been used. The I2 test gives a measure of how much variation between the studies is due to heterogeneity. A numerical value is presented for I2 which can range from 0%, indicating no heterogeneity, up to 100% indicating that all of the variation between the studies is due to heterogeneity. If an I2 value of 85% or more is seen, then the meta-analysis should not be performed since the studies are too different to combine together. In our example, the I2 value indicates that 45% of the variation is due to heterogeneity. 

Effect Methods
Choosing between Fixed and Random effects methods
There are two main types of methods that can be used when performing a meta-analysis. These are called the ‘fixed effect method’ and the ‘random effect method’. The choice over which method is more appropriate depends primarily on the level of heterogeneity between the individual studies. If low levels of heterogeneity are detected, indicated by an I2 value less than 40%, then it is appropriate to use a fixed effect method to pool the data. If there are moderate levels of heterogeneity between the study results, indicated by an I2 value between 40% and 84%, then it would be more appropriate to use a random effect method. A random effect method has the advantage that it allows for the heterogeneity between the studies to be taken into account. This is shown by the width of the confidence intervals for the pooled result getting wider than would be seen if a fixed effect method was considered. However, if an I2 value of 85% or more is seen in a meta-analysis, this would indicate that high levels of heterogeneity are present in the meta-analysis. In this situation even the random effect method can not account for the heterogeneity and so the meta-analysis should be abandoned since the individual results are too different to combine together. In our example, the I2 value was 45%. Therefore it would be more appropriate to use a random effect method. If we change our method from a fixed effect to a random effect we can see that 95% confidence intervals have got slightly wider. 

Resources

Meta-Analysis (RLO)	This RLO provides an introduction to the basic concepts of meta-analysis, which is an important and valuable tool for summarising data from multiple studies.	
Steps in Conducting a Systematic Review (RLO)	This RLO outlines the five fundamental steps to conducting a systematic review of health care research so as identify, select and critically appraise relevant research.	
Egger M, Davey Smith G, Phillips AN. Meta-analysis: Principles and Procedures. BMJ 1997;315:1533-1537	This publication is one in a series of seven articles examining the procedures in conducting reliable meta-analysis in medical research	
Cochrane Collaboration	The Cochrane Library contains high-quality, independent evidence to inform healthcare decision-making. It includes reliable evidence from Cochrane and other systematic reviews.	
Centre for Reviews and Dissemination	The CRD undertakes reviews of research about the effects of interventions used In health and social care.	
NHS Health Technology Assessment Programme	The HTA programme works to provide all those who make decisions in the NHS with high-quality information on the costs, effectiveness and broader impact of health care treatments and tests.	

Glossary
•	Risk ratio - the risk ratio determines the strength of the association between the intervention group (or risk factor) and a binary outcome. It can be estimated by dividing the risk of the outcome in the intervention group (or exposed group) by the risk of the outcome in the control group (or unexposed group). The risk of the outcome is the proportion or percentage of people with the outcome.
•	Risk difference - the risk difference is the estimated by subtracting the risks of the outcome in the two groups, i.e. risk of outcome in the intervention group (or exposed group) minus risk of outcome in the control group (or unexposed group).
•	Mean difference - the mean difference is used for continuous outcomes, and is estimated as the difference in the means of the outcome between the two groups i.e. the mean of the outcome in the intervention group (or exposed group) minus the mean of the outcome in the control group (or unexposed group).
•	Standardised mean difference - this is similar to the mean difference but used when the units of the continuous outcome are not the same across the studies. This is estimated as: [mean of the outcome in the intervention group (or exposed group) minus the mean of the outcome in the control group (or unexposed group)] all divided by the pooled standard deviation for the two groups.
Diagrams
The diagrams below, referred to in the transcript, illustrate the example of the effect of corticosteroid before preterm delivery on neonatal death. 

Fixed effect model:

Random effect model:

Activity screens

With one study

With two studies

With three studies

With four studies

With all 5 studies fixed effect method

With all 5 studies random effect method

Document updated: 28/06/2007