High quality up-to-date systematic reviews are essential in order to help healthcare practitioners and researchers keep up-to-date with a large and rapidly growing body of evidence. Systematic reviews answer pre-defined research questions using explicit, reproducible methods to identify, critically appraise and combine results of primary research studies. Key stages in the production of systematic reviews include clarification of aims and methods in a protocol, finding relevant research, collecting data, assessing study quality, synthesizing evidence, and interpreting findings. Systematic reviews may address different types of questions, such as questions about effectiveness of interventions, diagnostic test accuracy, prognosis, prevalence or incidence of disease, accuracy of measurement instruments, or qualitative data. For all reviews, it is important to define criteria such as the population, intervention, comparison and outcomes, and to identify potential risks of bias. Reviews of the effect of rehabilitation interventions or reviews of data from observational studies, diagnostic test accuracy, or qualitative data may be more methodologically challenging than reviews of effectiveness of drugs for the prevention or treatment of stroke. Challenges in reviews of stroke rehabilitation can include poor definition of complex interventions, use of outcome measures that have not been validated, and poor generalizability of results. There may also be challenges with bias because the effects are dependent on the persons delivering the intervention, and because masking of participants and investigators may not be possible. There are a wide range of resources which can support the planning and completion of systematic reviews, and these should be considered when planning a systematic review relating to stroke.
Introduction
Why do a systematic review in stroke?
In order to provide patients with the best possible care and treatment, healthcare decisions should be based on up-to-date, high-quality research evidence.1,2 However, there is an unmanageably large and continually growing body of research evidence, and healthcare practitioners do not have time to keep up to date with this evidence base.3,4 There has also been an exponential increase in the amount of stroke research over the last 50 years (see Figure 1). In April 2017, the Cochrane Stroke Group trials register contained 24,084 references to 9975 randomized or controlled clinical trials relating to stroke, and a search of any key electronic bibliographic database reveals that there are tens of thousands of non-randomized studies relating to stroke. Yet, despite important advances in evidence-based stroke care,5,6 stroke survivors still do not always get the best possible care.7 High quality up-to-date systematic reviews of primary research studies, addressing questions which are of recognized importance to stroke survivors, carers, and clinicians are therefore essential.8,9 Systematic reviews are also important for the avoidance of research waste, by ensuring that new primary research is done with full knowledge of what has already been done, and that new research evidence is interpreted in the light of what is already known.10–12
Figure 1. Number of reviews and trials registered by Cochrane Stroke Group, by year.
What is a systematic review?
A systematic review aims to bring evidence together to answer a pre-defined research question. This involves the identification of all primary research relevant to the defined review question, the critical appraisal of this research, and the synthesis of the findings.13 Systematic reviews may combine data from different research studies in order to produce a new integrated result or conclusion, or they may bring together different types of evidence in order to explore or explain meaning.14
Systematic reviews can address any defined research question. Table 1 provides examples of questions that have been addressed in published reviews relating to stroke, and examples of resources relating to different types of reviews. The table illustrates that there are different types and methods of systematic review for different types of questions. This is the same as when selecting a method for primary research, where the type of research question influences selection of an appropriate method (e.g. a question about the effect of an intervention may be best answered by a randomized controlled trial, or a question about prognosis best answered by an observational cohort study). A high quality systematic review will try to identify all primary research studies that are relevant, both published and unpublished, carried out all over the world and written in different languages. The quality of the identified research will be critically appraised, and the results of studies will be systematically brought together in order to provide the best possible answer to the review question; this process may involve the statistical combination of study results (meta-analysis) or other approaches to data synthesis. In this way, a systematic review of evidence should support the delivery of optimal healthcare interventions and research.
Table 1. Types of systematic reviews, frameworks for review questions, and resources to support protocol development, quality assessment of studies and review reporting
Type of research question
Type of systematic review
Published examples from the field of stroke
Framework for systematic review questions
Resources for protocol development
Tools for quality assessment of included studies
Reporting guidelines
What is the effectiveness of an intervention (e.g. a treatment, service or policy)?
Intervention review
What is the safety and effectiveness of thrombolytic therapy for the treatment of acute ischaemia stroke?15 What is the effect of stroke unit care, as compared to alternative forms of care for people following a stroke?16 Does fitness training after stroke reduce death, dependence and disability?17
Cochrane Handbook for intervention reviews19 Joanna Briggs Institute (JBI) Reviewers’ manual20 Standards for Systematic Reviews of Comparative Effectiveness Research21 Methodological Expectations of Cochrane Intervention reviews (MECIR)22
Cochrane risk of bias tool23 JBI Critical appraisal tools24
Preferred reporting items for systematic reviews and meta-analyses (PRISMA)25 Methodological expectations of cochrane intervention reviews (MECIR)22
What is the accuracy (sensitivity or specificity) of a diagnostic test?
Diagnostic test accuracy (DTA) review
What is the accuracy of MRI for the detection of acute hemorrhagic lesions within 12 hours of stroke symptoms?26 What is the accuracy of cognitive diagnosis of multidomain, cognitive impairment/dementia in stroke survivors?27
PICOT: Population, index test, comparator, outcome, target condition PPIRT: Population, prior tests, index tests, reference standard, target condition
Quality assessment of diagnostic accuracy studies (QUADAS-2) tool29 Critical Appraisal Skills Program (CASP) diagnostic checklist30 Centre for evidence based medicine (CEBM) diagnostic study appraisal worksheet31 JBI critical appraisal tools24
PRISMA-DTA: Checklist for reporting of diagnostic test accuracy systematic reviews (in development)32
What is the prognosis / prevalence / predictors of recovery of a condition?
Observational studies review
What is the worldwide incidence of stroke?33 What is the prevalence of pre-stroke dementia and the prevalence and incidence of post-stroke dementia and their associated risk factors?34 What are the predictors of upper limb recovery following stroke?35
Meta-analysis of observational studies in epidemiology (MOOSE)37
What is the accuracy of an outcome assessment or measurement tool?
Review of measurement instruments
What is the validity and reliability of the Modified Rankin Scale?38 What are the psychometric properties of outcome measures used in stroke self-management interventions?39
De Vet 2011. Chapter 9: Systematic reviews of measurement properties40 COnsensus-based Standards for the selection of health Measurement Instruments (COSMIN)41
COSMIN checklist41: Also potentially relevant: OMERACT filter42
Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ49 Realist and meta-narrative evidence syntheses – evolving standards (RAMESES) publication standards50
Essential features of systematic reviews include explicit, reproducible methods for identification of primary research studies and critical assessment and synthesis of studies that meet the eligibility criteria.3,20,51,52 Systematic reviews should be distinguished from “non-systematic” reviews which do not have these features, and which are sometimes also described as a “conventional literature review,”53 “scoping review,”54 or “narrative review.”55 In the past, there has been considerable confusion and inconsistency in the terminology used around systematic reviews,56 in part because historically the term “systematic review” had often been associated specifically with the bringing together of data from quantitative research studies. However, it is now widely recognized that a “systematic review” refers to the process of systematically bringing together the results of any research, including qualitative or mixed methods research studies.57,58
There is growing recognition of the importance of patient and public involvement to the value and relevance of systematic reviews,59 and some key organizations now identify patient and public involvement as an essential feature of a systematic review (e.g. EPPI Centre51). Patient and public involvement within systematic reviews is increasingly asked for by funders of health research (e.g. NIHR60 and European Science Foundation,61 including stroke research (e.g. Stroke Foundation62).
Systematic reviews in stroke
Most systematic reviews in stroke are reviews of interventions for prevention, acute treatment, and rehabilitation.63 While reviews of the effectiveness of drugs to prevent or treat stroke may arguably be relatively straight-forward, reviews of complex interventions, such as rehabilitation, are more complicated, as are reviews of diagnostic test accuracy or qualitative data. Challenges in reviews of stroke rehabilitation can include poor definition, implementation, and description of complex rehabilitation interventions64–67; inconsistent use of outcome measures, or use of outcome measures that have not been validated68; or poor generalizability of results (for example, because of exclusion of participants with aphasia or cognitive impairment69). Furthermore stroke rehabilitation research has particular challenges because the effects are dependent on the person delivering the intervention, and blinding of participants and staff to randomized interventions may not be possible within some studies.
In this article, our objective is to outline the systematic review process, from the planning of the review, through the writing of the protocol and the completion, publication, and dissemination of the review (Figure 2). We focus primarily on reviews of the effect of interventions for prevention, acute treatment and rehabilitation of stroke, but we also incorporate and discuss other types of systematic reviews, such as reviews of diagnostic accuracy and reviews of qualitative data. We use an example from stroke rehabilitation70 to illustrate methodological challenges, since reviews of rehabilitation are often more methodologically complex than reviews of prevention and acute treatment.
Figure 2. Systematic review process.
Planning a systematic review
What is the research question?
A systematic review should be prompted by an interest in a topic, and a wish to answer a specific question. The question should clarify the problem to be addressed, specifying the particular population to which the question applies, as well as any intervention and outcomes of interest. How to form a systematic review question is considered further below. Box 1 illustrates how an initial interest in the effect of rehabilitation interventions was formulated into a research question.
Box 1. What is the research question?
Background and question: My patient has recently had a stroke, and can only walk with assistance. Many physiotherapists have a preference for a specific approach to rehabilitation.71,72 These approaches include the Bobath approach73,74 and the motor learning approach.75 What specific physiotherapy approach should I use in order to best improve the walking of my patient?
Forming the PICO question: • Patient: Patients with acute stroke (less than six weeks) with reduced mobility. • Intervention: Any specific approach to physiotherapy. • Control: No physiotherapy. • Outcome: Independence in activities of daily living; ability to walk independently.
PICO question: In patients with a recent acute stroke (less than 6 weeks) with reduced mobility, is any specific physiotherapy approach method more beneficial than no physiotherapy at improving independence in activities of daily living and gait speed?
Is a systematic review needed?
For any research to be justified, including systematic reviews, the research question must address what is important to patients and clinicians.11 If a research question is of low priority to the people affected by the condition, or important outcomes are not considered, or the intervention is considered unacceptable to patients, or too costly to deliver, then further research can be wasteful.10,11 There are a number of reports which highlight key topics and research questions which are considered of greatest importance by stroke survivors, carers, and health professionals working in stroke care.76–81
In addition, if a research question has already been answered by a systematic review, another review of the same evidence will be wasteful and creates challenges for clinicians and policy makers seeking systematic reviews to inform their clinical decision making.82 There are currently (March 2017) at least 1385 systematic reviews relating to stroke.63 An overview of reviews relating to stroke upper limb rehabilitation identified multiple overlapping reviews, with over 10 published systematic reviews of evidence relating to constraint-induced movement therapy and electrical stimulation.82
It is sometimes argued that an additional criterion to consider is whether there is published research relevant to the research question. A systematic review which does not find and include any relevant studies can be referred to as an “empty review.”83 These empty reviews arguably are of little value in aiding clinical decisions, and subsequently careful consideration should be given to embarking on what may be an empty review. However, where the intention is to complete a systematic review in order to confirm the absence of primary research, prior to the planning and conduct of a primary research study, there remains clear justification for a systematic review.
Feasibility and scope of the systematic review
It is estimated that a typical systematic review will take at least 12 months to complete, although this could be less, depending on the review and the available resources.84 Data from the Cochrane Stroke Group demonstrate that completion time for a Cochrane systematic review, from initial registration of a title to publication of a completed review, is a median of 158 weeks (interquartile range 105 to 209). Although the scope of a systematic review will largely be determined by the research question which has been formulated, there may be opportunities to broaden or narrow a research question in an attempt to make the planned review manageable within the available time and resources.85 A broader review question (sometimes known as a “lumping” review) has the advantage that it will be applicable to a wider range of settings or populations (or interventions or outcomes), and provides greater potential for exploration of consistency of research findings, with less opportunities for chance findings.86,87 Furthermore, broad reviews arguably make systematic review findings more accessible to clinical decision makers, who often have to choose between a variety of interventions for delivery to a number of different patients. However, when resources are limited, a narrower review (or “splitting” a review) may make completion more feasible, and the increased homogeneity of the included studies may provide a more focused answer to the specific (narrow) research question.87Box 2 gives arguments for a broad and for a narrow review, using the example of rehabilitation interventions.
Box 2. Should the review be broad or narrow?
PICO question: In patients with a recent acute stroke (less than six weeks) with reduced mobility, is any specific physiotherapy approach more beneficial than no physiotherapy at improving independence in activities of daily living and gait speed?
Arguments in favor of a broad review: • Limiting the review to patients who had a stroke during the last six weeks will arguably result in a fairly “narrow” review, and potentially large volumes of evidence arising from other patients would be excluded. A broader review would result in a review of a greater volume of evidence. • Assessing the effects of different physiotherapy approaches (not only the Bobath approach) will be clinically relevant to clinicians, who have to consider all available approaches when reaching a treatment decision. Limiting the review to only one specific approach (e.g. the Bobath approach) does not answer the clinical question relating to the relative effects of different approaches. • Considering control groups other than just a “no physiotherapy” control group will reflect the choice faced by many clinicians, who have to choose between two or more different approaches, rather than between one approach or no physiotherapy. • A broader review will have more data from additional studies, making it possible to perform meaningful subgroup analyses. Example of a broader review question: In patients with stroke with reduced mobility, is any specific approach to physiotherapy more beneficial than no physiotherapy or any other physiotherapy approach at improving independence in activities of daily living and gait speed?
Arguments in favor of a narrow review: • The broad review would be more work (more articles to screen, more data to extract, more analyses to be done, more results to discuss). • There would be a need to consider the generalizability of results arising from this broad population to the sub-population of primary interest for this review (patients with stroke during the last six weeks). • A review focused on just one physiotherapy approach (e.g. the Bobath approach) will be more concise and of greater interest for readers interested in this specific approach. Example of a more narrow review question: In patients with a recent acute stroke (less than six weeks) with reduced mobility, is the Bobath approach more beneficial than the motor learning approach at improving independence in activities of daily living and gait speed?
What sort of systematic review best suits the research question?
The type of research question which has been asked will be central to determining the most appropriate type of systematic review (Table 1). The research questions in Box 1 and Box 2 require an intervention review.
Write and publish a protocol
A protocol is an essential part of the review process,20,25,88–90 and should include sufficient information to enable independent replication of the methods. Adherence to a pre-defined protocol is a key method with which to avoid the introduction of selection bias, as it ensures that all important decisions have been made in advance of knowledge of the results.25,89–91 Peer review and feedback from key stakeholders are important,20,52,90 and a protocol should be published prior to starting on the systematic review, for example in a repository, electronic library (e.g. within the Cochrane Library,92 PROSPERO,93 or Joanna Briggs Database94), or in a journal. Publication helps ensure transparency within the review process, enabling any deviation from review protocol to be easily identified.21,90,91 For example, prior publication of a protocol will enable selective outcome reporting to be identified if this occurs within the final review.88,89,91 Furthermore, publication is a key step to avoid research duplication and waste, ensuring that other researchers are aware that the review is being completed.89–91Figure 3 illustrates the key stages for writing a protocol and completing a systematic review. Each stage is briefly discussed below, and key resources highlighted (Table 1).
Figure 3. Key stages for a protocol and systematic review completion.
Clarify review aims and objectives
A clear research question, like the one in Box 1, will help clarify the eligibility criteria for inclusion of relevant studies (and exclusion of irrelevant studies). For relatively simple systematic reviews of effectiveness interventions, the systematic review question is often informed by the “PICO” framework, but there are a range of other frameworks which can inform the questions for more complex reviews (Table 1).95
There are some specific considerations relating to systematic reviews in stroke. For example, when defining the population (P), it may be important to state how stroke is defined or diagnosed, or to define a specific subset of participants (e.g. participants with aphasia), or those within a specific care setting. Sometimes it may be appropriate to broaden the scope of the review by including other relevant populations in addition to stroke (e.g. other non-progressive brain diseases/injuries).
Defining interventions (I) used in stroke care, particularly non-pharmacological interventions, can be complex, and careful consideration should be given to describing the key components of the intervention. The TIDieR checklist96 may provide a useful guide to clarifying the intervention, and ensuring a structured definition. Careful consideration should be given to the “dose” of complex interventions, clarifying how this will be defined, and acknowledging that this can be a complex combination of total number of treatment sessions over the study duration, number of treatment sessions per day, week or month, length of treatment sessions, intensity of treatment (possibly measured in a range of ways such as number of repetitions, or a measure of exertion). Where a comparison or control (C) intervention is defined, it is important to consider that within some stroke research studies, a control group which receives no active treatment may be unlikely (perhaps for ethical reasons), and consequently an active intervention may be compared to a variety of alternative interventions. These could include “standard care” (which would need to be defined fully for the purposes of the review) or another active intervention, or the same active intervention delivered at a different dose or intensity. It is important that the protocol states whether studies which deliver interventions in combination (e.g. constraint induced movement therapy plus electrical stimulation) will be eligible and, if so, how these studies with combined interventions will be brought together with studies of single interventions.
Outcomes (O) that are of interest to the research question should be defined; these ought to be outcomes which are meaningful to patients and other key stakeholders, and it may be appropriate to consider the views of stroke survivors, carers and/or health professionals when determining what outcomes are most important.97 Acceptable methods for measuring an outcome should be stated, including any objective measures (e.g. blood pressure, number of strokes, number of falls, walking speed) or subjective scales (e.g. Barthel Index, Fugl–Meyer Assessment, quality of life scales). To avoid the introduction of bias, the outcome of greatest interest should be defined as the primary outcome, and additional outcomes as secondary outcomes. The timing of the outcome of interest should be clearly defined, and consideration given to how measurements taken at different times in the research study, and at different times post stroke, will be included.
Another key parameter to be defined is the types of study design which will be included in the systematic review. For Cochrane intervention reviews, this is often limited to randomized controlled trials, but other reviews may include other types of study (e.g. observational studies). For example, considering the question relating to physical rehabilitation in Box 1, the question could be broadened to consider issues relating to stroke survivors’ views and experiences of rehabilitation therapies, resulting in the inclusion of qualitative research studies (e.g. studies reporting results from interviews and/or discussions in focus groups).
Find relevant research
The protocol should include the full search strategy, which ought to be developed with appropriate expert advice or support from an information specialist, and description of electronic databases, and any other sources, which are to be searched. There are a wide-range of health-related bibliographic databases, some covering broad areas of healthcare research (e.g. MEDLINE98 and EMBASE99), while some focus on specific study designs (e.g. the Cochrane Database of Systematic Reviews100), more narrow specialist areas (e.g. PsychINFO for behavioural and social science research,101 PEDro for physiotherapy related trials, reviews and guidelines,102 REHABDATA for rehabilitation research103) or a particular language or geographical area of publication (e.g. Wangfangdata, a database of Chinese studies104). In general, multiple electronic databases should be used, in an attempt to be comprehensive and avoid introduction of reporting bias.105
Consideration should be given to how the search results will be managed, including use of any bibliographic or data management software. For example, adequate records of the results of the search and application of eligibility criteria must be kept, in order to complete a detailed PRISMA flowchart (Figure 4). The methods for identifying studies for inclusion should detail processes for screening of titles or abstracts in order to remove irrelevant reports, application of eligibility criteria to abstracts or full texts, and final decision making. It should be clear which of these processes will be carried out by two independent reviewers, and if there are independent reviewers what the process will be if there is disagreement. The use of two independent reviewers at key stages in the review process is considered an important approach in order to avoid one single reviewer introducing a biased (or flawed) interpretation of review criteria. At the end of this stage of the review process, the final list of included studies will have been identified.
Figure 4. PRISMA flow diagram from example review (Pollock et al.70)
Adapted from Moher127 *Data summarised from results of the search from example Cochrane review (Pollock et al.70)
Collect data
“Data” refer to any information within the included studies, including information relating to the characteristics of the study as well as to quantitative and/or qualitative results. The protocol should define the data which will be extracted from each study, who will extract it and in what format. Methods to avoid the introduction of errors (e.g. entering wrong numerals into a spreadsheet; failure to identify required data from a study report) or bias should be considered, and may involve the use of two independent reviewers. Information extracted relating to stroke populations could incorporate data associated with stroke diagnosis (e.g. type, severity of stroke; lesion site; date or time since stroke; measures of initial impairment or disability) and demographic variables (e.g. age, gender, socioeconomic status, level of education, handedness). The TIDieR template96 may be a useful tool for extraction of data relating to complex interventions, and could be incorporated into data extraction plans. Data should also be systematically collected relating to the design and conduct of the research study, such as the method of randomization and allocation concealment in the case of a randomized controlled trial. The protocol should also state which specific statistical variables (e.g. mean, confidence intervals, standard deviation) will be extracted.
Assess quality of included studies
A key stage within a systematic review is the assessment of the methodological quality of the included studies. This process involves critical appraisal and judgment relating to whether there were any potential risks of bias within the study. A bias is a “systematic error, or deviation from the truth, in results or inferences,”23 and this can lead to findings which do not reflect the true result.106Table 2 summarizes common sources of bias, summarizing methods which can be used to avoid or limit the introduction of bias, and giving examples of bias identified in studies included in our example review.70 Within stroke research, bias is a common risk when masking of study participants and investigators is not possible, as is the case when testing many non-pharmacological interventions. For example, outcome assessors may inadvertently provide greater encouragement during the measurement of walking speed in the intervention group than in the control group (performance bias), or may record more positive outcomes for those in the treatment group when using a subjective rating scale or questionnaire (detection bias). Bias can also result if dropouts (for example, due to death or subsequent stroke) occur more often in one group than the other (attrition bias), or if studies or outcomes are reported selectively, depending on the results (reporting bias).
Table 2. Common sources of bias in stroke research
Examples from review of physical rehabilitation approaches70
Selection bias
The groups of participants who are being compared have differences at baseline, due to the way that participants have been allocated to groups.
Randomization (allocation of participants to groups based on a random process, or sequence, with the order of allocation concealed from all people involved in the study)
Zhu 2006a allocated participants to groups “according to time of hospital admission.” This method introduced a risk of selection bias as the characteristics of participants could vary according to time, and the researchers could potentially influence the allocation of a participant to a specific treatment group.
Performance bias
The groups of participants receive differences in care, other than differences in the intervention which is being tested.
Masking (blinding) of participants and personnel (concealment) to the study treatment being delivered
Dean 2000, Chan 2006 and Gelber 1995a did not use masking of person delivering the intervention, who could therefore be more enthusiastic and encouraging towards patients in the intervention group than the control group.
Detection bias
The way outcomes are measured in the groups of participants differs.
Masking (blinding) of outcome assessor to the study treatment being delivered
Salbach 2004a un-masked outcome assessors, introducing a high risk of detection bias.
Attrition bias
There are differences in retention / withdrawals between the groups of participants.
Complete data collection in both groups The reasons for missing data must be reported for each treatment group, so that any differences between groups can be explored.
Fang 2003a had more drop-outs from one group than the other.
Reporting bias (including publication bias, and selective outcome reporting)
There are systematic differences between reported and unreported findings.
Comprehensive searching for all eligible studies (regardless of publication status) can help avoid publication bias. Pre-specification of outcome measures within a published protocol can help avoid selective outcome reporting. Statistical methods can be used to aid detection of reporting biases (funnel plots and sensitivity analyses).
Trials published in non-English language and in Chinese journals may not all have been identified70
There are a large number of tools available to support critical appraisal of study quality (Table 1).107 These tools can be “scales” which score quality components and provide a summary score. Despite the existence of a wide number of scales to assess quality,108 the use of scales is explicitly discouraged by Cochrane23 as the validity and transparency of such summative scales can be questioned.109 The Cochrane risk of bias tool is now recommended for use within all Cochrane reviews, and is widely used by non-Cochrane reviews of randomized controlled trials. Figure 5 shows how risk of bias can be presented, using our example review.70
Figure 5. Risk of bias graph from example review (Pollock et al.70)
The risk of reporting bias can be assessed using a funnel plot, which shows estimates of effect size from included studies against a measure of each study’s size or precision. An asymmetrical funnel plot can indicate reporting bias, for example if a search strategy had failed to identify small unpublished studies which did not show statistically significant effects, while larger published studies with statistically significant effects were identified. The presence of funnel plot asymmetry is often judged subjectively through visual inspection, but a number of statistical tests have been proposed.110 It is important to note that asymmetry within a funnel plot can be due to reasons other than reporting bias, for example poor methodological quality.110 Funnel plots are not recommended if there are less than 10 studies in a meta-analysis,110 and in these cases, the potential impact of reporting bias should be considered without statistical analysis.
Assessment of quality requires adequate reporting of information in the individual study reports. The protocol should detail how absence of information (i.e. lack of reporting) will be incorporated into the assessment of risk of bias, and consider how to distinguish between a study for which risk of bias is unclear, and a study for which there is clear evidence of specific bias. The protocol may also describe methods to attempt to seek missing information, such as contacting research authors or imputing alternative values.
In addition to describing how risk of bias will be assessed, the protocol should also state how the risk of bias assessment will be used. Some systematic reviews may exclude studies which are judged to be of poor methodological quality, or at high risk of bias, from subsequent synthesis. However, a more comprehensive and transparent approach is arguably to maintain all included studies and perform sensitivity analyses to explore the impact of excluding studies which have been judged to be at high risk of bias.
Synthesize evidence
All systematic reviews should include a synthesis of the data that have been found. Data synthesis can involve summarizing results (quantitative and/or qualitative findings) in tables, or producing narrative summaries.
Systematic reviews of quantitative data may include statistical pooling (meta-analysis). Figure 6 shows a typical Forest plot used in most reviews of quantitative data, using our example review.70 In this example, the data being combined comprise a number of continuous outcome scales and the statistical effect measure determined is the standardized mean difference; if only one outcome scale was used, a mean difference may be calculated and if outcome data are dichotomous, the effect measure selected may be an odds or risk ratio, or risk difference. Key components which ought to be specified at the protocol stage include; the comparisons (meta-analyses) which are planned, the types of data and outcome measures which will be combined, the statistical method for pooling data and effect measure which will be used, and how heterogeneity will be assessed and interpreted. For example, the choice of statistical method for pooling data will depend on whether the heterogeneity between effect estimates is most likely due to clinical or methodological diversity between studies (in which case a fixed-effect method should be used), or whether it is most likely due to random variation (in which case a random-effect method should be used).111 The protocol should also specify which subgroup and sensitivity analyses are planned. Examples of subgroup analyses that were considered relevant and carried out within our example review,70 are given in Box 3. Where statistical pooling is not planned (for example within systematic reviews of observational studies), tables summarizing results of individual studies can be useful.
Figure 6. Forest plot from example review (Pollock et al.70).
Box 3. What subgroup analyses are relevant?
PICO question: In patients with a recent acute stroke (less than six weeks) with reduced mobility, is any specific physiotherapy approach more beneficial than no physiotherapy at improving independence in activities of daily living and gait speed?
Relevant subgroup analyses to consider: • Effects of therapy given at different times after stroke (<1 week, 1–3 weeks, or 3–6 weeks). • Effects of therapy in different parts of the world (Europe, Australasia, America, Asia). • Effects of therapy at different doses/intensities (> 45 min/day, 30–45 min/day, 15–30 min/day, <5 sessions/week, <2 sessions/week). • Effects of therapy delivered by different professions (physiotherapist, nurse, assistant therapist, carer/family member). • Effect of different specific therapy approaches (e.g. Bobath approach, motor learning approach, orthopedic methods).
Systematic reviews including qualitative studies may adopt a number of different formal synthesis methods.57,58,112–114 Readers are referred to appropriate texts relating to specific synthesis techniques; however, there is considerable confusion in the published literature in relation to the terminology used to describe methods of synthesizing qualitative or mixed method studies.56,115 Cochrane does not recommend a specific synthesis approach for inclusion of qualitative evidence, highlighting that evaluation of the robustness of different methods is lacking.48
Interpret findings
A plan for summarizing key findings is an essential part of a systematic review. This is often in the form of a table that summarizes the key findings and the overall quality of the data, and it is good practice to decide what will go in this at the protocol stage.116 The GRADE approach is being widely used within systematic reviews,117,118 but other approaches are available (e.g. Weight of Evidence framework106,119). Table 3 shows a summary of findings table from our example review, using the GRADE approach.
Table 3. Summary of findings table from example review
PICO question: In patients with a recent acute stroke (less than six weeks) with reduced mobility is any specific physiotherapy method more beneficial than no physiotherapy at improving independence in activities of daily living and gait speed?
Selected outcomes
Standardized mean difference (95% CI)
No. of participants (studies)
Quality of the evidence (GRADE)
Comments
Independence in ADL scales
0.58 (0.11 to 1.04)
9 studies 540 participants
⊕⊕⊕ ⊖moderate
Quality of evidence downgraded as there was substantial statistical heterogeneity in results (I2 = 85%)
Gait velocity
−0.06 (−0.29 to 0.18)
3 studies 271 participants
⊕⊕⊖⊖low
Quality of evidence downgraded twice as dose of physiotherapy varied substantially between studies, and 1/3 studies were carried out in China (and a significant subgroup effect relating to geographical location of the study was identified)
Note: This table is adapted from the summary of findings table within our example review,70 for the purpose of this article.
Complete the systematic review
Following peer review and publication of the systematic review protocol, the review can be carried out, informed by the methods described in the protocol. Ideally, there will not be any differences between the protocol and review; however if there are any deviations from protocol, then these should be clearly documented, justified, and reported within the final systematic review.89
A discussion within a completed systematic review should address a number of key points, including the quality and completeness of the data. Any potential biases in the review process and any deviations from protocol should be discussed, as should any agreements or disagreements between the review findings and other relevant reviews, guidelines or policies. The generalizability of the evidence to the original research question, and the implications of the review findings to clinical practice should be considered. However, systematic reviews ought to avoid giving specific recommendations for clinical practice, since local circumstances, such as status of the health care system, costs of the intervention, and patients’ preferences must always be considered when making clinical decisions based on systematic review evidence. Involvement of key stakeholders, including patients, carers and health professionals, may be beneficial at this stage, helping to interpret findings in a way that is meaningful to the users of the review.120–122 Often a systematic review will highlight gaps in the evidence, or in the quality of the evidence, in which case specific implications for research should be derived. This should move beyond a statement that “more research is needed” discussing the need for different types of study designs and proposing research questions which need to be addressed.
After review completion: Publication, dissemination and up-dating
After the systematic review has been reviewed and approved, it should be made freely available through publication in a journal, electronic repository, or other resource. The publication should highlight any sources of funding for the review, and any competing interests between the review authors, funders, or other related organizations in the review production. Systematic review authors should consider strategies for effective dissemination,123 and involvement of patients and the public seems to be important for successful implementation of research findings.61,121,122 The need for further work to highlight effective strategies for implementation in the field of stroke care has been highlighted.124,125
Ideally, systematic reviews will be updated regularly in order to incorporate new studies, although decisions to update have to take many factors into account, including the importance of the topic, whether there is new evidence and the likelihood of this changing the conclusions of the review.126 Regular updating of a systematic review is generally recommended and is a more efficient use of resources than embarking on a new review addressing the same question.126
Conclusions
Systematic reviews in stroke are necessary to ensure that healthcare and research decisions are informed by the best possible, up-to-date research evidence, and that patients are provided with the best possible care. A protocol is an essential part of all systematic reviews, ensuring rigorous, transparent methods. Key stages in systematic reviews include the formulation of the research question, the identification of relevant research, data extraction, assessment of risk of bias, data synthesis, summary and interpretation of the findings. The review process should include strategies for dissemination. Updating a systematic review after completion is important to ensure that the conclusions remain valid.
Acknowledgements
Hazel Fraser and Alison McInnes from Cochrane Stroke provided data from the Cochrane Stroke specialised register of stroke trials and reviews.
Alex Pollock is employed by the NMAHP Research Unit, which is funded by the Chief Scientist Office of the Scottish Government.
Declaration of conflicting interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Alex Pollock and Eivind Berge are both Associate Editors for Cochrane Stroke.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Permissions
Permission has been provided by Wiley for use of Figures 5 and 6.
References
1. Sackett DL, Rosenberg WM, Gray JAM, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ 1996; 312: 71–72.
3. Green S, Higgins JPT, Alderson P, Clarke M, Mulrow CD, Oxman AD. Chapter 1: Introduction. In: Higgins JPT, Green S (eds) Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.handbook.cochrane.org.
6. Lindsay P, Furie KL, Davis SM, Donnan GA, Norrving B. World Stroke Organization global stroke services guidelines and action plan. Int J Stroke 20149 Suppl A100): 4–13.
7. Intercollegiate Stroke Working Party. Sentinel stroke national audit programme (SSNAP) post-acute stroke service commissioning audit. London: Royal College of Physicians, 2015.
8. Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open 2017; 7: e012545.
14. Snilstveit B, Oliver S, Vojtkova M. Narrative approaches to systematic review and synthesis of evidence for international development policy and practice. J Dev Effect 2012; 4: 409–429.
18. Riva JJ, Malik KMP, Burnie SJ, Endicott AR, Busse JW. What is your research question? An introduction to the PICOT format for clinicians. J Canad Chiropract Assoc 2012; 56: 167–171.
19. Higgins JPT and Green S (eds). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.handbook.cochrane.org.
21. Eden J, Levit L, Berg A, et al. (eds) Finding what works in health care. Standards for systematic reviews. Washington: National Academies Press, 2011.
23. Higgins JPT, Altman DG and Sterne JAC. Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT and Green S (eds) Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.handbook.cochrane.org.
26. Brazzelli M, Sandercock PA, Chappell FM, et al. Magnetic resonance imaging versus computed tomography for detection of acute vascular lesions in patients presenting with stroke symptoms. Cochrane Database Syst Rev 2009; 4: CD007424.
27. Lees R, Selvarajah J, Fenton C, et al. Test accuracy of cognitive screening tests for diagnosis of dementia and multidomain cognitive impairment in stroke. Stroke 2014; 45: 3008–3018.
29. Whiting PF, Rutjes AW, Westwood ME, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011; 155: 529–536.
33. Feigin VL, Lawes CM, Bennett DA, Barker-Collo SL, Parag V. Worldwide stroke incidence and early case fatality reported in 56 population-based studies: a systematic review. Lancet Neurol 2009; 8: 355–369.
34. Pendlebury ST, Rothwell PM. Prevalence, incidence, and factors associated with pre-stroke and post-stroke dementia: a systematic review and meta-analysis. Lancet Neurol 2009; 8: 1006–1018.
35. Coupar F, Pollock A, Rowe P, Weir C, Langhorne P. Predictors of upper limb recovery after stroke: a systematic review and meta-analysis. Clin Rehabil 2012; 26: 291–313.
37. Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis of Observational studies in epidemiology (MOOSE) group. JAMA 2000; 283: 2008–2012.
38. Banks JL, Marotta CA. Outcomes validity and reliability of the modified Rankin scale: implications for stroke clinical trials: a literature review and synthesis. Stroke 2007; 38: 1091–1096.
39. Boger EJ, Demain S, Latter S. Self-management: a systematic review of outcome measures adopted in self-management interventions for stroke. Disabil Rehabil 2013; 35: 1415–1428.
40. De Vet HCW, Terwee CB, Mokkink LB and Knol DL. Chapter 9. Systematic reviews of measurement properties. In: Measurement in Medicine. A Practical Guide. Cambridge: Cambridge University Press, 2011.
44. Greenwood N, Mackenzie A, Cloud GC, Wilson N. Informal primary carers of stroke survivors living at home-challenges, satisfactions and coping: a systematic review of qualitative studies. Disabil Rehabil 2009; 31: 337–351.
47. Wildridge V, Bell L. How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management information. Health Inform Libr J 2002; 19: 113–115.
48. Noyes J, Popay J, Pearson A, Hannes K and Booth A. Chapter 20: Qualitative research and Cochrane reviews. In: Higgins JPT and Green S (eds) Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.handbook.cochrane.org.
49. Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol 2012; 12: 181.
50. Wong G, Greenhaulgh T, Westhorp G, Pawson R. Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (realist and meta-narrative evidence syntheses: evolving standards) project. Health Serv Deliv Res 2014; 2: 1–251.
53. Waddington H, White H, Snilstveit B, et al. How to do a good systematic review of effects in international development: a tool kit. J Dev Effect 2012; 4: 359–387.
55. Green BN, Johnson CD, Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J Chiropr Med 2006; 5: 101–117.
57. Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review, London: Evidence for Policy and Practice Information and Coordinating (EPPI) Centre, 2009.
59. Morley RF, Norman G, Golder S, Griffith P. A systematic scoping review of the evidence for consumer involvement in organisations undertaking systematic reviews: focus on cochrane. Res Involve Engage 2016; 2: 36.
64. DeJong G, Horn SD, Conroy B, Nichols D and Healton EB. Opening the black box of post-stroke rehabilitation: stroke rehabilitation patients, processes, and outcomes. Arch Phys Med Rehabil 2005; 86(12 Suppl 2): S1–S7.
67. Hoffmann TC, Walker MF, Langhorne P, Eames S, Thomas E, Glasziou P. What's in a name? The challenge of describing interventions in systematic reviews: analysis of a random sample of reviews of non-pharmacological stroke interventions. BMJ Open 2015; 5: e009051.
69. Brady MC, Fredrick A, Williams B. People with aphasia: capacity to consent, research participation and intervention inequalities. Int J Stroke 2013; 8: 193–196.
70. Pollock A, Baer G, Campbell P, et al. Physical rehabilitation approaches for the recovery of function and mobility following stroke. Cochrane Database Syst Rev 2014; 4: CD001920.
71. Tyson SF, Connell LA, Lennon S, Busse ME. What treatment packages do UK physiotherapists use to treat postural control and mobility problems after stroke? Disab Rehab 2009; 31: 1494–1500.
72. Tyson SF, Connell LA, Busse ME, Lennon S. What is Bobath? A survey of UK stroke physiotherapists’ perceptions of the content of the Bobath concept to treat postural control and mobility problems after stroke. Disab Rehab 2009; 31: 448–457.
77. Pollock A, St George B, Fenton M, Firkins L. Top 10 research priorities relating to life after stroke – consensus from stroke survivors, caregivers, and health professionals. Int J Stroke 2014; 9: 313–320.
78. Pollock A, St George B, Rowat A, Scottish Stroke Nurses F. Top 10 research priorities relating to stroke nursing. Int J Stroke 2015. October(10 Suppl A100): 164.
79. Rowat A, Pollock A, St George B, et al. Top 10 research priorities relating to stroke nursing: a rigorous approach to establish a national nurse-led research agenda. J Adv Nurs 2016; 72: 2831–2843.
80. Vickrey BG, Brott TG, Koroshetz WJ, Stroke Research Priorities Meeting Steering C. the National Advisory Neurological Dsorders and Stroke Council. Research priority setting: a summary of the 2012 NINDS stroke planning meeting report. Stroke 2013; 44: 2338–2342.
81. Franklin S, Harhen D, Hayes M, S. M and Pollock A. Stakeholder derived top research priorities for aphasia following stroke. In: World congress of the international associations of logopedics and phoniatrics, Dublin, August 21–25 2016.
82. Pollock A, Farmer SE, Brady MC, et al. Interventions for improving upper limb function after stroke. Cochrane Database Syst Rev 2014; 12: CD010820.
83. Yaffe J, Montgomery P, Hopewell S, Shepard LD. Empty reviews: a description and consideration of Cochrane systematic reviews with no included studies. PLoS One 2012; 7: e36626.
85. Grimshaw J, McAuley LM, Bero LA, et al. Systematic reviews of the effectiveness of quality improvement strategies and programmes. Qual Saf Health Care 2003; 12: 298–303.
87. Weir MC, Grimshaw JM, Mayhew A, Fergusson D. Decisions about lumping vs. splitting of the scope of systematic reviews of complex interventions are not well justified: a case study in systematic reviews of health care professional reminders. J Clin Epidemiol 2012; 65: 756–763.
89. Silagy CA, Middleton P, Hopewell S. Publishing protocols of systematic reviews: comparing what was done to what was planned. JAMA 2002; 287: 2831–2834.
90. Green S and Higgins JPT. Chapter 2: Preparing a Cochrane review. In: Higgins JPT and Green S (eds) Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.handbook.cochrane.org.
93. NIHR. PROSPERO International prospective register of systematic reviews University of York: NHS National Institute for Health Research, Centre for reviews and dissemination, www.crd.york.ac.uk/PROSPERO/ (accessed 28 April 2017).
95. Davies S. Formulating the evidence based practice question: a review of the frameworks. Evidence Based Library and Information Practice 2011; 6: 75–80.
96. Hoffmann TC, Glasziou PP, Boutron I, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014; 348: g1687.
101. PsychINFO. PsychInfo. A world-class resource for abstractions and citations of behavioral and social science research. American Psychological Association, www.apa.org/pubs/databases/psycinfo/index.aspx (accessed 21 August 2017).
102. PEDro. Physiotherapy Evidence Database Musculoskeletal Health Sydney, School of Public Health: University of Sydney, www.pedro.org.au/ (accessed 21 August 2017).
104. Wanfangdata. E-Resources for China studies. Beijing, China: Chinese Ministry of Science & Technology, www.wanfangdata.com/ (accessed 21 August 2017).
105. Rathbone J, Carter M, Hoffmann T, Glasziou P. A comparison of the performance of seven key bibliographic databases in identifying all relevant systematic reviews of interventions for hypertension. Syst Rev 2016; 5: 27.
106. Harden A, Gough D Quality and relevance appraisal. In: Gough D, Oliver S, Thomas J (eds). An introduction to systematic reviews, London: Sage, 2012, pp. 153–178.
107. Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol 2004; 4: 22.
108. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials 1995; 16: 62–73.
110. Sterne JAC, Egger M, Moher D and Boutron I. Chapter 10: Addressing reporting biases. In: Higgins JPT, Churchill R, Chandler J, Cumpston MS, (eds) Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0 (updated June 2017), Cochrane, 2017. Available from www.training.cochrane.org/handbook.
111. Deeks JJ, Higgins JPT, Altman DG; on behalf of the Cochrane Statistical Methods Group. Chapter 9: Analysing data and undertaking metaanalyses. In: Higgins JPT, Churchill R, Chandler J, Cumpston MS, (eds) Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0 (updated June 2017), Cochrane, 2017. Available from www.training.cochrane.org/handbook.
113. Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy 2005; 10: 45–53.
114. Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews, London: ESRC National Centre for Research Methods, 2007.
115. Kastner M, Tricco AC, Soobiah C, et al. What is the most appropriate knowledge synthesis method to conduct a review? Protocol for a scoping review. BMC Med Res Methodol 2012; 12: 114.
116. Schünnemann HJ, Oxman AD, Higgins JPT, Vist GE, Glasziou P and Guyatt GH. Chapter 11: Presenting results and ‘Summary of findings’ tables. In: Higgins JPT and Green S (eds) Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.handbook.cochrane.org.
117. Guyatt GH, Oxman AD, Schunemann HJ, Tugwell P, Knottnerus A. GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol 2011; 64: 380–382.
118. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008; 336: 924–926.
120. Pollock A, Campbell P, Baer G, Choo PL, Morris J, Forster A. User involvement in a Cochrane systematic review: using structured methods to enhance the clinical relevance, usefulness and usability of a systematic review update. Syst Rev 2015; 4: 55.
121. Kreis J, Puhan MA, Schunemann HJ, Dickersin K. Consumer involvement in systematic reviews of comparative effectiveness research. Health Expect 2013; 16: 323–337.
123. Craig P, Dieppe P, Macintyre S, et al. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008; 337: a1655.
124. Walker MF, Fisher RJ, Korner-Bitensky N, McCluskey A, Carey LM. From what we know to what we do: translating stroke rehabilitation research into practice. Int J Stroke 2013; 8: 11–17.
125. Cahill LS, Carey LM, Lannin NA, Turville M, O'Connor D. Implementation interventions to promote the uptake of evidence-based practices in stroke rehabilitation. Cochrane Database of Syst Rev 2017.
127. Moher D, Liberati A, Tetzlaff J, Altman DG. The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 2009; 6: e1000097. doi:10.1371/journal.pmed1000097 Available from: www.prisma-statement.org/documents/PRISMA%202009%20flow%20diagram.doc (accessed 4 April 2017).
Alex Pollock, Nursing Midwifery and Allied Health Professions (NMAHP) Research Unit, Glasgow Caledonian University, Glasgow, UK. Email: [email protected]
Metrics and citations
Metrics
Journals metrics
This article was published in International Journal of Stroke.
I can access personal subscriptions, purchases, paired institutional access and free tools such as favourite journals, email alerts and saved searches.
