Workflow

Planning, performing, protocolling, analysing and reporting a systematic search is a challenging and time-consuming task, especially if it is intended to culminate in a dedicated systematic review article, which is -in full rigour- the most comprehensive, transparent and unbiased analysis, synthesis and summary of the current state of research. The whole process might need the distribution of the workload amongst several scientists. Inclusion of an information specialist should be considered.

Originating in health sciences, the guidelines "Preferred Reporting Items for Systematic Reviews and Meta-Analyses" (PRISMA) and its extensions (e.g. for ecology or evolutionary biology) are the most commonly used resources to follow on what a full systematic review should include. Some disciplines have adopted their own reporting framework such as the "RepOrting standards for Systematic Evidence Syntheses" (ROSES) in environmental science and related fields.

The individual steps can be outlined in various ways but they always follow a similar path. Often, supporting tools are available.

  • Formulating a research question for a systematic search requires precision, clarity, and a structured approach to ensure the question is specific and researchable.

    Define the scope of your research

    • Identify the problem or gap: What specific issue, challenge, or topic are you addressing?
    • Set boundaries: Determine the breadth or depth of your topic. Is it broad (e.g. "renewable energy") or narrow (e.g. "optimization of solar panel efficiency")?

    Use a question framework

    Question frameworks help break down your research question into manageable parts. Many frameworks originate from evidence based research in health sciences but can readily be adapted for other disciplines. Typical frameworks include:

    PICO (Population/Problem/Patient/Research Object, Intervention/Issue/Method, Comparison/Comparator/Control/Alternative Method, Outcome/Knowledge Generation)

    • Often used in biomedical and experimental sciences.
    • Example Question: "How do solar tracking systems (I) compare to fixed solar panels (C) in enhancing energy efficiency (O) in urban environments (P)?"

    SPICE (Setting, Population/Perspective, Intervention, Comparison, Evaluation)

    • Focuses on contextual and evaluative aspects.
    • Example question: "In urban transportation systems with high vehicle congestion (S), how effective are adaptive traffic signal control systems using AI (I) compared to traditional fixed-time traffic signals (C) in reducing vehicle fuel consumption and emissions (E)?"

    SPIDER (Sample, Phenomenon of interest, Design, Evaluation, Research Type)

    • Encourages deeper engagement by addressing multiple dimensions of an issue
    • Example question: "Among civil engineers working on smart city infrastructure projects (S), what are the challenges and perceptions (P) regarding the adoption of IoT-enabled technologies, as explored through semi-structured interviews (D) and evaluated in terms of barriers and strategies for overcoming them (E) in a qualitative study (R)?"

    PEO (Population/Problem/Patient, Exposure/Issue/Intervention, Outcome)

    • Often used in observational studies.
    • Example question: "What are the effects of exposure to microplastics (E) on marine organisms (P) in terms of reproductive health (O)?"

    Refine and specify your question

    Be specific: Avoid overly broad or vague terms. Focus on variables: Clearly identify independent and dependent variables if applicable.

    Example of Refinement:

    • Broad: "How do wind turbines work?"
    • Specific: "What are the aerodynamic improvements in wind turbine blades that increase energy efficiency in low-wind regions?"

    Test the question

    • Ensure the question aligns with your research goals.
    • Check if it's clear, measurable, and answerable through systematic searching (recent AI-based research assistant can help with this).
    • Example Checklist:
      Does it address your specific problem?
      Is it feasible, i.e. researchable with available data or resources?
      Is it unbiased and open-ended?
      Is it ethical?

    Example Questions

    • "What are the environmental impacts of lithium-ion battery recycling technologies?"
    • "How do machine learning algorithms optimize protein structure prediction compared to traditional methods?"
    • "What are the key factors influencing the stability of perovskite solar cells under high humidity conditions?"

  • Developing a design for the systematic review helps to plan and outline the study methodology.
    The design should address the following points:

    Rationale/scope of the review and the research question

    Inclusion and exclusion criteria

    Define clear criteria to filter relevant studies, such as:

    • Subject area: Limit to specific disciplines (e.g. engineering, biology).
    • Publication type: Include peer-reviewed articles, exclude conference abstracts.
    • Language: Restrict to languages you can analyse effectively.
    • Time frame: Decide on a publication date range.

    Selection of data sources

    Include a mix of:

    Comprehensive search strategy

    • Break the research question into key concepts.
    • List synonyms, alternate spellings, related, super- and subordinate terms.
    • Use truncation and wildcards like * or ? to include variations, e.g., diagnos* for diagnose, diagnostic, diagnostics
    • Use quotation marks for exact phrases, e.g., "artificial intelligence".
    • If available use controlled vocabulary, i.e. database-specific indexing terms like MeSH terms in PubMed.
    • Use Boolean operators (AND, OR, NOT) to combine terms.
    • Consider expansion of the search by including the reference lists of the studies obtained (backward search) and also those citing these studies (forward search).
    • Example: ("Artificial Intelligence" OR "Machine Learning" OR "Deep Learning") AND (Healthcare OR Medicine OR Diagnostics) AND (Accuracy OR Efficiency OR Prediction)

    Also refer Lib4RI's Info sheet Topic Search.

     

    Screening and analysing the studies

    No search strategy leads to 100% relevant hits. So, there will always be the need to first separate the irrelevant ones applying predefined selection criteria. Selection and the subsequent deeper analysis and assessment of the relevant studies is usually the most time consuming part of the review. Appropriate software tools as well as sharing the workload need to be addressed.

    • Decide how the screening and selection process is performed (who and how?)
    • State methods for quality assessment
    • Outline data extraction and synthesis procedures

  • Albeit somewhat depending on the actual number of records received the screening process typically is a 2-step process with the aim of excluding the non-relevant studies based on titles and abstracts (1) and full texts (2).

  • Extracting and synthesizing data

    In the data extraction process key information is systematically collected from the studies ensuring that the data is organized, relevant and comparable across studies, enabling researchers to synthesize findings effectively.

    • Develop a data extraction form to capture key information from the selected studies including e.g. author, publication year, methods, study design, results and conclusions.
    • Synthesize findings qualitatively (thematic analysis) or quantitatively (meta-analysis)
    • Tools such as Covidence, DistillerSR, Eppi-Reviewer or Rayyan can also be useful here. The visualisation tool VosViewer (originally developed for bibliometric analysis) can, for example, be used to construct and visualize important terms extracted from a body of scientific literature. Other text and data mining tools may also be applicable.

    Assessing the quality of the studies

    Critically evaluate the methodological rigour of the included studies. This helps assess the trustworthiness of their findings and the overall strength of the evidence. Use standard appraisal tools such as CASP, GRADE, Cochrane Risk of Bias tool, STROBE or AMSTAR.

  • Summarise and discuss the results according to guidelines such as PRISMA or ROSES for systematic reviews or meta-analyses. This should typically include:

    • Present search methodology, study selection and findings comprehensively
    • State inclusion/exclusion criteria (e.g. date, geographic location, participants, type of publication, etc.)
    • Determine an overall result
    • Discussion of the studies and your own approach (incl. risk of bias, reliability limits)
    • If a meta-analysis was conducted: synthesis, handling, combination, data consistency

  • A systematic review could for example be structured as follows:

    • Introduction to the topic, derivation of the research question
    • Background
      Relevant definitions, concepts and theories, relevance of the research question
    • Research design
    • Methods
      Literature research and selection, selection and treatment of the results from the studies analysed
    • Results
      Precise and critical description of the individual studies, comparison and discussion
    • Discussion of own approach
    • Conclusion