2  Introduction to Planning

2.1 Overview

Planning a systematic review works in the opposite order of conducting it. Specifically, when planning, you and your research team have to achieve consensus on the following matters, in this order:

  1. The goals and/or research question(s);

  2. As a function of this, the entities you will extract (see the Planning: Extraction chapter, Chapter 5 in this version of the book);

  3. As a function of these entities and the goal/research questions, the exclusion criteria you will use during screening;

  4. As a function of the goal/research questions and the exclusion criteria, your search strategy, which includes:

    1. the conceptual form of the query you will use;

    2. which database(s) and interface(s) you will use;

    3. the conceptual query translated to each database/interface combination;

    4. additional strategies, e.g. forward and backward citation searches;

This will determine the scope of your review. Although these steps depend on the previous steps’ output, in practice, this process is often nonlinear and iterative. For example, you often test draft queries in your interface/database combinations to see how many hits you obtain, potentially deciding to adjust your exclusion criteria or even your goals or research questions depending on what you find to ensure that the systematic review stays within the scope determined by your resources (time, funding).

2.2 Be explicit for redundancy, transparency, and future you

To err is human, and therefore, in scientific endeavors, it is best to never count on a single human not erring. The solution to this is two-fold. First, don’t let humans perform tasks that computers can perform; and second, implement redundancy. That means that for tasks that have to be performed by humans, always have at least two people perform every task independently and check for consistency in the results.

If you can afford such redundancy, there is a clear penalty for sloppy planning. If your definitions, descriptions, and instructions leave room for interpretation, the laws of probability decree that that room will inevitably be taken sooner or later. This will manifest as heterogeneous results that will be labor-intensive to reconcile. In a non-trivial proportion of cases, such divergent results may prove almost impossible to reconcile, as they may bring to light fundamental problems with the tasks, definitions, and procedures you specified during your planning.

However, in many cases, full redundancy is not practically attainable. For example, when a systematic review is conducted in the context of a bachelor’s thesis, a master’s thesis, or a PhD. thesis, only one screener and one extractor may be available. Similarly, in many projects, having multiple independent synthesists is not feasible. In such cases, sloppy planning and sloppy documentation of the planning is not penalized as explicitly and as acutely.

Therefore, in such cases, it is particularly important to pay special attention to clearly documenting your plans, definitions, decisions, and their justifications. In addition, it is important to not forego developing instructions for each task as if you were not the only person who will conduct them. Although redundancy may not require this, there are two reasons to do this nonetheless.

First, as the transition towards open science has shown, there is much to gain from exercising transparency in science, both epistemologically and operationally. Epistemological benefits include error-detection, easy and accurate identification of risks of bias, and availability of process information for meta-scientific interrogation. Operational benefits include facilitating learning from other researchers and prompting more elaboration through the awareness that one works in public.

Second, systematic reviews tend to be both very valuable and take a lot of time, and the process tends to be very similar every time. These characteristics mean that you will likely come back to your earlier documentation and plans, either to remember what exactly you did and why, or because you want to adapt and re-use some elements of your process. You will save Future You a lot of time and effort by exercising some minimal hygiene throughout your project in terms of data management and clear documentation, including instructions that may seen unneccessary at the time.