Meta-Psychology, 2023, vol 7, MP.2021.2840 https://doi.org/10.15626/MP.2021.2840 Article type: Original Article Published under the CC-BY4.0 license Open data: Not Applicable Open materials: Not Applicable Open and reproducible analysis: Not Applicable Open reviews and editorial process: Yes Preregistration: No Edited by: Rickard Carlsson Reviewed by: Christina Bergmann, Matthew Page Analysis reproduced by: Not Applicable All supplementary files can be accessed at OSF: https://doi.org/10.17605/OSF.IO/FG8TZ An integrative framework for planning and conducting Non-Intervention, Reproducible, and Open Systematic Reviews (NIRO-SR) Marta K Topor*1,2, Jade S Pickering*3, Ana Barbosa Mendes4, Dorothy Bishop5, Fionn Büttner6, Mahmoud M Elsherif7, Thomas Rhys Evans8, Emma L Henderson2, Tamara Kalandadze9, Faye T Nitschke10, Janneke P C Staaks11, Olmo R van den Akker12, Siu Kit Yeung13,14, Mirela Zaneva5, Alison Lam15, Christopher R Madan16, David Moreau17, Aoife O’Mahony18, Adam J Parker19, Amy Riegelman20, Meghan Testerman21, and Samuel Westwood22,23 1University of Copenhagen, Denmark 2University of Surrey, UK 3University of Manchester, UK 4Erasmus University Rotterdam, Netherlands 5University of Oxford, UK 6University College Dublin, Ireland 7University of Birmingham, UK 8University of Greenwich, UK 9Østfold University College, Norway 10University of Newcastle, Australia 11University of Amsterdam, Netherlands 12Tilburg University, Netherlands 13The University of Hong Kong, HKSAR, China 14The Chinese University of Hong Kong, HKSAR, China 15University of Liverpool, UK 16University of Nottingham, UK 17University of Auckland, NZ 18Cardiff University, UK 19University College London, UK 20University of Minnesota, MN, USA 21Princeton University, NJ, USA 22University of Westminster, UK 23King’s College London, UK *Joint first authors Most of the commonly used and endorsed guidelines for systematic review protocols and reporting standards have been developed for intervention research. These excel- lent guidelines have been adopted as the gold-standard for systematic reviews as an evidence synthesis method. In the current paper, we highlight some issues that may arise from adopting these guidelines beyond intervention designs, including in basic behavioural, cognitive, experimental, and exploratory research. We have adapted and built upon the existing guidelines to establish a complementary, comprehensive, and ac- cessible tool for designing, conducting, and reporting Non-Intervention, Reproducible, and Open Systematic Reviews (NIRO-SR). NIRO-SR is a checklist composed of two parts that provide itemised guidance on the preparation of a systematic review proto- col for pre-registration (Part A) and reporting the review (Part B) in a reproducible and transparent manner. This paper, the tool, and an open repository osf.io/f3brw provide a comprehensive resource for those who aim to conduct a high quality, reproducible, and transparent systematic review of non-intervention studies. Keywords: guidelines, non-intervention research, open research, reproducibility, systematic reviews, transparency https://doi.org/10.15626/MP.2021.2840 https://doi.org/10.17605/OSF.IO/FG8TZ https://osf.io/f3brw/ Introduction Systematic literature reviews are a widely used method for rigorously synthesising existing evidence to answer research questions and to inform best practice and policy-making. The quality of systematic reviews is contingent upon comprehensive, systematic, and trans- parent identification of all the relevant literature, fol- lowed by a balanced critical evaluation and synthesis of the data extracted from that literature. Rigorous implementation can minimise biases and questionable reporting practices that can lead to misleading or in- consistent conclusions (Ioannidis, 2016; Moher et al., 2009; Siddaway et al., 2019). However, the most popu- lar guidelines for designing, reporting, conducting, and critically appraising systematic reviews to date have been designed for the synthesis of healthcare, medi- cal, and intervention-based research. These include the PROSPERO protocol pre-registration system and tem- plate (Booth et al., 2012); the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA; Page et al., 2021); Cochrane Handbook for Systematic Reviews of Interventions (Higgins et al., 2019); and the Assessing the Methodological Quality of Systematic Re- views tool (AMSTAR; Shea et al., 2017). The popularity of these tools is evident through endorsement- from a number of journals (see PRISMA endorsers for an exam- ple), university libraries, and collaborative groups spe- cialised in conducting systematic reviews (see the list of recommended systematic review tools by the EQUA- TOR network). Therefore, these tools are widely chosen by authors of systematic reviews through recommen- dations, journal requirements, good findability, and/or greater accessibility. Moreover, they are a likely choice for authors who conduct systematic reviews based on studies other than interventions who reach for these tools through similar routes. For instance, some more general and multidisciplinary journals that publish vari- ous types of studies encourage or require that all sub- mitted systematic reviews must follow the PRISMA guidelines intended for intervention studies (e.g. Sys- tematic Reviews, PeerJ or PLOS One), which may not al- ways be well suited. Intervention studies focus on assessing the efficacy or effectiveness of, for example, healthcare interven- tions and clinical trials that a priori assign participants to different intervention groups (Committee of Medi- cal Journal Editors, 2021). As conceptualised by Glass (1972), the essential aim of intervention studies is to evaluate the proposed intervention and its effects. Many other types of research, such as basic, experimental, and exploratory research in the social, cognitive, and behavioural sciences, do not share the same aims as in- tervention research, and instead aim to explore and ex- plain mechanisms, and thus evidence synthesis of such papers must be approached from a different perspective. Research that does not fit the scope of intervention, such as explanatory, experimental, and basic research, should also adopt rigorous and transparent practises of conducting evidence synthesis, particularly in the con- text of the ongoing paradigm shift that places empha- sis on replicable and reproducible research (Munafò et al., 2017). However, researchers conducting systematic reviews of non-intervention research who wish to fol- low established guidelines must often resort to adapt- ing the criteria of less applicable guidelines to make it appropriate to assess these types of studies, leading to ad hoc solutions such as filtering, combining, or cus- tomising practices from several sources (Macpherson and Jones, 2010). For instance, one popular tool is the “PICOS” framework (Population, Intervention, Compar- ison, Outcome, Study design) which aids the develop- ment of a research question and eligibility criteria for evidence synthesis. PICOS is an important component of the PROSPERO template, Cochrane guidelines, and the AMSTAR tool, and it was only recently removed from PRISMA following the 2020 update. This frame- work cannot always be directly applied to diverse re- search designs (Bramer, 2015) and many alternatives have been developed (Booth et al., 2019); for example the SPIDER framework (Cooke et al., 2012) for system- atic reviews of research using qualitative methods. More general guidelines which are not limited to in- tervention designs also exist. In the field of psychol- ogy a comprehensive tool, the Meta-Analysis Report- ing Standards (MARS; Appelbaum et al., 2018), was recently proposed by the American Psychological Asso- ciation (APA) Working Group on Quantitative Research Reporting Standards. The tool advances standards, but there are barriers to its implementation. Not all sys- tematic reviews include meta-analyses, thus for many authors who decide not to include a meta-analysis com- ponent, MARS may initially be considered unsuitable. In addition, accessibility of MARS as a tool is limited because it is not an open-access resource. In fact, the uptake of MARS for evidence synthesis has been very limited and described as “non-existent” in a recent re- view (Hohn et al., 2020). Lastly, MARS is a reporting guideline, which in practice means that researchers may follow it retrospectively for reporting purposes only and are less likely to use it to inform the design of their study. In summary, although valuable resources exist for guiding the design and reporting of systematic reviews, researchers have a limited choice when it comes to se- lecting an appropriate and accessible tool for systematic reviews beyond interventional research. The utility of existing guidelines for high quality sys- https://www.crd.york.ac.uk/prospero/ http://www.prisma-statement.org/Endorsement/PRISMAEndorsers https://www.equator-network.org/?post_type=eq_guidelines&eq_guidelines_study_design=systematic-reviews-and-meta-analyses&eq_guidelines_clinical_specialty=0&eq_guidelines_report_section=0&s=+ https://www.equator-network.org/?post_type=eq_guidelines&eq_guidelines_study_design=systematic-reviews-and-meta-analyses&eq_guidelines_clinical_specialty=0&eq_guidelines_report_section=0&s=+ https://www.equator-network.org/?post_type=eq_guidelines&eq_guidelines_study_design=systematic-reviews-and-meta-analyses&eq_guidelines_clinical_specialty=0&eq_guidelines_report_section=0&s=+ 2 Table 1 Definitions of terminology Term Definition Intervention Research A study which aims to evaluate the effects of an intervention, often against another intervention, on primary or secondary outcomes of interest. Non-Intervention Research A study which aims to provide an explanatory framework of an empirical phenomenon, or to provide supporting evidence for a theoretical paradigm (Glass, 1972). Intervention Systematic Review A systematic review which synthesises results from intervention research. Non-Intervention Systematic Review A systematic review which synthesises results from non-intervention re- search. Systematic Review Protocol A protocol (ideally pre-registered, see “Systematic Review Pre- registration” below) which outlines specific plans for conducting the sys- tematic review. It may be understood as a ‘recipe’ for the systematic re- view. Systematic Review Pre-registration A systematic review can be pre-registered by submitting the finished pro- tocol to a pre-registration platform, such as the Open Science Framework. Systematic Map A report of the ongoing research activity on a particular topic, informed by a systematic search and screening strategy, which can be used to identify gaps in research. Methodological Systematic Review Informed by a systematic search, this review summarises methodological practices or questions in a given area. tematic reviews is limited by whether they are correctly applied. General problems with adherence to guidelines have been highlighted in the 2009 version of PRISMA (Page and Moher, 2017) but also in systematic reviews in psychology as a field (Hohn et al., 2020). Page et al. (2021) and Hohn et al. (2020) suggested that low adherence could be related to possible lack of guideline adherence checks during peer review, relaxed demands for adherence by journals, or variation in how checklist items are interpreted by the systematic reviewers. How- ever, adherence rates might be significantly impacted by the guideline’s appropriateness to specific fields. Prob- lems with the use of guidelines differ across disciplines and may be driven by discipline-specific interpretation of items which can be further exacerbated by ambiguity and lack of clarity regarding item wording (Rethlefsen et al., 2021). This is specifically problematic for fields where non-intervention research is common (Gates and March, 2016). Therefore, the development of system- atic review guidelines that cater beyond interventional designs and are appropriate for explanatory, experimen- tal, and basic research could help to improve a guide- line’s adherence rate in fields such as psychology, neu- roscience, and economics. The lack of sufficient instructions accompanying guidelines may also contribute to the low adherence problem especially with regards to items designed to facilitate transparency and robustness of systematic re- views. For example, protocol pre-registration is one of the PRISMA items with a very low adherence rate. Considering that pre-registration is widely understood to be an important measure to constrain reporting bias (Nosek et al., 2018), it is of particular concern that this item is only adhered to by 21% of systematic reviews published using PRISMA between 2010 and 2017 (Page & Moher, 2017). This low adherence may be partly due to the uncertainty that surrounds the writing of system- atic reviews protocols, their pre-registration, and how to transparently report and justify deviations from pro- tocol when necessary. For example, it is often consid- ered unclear how immutable a pre-registered protocol is, and when and how systematic reviewers can appro- priately deviate from protocol and subsequently report this transparently (DeHaven, 2017). In addition, sys- tematic reviews tend not to report specific search results (48%), or screening and extraction procedures (abstract screening: 18%; full text screening: 20%). Further- more, specifically in meta-analyses, systematic reviews reported the effect-size in 62% and moderator informa- tion in 58%. Finally, only 11% of systematic reviews contained the statistical code required for reproducibil- ity of the analysis (Polanin et al., 2020). This report- ing is necessary not only to give context to any addi- tional decisions made during the analysis, but also to https://osf.io 3 give others the information to evaluate key decisions made within the planned review, and improve the re- producibility of evidence synthesis (which is known to be low; Maassen et al., 2020). Given these issues surrounding uptake, adherence, accessibility, and relevance of existing guidelines, the Non-Intervention, Open, and Reproducible Evidence Synthesis (NIROES) collaboration was set up to cre- ate a suite of accessible tools designed to facilitate evi- dence synthesis of non-intervention research, while also minimising the limitations of existing guidelines. In particular, it is designed to have high utility amongst novice systematic reviewers. This paper presents the Non-Intervention, Open, and Reproducible tool for sys- tematic reviews (NIRO-SR), which is designed with the specific purpose of providing guidelines and a frame- work for researchers to conduct a systematic review of non-intervention research in line with best practice. We believe this to be particularly applicable to the social, cognitive, and behavioural sciences, as those are the perspectives from which the majority of co-authors have approached the problem, but the guidelines may well prove useful to a wider range of fields given the non- specificity of the items. We acknowledge the impor- tance of conducting meta-analyses as part of systematic reviews, however it is not a strictly necessary part of a systematic review and so this is outside of the scope of the current paper. Our tools provide guidance for creating, planning, and pre-registering a systematic re- view protocol (Part A), and conducting and reporting a systematic review (Part B), with the goal of making evidence synthesis as open and reproducible as possi- ble, thereby improving the credibility of the systematic review and reducing the likelihood of biased outcomes and conclusions. Method Item bank Search and Information Sources. The refinement and specification of the aims and the scope of the project (as reflected in the introduc- tion) occurred during conferences and working groups that engaged researchers, librarians, and journal edi- tors predominantly representing experimental and basic behavioural/cognitive fields from January to December 2019 (e.g. Advanced Methods for Reproducible Science Workshop, UK Reproducibility Network 2019; Society for the Improvement of Psychological Science Confer- ence, 2019; NIROES Online Collaborative Hackathon, August 2019). Participants were at different career levels and with varied experience of applying system- atic reviews in their work. Discussions during these meetings unveiled personal experiences of barriers for conducting systematic reviews beyond intervention re- search. In addition, many pre-existing tools to guide systematic reviews across experimental, behavioural and cognitive fields were shared, forming the first step for compiling relevant existing tools that would in- form the development of NIRO-SR. Talks and presen- tations given about the project to date can be found through the project’s Open Science Framework page (osf.io/8seby/). The initial list of relevant systematic review guide- lines was expanded by two authors (MKT and JSP) who conducted a search of existing guidelines for writing, reporting and quality assessment of systematic reviews, systematic maps, and meta-analyses. This was facili- tated through extensive web searches (e.g. “systematic review checklist”, “systematic review guidelines”, “sys- tematic review reporting”), resources from the EQUA- TOR network website and further collaborative sessions with the NIROES team until we reached saturation, i.e. we could not find any more relevant tools using this method. Our search identified 19 guidelines (Appendix A) that provided quality assessment and protocols for systematic reviews, with a total of 517 items. Item Extraction. All items and explanatory text were extracted verba- tim from the 19 sourced guidelines to create an item bank. The PRISMA 2020 update (Page et al., 2020) and accompanying item bank were published after our item bank was compiled and, therefore, was not included in our item bank. We cross-referenced our own with those from PRISMA 2020 and identified 55 items from various additional guidelines that added value to the items we had already included. The final item bank contained 572 items extracted from all sources. The flowchart for this process is presented in Figure 1. Eligibility was determined by two authors (MKT and JSP) who independently coded each item for poten- tial inclusion as “Yes”, “No” or “Maybe” depending on its broad relevance and application for systematic re- views of non-intervention studies. “Maybe” was de- fined as having components that were useful but with- out being directly applicable as a whole item. Exclusion criteria included application (e.g., applicable to meta- analyses and/or to systematic maps only); relevance (items that were relevant to clinical/intervention re- search and not adaptable for non-intervention research systematic reviews), formatting and presentation (items which suggested formatting that was not specific to sys- tematic reviews, for instance, if they referred only to systematic maps), and ambiguity (e.g., items that had https://osf.io/8seby/ https://www.equator-network.org/ https://www.equator-network.org/ 4 Figure 1 Flowchart showing the records identified from searching, and the records included/excluded during screening throughout the development of NIRO-SR. 5 a lack of clarity or incomplete guidance). Disagree- ments were resolved by consensus, and irreconcilable disagreements were re-evaluated at a later stage of the NIRO-SR tool development following discussions with a larger group of collaborators and experts in systematic review methodology. The final item bank, including de- cisions about the inclusion and exclusion of items, can be found in the project’s OSF repository (osf.io/p2v34). Item Development First, eligible items were categorised into the section of a systematic review that was most applicable, which included abstract, title, protocol, introduction, aims, re- search question, search strategy, screening, data extrac- tion, risk of bias and quality assessment, synthesis, re- sults, transparency, discussion, and miscellaneous items (see item bank tab “included items by category”). Sec- ond, items were further divided to form two parts of the NIRO-SR tool, the protocol (Part A) and the review (Part B). Protocol items were applicable when devising and pre-registering a prospective systematic review pro- tocol of nonintervention studies, and review items were applicable for guiding the process of conducting a sys- tematic review and writing a report for publication. Fi- nally, each group of items was either rewritten for clar- ity or adapted for general use in non-intervention re- search. This process of rewriting items, splitting com- plex items, and merging similar items was conducted iteratively and collaboratively over several months and alongside other feedback methods (see section 2.3 and section 2.4). The resulting items resemble the original curated items in theme, depth, scope. An example of an adapted item is provided in Table 2. Please note that items addressing the risk of bias and heterogeneity of reviewed studies were included in the NIRO-SR tool, but to a limited extent. This is because a separate, com- plementary tool for guiding the assessment of bias and quality in non-intervention research systematic reviews is under development by NIROES. Initial Feedback; Accessibility and Understandability One aim of NIRO-SR was to make it accessible to researchers who had never conducted a systematic re- view before. In December 2019, feedback on the ini- tial version of the tool was sought from a convenience sample of students and staff (N = 9) in the School of Psychology, University of Surrey (all materials and feed- back available on osf.io/f3brw). None of the partici- pants had published a systematic review at the time of response, and they had little previous experience with conducting systematic reviews, relatively low con- fidence in this method, and their research areas were non-interventional. Participants were asked to provide general ratings of NIRO-SR using a three-point scale (“1 - Not good enough”, “2 - Could be improved” and “3 - Good”) across five separate categories: clarity (mean rating = 2.56, SD = 0.53), structure (mean rating = 2.89, SD = 0.33), practicality (mean rating = 2.61, SD = 0.49), relevance (mean rating = 2.86, SD = 0.38), and simplicity (mean rating = 2.44, 0.52). Comments were overall positive about the tool’s utility, with sug- gested revisions limited to improvements in clarity and further guidance in a minority of items. All partici- pants reported that they would want to use this tool when conducting relevant systematic reviews in the fu- ture. The feedback guided some initial changes to im- prove the tool’s clarity for non-expert users, which in- cluded adding an explanation of the purpose and pro- cedures of pre-registration at the beginning of the tool, and explaining items in further detail. The study proce- dures involving human participants have been reviewed against the guidelines set out by the Ethics Committee of Faculty of Health and Medical Sciences, University of Surrey and carried out in accordance with the Uni- versity of Surrey’s Code of Conduct on Good Research Practice and the Declaration of Helsinki. Final Edits; Collaborator Feedback In March 2020, a virtual hackathon was hosted to in- vite final feedback on the tool from a multidisciplinary team of both existing and new collaborators compris- ing expert researchers and librarians experienced in sys- tematic reviews, systematic maps, and meta-analyses as well as more novice researchers with little experience of evidence synthesis. Expert researchers revised the tool to ensure that it covered the breadth of knowl- edge needed to conduct a systematic review, including adding details that were missing based on their own experiences of preparing pre-registration protocols and writing non-intervention systematic reviews. Novice contributors refined the tool with the aim of making it as accessible and understandable as possible to users of all levels of expertise in reporting and conducting system- atic reviews. In the cases where new items were appli- cable to only certain types of non-intervention studies, they were marked as optional. Finally, it was identified that certain items could ben- efit from additional illustrative examples, templates, or detailed guidance. These included: • A full example of a search strategy • A decision log template to track the decisions made during the screening and data extraction stages • An example of a screening manual https://osf.io/p2v34 https://osf.io/f3brw 6 Table 2 The table below presents an item which guides authors on how to prepare and report systematic review research ques- tions. On the left, the PICOS framework sourced from the PRISMA statement. On the right, the same framework is adapted for non-intervention research in NIRO-SR. In the adapted version, the language clearly guides the researchers to state their dependent and independent variables. “Interventions” are excluded from the item and there is an added optional position on the consideration of covariates. PICOS, PRISMA statement; Moher et al., (2009) Item 3, NIRO-SR (Part A) Provide an explicit statement of questions being addressed with reference to: What is the primary review question? The review question must be clearly defined and include the following: • participants, • The primary outcome measure(s) of interest (the • interventions, dependent variables(s); DV) • comparisons, • The primary independent variables (IVs) of interest • outcomes, • The population/participants of interest (e.g., • study design undergraduate students, participants with a specific diagnosis, school-age children etc.) • (optional) Study design(s) of interest, for example: i. observational - measured variables at one time-point ii. cross-sectional - measured variables with different individuals at different timepoints/variables iii. longitudinal - same individuals followed over time; could be prospective or retrospective iv. experimental - examining effect of specific manipulation • (optional) Any covariates of interest or variables you want to control for (e.g. participant age) NB. If you find that your research question does not fit the above, for instance in exploratory or methodological systematic reviews, you should state this in the protocol for transparency. If you cannot operationalise the DV and IV make sure to clearly define the focus (e.g. methodological variation) and the context (e.g. in working memory research) of your investigation. • A template for data extraction forms • A risk-of-bias assessment tool to help with the assessment of credibility of included non- intervention studies These are outside of the scope of the current pa- per, but represent the need for further information and guidance. Following this feedback process, NIRO- SR Version 0.1 (and version 0.1.1 for subsequent mi- nor fixes) was uploaded to osf.io/c9wer for any re- searcher who wanted to use it to guide their system- atic review projects. The NIRO-SR tool has already been used by several projects to inform pre-registration and the guidelines have been implemented in some cur- riculums, including the University of Coventry and the University of the Philippines Diliman. Feedback from users has been very positive, and they provided fur- ther suggestions to improve the tool and increase clar- ity. These changes were implemented, and the current paper presents the finalised NIRO-SR Version 1.1. https://osf.io/c9wer/ 7 Results NIRO-SR Version 1.0 NIRO-SR comprises two parts (osf.io/c9wer), A and B. Part A is a guide for pre-registering a systematic re- view protocol composed of 30 items, of which 26 items are required and 4 items are recommended for best practice. The items are divided into eight sections: Ti- tle, Description and Aims, Research Question, Search Strategy, Screening, Data Extraction, Critical Appraisal, Synthesis, and Transparency. Part B is a 38-item guide for high standards of reporting for non-intervention systematic reviews with the following sections: Title, Abstract, Introduction, Method (Deviations from pro- tocol, Search Strategy, Screening Methods, Data Ex- traction Method, Critical Appraisal Method, Synthe- sis Method), Results (Extracted Records Results, Crit- ical Appraisal Results, Synthesis Results), Discussion, and Transparency. If Part A cannot be completed, re- searchers must give a justification why this is the case and are advised to include as much relevant content from Part A as possible in the final systematic review publication. Discussion NIRO-SR aims to firstly provide guidelines for con- ducting systematic reviews of research that do not clearly fit the definition of intervention research, such as explanatory, experimental, and basic research. The guidelines are intended to be particularly applicable to the behavioural sciences and related fields, but may also be used in other fields outside the expertise of the au- thors of this paper. Secondly, NIRO-SR aims to place emphasis on reproducibility, openness and transparency of systematic reviews. Part A provides guidance for de- veloping and pre-registering a comprehensive review protocol, and Part B guides authors in writing and re- porting systematic reviews. Both parts of the guidelines are designed to be usable on their own, but can also complement existing tools such as PRISMA 2020. NIRO-SR may particularly benefit psychologists and experimental and behavioural scientists who focus on non-intervention research in their systematic reviews, by providing specific advice on how to develop a review protocol, and to conduct and report a rigorous system- atic review. It provides guidance to authors on defin- ing primary review questions (item A3), secondary re- search questions (item A4), hypotheses (item A5), in- clusion and exclusion criteria (item A13), and data ex- traction processes (items A15 to A17). These are the areas where existing systematic review guidelines are often inapplicable for non-intervention research. NIRO- SR provides a framework that places particular empha- sis on the operationalisation of variables of interest (e.g. IVs and DVs) and covariates, whilst still maintaining focus on relevant study designs and participant groups (see Table 2 and item A3). It is hoped that by providing specific advice for con- ducting comprehensive systematic reviews of basic re- search in the behavioural sciences, NIRO-SR will help to begin to standardise and improve the contents of non- intervention systematic reviews protocols. NIRO-SR may help prevent author bias (which is usu- ally unintentional) through its emphasis on the develop- ment and pre-registration of a protocol before conduct- ing a systematic review. NIRO-SR does not make the distinction as to whether the protocol should be pub- licly available from the outset or upon publication of the review (for example, by pre-registering with an em- bargo period on the Open Science Framework), but it places importance on the availability and transparency of the public record. NIRO-SR advises that the protocol should be available together with the final review and include a statement of transparency which specifies the date of pre-registration and point in the review process at which the protocol was pre-registered (e.g., before the final search was completed, or before data extrac- tion began; see item A26). The protocol benefits the authors as it sets out a detailed and transparent plan for the systematic review, and benefits the reader who can more confidently reflect on how different decisions made throughout the process of conducting the system- atic review may have influenced its outcomes. NIRO-SR also emphasises the importance of reporting all devia- tions from the original protocol. We acknowledge that such deviations are often necessary, so we recommend that they are justified and transparently declared in the eventual report of the systematic review (item B5). NIRO-SR recommends a multiple-author approach when conducting systematic reviews, in line with best practice recommendations (Page et al., 2021; Watts and Li, 2019). For example, multiple team members should independently screen the titles and abstracts and full texts, and have a clear procedure for solving potential disagreements between systematic reviewers, as well as report a quantitative measure of inter-reviewer reliabil- ity (items B13, B14, B18, B20 and B21). This helps fa- cilitate reproducibility by increasing the likelihood that a separate team of researchers could follow the exact steps of the original review and reach the same con- clusions (i.e., same data, same method, same results; Barba, 2018). Researchers should be able to use the same method (i.e., search strategy, screening process and inclusion/exclusion criteria), on the same data (i.e., the databases and search results) and arrive at the same results (i.e., the final set of papers and the extracted https://osf.io/c9wer 8 data). However, subjective decisions must still be made throughout this process and so, where full reproducibil- ity is not possible, NIRO-SR emphasises the importance of transparency. We recommend that a decision log is made available that catalogues important decisions (items B14, B15 and B36). The decision log allows any- one trying to reproduce the results to identify and eval- uate the subjective decisions behind any discrepancies. NIRO-SR was developed to both alleviate the barri- ers preventing researchers from conducting systematic reviews and to encourage novice researchers to conduct systematic review in fields where specific guidelines are currently lacking. We strived to ensure that NIRO-SR is comprehensive, clear, and openly accessible to enable researchers to improve their literature review method- ology with a systematic and transparent approach. Methodological Limitations NIRO-SR was developed without a pre-registered protocol or previously published methodological guid- ance for the development of such tools, which could introduce biases at the item selection stage of the tool development. Unfortunately, the lack of pre-registration was due to the fact that––as far as we are aware––there was no pre-registration template that could serve as an adequate template for developing NIRO-SR. Our web searches to identify appropriate guidelines and tools were therefore not systematic. We minimised biases with the breadth of the collaboration team and, al- though the sample of nine researchers providing initial feedback was small, we additionally sought input from multiple, independent contributors comprising a cross- discipline mix of academics and librarians with exten- sive experience of conducting and teaching interven- tion or non-intervention research systematic reviews. Further, we chose to develop NIRO-SR based on exist- ing, peer-reviewed, consensus-based guidelines of ro- bust methods for rigorous and transparent reporting (see Appendix A). As with all guidelines, some limitations may only be fully known when NIRO-SR has been widely adopted. Furthermore, whilst the NIROES collaboration repre- sents multiple disciplines and research fields, the dom- inant field of the authors is the experimental and be- havioural sciences, which may reduce its applicability to some fields. Whilst we believe the tool to be partic- ularly applicable to explanatory experimental and basic behavioural/cognitive research, we cannot confidently assess its use for other fields. This paper accompanies the release of NIRO-SR Version 1.1, and we anticipate that further updates will be necessary and may affect the structure, content, and wording of the items. To retain standardisation, these are anticipated to be infre- quent. To facilitate future updates, users of NIRO-SR are encouraged to provide feedback to the correspond- ing authors. Implications and Future Directions NIRO-SR addresses an important gap in the avail- able guidelines to help reviewers produce high qual- ity systematic reviews for research in experimental and behavioural sciences. The project was conceptu- alised through a collaborative effort during multiple method and metascience oriented meetings including the Society for the Improvement of Psychological Sci- ence 2019 conference, Advanced Methods for Repro- ducible Science 2019 and 2020 workshops and Repro- ducibiliTea meetings. The growing demand for the tool is also reflected through many presentations about NIRO-SR delivered at psychology-focused or interdis- ciplinary meetings and conferences including The Or- ganisation for Human Brain Mapping 2020 conference, Metascience 2021 conference and UK Reproducibility Network’s meeting for Open Science Working Groups in 2020. A number of pre-registered protocols have al- ready been completed using NIRO-SR, some of which can be found on the OSF (osf.io/f3brw). Therefore, we expect a further increase in use of the NIRO-SR tool, which we hope will have a significant impact on the quality of systematic reviews in non-intervention re- search, reducing the need for bespoke customisations of existing guidelines in order to answer specific research questions. A few years after release, we plan to assess the implementation of the NIRO-SR guidelines to fur- ther understand the challenges of conducting systematic reviews in our field, as well as to inform future updates. Specifically, we would like to provide an evidence-base for whether there is a demand for the tool as we have anecdotally observed already, and whether reviews us- ing NIRO-SR are of comparable or greater quality to the high quality systematic reviews that have used other pre-existing tools. The further standardisation of systematic reviews outside of intervention research will also allow for bet- ter meta-scientific approaches and comparison of out- comes across multiple systematic reviews in the future. Further, NIRO-SR provides a solid basis for conduct- ing systematic reviews with a meta-analysis component. Whilst NIRO-SR does not advise on the methodology specific to meta-analyses, it will help to raise the stan- dard of the systematic approach such as the establish- ment of the research question, pre-registration, search strategy, inclusion/exclusion criteria, and logging deci- sion making. Finally, NIRO-SR is tailored for systematic reviews of https://osf.io/f3brw/ 9 experimental, cognitive and behavioural research, but future additions to the project could include “plug-ins” for the tool that enhance its existing features (released as needed on the OSF page; osf.io/f3brw). For exam- ple, additional optional items could assist with reviews of other study designs such as qualitative studies or lon- gitudinal studies, or specific items could be created for other approaches to evidence synthesis such as meta- analyses or systematic maps. Additionally, extensions of the NIRO-SR are currently under development, includ- ing further guidance for risk of bias and quality assess- ment (related, but not necessarily synonymous, endeav- ours). There are elements of a study that may not di- rectly introduce bias but which are nevertheless impor- tant indicators of quality, for example incompleteness in the reporting of the methodology which can lead to problems with replicability. Conclusions NIRO-SR is a new tool that will allow researchers to follow standardised guidelines for systematic reviews of basic cognitive and behavioural research. It fills an im- portant gap in methodological standards and we hope it will contribute to the improvement of the quality of systematic reviews of research that does not form an intervention. Author Contributions The authorship for this project was determined using the CRediT Taxonomy and the authorship agreement for the NIROES collaboration (available on the OSF). For the current project, authors were divided into four relevant tiers as specified in the authorship agreement. The first tier, “Project Management” specifies the joint lead authors and project co-leads, M.K. Topor and J.S. Pickering. Within each subsequent tier, authors were listed in an alphabetical order as follows: Tier 2 “Ma- jor Contributions” (data curation, formal analysis, in- vestigation, methodology, visualisation, writing - orig- inal draft, miscellaneous input into creating the tool): A. Barbosa Mendes, D.V.M. Bishop, F.C. Büttner, M.M. Elsherif, T.R. Evans, E.L. Henderson, T. Kalandadze, F.T. Nitschke, J.P.C. Staaks, O. van den Akker, S.K. Yeung, M. Zaneva; Tier 3 “Feedback and Review” (conceptu- alisation, writing - review & editing, feedback on the tool): A. Lam, C.R. Madan, D. Moreau, A. O’Mahony, A.J. Parker, A. Riegelman, M. Testerman; and Tier 5 “Se- nior Supervision” (in addition to Tier 2 “Major Contri- butions”): S.J. Westwood. Acknowledgements Special thanks to the attendees of the Society for the Improvement of Psychological Science meeting in Rot- terdam, Netherlands, July 2019, as well as attendees of the Advanced Methods for Reproducible Science work- shop in Windsor, UK, January 2020. We are also very grateful to all researchers at the University of Surrey for providing their feedback. Additional thanks to: Katie Corker, Margriet Groen, Matt Jaquiery, Sayaka Kidby, Emily Kothe, Marta Ma- jewska, Marissa McBride, and James Montilla Doble. Supplemental Material The NIRO-SR tool is available at osf.io/c9wer. All data and supplementary materials have been deposited in an open repository on the Open Science Framework. Relevant links have been provided throughout the paper for access to specific materials. All of these materials are hosted on the open NIRO-SR OSF page (osf.io/f3brw; Topor et al., 2023). Conflict of Interest and Funding No funding has been received for the realisation of this project. Jade Pickering was on the advisory board at Meta-Psychology at the point of submission. No other authors declare any conflicts of interest. Author Contact Corresponding author: Marta K. Topor, marta.topor@hotmail.co.uk (ORCID: https://orcid.org/0000-0003-3761-392X). MKT and JSP contributed equally and are joint first authors. Open Science Practices This article is purely theoretical and as such is not eligible for Open Science badges. The entire editorial process, including the open reviews, is published in the online supplement. References Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Jour- nal article reporting standards for quantitative research in psychology: The APA publications and communications board task force report. The American Psychologist, 73(1), 3–25. https : //doi.org/10.1037/amp0000191 Barba, L. A. (2018). Terminologies for reproducible re- search. arXiv. Retrieved October 30, 2020, from https://arxiv.org/abs/1802.03311 https://osf.io/f3brw/ https://credit.niso.org/ https://osf.io/c9wer https://osf.io/f3brw https://doi.org/10.1037/amp0000191 https://doi.org/10.1037/amp0000191 https://arxiv.org/abs/1802.03311 10 Booth, A., Clarke, M., Dooley, G., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2012). The nuts and bolts of PROSPERO: An international prospective register of systematic reviews. Sys- tematic Reviews, 1, 2. https://doi.org/10.1186/ 2046-4053-1-2 Booth, A., Noyes, J., Flemming, K., Moore, G., Tunçalp, Ö., & Shakibazadeh, E. (2019). Formulat- ing questions to explore complex interven- tions within qualitative evidence synthesis. BMJ Global Health, 4(Suppl 1), e001107. https : / / doi.org/10.1136/bmjgh-2018-001107 Bramer, W. M. (2015). Patient, intervention, compari- son, outcome (PICO): An overrated tool. MLA News, 55(2). Campbell, M., McKenzie, J. E., Sowden, A., Katikireddi, S. V., Brennan, S. E., Ellis, S., Hartmann-Boyce, J., Ryan, R., Shepperd, S., Thomas, J., Welch, V., & Thomson, H. (2020). Synthesis with- out meta-analysis (SWiM) in systematic re- views: Reporting guideline. BMJ (Clinical Re- search Ed.), 368, l6890. https : / / doi . org / 10 . 1136/bmj.l6890 Coeytaux, R. R., McDuffie, J., Goode, A., Cassel, S., Porter, W. D., Sharma, P., Meleth, S., Minnella, H., Nagi, A., & John W Williams, J. (2014). Criteria used in quality assessment of system- atic reviews (tech. rep.). Department of Veter- ans Affairs (US). Retrieved March 3, 2021, from https : / / www . ncbi . nlm . nih . gov / books / %7BNBK242394%7D/ Collins, A., Vercammen, A., McBride, M., Carling, C., & Burgman, M. (n.d.). Reproducibility of system- atic reviews in environmental and conservation science. Committee of Medical Journal Editors. (2021). Clini- cal trials. Retrieved March 3, 2020, from http: //www.icmje.org/recommendations/browse/ publishing - and - editorial - issues / clinical - trial - registration.html Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: The SPIDER tool for qualitative evidence synthesis. Qualitative Health Research, 22(10), 1435–1443. https : / / doi . org / 10 . 1177 / 1049732312452938 Critical Appraisal Skills Program. (n.d.). CASP system- atic review checklist. Retrieved December 2, 2020, from https : / / casp - uk . net / casp - tools - checklists/ DeHaven, A. (2017). Preregistration: A plan, not a prison. Retrieved March 3, 2020, from https : //www.cos.io/blog/preregistration- plan- not- prison Gates, N. J., & March, E. G. (2016). A neuropsy- chologist’s guide to undertaking a systematic review for publication: Making the most of PRISMA guidelines. Neuropsychology Review, 26(2), 109–120. https : / / doi . org / 10 . 1007 / s11065-016-9318-0 Glass, G. V. (1972). The wisdom of scientific inquiry on education. Journal of Research in Science Teach- ing, 9(1), 1–18. https://doi.org/10.1002/tea. 3660090103 Haddaway, N. R., Macura, B., Whaley, P., & Pullin, A. S. (2018). ROSES RepOrting standards for sys- tematic evidence syntheses: Pro forma, flow- diagram and descriptive summary of the plan and conduct of environmental systematic re- views and systematic maps. Environmental Ev- idence, 7(1), 7. https : / / doi . org / 10 . 1186 / s13750-018-0121-7 Higgins, J. P., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (Eds.). (2019). Cochrane handbook for systematic reviews of in- terventions (Version 6). Cochrane. https://doi. org/10.1002/9781119536604 Hohn, R. E., Slaney, K. L., & Tafreshi, D. (2020). An em- pirical review of research and reporting prac- tices in psychological meta-analyses. Review of General Psychology, 108926802091884. https : //doi.org/10.1177/1089268020918844 Ioannidis, J. P. A. (2016). The mass production of re- dundant, misleading, and conflicted systematic reviews and meta-analyses. The Milbank Quar- terly, 94(3), 485–514. https : / / doi . org / 10 . 1111/1468-0009.12210 Joanna Briggs Institute. (n.d.). Critical appraisal tools. Retrieved December 2, 2020, from https : / / joannabriggs.org/critical-appraisal-tools Maassen, E., van Assen, M. A. L. M., Nuijten, M. B., Olsson-Collentine, A., & Wicherts, J. M. (2020). Reproducibility of individual effect sizes in meta-analyses in psychology. Plos One, 15(5), e0233107. https://doi.org/10.1371/journal. pone.0233107 Macpherson, A., & Jones, O. (2010). Editorial: Strate- gies for the development of international jour- nal of management reviews. Wiley. https://doi. org/10.1111/j.1468-2370.2010.00282.x Methodological expectations of campbell collaboration intervention reviews: Conduct standards (tech. rep.). (2019). The Campbell Collaboration. https://doi.org/10.4073/cpg.2016.3 Methodological expectations of campbell collaboration in- tervention reviews: Reporting standards (tech. https://doi.org/10.1186/2046-4053-1-2 https://doi.org/10.1186/2046-4053-1-2 https://doi.org/10.1136/bmjgh-2018-001107 https://doi.org/10.1136/bmjgh-2018-001107 https://doi.org/10.1136/bmj.l6890 https://doi.org/10.1136/bmj.l6890 https://www.ncbi.nlm.nih.gov/books/%7BNBK242394%7D/ https://www.ncbi.nlm.nih.gov/books/%7BNBK242394%7D/ http://www.icmje.org/recommendations/browse/publishing-and-editorial-issues/clinical-trial-registration.html http://www.icmje.org/recommendations/browse/publishing-and-editorial-issues/clinical-trial-registration.html http://www.icmje.org/recommendations/browse/publishing-and-editorial-issues/clinical-trial-registration.html http://www.icmje.org/recommendations/browse/publishing-and-editorial-issues/clinical-trial-registration.html https://doi.org/10.1177/1049732312452938 https://doi.org/10.1177/1049732312452938 https://casp-uk.net/casp-tools-checklists/ https://casp-uk.net/casp-tools-checklists/ https://www.cos.io/blog/preregistration-plan-not-prison https://www.cos.io/blog/preregistration-plan-not-prison https://www.cos.io/blog/preregistration-plan-not-prison https://doi.org/10.1007/s11065-016-9318-0 https://doi.org/10.1007/s11065-016-9318-0 https://doi.org/10.1002/tea.3660090103 https://doi.org/10.1002/tea.3660090103 https://doi.org/10.1186/s13750-018-0121-7 https://doi.org/10.1186/s13750-018-0121-7 https://doi.org/10.1002/9781119536604 https://doi.org/10.1002/9781119536604 https://doi.org/10.1177/1089268020918844 https://doi.org/10.1177/1089268020918844 https://doi.org/10.1111/1468-0009.12210 https://doi.org/10.1111/1468-0009.12210 https://joannabriggs.org/critical-appraisal-tools https://joannabriggs.org/critical-appraisal-tools https://doi.org/10.1371/journal.pone.0233107 https://doi.org/10.1371/journal.pone.0233107 https://doi.org/10.1111/j.1468-2370.2010.00282.x https://doi.org/10.1111/j.1468-2370.2010.00282.x https://doi.org/10.4073/cpg.2016.3 11 rep.). (2019). The Campbell Collaboration. https://doi.org/10.4073/cpg.2016.4 Miller, J. (2002). The scottish intercollegiate guidelines network (SIGN). The British Journal of Diabetes & Vascular Disease, 2(1), 47–49. https : / / doi . org/10.1177/14746514020020010401 Moher, D., Liberati, A., Tetzlaff, J., Altman, D., & Group, P. (2009). Preferred reporting items for system- atic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097. https : / / doi . org / 10 . 1371 / journal . pmed . 1000097 Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & Group, P.-P. (2015). Preferred reporting items for systematic review and meta-analysis pro- tocols (PRISMA-p) 2015 statement. Systematic Reviews, 4(1), 1. https : / / doi . org / 10 . 1186 / 2046-4053-4-1 Munafò, M. R., Nosek, B. A., Bishop, D. V. M., But- ton, K. S., Chambers, C. D., du Sert, N. P., Si- monsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. https://doi.org/10.1038/s41562-016- 0021 National Heart, Lung, and Blood Institute. (n.d.). Study quality assessment tools. Retrieved December 2, 2020, from https : / / www. nhlbi . nih . gov / health-topics/study-quality-assessment-tools Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mel- lor, D. T. (2018). The preregistration revolu- tion. Proceedings of the National Academy of Sci- ences of the United States of America, 115(11), 2600–2606. https : / / doi . org / 10 . 1073 / pnas . 1708274114 Oxman, A. D., & Guyatt, G. H. (1991). Validation of an index of the quality of review articles. Journal of Clinical Epidemiology, 44(11), 1271–1278. https : / / doi . org / 10 . 1016 / 0895 - 4356(91 ) 90160-b Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hrób- jartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., . . . Moher, D. (2021). The PRISMA 2020 statement: An up- dated guideline for reporting systematic re- views. Systematic Reviews, 10(1), 89. https : / / doi.org/10.1186/s13643-021-01626-4 Page, M. J., & Moher, D. (2017). Evaluations of the uptake and impact of the preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement and extensions: A scoping review. Systematic Reviews, 6(1), 263. https:// doi.org/10.1186/s13643-017-0663-8 Polanin, J. R., Hennessy, E. A., & Tsuji, S. (2020). Trans- parency and reproducibility of meta-analyses in psychology: A meta-review. Perspectives on Psy- chological Science, 15(4), 1026–1041. https:// doi.org/10.1177/1745691620906416 Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., & Group, P.-S. (2021). PRISMA-s: An extension to the PRISMA statement for reporting litera- ture searches in systematic reviews. Systematic Reviews, 10(1), 39. https://doi.org/10.1186/ s13643-020-01542-z Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., & Henry, D. A. (2017). AMSTAR 2: A critical appraisal tool for sys- tematic reviews that include randomised or non-randomised studies of healthcare interven- tions, or both. BMJ (Clinical Research Ed.), 358, j4008. https://doi.org/10.1136/bmj.j4008 Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019). How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annual Review of Psychology, 70, 747–770. https : / / doi . org / 10 . 1146 / annurev - psych - 010418-102803 Stroup, D. F., Berlin, J. A., Morton, S. C., Olkin, I., Williamson, G. D., Rennie, D., Moher, D., Becker, B. J., Sipe, T. A., & Thacker, S. B. (2000). Meta-analysis of observational stud- ies in epidemiology: A proposal for reporting. meta-analysis of observational studies in epi- demiology (MOOSE) group. The Journal of the American Medical Association, 283(15), 2008– 2012. https://doi.org/10.1001/jama.283.15. 2008 Topor, M., Pickering, J. S., Barbosa Mendes, A., Bishop, D. V. M., Büttner, F. C., Elsherif, M. M., Evans, T. R., Henderson, E. L., Kalandadze, T., Nitschke, F. T., Staaks, J., van den Akker, O., Yeung, S. K., Zaneva, M., Lam, A., Madan, C., Moreau, D., O’Mahony, A., Parker, A. J., . . . Westwood, S. (2023). Non-interventional, re- producible, and open systematic review (niro- sr) guidelines. Retrieved March 27, 2023, from https://osf.io/f3brw/ Watts, R. D., & Li, I. W. (2019). Use of checklists in re- views of health economic evaluations, 2010 to https://doi.org/10.4073/cpg.2016.4 https://doi.org/10.1177/14746514020020010401 https://doi.org/10.1177/14746514020020010401 https://doi.org/10.1371/journal.pmed.1000097 https://doi.org/10.1371/journal.pmed.1000097 https://doi.org/10.1186/2046-4053-4-1 https://doi.org/10.1186/2046-4053-4-1 https://doi.org/10.1038/s41562-016-0021 https://doi.org/10.1038/s41562-016-0021 https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools https://doi.org/10.1073/pnas.1708274114 https://doi.org/10.1073/pnas.1708274114 https://doi.org/10.1016/0895-4356(91)90160-b https://doi.org/10.1016/0895-4356(91)90160-b https://doi.org/10.1186/s13643-021-01626-4 https://doi.org/10.1186/s13643-021-01626-4 https://doi.org/10.1186/s13643-017-0663-8 https://doi.org/10.1186/s13643-017-0663-8 https://doi.org/10.1177/1745691620906416 https://doi.org/10.1177/1745691620906416 https://doi.org/10.1186/s13643-020-01542-z https://doi.org/10.1186/s13643-020-01542-z https://doi.org/10.1136/bmj.j4008 https://doi.org/10.1146/annurev-psych-010418-102803 https://doi.org/10.1146/annurev-psych-010418-102803 https://doi.org/10.1001/jama.283.15.2008 https://doi.org/10.1001/jama.283.15.2008 https://osf.io/f3brw/ 12 2018. Value in Health, 22(3), 377–382. https : //doi.org/10.1016/j.jval.2018.10.006 Whiting, P., Savović, J., Higgins, J. P. T., Caldwell, D. M., Reeves, B. C., Shea, B., Davies, P., Kleijnen, J., Churchill, R., & group, R. (2016). ROBIS: A new tool to assess risk of bias in systematic re- views was developed. Journal of Clinical Epi- demiology, 69, 225–234. https : / / doi . org / 10 . 1016/j.jclinepi.2015.06.005 https://doi.org/10.1016/j.jval.2018.10.006 https://doi.org/10.1016/j.jval.2018.10.006 https://doi.org/10.1016/j.jclinepi.2015.06.005 https://doi.org/10.1016/j.jclinepi.2015.06.005 13 Appendix The list of guidelines used to extract items for curation and preparation of NIRO-SR. 517 items have been extracted verbatim from the guidelines below: • AMSTAR systematic review quality checklist (Shea et al., 2017) • CASP Checklist for Systematic Reviews (Critical Appraisal Skills Program, n.d.) • Criteria Used in Quality Assessment of Systematic Reviews (Coeytaux et al., 2014) • Joanna Briggs Institute Checklist for Systematic Reviews (Joanna Briggs Institute, n.d.) • MECCIR: Conduct standards (Methodological expectations of Campbell Collaboration intervention reviews: Con- duct standards, 2019) • MECCIR: Reporting standards (Methodological expectations of Campbell Collaboration intervention reviews: Re- porting standards, 2019) • MOOSE: Reporting guidelines for Meta-analysis of Observational Studies in Epidemiology (Stroup et al., 2000) • National Heart Lung and Blood Institute Checklist for Systematic Reviews (National Heart, Lung, and Blood Institute, n.d.) • Overview Quality Assessment Questionnaire (Oxman and Guyatt, 1991) • PRISMA Protocols (Moher et al., 2015) • PRISMA Statement (Moher et al., 2009) • PRISMA 2020 update item bank (Page et al., 2021) • PROSPERO (Booth et al., 2012) • Reproducibility of systematic reviews in environmental and conservation science (Collins et al., n.d.) • ROBIS: Tool to assess risk of bias in systematic reviews (1.2) (Whiting et al., 2016) • ROSES (Haddaway et al., 2018) • SIGN Tool based on AMSTAR (Miller, 2002) • SPIDER - alternative to PICO for qualitative and mixed research (Cooke et al., 2012) • Synthesis without meta-analysis (SWiM) in systematic reviews (Campbell et al., 2020) An additional 55 relevant items were extracted from close inspection of the PRISMA 2020 update item bank (https://osf.io/kbj6v/, Page et al., 2021) which included a number of additional tools and guidelines used across different fields. https://osf.io/kbj6v/