Paths towards greater consensus building in experimental biology

Tackling the myriad of problems facing global biodiversity resulting from a changing climate increasingly requires an integrative, cross-disciplinary, and forward-thinking approach (Franklin and Hoppeler, 2021; Hof, 2021). Studying the impacts of climate change on organisms was once firmly in the realm of ecologists, but researchers from all disciplines are now called upon to bring their expertise forward to address this urgent, global concern. The last two decades have seen an increasing number of comparative physiologists and behavioural ecologists (hereafter ‘experimental biologists’) design studies that propose and test mechanisms underlying the impacts of global change, such as warming temperatures and ocean acidification (OA), on a range of organisms (Schulte, 2015; Tresguerres and Hamilton, 2017). These contributions have advanced our proximate understanding of the challenges posed by climate change to wild organisms, especially aquatic ectotherms, and are now being used by conservation scientists and practitioners to predict the impacts of future change on areas and species most at risk (Cooke et al., 2017; Stillman, 2019). Despite experimental biologists embracing this new role, it remains unclear how ongoing debates and conflicts in our understanding and interpretation of important climate-related phenomena should be resolved.

In a recent editorial in the Journal of Experimental Biology, Franklin and Hoppeler (2021) argued that three practices – consensus building, data sharing, and cross-disciplinary integration – are essential to elucidate mechanisms that will help scientists forecast the effects of global change. Here, we expand on the challenges and benefits of consensus building to resolve uncertainty with a focus on three environmental change paradigms currently being debated by experimental biologists: the oxygen- and capacity-limited thermal tolerance (OCLTT) hypothesis, gill-oxygen limitation (GOL) hypothesis, and OA-driven behavioural changes (see Box 1). We discuss practical strategies for improving cross-disciplinary integration and transparency in research that could help consensus building among experimental biologists conducting global change research.

Understanding how environmental change affects organisms (e.g. Franklin and Hoppeler, 2021) and using these insights to inform conservation strategies (e.g. Cooke et al., 2014) is a process that requires consensus building among scientists. Scientific consensus can be understood as agreement on a topic by most scientists in a particular field of study at a given point in time (Pinch and Bijker, 1984). Ideally, agreement is achieved through the contributions of many different research groups followed by open debate within the relevant scientific community (Shwed and Bearman, 2010). Debate necessarily stems from disagreement, which is a common phenomenon in science (Lamers et al., 2021; Oreskes, 2004) and an essential step in generating knowledge that is both unbiased and reliable (cf. falsification; Popper, 1959).

Researchers studying a phenomenon may disagree for many reasons, including diverging opinions over the suitability of methodologies, the quality of datasets, or the interpretation of results (Dieckmann and Johnson, 2019; Hof, 2021). Provided they are respectful and done in good faith, disagreement and debate over scientific observations and their underlying mechanisms are healthy parts of the scientific process. Indeed, consensus can be built through attempting to understand and improve on weaknesses, biases and misinterpretations in the published literature; for example, by spurring inter- and/or multi-laboratory collaborations (see ‘Cross-disciplinary integration’, below) and improving how studies are designed and reported (see ‘Transparency and reproducibility’, below).

One key challenge to consensus building has been the bias of scientific journals towards the publication of ‘positive’ results (findings that support a hypothesis) (Loehle, 1987). Such publication bias hampers debate by preventing the dissemination of contradictory evidence that does not support a hypothesis (the ‘file drawer effect’) (Mehta, 2019; Nissen et al., 2016). As a result, publishing trends that often follow the discovery of an exciting, new result can give the impression that there is consensus on a topic when, in reality, negative results are simply less likely to be published (Browman, 2016; Fanelli, 2012) or attract reader attention due to publication in lower-profile journals. Decline effects – the tendency of strong initial scientific findings to lose strength over time – are an acute symptom of this problem (Clements et al., 2022; Schooler, 2011), but often go unnoticed to those not intimately familiar with the field of research in question.

Another challenge to consensus building is the process of validating results through replication studies (preferably transparently conducted and reported – see ‘Transparency and reproducibility’, below). Until recently, replications have been ostensibly devalued by the scientific community because funding agencies and journals have a long-standing record of prioritizing novelty over validation of published results (Brembs, 2019). Consequently, independent replication studies are difficult to fund and publish, with few incentives for researchers to undertake them (Fraser et al., 2019). In ecology and evolution, for instance, only 0.02% of published papers are described by their authors as replications (28,000 papers examined across 160 journals; Kelly, 2019). In reality, the percentage of papers that implicitly contain some form of conceptual replication is much higher than 0.02%, but researchers typically emphasize novelty rather than present their work as a replication. The importance of validating both novel and established findings is clear, considering the many instances of foundational studies failing to replicate in fields such as psychology, medicine, economics and biology (Ritchie, 2020). Fortunately, recent work has contributed to identifying experimental factors that increase replicability, namely large sample sizes, and method and experimenter heterogeneity (Milcu et al., 2018; Usui et al., 2021; Voelkl et al., 2018; Voelkl et al., 2021). As such, the replication potential and generality of highly controlled experimental studies is likely to be limited, and their results should be interpreted accordingly.

Faced with these realizations, the scientific community is increasingly acknowledging the need for journals and funders to foster healthy scientific debate by valuing negative results (i.e. avoiding publication bias) and by supporting efforts to validate published findings (i.e. supporting replication studies). For example, in a 2017 editorial, the journal Nature explicitly acknowledged the importance of independent replications and flagged missing information in papers as a major impediment to the self-correcting nature of science (Nature Editorial, 2017). In the Netherlands, funders are leading in this regard and are taking a strong stance towards promoting research transparency and replication: in 2018, the country's main funding body published an advisory report arguing that more funds must be allotted to replication studies, and that researchers who undertake replication efforts must be given greater credit by their institutions (Royal Netherlands Academy of Arts and Sciences, 2018).

Replications are beginning to be valued more highly than in the past and are an important part of resolving scientific debates. However, debates themselves have benefits and drawbacks, depending on how they unfold. For example, debates about the OCLTT, GOL and OA-driven behavioural change, among others, have led to stalemates rather than efforts by interested parties to seek consensus (e.g. Clark et al., 2020a; Munday et al., 2014b; Pörtner et al., 2018; Williamson et al., 2021). Various reasons might explain this outcome, and the failure of debates to achieve a constructive and open-minded tone is not unique to these examples, this generation of scientists, or to the natural sciences specifically (e.g. the Bone wars in paleontology in the late 19th century, or Monetarist–Keynesian debates in economics). For example, researchers setting out to independently validate or replicate others’ results may be perceived negatively by the original study researchers and other scientists (Vazire, 2019). In these situations, researchers with valid scientific intentions risk being portrayed as contrarians (Jamieson et al., 2019). Ultimately, however, generating, (re)testing, and challenging hypotheses are all essential and legitimate scientific endeavours (Merton, 1973; Oreskes, 2004). Scientists of all career stages should welcome transparent, independent replication of their work. Each of these steps must be done in good faith and in a spirit of openness and transparency to ensure that the information produced is robust and credible (Browman, 2016; Song et al., 2021).

Scientists must remind themselves that the fundamental objective of research is to advance knowledge (O'Dea et al., 2021). A productive debate is one that produces testable hypotheses and trusted results through rigorous and transparent empirical evidence (Jamieson et al., 2019). As a general rule, consensus building is possible when proponents and detractors of scientific ideas abide by this principle.

Recognition that climates are changing, species are disappearing, and organismal ranges are shifting has led to the emergence of new fields such as global change biology, conservation physiology, and conservation behaviour. These fields explicitly recognize the importance of mechanistic approaches to understand how organisms are affected by human activities and to inform policy and practice for conserving them (Buchholz, 2007; Cooke et al., 2014). Methods in experimental biology are increasingly being applied to address complex questions across disciplines and levels of biological organization (Hof, 2021). This integration allows researchers to link broad biological patterns with underlying physiological mechanisms to better understand and predict how organisms will respond to environmental change (Pörtner et al., 2006; Stillman, 2019).

Despite these benefits, cross-disciplinary integration also has risks. One risk is researchers venturing into new fields without the necessary expertise, which can lead to poorly designed experiments, inappropriate methods, and misinterpreted results. The growing number of papers, special issues and journals dedicated to method descriptions and validations speaks to the frustration felt by many experimental biologists at the poor execution and reporting of many experiments in global change biology (e.g. Chabot et al., 2016; Clark et al., 2013; Cornwall and Hurd, 2016; Jutfelt et al., 2017; Killen et al., 2021; Moran, 2014; Roche et al., 2020). In the context of the three paradigms discussed in Box 1, debates between their proponents and detractors have highlighted the need for more rigorous and collaborative interdisciplinary work to inform research questions.

Firstly, cross-disciplinary integration can be problematic when mechanistic explanations developed to test ideas in a specific field of study are then generalized broadly across disciplines. For example, the OCLTT hypothesis was originally formulated to make predictions about the metabolic performance of ectotherms as a function of temperature; however, it has since considerably expanded in scope, with proponents suggesting that a wide range of taxa, including endotherms, humans and their societies, may follow the same general rules (Pörtner, 2021). Despite the appeal of OCLTT as a simple, over-arching mechanism, critics have argued that the concepts and terminology at its core are becoming increasingly vague and untestable (Jutfelt et al., 2018). Resolving this debate will benefit from greater cross-disciplinary integration and ‘building bridges between disciplines such as physiology and ecology, to develop more precise conclusions and to avoid conceptual ambiguities’ (Pörtner, 2021).

A second risk of cross-disciplinary integration is when potentially erroneous mechanistic explanations are put forth by non-specialists as a ‘silver bullet’ to explain complex global phenomena. For instance, the GOL hypothesis was developed in the field of fisheries research to explain observed trends in fish size from principles of fish morphology and physiology. Although the idea has received considerable attention in high-impact, generalist journals and from fisheries biologists, the underlying principles have been questioned by fish physiologists, highlighting a lack of effective cross-disciplinary integration (Box 1). Detractors of the GOL hypothesis argue that adopting a rigorous interdisciplinary approach – in this case contributing a fundamental understanding of physiological principles to fisheries research – can inform whether climate-related hypotheses such as GOL are scientifically sound and worth pursuing (Lefevre et al., 2021).

Third, cross-disciplinary integration can be problematic when researchers apply methods and techniques from many different fields to test hypotheses without involving experts from each field or using methodological best practices. For example, research examining the effects of OA on fish behaviour (particularly in coral reef fishes) spans multiple disciplines including behavioural ecology (Munday et al., 2014a; Welch et al., 2014), ecophysiology (Laubenstein et al., 2019; Munday et al., 2009a), sensory biology (Chung et al., 2014; Ferrari et al., 2012; Munday et al., 2009b), neurobiology (Chivers et al., 2014; Heuer et al., 2016; Schunter et al., 2019), chemical ecology (Dixson et al., 2010; Ferrari et al., 2011) and molecular biology (Schunter et al., 2018; Tsang et al., 2020). Thus, the potential for cross-disciplinary integration in this field is considerable, with researchers having worked to gain a broad understanding of OA-related patterns and mechanisms. Yet, despite the benefits of this integrative approach, concerns have been raised over the rigour and reliability of experimental studies in the field (e.g. Browman, 2016; Clark et al., 2020a). These issues include flawed experimental designs (Cornwall and Hurd, 2016), inaccurate water chemistry measurements (Moran, 2014), lack of video recording and automated tracking (to remove experimenter bias and increase reproducibility; Clark, 2017), flawed methodologies (e.g. closed respirometry and unreliable choice tests; Clark et al., 2013; Jutfelt et al., 2017), and inadequate statistical analyses (Roche et al., 2020). These problems can (and should) be avoided by collaborating with researchers with relevant expertise and/or investing time and effort to adequately learn methods used in other fields.

Arguably, the most effective way to engage in cross-disciplinary integration is to mindfully seek out collaborations with researchers possessing complementary skill sets (Hof, 2021). This sounds easy in theory; however, in practice, forging new collaborations with experts can be difficult, in part because opportunities to interact with researchers across disciplines and geographical regions are limited. Scientific conferences are often specialized, and thus not attended by researchers across disciplines. Large conferences may attract more participants, but talk sessions still often revolve around a narrow theme of interest. As a result, researchers from different disciplines often attend different conferences or conference sessions, read and publish in different journals, and use their own field-specific jargon when referring to similar phenomena. In addition, attending international conferences comes with a large price tag, which often limits participation from researchers from under-funded institutions and countries (Sarabipour et al., 2021). Fortunately, the recent and rapid rise in communications technologies, particularly during the COVID-19 pandemic, has normalized virtual meetings and unconferences (e.g. hackathons), allowing researchers to more easily reach out to colleagues, exchange ideas, and develop diverse networks of mentors and collaborators from across a range of geographical regions (Huppenkothen et al., 2018; Korbel and Stegle, 2020; Sarabipour et al., 2021).

Several other avenues are available for experimental biologists to engage in cross-disciplinary research when collaboration is not possible. The emergence of open platforms dedicated to sharing methodological information (e.g. protocols.io, Protocol Exchange, STAR protocols), the release of open software to acquire and process data (e.g. Harianto et al., 2019; Sridhar et al., 2019), and a growing number of articles outlining methods and reporting guidelines (e.g. Chabot et al., 2016; Killen et al., 2021; Norin and Clark, 2016) all provide readily available resources for researchers to incorporate best practices and new techniques into their skill set (Baker, 2021). While those resources offer useful starting points, it can often be useful to consult directly with experts for detailed technical advice when using a technique or tool for the first time.

Consulting with colleagues is one valid option to obtain feedback, but more formal avenues are now available for researchers seeking independent expert comments on planned experiments before they begin. First, authors can submit a registered report. Registered reports are a publication format currently offered by over 300 journals (https://www.cos.io/initiatives/registered-reports); they focus on the validity of the research question and methodology rather than the study results. Peer review begins prior to the data collection and rigorous protocols are provisionally accepted for publication if the authors follow through with their registered methodology (Parker et al., 2019). Second, authors can hire an independent ‘red team’, the concept of which originates from cyber security, where ethical hackers (the red team) challenge a blue team of security professionals charged with maintaining network defences against cyber attacks. In science, researchers (the blue team) send methods, analyses, or an entire manuscript to a red team service (e.g. https://redteammarket.com), which designates field-specific experts to identify potential weaknesses in the work. Calling on a red team for feedback is more flexible than submitting a registered report because users pay a fee to receive an independent, expert evaluation of their work at any point in the study development. Whereas registered reports are intended to evaluate a research project from start to finish, the red team approach allows users to seek advice on any component of their project (e.g. experimental design, statistical analyses, laboratory manipulations) based on their specific needs and budget. In both cases, having research protocols expertly assessed prior to data collection helps researchers who venture outside their field of expertise to ensure that their methods are rigorous and appropriate to answer their question of interest.

Ultimately, engaging in effective cross-disciplinary integration requires that researchers be mindful of their own expertise limits and, when possible, seek out knowledgeable collaborators when venturing outside their field. Open science initiatives, such as open protocols and software, registered reports, and red teams, can facilitate integrative research and consensus building by promoting rigorous, objective and transparent research. Some of our proposed solutions may be more available to better-funded research teams. As a community, it is important that we recognize how differential access to resources may impact the ability to implement the changes we suggest, and to mindfully develop ways in which we can more actively engage and encourage participation with our under-represented and under-funded colleagues (e.g. Caravaggi et al., 2021; Sarabipour et al., 2021).

Working collaboratively to increase the reliability and impact of studies in experimental biology requires not only more mindful cross-disciplinary integration but also greater transparency throughout the process of data generation, extraction, analysis and interpretation. Experimental biology is an empirical discipline. Thus, debates may be more easily resolved with data. When data are collected and shared transparently, this fosters trust (Jamieson et al., 2019) and helps science advance more rapidly and reliably by allowing independent research groups to more easily understand, validate and build on previous results (Moher et al., 2018; Nosek et al., 2015; Parker et al., 2016). For these reasons, many scientists (O'Dea et al., 2021), funding agencies (Schiermeier, 2018) and journal editors (Bakker and Traniello, 2020; Simmons, 2016) now strongly advocate measures that encourage and/or require greater transparency in published research. This includes publicly sharing data underlying published results (i.e. open data; Caetano and Aisenberg, 2014; Ihle et al., 2017), preferably at the time of peer review so analytical errors can be identified and fixed prior to publication (Fernández-Juricic, 2021). Recent surveys report that 55% of researchers believe funders should make data sharing part of their grant requirements (n>4500 international respondents; Digital Science et al., 2020) and 78% of Canada-based faculty members in ecology and evolution support mandatory open data policies by journals (n=140 respondents at Canadian Universities; Soeharjono and Roche, 2021). This growing support for wider data sharing mandates is spurred, in large part, by the continuing reproducibility crisis in science (Ritchie, 2020) and a sense that, despite the highly competitive nature of academia, a collaborative and open approach to research ultimately benefits both society and individual researchers (Hunt, 2019; McKiernan et al., 2016).

To date, efforts to increase transparency in research have primarily focused on open data, with less attention devoted to other, yet equally important reproducibility materials. One such material is the script or code used for processing and analysing data (i.e. open code) – for example to produce the tables, figures and statistics presented in a paper (Culina et al., 2020; Mislan et al., 2016). Until recently, technological limitations made it challenging for researchers to share the decisions and steps involved in their data analysis. For example, many commonly used statistical software packages rely on a graphical user interface (GUI), making it difficult for users to share and reproduce analyses (e.g. SPSS, JMP, Minitab, SigmaPlot). However, the last two decades have witnessed a surge in the uptake of command-based software to analyse biological data, readily allowing researchers to openly share annotated analysis code (Mislan et al., 2016). In ecology, for example, the use of the R programming language for data analysis increased linearly from 11.4 to 58.0% between 2008 and 2017 (Lai et al., 2019). When combined with open data, open code is a powerful tool for validating results and building on previous studies to answer new and often broader research questions (Barnes, 2010; Stodden et al., 2018).

To examine the contributions of reproducible research practices for advancing consensus building in experimental biology, we investigated the prevalence of open data and code associated with empirical studies on the OCLTT, GOL and OA-driven behavioural change paradigms (Box 1) as well as in eight journals publishing research in comparative physiology (Journal of Experimental Biology, Journal of Experimental Zoology Part A, Physiological and Biochemical Zoology and Comparative Biochemistry and Physiology Part B) and behavioural ecology (Animal Behaviour, Behavioral Ecology, Behavioral Ecology and Sociobiology and Ethology). We identified 532 published studies on the OCLTT hypothesis, 10 on the GOL hypothesis, and 186 on OA-driven behavioural change between 2001 and 2021 (see ‘Methods’, below; note that 66% of studies were published after 2014). Approximately one-fifth (19.0%) of studies we examined on these paradigms had associated open data, and fewer than one in 30 studies (3.2%) had open code (see Table 1 for a breakdown by paradigm). These numbers are slightly higher than the overall availability of open data (12.1%) and comparable to the availability of open code (3.1%) for studies published in the eight leading comparative physiology and behavioural ecology journals between 2010 and 2021 (Table 1; Table S1). In contrast, they are markedly lower than the prevalence of open data (73%) and code (27%) reported for ecological studies on other topics published within a comparable time frame (n=346 studies published in 14 journals that recommend or require open data/code between 2015 and 2019; Culina et al., 2020), highlighting notable differences among disciplines.

Encouragingly, our data show that the percentage of empirical papers with open data in comparative physiology, behavioural ecology, and on the OCLTT hypothesis and OA-driven behavioural change is increasing over time (Figs 1 and 2). Notable differences exist among journals (Fig. 2), but the overall trend suggests that a growing number of experimental biologists working on these topics are complying with editorial open data policies and/or supporting efforts to improve transparency. One caveat to this finding is that not all open data are FAIR (i.e. Findable, Accessible, Interoperable, and Re-usable; Box 2). Indeed, recent studies suggest that approximately 50% of open datasets shared by researchers in ecology and evolution are incomplete or uninterpretable (Roche et al., 2015; Roche et al., 2021 preprint), which considerably limits transparency and computational reproducibility (i.e. the ability to reproduce a study's results with the associated data and code). This consideration effectively means that fewer than 10% of studies in experimental biology have open data that would allow an independent assessment of their conclusions.

Without access to data and code, readers must rely on the authors’ interpretation of a study's data (Fig. 3). This is problematic for all studies, but particularly in the context of debated topics because transparent and trusted evidence is critical for consensus building (see Jamieson et al., 2019). Although open data is increasingly the norm in many disciplines in biology, it remains a relatively new practice. As a result, many more established researchers have not received formal training in best practices for archiving high quality, FAIR data. Research on data sharing suggests that journal policies lead to more open data (Vines et al., 2013), but that education, training and technical support are important to ensure high quality, FAIR sharing (Soeharjono and Roche, 2021; Roche et al., 2021 preprint). Fortunately, an increasing number of courses and workshops aimed at teaching reproducible research practices are available for researchers at all career stages (Table S2). It is important to note that engaging in research practices that promote transparency and reproducibility can be done in a stepwise fashion. Researchers cannot be expected to go from zero to hero overnight – rather, a sensible approach is to adopt practices with the fewest logistic and monetary hurdles (e.g. open data) and progress from there based on access to resources, training and comfort level (see resources in Box 3). We must also be careful as a community not to shame researchers who engage in open and transparent research practices in good faith if problems or omissions in shared materials are detected. Recognizing that adopting transparent and reproducible research practices can be a steep learning curve is necessary to foster a culture of trust, belonging and pride as experimental biology moves towards increased openness.

Experimental biology is now firmly at the forefront in predicting the ecological impacts of environmental change on organisms. However, debates among researchers on topics such as OCLTT, GOL and OA-driven behavioural change linger and are beginning to show signs of ‘theory tenacity’: the persistent belief in a theory even in the face of contradictory evidence (Duarte et al., 2015; Loehle, 1987). Rather than embodying the open mind expected of scientists, ‘theory tenacity tends to make opposing camps dig their trenches deeper’ (Loehle, 1987). Resolving debate and building consensus on these topics requires that scientific discussions and research be undertaken in a spirit of good faith, with the objective of resolving disagreement using robust empirical evidence and best practices in experimental design and analysis (see Hoekstra and Vazire, 2021). Achieving this requires all experimental biologists to engage in mindful cross-disciplinary integration and adopt collaborative research practices that foster transparency and reproducibility. In some instances, the perception that parties involved in a debate are not engaging in good faith can lead to a breakdown in trust and outright conflict. In such situations, when collaboration is no longer possible, transparency and reproducibility become the most important arbiters of debate.

We hope that a greater adoption of open science practices, with a particular focus on FAIR data and code, will lead to a much-needed paradigm shift towards improved transparency, cross-disciplinary integration, and consensus building in experimental biology to maximize the contributions of researchers in addressing the impacts of climate change on organisms.

We investigated the availability of open data and analysis code associated with: (1) studies that contribute to advancing our understanding of three debated paradigms investigating the impacts of climate change on aquatic ectotherms (Box 1), and (2) studies published in leading comparative physiology and behavioural ecology journals between 2000 and 2021 (Table 1).

Regarding the three paradigms of interest, studies were systematically identified via a forward reference search of journal articles citing foundational papers for each of the three paradigms: OCLTT (Deutsch et al., 2015; Frederich and Pörtner, 2000; Pörtner, 2001, 2010, 2012; Pörtner and Farrell, 2008; Pörtner and Knust, 2007; Pörtner and Peck, 2010; Pörtner et al., 2017), GOL (Cheung et al., 2013; Lefevre et al., 2017; Pauly, 1998; Pauly and Cheung, 2018b), and OA-driven behavioural changes (Dixson et al., 2010; Munday et al., 2009b, 2010; Nilsson et al., 2012). The forward searches were conducted in Web of Science on 4 June 2021 and yielded a total of 3541, 543 and 804 citations for each paradigm, respectively, after removing duplicates (Table S3). We used the software Rayyan (https://rayyan.ai; Ouzzani et al., 2016) to narrow our search and identify relevant papers from within each list using keywords present in article titles and abstracts. The keywords used to screen papers are listed in Table S4. When a keyword was present in a paper's title and/or abstract, we manually reviewed the article to determine its suitability for our analysis. An article was deemed suitable when it reported data and results considered to advance our understanding of one of the three paradigms mentioned above. Several articles identified in our forward reference searches contributed relevant and valuable evidence for testing the three paradigms without the authors explicitly stating that they had tested those paradigms. Meta-analyses and theoretical studies using empirical data were included in our analysis; we excluded theoretical studies based on simulations (as these rely on code but not empirical data), and papers where there would be no expectation of underlying data, such as reviews, comments and perspectives.

For studies in comparative physiology and behavioural ecology, we chose four representative journals in each of the two disciplines (Table 1) and examined the availability of data and code for the first 10 articles published each year between 2010 and 2021. We excluded article types such as reviews, perspectives and other types of contributions where there would be no expectation of underlying data.

Once the final list of articles was completed in both cases, we manually screened each article and its associated supplementary material for the presence of: (1) a data availability (or accessibility) statement; (2) open data (i.e. publicly available data that could be inputed into software to produce tables, figures or statistical results); and (3) open code (publicly available script or code used to produce tables, figures or descriptive/inferential statistics). We only recorded the availability of open data and code, not whether these materials were complete (i.e. contained all the variables measured and used in all the analyses presented in a study) and interoperable/easily re-usable (i.e. had associated metadata and were shared in a non-proprietary, machine-readable format).

We thank the many colleagues who participated in discussions that led to this paper, as well as Andrea Morash and an anonymous reviewer for helpful comments on the manuscript. We acknowledge the Traditional Custodians of the unceded lands and waterways on which several of our institutions now operate.