In a recent editorial, the Editors-in-Chief of Journal of Experimental Biology argued that consensus building, data sharing, and better integration across disciplines are needed to address the urgent scientific challenges posed by climate change. We agree and expand on the importance of cross-disciplinary integration and transparency to improve consensus building and advance climate change research in experimental biology. We investigated reproducible research practices in experimental biology through a review of open data and analysis code associated with empirical studies on three debated paradigms and for unrelated studies published in leading journals in comparative physiology and behavioural ecology over the last 10 years. Nineteen per cent of studies on the three paradigms had open data, and 3.2% had open code. Similarly, 12.1% of studies in the journals we examined had open data, and 3.1% had open code. Previous research indicates that only 50% of shared datasets are complete and re-usable, suggesting that fewer than 10% of studies in experimental biology have usable open data. Encouragingly, our results indicate that reproducible research practices are increasing over time, with data sharing rates in some journals reaching 75% in recent years. Rigorous empirical research in experimental biology is key to understanding the mechanisms by which climate change affects organisms, and ultimately promotes evidence-based conservation policy and practice. We argue that a greater adoption of open science practices, with a particular focus on FAIR (Findable, Accessible, Interoperable, Re-usable) data and code, represents a much-needed paradigm shift towards improved transparency, cross-disciplinary integration, and consensus building to maximize the contributions of experimental biologists in addressing the impacts of environmental change on living organisms.
Tackling the myriad of problems facing global biodiversity resulting from a changing climate increasingly requires an integrative, cross-disciplinary, and forward-thinking approach (Franklin and Hoppeler, 2021; Hof, 2021). Studying the impacts of climate change on organisms was once firmly in the realm of ecologists, but researchers from all disciplines are now called upon to bring their expertise forward to address this urgent, global concern. The last two decades have seen an increasing number of comparative physiologists and behavioural ecologists (hereafter ‘experimental biologists’) design studies that propose and test mechanisms underlying the impacts of global change, such as warming temperatures and ocean acidification (OA), on a range of organisms (Schulte, 2015; Tresguerres and Hamilton, 2017). These contributions have advanced our proximate understanding of the challenges posed by climate change to wild organisms, especially aquatic ectotherms, and are now being used by conservation scientists and practitioners to predict the impacts of future change on areas and species most at risk (Cooke et al., 2017; Stillman, 2019). Despite experimental biologists embracing this new role, it remains unclear how ongoing debates and conflicts in our understanding and interpretation of important climate-related phenomena should be resolved.
In a recent editorial in the Journal of Experimental Biology, Franklin and Hoppeler (2021) argued that three practices – consensus building, data sharing, and cross-disciplinary integration – are essential to elucidate mechanisms that will help scientists forecast the effects of global change. Here, we expand on the challenges and benefits of consensus building to resolve uncertainty with a focus on three environmental change paradigms currently being debated by experimental biologists: the oxygen- and capacity-limited thermal tolerance (OCLTT) hypothesis, gill-oxygen limitation (GOL) hypothesis, and OA-driven behavioural changes (see Box 1). We discuss practical strategies for improving cross-disciplinary integration and transparency in research that could help consensus building among experimental biologists conducting global change research.
(A) Oxygen- and capacity-limited thermal tolerance (OCLTT)
The concept of OCLTT proposes a causal link between whole-animal performance (fitness) and the capacity of the cardiorespiratory system to supply tissues with oxygen for physiological functions (Pörtner, 2010). OCLTT is based on the Fry paradigm (Fry, 1947), which describes how temperature affects aerobic scope, using rates of oxygen consumption as a proxy for aerobic metabolism. Aerobic scope is the difference between the maximum oxygen consumption rate and the standard oxygen consumption rate, and it quantifies the oxygen available for physiological functions beyond standard maintenance functions. OCLTT was first proposed as a ‘unifying concept’ in 2001 (Pörtner, 2001). Since then, the proposed causal link between oxygen supply capacity and animal performance has been extended to also include population performance at the ecosystem level (Pörtner, 2010; Pörtner, 2021; Pörtner and Farrell, 2008; Pörtner and Knust, 2007; Pörtner et al., 2017). As a result, OCLTT has been incorporated into ecological species distribution models and reports by the Intergovernmental Panel on Climate Change (Pörtner and Peck, 2010; Pörtner et al., 2014). However, the causal links proposed in OCLTT and the general applicability of the concept across aquatic ectotherms are debated (Clark et al., 2013; Ern et al., 2016; Jutfelt et al., 2018; Norin et al., 2014).
(B) Gill-oxygen limitation (GOL)
The GOL hypothesis (or ‘theory’, GOLT) was first proposed decades ago (Pauly, 1981), but interest in the concept has increased in recent years as a potentially universal explanation for the observed decline in body size of fishes and other aquatic ectotherms with warming (termed the temperature–size rule, TSR; Angilletta and Dunham, 2003; Audzijonyte et al., 2016; Pauly, 2021; Pauly and Cheung, 2018a). The GOL hypothesis proposes that as aquatic ectotherms, like fishes, get larger (within and across species), their gill surface area is unable to increase sufficiently with their body size and metabolic rate; this inhibits oxygen uptake capacity relative to metabolic requirements, which in turns slows the animal's growth as they get larger (Pauly and Cheung, 2018b; Pauly, 2021). The GOL hypothesis is closely related to the OCLTT hypothesis in that they both posit that the amount of oxygen supply to tissues is an important mechanism that determines fitness, with a specific focus on growth in the case of GOL. However, comparative physiologists have argued that there is sufficient evidence in the physiological literature to reject the GOL hypothesis (Lefevre et al., 2017). Subsequent papers have continued the discussion about the merits of the GOL hypothesis (Audzijonyte et al., 2019; Lefevre et al., 2018; Marshall and White, 2019; Pauly, 2021; Pauly and Cheung, 2018a,b), and researchers have since begun directly and indirectly testing the hypothesis using new experiments and previously collected data (Bigman et al., 2021; Christensen et al., 2020; Meyer and Schill, 2021; Scheuffele et al., 2021; Shapiro Goldberg et al., 2019).
(C) Ocean acidification (OA)-driven behavioural change
After decades of research showing that fish were resilient to very elevated PCO2 levels (e.g. 10 times end-of-century forecasts) (Ishimatsu et al., 2005; Melzner et al., 2009), renewed interest in this field grew from a series of publications reporting profoundly maladaptive effects of short-term exposures to end-of-century PCO2 (∼1000 μatm) on the behaviour of coral reef fishes, particularly homing and predator avoidance (Dixson et al., 2010; Munday et al., 2009a; Munday et al., 2010). In subsequent years, mechanistic drivers of these maladaptive behavioural effects were proposed, including alterations to GABAA neurotransmission and/or olfactory organ physiology (Leduc et al., 2013; Nilsson et al., 2012), along with molecular changes to chemical cues in the environment (Roggatz et al., 2016). More recent studies have cast doubt on the severity and universality of acidification effects on fish behaviour (Clark et al., 2020a), and a decade of work in the field exhibits signs of a ‘decline effect’, whereby the strong initial effect sizes reported in early studies have all but disappeared over the past 5 years (Clements et al., 2022). The extent and degree to which aquatic acidification affects the behaviour of ectotherms remains a debate in the field (Clark et al., 2020b; Munday et al., 2020; Williamson et al., 2021).
Building consensus in experimental biology
Understanding how environmental change affects organisms (e.g. Franklin and Hoppeler, 2021) and using these insights to inform conservation strategies (e.g. Cooke et al., 2014) is a process that requires consensus building among scientists. Scientific consensus can be understood as agreement on a topic by most scientists in a particular field of study at a given point in time (Pinch and Bijker, 1984). Ideally, agreement is achieved through the contributions of many different research groups followed by open debate within the relevant scientific community (Shwed and Bearman, 2010). Debate necessarily stems from disagreement, which is a common phenomenon in science (Lamers et al., 2021; Oreskes, 2004) and an essential step in generating knowledge that is both unbiased and reliable (cf. falsification; Popper, 1959).
Researchers studying a phenomenon may disagree for many reasons, including diverging opinions over the suitability of methodologies, the quality of datasets, or the interpretation of results (Dieckmann and Johnson, 2019; Hof, 2021). Provided they are respectful and done in good faith, disagreement and debate over scientific observations and their underlying mechanisms are healthy parts of the scientific process. Indeed, consensus can be built through attempting to understand and improve on weaknesses, biases and misinterpretations in the published literature; for example, by spurring inter- and/or multi-laboratory collaborations (see ‘Cross-disciplinary integration’, below) and improving how studies are designed and reported (see ‘Transparency and reproducibility’, below).
One key challenge to consensus building has been the bias of scientific journals towards the publication of ‘positive’ results (findings that support a hypothesis) (Loehle, 1987). Such publication bias hampers debate by preventing the dissemination of contradictory evidence that does not support a hypothesis (the ‘file drawer effect’) (Mehta, 2019; Nissen et al., 2016). As a result, publishing trends that often follow the discovery of an exciting, new result can give the impression that there is consensus on a topic when, in reality, negative results are simply less likely to be published (Browman, 2016; Fanelli, 2012) or attract reader attention due to publication in lower-profile journals. Decline effects – the tendency of strong initial scientific findings to lose strength over time – are an acute symptom of this problem (Clements et al., 2022; Schooler, 2011), but often go unnoticed to those not intimately familiar with the field of research in question.
Another challenge to consensus building is the process of validating results through replication studies (preferably transparently conducted and reported – see ‘Transparency and reproducibility’, below). Until recently, replications have been ostensibly devalued by the scientific community because funding agencies and journals have a long-standing record of prioritizing novelty over validation of published results (Brembs, 2019). Consequently, independent replication studies are difficult to fund and publish, with few incentives for researchers to undertake them (Fraser et al., 2019). In ecology and evolution, for instance, only 0.02% of published papers are described by their authors as replications (28,000 papers examined across 160 journals; Kelly, 2019). In reality, the percentage of papers that implicitly contain some form of conceptual replication is much higher than 0.02%, but researchers typically emphasize novelty rather than present their work as a replication. The importance of validating both novel and established findings is clear, considering the many instances of foundational studies failing to replicate in fields such as psychology, medicine, economics and biology (Ritchie, 2020). Fortunately, recent work has contributed to identifying experimental factors that increase replicability, namely large sample sizes, and method and experimenter heterogeneity (Milcu et al., 2018; Usui et al., 2021; Voelkl et al., 2018; Voelkl et al., 2021). As such, the replication potential and generality of highly controlled experimental studies is likely to be limited, and their results should be interpreted accordingly.
Faced with these realizations, the scientific community is increasingly acknowledging the need for journals and funders to foster healthy scientific debate by valuing negative results (i.e. avoiding publication bias) and by supporting efforts to validate published findings (i.e. supporting replication studies). For example, in a 2017 editorial, the journal Nature explicitly acknowledged the importance of independent replications and flagged missing information in papers as a major impediment to the self-correcting nature of science (Nature Editorial, 2017). In the Netherlands, funders are leading in this regard and are taking a strong stance towards promoting research transparency and replication: in 2018, the country's main funding body published an advisory report arguing that more funds must be allotted to replication studies, and that researchers who undertake replication efforts must be given greater credit by their institutions (Royal Netherlands Academy of Arts and Sciences, 2018).
Replications are beginning to be valued more highly than in the past and are an important part of resolving scientific debates. However, debates themselves have benefits and drawbacks, depending on how they unfold. For example, debates about the OCLTT, GOL and OA-driven behavioural change, among others, have led to stalemates rather than efforts by interested parties to seek consensus (e.g. Clark et al., 2020a; Munday et al., 2014b; Pörtner et al., 2018; Williamson et al., 2021). Various reasons might explain this outcome, and the failure of debates to achieve a constructive and open-minded tone is not unique to these examples, this generation of scientists, or to the natural sciences specifically (e.g. the Bone wars in paleontology in the late 19th century, or Monetarist–Keynesian debates in economics). For example, researchers setting out to independently validate or replicate others’ results may be perceived negatively by the original study researchers and other scientists (Vazire, 2019). In these situations, researchers with valid scientific intentions risk being portrayed as contrarians (Jamieson et al., 2019). Ultimately, however, generating, (re)testing, and challenging hypotheses are all essential and legitimate scientific endeavours (Merton, 1973; Oreskes, 2004). Scientists of all career stages should welcome transparent, independent replication of their work. Each of these steps must be done in good faith and in a spirit of openness and transparency to ensure that the information produced is robust and credible (Browman, 2016; Song et al., 2021).
Scientists must remind themselves that the fundamental objective of research is to advance knowledge (O'Dea et al., 2021). A productive debate is one that produces testable hypotheses and trusted results through rigorous and transparent empirical evidence (Jamieson et al., 2019). As a general rule, consensus building is possible when proponents and detractors of scientific ideas abide by this principle.
Recognition that climates are changing, species are disappearing, and organismal ranges are shifting has led to the emergence of new fields such as global change biology, conservation physiology, and conservation behaviour. These fields explicitly recognize the importance of mechanistic approaches to understand how organisms are affected by human activities and to inform policy and practice for conserving them (Buchholz, 2007; Cooke et al., 2014). Methods in experimental biology are increasingly being applied to address complex questions across disciplines and levels of biological organization (Hof, 2021). This integration allows researchers to link broad biological patterns with underlying physiological mechanisms to better understand and predict how organisms will respond to environmental change (Pörtner et al., 2006; Stillman, 2019).
Despite these benefits, cross-disciplinary integration also has risks. One risk is researchers venturing into new fields without the necessary expertise, which can lead to poorly designed experiments, inappropriate methods, and misinterpreted results. The growing number of papers, special issues and journals dedicated to method descriptions and validations speaks to the frustration felt by many experimental biologists at the poor execution and reporting of many experiments in global change biology (e.g. Chabot et al., 2016; Clark et al., 2013; Cornwall and Hurd, 2016; Jutfelt et al., 2017; Killen et al., 2021; Moran, 2014; Roche et al., 2020). In the context of the three paradigms discussed in Box 1, debates between their proponents and detractors have highlighted the need for more rigorous and collaborative interdisciplinary work to inform research questions.
Firstly, cross-disciplinary integration can be problematic when mechanistic explanations developed to test ideas in a specific field of study are then generalized broadly across disciplines. For example, the OCLTT hypothesis was originally formulated to make predictions about the metabolic performance of ectotherms as a function of temperature; however, it has since considerably expanded in scope, with proponents suggesting that a wide range of taxa, including endotherms, humans and their societies, may follow the same general rules (Pörtner, 2021). Despite the appeal of OCLTT as a simple, over-arching mechanism, critics have argued that the concepts and terminology at its core are becoming increasingly vague and untestable (Jutfelt et al., 2018). Resolving this debate will benefit from greater cross-disciplinary integration and ‘building bridges between disciplines such as physiology and ecology, to develop more precise conclusions and to avoid conceptual ambiguities’ (Pörtner, 2021).
A second risk of cross-disciplinary integration is when potentially erroneous mechanistic explanations are put forth by non-specialists as a ‘silver bullet’ to explain complex global phenomena. For instance, the GOL hypothesis was developed in the field of fisheries research to explain observed trends in fish size from principles of fish morphology and physiology. Although the idea has received considerable attention in high-impact, generalist journals and from fisheries biologists, the underlying principles have been questioned by fish physiologists, highlighting a lack of effective cross-disciplinary integration (Box 1). Detractors of the GOL hypothesis argue that adopting a rigorous interdisciplinary approach – in this case contributing a fundamental understanding of physiological principles to fisheries research – can inform whether climate-related hypotheses such as GOL are scientifically sound and worth pursuing (Lefevre et al., 2021).
Third, cross-disciplinary integration can be problematic when researchers apply methods and techniques from many different fields to test hypotheses without involving experts from each field or using methodological best practices. For example, research examining the effects of OA on fish behaviour (particularly in coral reef fishes) spans multiple disciplines including behavioural ecology (Munday et al., 2014a; Welch et al., 2014), ecophysiology (Laubenstein et al., 2019; Munday et al., 2009a), sensory biology (Chung et al., 2014; Ferrari et al., 2012; Munday et al., 2009b), neurobiology (Chivers et al., 2014; Heuer et al., 2016; Schunter et al., 2019), chemical ecology (Dixson et al., 2010; Ferrari et al., 2011) and molecular biology (Schunter et al., 2018; Tsang et al., 2020). Thus, the potential for cross-disciplinary integration in this field is considerable, with researchers having worked to gain a broad understanding of OA-related patterns and mechanisms. Yet, despite the benefits of this integrative approach, concerns have been raised over the rigour and reliability of experimental studies in the field (e.g. Browman, 2016; Clark et al., 2020a). These issues include flawed experimental designs (Cornwall and Hurd, 2016), inaccurate water chemistry measurements (Moran, 2014), lack of video recording and automated tracking (to remove experimenter bias and increase reproducibility; Clark, 2017), flawed methodologies (e.g. closed respirometry and unreliable choice tests; Clark et al., 2013; Jutfelt et al., 2017), and inadequate statistical analyses (Roche et al., 2020). These problems can (and should) be avoided by collaborating with researchers with relevant expertise and/or investing time and effort to adequately learn methods used in other fields.
Arguably, the most effective way to engage in cross-disciplinary integration is to mindfully seek out collaborations with researchers possessing complementary skill sets (Hof, 2021). This sounds easy in theory; however, in practice, forging new collaborations with experts can be difficult, in part because opportunities to interact with researchers across disciplines and geographical regions are limited. Scientific conferences are often specialized, and thus not attended by researchers across disciplines. Large conferences may attract more participants, but talk sessions still often revolve around a narrow theme of interest. As a result, researchers from different disciplines often attend different conferences or conference sessions, read and publish in different journals, and use their own field-specific jargon when referring to similar phenomena. In addition, attending international conferences comes with a large price tag, which often limits participation from researchers from under-funded institutions and countries (Sarabipour et al., 2021). Fortunately, the recent and rapid rise in communications technologies, particularly during the COVID-19 pandemic, has normalized virtual meetings and unconferences (e.g. hackathons), allowing researchers to more easily reach out to colleagues, exchange ideas, and develop diverse networks of mentors and collaborators from across a range of geographical regions (Huppenkothen et al., 2018; Korbel and Stegle, 2020; Sarabipour et al., 2021).
Several other avenues are available for experimental biologists to engage in cross-disciplinary research when collaboration is not possible. The emergence of open platforms dedicated to sharing methodological information (e.g. protocols.io, Protocol Exchange, STAR protocols), the release of open software to acquire and process data (e.g. Harianto et al., 2019; Sridhar et al., 2019), and a growing number of articles outlining methods and reporting guidelines (e.g. Chabot et al., 2016; Killen et al., 2021; Norin and Clark, 2016) all provide readily available resources for researchers to incorporate best practices and new techniques into their skill set (Baker, 2021). While those resources offer useful starting points, it can often be useful to consult directly with experts for detailed technical advice when using a technique or tool for the first time.
Consulting with colleagues is one valid option to obtain feedback, but more formal avenues are now available for researchers seeking independent expert comments on planned experiments before they begin. First, authors can submit a registered report. Registered reports are a publication format currently offered by over 300 journals (https://www.cos.io/initiatives/registered-reports); they focus on the validity of the research question and methodology rather than the study results. Peer review begins prior to the data collection and rigorous protocols are provisionally accepted for publication if the authors follow through with their registered methodology (Parker et al., 2019). Second, authors can hire an independent ‘red team’, the concept of which originates from cyber security, where ethical hackers (the red team) challenge a blue team of security professionals charged with maintaining network defences against cyber attacks. In science, researchers (the blue team) send methods, analyses, or an entire manuscript to a red team service (e.g. https://redteammarket.com), which designates field-specific experts to identify potential weaknesses in the work. Calling on a red team for feedback is more flexible than submitting a registered report because users pay a fee to receive an independent, expert evaluation of their work at any point in the study development. Whereas registered reports are intended to evaluate a research project from start to finish, the red team approach allows users to seek advice on any component of their project (e.g. experimental design, statistical analyses, laboratory manipulations) based on their specific needs and budget. In both cases, having research protocols expertly assessed prior to data collection helps researchers who venture outside their field of expertise to ensure that their methods are rigorous and appropriate to answer their question of interest.
Ultimately, engaging in effective cross-disciplinary integration requires that researchers be mindful of their own expertise limits and, when possible, seek out knowledgeable collaborators when venturing outside their field. Open science initiatives, such as open protocols and software, registered reports, and red teams, can facilitate integrative research and consensus building by promoting rigorous, objective and transparent research. Some of our proposed solutions may be more available to better-funded research teams. As a community, it is important that we recognize how differential access to resources may impact the ability to implement the changes we suggest, and to mindfully develop ways in which we can more actively engage and encourage participation with our under-represented and under-funded colleagues (e.g. Caravaggi et al., 2021; Sarabipour et al., 2021).
Transparency and reproducibility
Working collaboratively to increase the reliability and impact of studies in experimental biology requires not only more mindful cross-disciplinary integration but also greater transparency throughout the process of data generation, extraction, analysis and interpretation. Experimental biology is an empirical discipline. Thus, debates may be more easily resolved with data. When data are collected and shared transparently, this fosters trust (Jamieson et al., 2019) and helps science advance more rapidly and reliably by allowing independent research groups to more easily understand, validate and build on previous results (Moher et al., 2018; Nosek et al., 2015; Parker et al., 2016). For these reasons, many scientists (O'Dea et al., 2021), funding agencies (Schiermeier, 2018) and journal editors (Bakker and Traniello, 2020; Simmons, 2016) now strongly advocate measures that encourage and/or require greater transparency in published research. This includes publicly sharing data underlying published results (i.e. open data; Caetano and Aisenberg, 2014; Ihle et al., 2017), preferably at the time of peer review so analytical errors can be identified and fixed prior to publication (Fernández-Juricic, 2021). Recent surveys report that 55% of researchers believe funders should make data sharing part of their grant requirements (n>4500 international respondents; Digital Science et al., 2020) and 78% of Canada-based faculty members in ecology and evolution support mandatory open data policies by journals (n=140 respondents at Canadian Universities; Soeharjono and Roche, 2021). This growing support for wider data sharing mandates is spurred, in large part, by the continuing reproducibility crisis in science (Ritchie, 2020) and a sense that, despite the highly competitive nature of academia, a collaborative and open approach to research ultimately benefits both society and individual researchers (Hunt, 2019; McKiernan et al., 2016).
To date, efforts to increase transparency in research have primarily focused on open data, with less attention devoted to other, yet equally important reproducibility materials. One such material is the script or code used for processing and analysing data (i.e. open code) – for example to produce the tables, figures and statistics presented in a paper (Culina et al., 2020; Mislan et al., 2016). Until recently, technological limitations made it challenging for researchers to share the decisions and steps involved in their data analysis. For example, many commonly used statistical software packages rely on a graphical user interface (GUI), making it difficult for users to share and reproduce analyses (e.g. SPSS, JMP, Minitab, SigmaPlot). However, the last two decades have witnessed a surge in the uptake of command-based software to analyse biological data, readily allowing researchers to openly share annotated analysis code (Mislan et al., 2016). In ecology, for example, the use of the R programming language for data analysis increased linearly from 11.4 to 58.0% between 2008 and 2017 (Lai et al., 2019). When combined with open data, open code is a powerful tool for validating results and building on previous studies to answer new and often broader research questions (Barnes, 2010; Stodden et al., 2018).
To examine the contributions of reproducible research practices for advancing consensus building in experimental biology, we investigated the prevalence of open data and code associated with empirical studies on the OCLTT, GOL and OA-driven behavioural change paradigms (Box 1) as well as in eight journals publishing research in comparative physiology (Journal of Experimental Biology, Journal of Experimental Zoology Part A, Physiological and Biochemical Zoology and Comparative Biochemistry and Physiology Part B) and behavioural ecology (Animal Behaviour, Behavioral Ecology, Behavioral Ecology and Sociobiology and Ethology). We identified 532 published studies on the OCLTT hypothesis, 10 on the GOL hypothesis, and 186 on OA-driven behavioural change between 2001 and 2021 (see ‘Methods’, below; note that 66% of studies were published after 2014). Approximately one-fifth (19.0%) of studies we examined on these paradigms had associated open data, and fewer than one in 30 studies (3.2%) had open code (see Table 1 for a breakdown by paradigm). These numbers are slightly higher than the overall availability of open data (12.1%) and comparable to the availability of open code (3.1%) for studies published in the eight leading comparative physiology and behavioural ecology journals between 2010 and 2021 (Table 1; Table S1). In contrast, they are markedly lower than the prevalence of open data (73%) and code (27%) reported for ecological studies on other topics published within a comparable time frame (n=346 studies published in 14 journals that recommend or require open data/code between 2015 and 2019; Culina et al., 2020), highlighting notable differences among disciplines.
Encouragingly, our data show that the percentage of empirical papers with open data in comparative physiology, behavioural ecology, and on the OCLTT hypothesis and OA-driven behavioural change is increasing over time (Figs 1 and 2). Notable differences exist among journals (Fig. 2), but the overall trend suggests that a growing number of experimental biologists working on these topics are complying with editorial open data policies and/or supporting efforts to improve transparency. One caveat to this finding is that not all open data are FAIR (i.e. Findable, Accessible, Interoperable, and Re-usable; Box 2). Indeed, recent studies suggest that approximately 50% of open datasets shared by researchers in ecology and evolution are incomplete or uninterpretable (Roche et al., 2015; Roche et al., 2021 preprint), which considerably limits transparency and computational reproducibility (i.e. the ability to reproduce a study's results with the associated data and code). This consideration effectively means that fewer than 10% of studies in experimental biology have open data that would allow an independent assessment of their conclusions.
The FAIR sharing principles were established in 2016 as guidelines to improve the Findability, Accessibility, Interoperability and Re-use of digital materials such as data and code (Wilkinson et al., 2016). The principles emphasize machine-actionability, and the capacity of computational systems to find, access, interoperate and re-use digital materials with minimal human intervention (see www.go-fair.org). For some fields and/or researchers, sharing machine-actionable data can be challenging, particularly if technical support is lacking. In these cases, researchers should strive to share open data that can readily be found, accessed and re-used by humans (i.e. human-actionable).
Making data ‘findable’ requires archiving them on an indexed, searchable public data repository. Preferred repositories comply with the TRUST Principles for digital repositories (Lin et al., 2020) and can readily be searched via data aggregators such as DataOne, DataCite or Google Dataset Search. Zenodo (https://zenodo.org) and Figshare (https://figshare.com) are examples of trusted repositories that accept data from any discipline at no cost to users. Archiving data on a public repository is preferred to sharing data as supplementary material because the latter are not easily findable and are often behind a publisher paywall (i.e. not accessible).
Making data ‘accessible’ requires that open datasets be complete (i.e. all the variables used in a study for the statistical analyses, figures, tables, etc. are present in the dataset) and easy to download. Archiving datasets using non-proprietary file formats (e.g. .txt or .csv) promotes accessibility as proprietary file formats (e.g. .mat, .sav, .arc) require a software licence to open and edit. When data for a study have been compiled from third party databases (e.g. meteorological data, distribution data, morphology data), researchers should archive the subset of the data used (licences often allow this; if unclear, permission can be sought) or archive the code used to extract the data from the database. Large databases are frequently updated with new data and can be difficult to navigate, making it difficult for users without sufficient information to access specific subsets of the data used in a study.
Making data ‘interoperable’ requires using file formats that are readily compatible with different software (e.g. .txt, .csv) and established standards for sharing data when available. For example, Darwin Core is a standard for sharing biological diversity data using agreed-upon identifiers, labels, and definitions (https://dwc.tdwg.org). The Ecological Metadata Language is a community-maintained specification with a vocabulary and syntax for sharing research data in environmental science (https://eml.ecoinformatics.org). Several other communities of practice have developed ‘minimum information standards’ with field and vocabulary standards to facilitate integrating datasets (e.g. Rund et al., 2019).
Making data ‘re-usable’ requires publishing the data under a licence that facilitates re-use (https://creativecommons.org/licenses) and providing comprehensive metadata (typically a text file describing the data) allowing (re)users to understand the data. Metadata are used to describe the provenance of the data (e.g. who created them, what other materials are they associated with) and information such as column headings, abbreviations, units of measurement, and missing data. Data re-usability also hinges on the data being in unprocessed form and many of the factors that promote interoperability, such as the use of non-proprietary file formats, vocabulary standards and spreadsheet organization (e.g. Tidy data; https://r4ds.had.co.nz/tidy-data.html). For example, colour coding and cell formatting in Excel files are problematic for re-usability because these features are lost or can lead to errors when Excel files are converted to other formats such as .csv and .txt.
Without access to data and code, readers must rely on the authors’ interpretation of a study's data (Fig. 3). This is problematic for all studies, but particularly in the context of debated topics because transparent and trusted evidence is critical for consensus building (see Jamieson et al., 2019). Although open data is increasingly the norm in many disciplines in biology, it remains a relatively new practice. As a result, many more established researchers have not received formal training in best practices for archiving high quality, FAIR data. Research on data sharing suggests that journal policies lead to more open data (Vines et al., 2013), but that education, training and technical support are important to ensure high quality, FAIR sharing (Soeharjono and Roche, 2021; Roche et al., 2021 preprint). Fortunately, an increasing number of courses and workshops aimed at teaching reproducible research practices are available for researchers at all career stages (Table S2). It is important to note that engaging in research practices that promote transparency and reproducibility can be done in a stepwise fashion. Researchers cannot be expected to go from zero to hero overnight – rather, a sensible approach is to adopt practices with the fewest logistic and monetary hurdles (e.g. open data) and progress from there based on access to resources, training and comfort level (see resources in Box 3). We must also be careful as a community not to shame researchers who engage in open and transparent research practices in good faith if problems or omissions in shared materials are detected. Recognizing that adopting transparent and reproducible research practices can be a steep learning curve is necessary to foster a culture of trust, belonging and pride as experimental biology moves towards increased openness.
Open science (general)
Kathawalla et al. (2021). Easing into open science: a guide for graduate students and their advisors. Collabra: Psychology7, 18684. doi:10.1525/collabra.18684
Alston and Rick (2020). A beginner's guide to conducting reproducible research. Bull. Ecol. Soc. Am.102, e01801. doi:10.1002/bes2.1801
Wittman and Aukema (2020). A guide and toolbox to replicability and open science in entomology. J. Insect Sci.20, 6. doi:10.1093/jisesa/ieaa036
Ihle et al. (2017). Striving for transparent and credible research: practical guidelines for behavioral ecologists. Behav. Ecol.28, 348-352. doi:10.1093/beheco/arx003
Towse et al. (2021). Making data meaningful: guidelines for good quality open data. J. Soc. Psychol.161, 395-402. doi:10.1080/00224545.2021.1938811
British Ecological Society (2019). Guides to Better Science: Data Management. https://www.britishecologicalsociety.org/publications/guides-to
Hart et al. (2016). Ten simple rules for digital data storage. PLoS Comput. Biol.12, e1005097. doi:10.1371/journal.pcbi.1005097
White et al. (2013). Nine simple ways to make it easier to (re) use your data. Ideas Ecol. Evol.6, 1-10. https://ojs.library.queensu.ca/index.php/IEE/article/view/4608
Destasio (2021). R best practices. https://kdestasio.github.io/post/r_best_practices/
British Ecological Society (2017). Guides to Better Science: Reproducible code. https://www.britishecologicalsociety.org/publications/guides-to
Eglen et al. (2017). Toward standard practices for sharing computer code and programs in neuroscience. Nat. Neurosci.20, 770-773. doi:10.1038/nn.4550
Wilson et al. (2017). Good enough practices in scientific computing. PLoS Comput. Biol.13, e1005510. doi:10.1371/journal.pcbi.1005510
Sandve et al. (2013). Ten simple rules for reproducible computational research. PLoS Comput. Biol.9, e1003285. doi:10.1371/journal.pcbi.1003285
Experimental biology is now firmly at the forefront in predicting the ecological impacts of environmental change on organisms. However, debates among researchers on topics such as OCLTT, GOL and OA-driven behavioural change linger and are beginning to show signs of ‘theory tenacity’: the persistent belief in a theory even in the face of contradictory evidence (Duarte et al., 2015; Loehle, 1987). Rather than embodying the open mind expected of scientists, ‘theory tenacity tends to make opposing camps dig their trenches deeper’ (Loehle, 1987). Resolving debate and building consensus on these topics requires that scientific discussions and research be undertaken in a spirit of good faith, with the objective of resolving disagreement using robust empirical evidence and best practices in experimental design and analysis (see Hoekstra and Vazire, 2021). Achieving this requires all experimental biologists to engage in mindful cross-disciplinary integration and adopt collaborative research practices that foster transparency and reproducibility. In some instances, the perception that parties involved in a debate are not engaging in good faith can lead to a breakdown in trust and outright conflict. In such situations, when collaboration is no longer possible, transparency and reproducibility become the most important arbiters of debate.
We hope that a greater adoption of open science practices, with a particular focus on FAIR data and code, will lead to a much-needed paradigm shift towards improved transparency, cross-disciplinary integration, and consensus building in experimental biology to maximize the contributions of researchers in addressing the impacts of climate change on organisms.
We investigated the availability of open data and analysis code associated with: (1) studies that contribute to advancing our understanding of three debated paradigms investigating the impacts of climate change on aquatic ectotherms (Box 1), and (2) studies published in leading comparative physiology and behavioural ecology journals between 2000 and 2021 (Table 1).
Regarding the three paradigms of interest, studies were systematically identified via a forward reference search of journal articles citing foundational papers for each of the three paradigms: OCLTT (Deutsch et al., 2015; Frederich and Pörtner, 2000; Pörtner, 2001, 2010, 2012; Pörtner and Farrell, 2008; Pörtner and Knust, 2007; Pörtner and Peck, 2010; Pörtner et al., 2017), GOL (Cheung et al., 2013; Lefevre et al., 2017; Pauly, 1998; Pauly and Cheung, 2018b), and OA-driven behavioural changes (Dixson et al., 2010; Munday et al., 2009b, 2010; Nilsson et al., 2012). The forward searches were conducted in Web of Science on 4 June 2021 and yielded a total of 3541, 543 and 804 citations for each paradigm, respectively, after removing duplicates (Table S3). We used the software Rayyan (https://rayyan.ai; Ouzzani et al., 2016) to narrow our search and identify relevant papers from within each list using keywords present in article titles and abstracts. The keywords used to screen papers are listed in Table S4. When a keyword was present in a paper's title and/or abstract, we manually reviewed the article to determine its suitability for our analysis. An article was deemed suitable when it reported data and results considered to advance our understanding of one of the three paradigms mentioned above. Several articles identified in our forward reference searches contributed relevant and valuable evidence for testing the three paradigms without the authors explicitly stating that they had tested those paradigms. Meta-analyses and theoretical studies using empirical data were included in our analysis; we excluded theoretical studies based on simulations (as these rely on code but not empirical data), and papers where there would be no expectation of underlying data, such as reviews, comments and perspectives.
For studies in comparative physiology and behavioural ecology, we chose four representative journals in each of the two disciplines (Table 1) and examined the availability of data and code for the first 10 articles published each year between 2010 and 2021. We excluded article types such as reviews, perspectives and other types of contributions where there would be no expectation of underlying data.
Once the final list of articles was completed in both cases, we manually screened each article and its associated supplementary material for the presence of: (1) a data availability (or accessibility) statement; (2) open data (i.e. publicly available data that could be inputed into software to produce tables, figures or statistical results); and (3) open code (publicly available script or code used to produce tables, figures or descriptive/inferential statistics). We only recorded the availability of open data and code, not whether these materials were complete (i.e. contained all the variables measured and used in all the analyses presented in a study) and interoperable/easily re-usable (i.e. had associated metadata and were shared in a non-proprietary, machine-readable format).
We thank the many colleagues who participated in discussions that led to this paper, as well as Andrea Morash and an anonymous reviewer for helpful comments on the manuscript. We acknowledge the Traditional Custodians of the unceded lands and waterways on which several of our institutions now operate.
D.G.R. is supported by the European Union's Horizon 2020 research and innovation programme under Marie Skłodowska-Curie grant agreement no. 838237-OPTIMISE. T.D.C. is supported by an Australian Research Council Future Fellowship (FT180100154) funded by the Australian Government. T.N. is funded by a research grant (40713) from Villum Fonden. G.D.R. and S.A.B. are supported by Discovery Grants from the Natural Sciences and Engineering Research Council of Canada.
The data and code to reproduce the results presented in this study are publicly available on the Open Science Framework (Roche et al., 2022): https://doi.org/10.17605/OSF.IO/F2DAZ. These were made available to the editors and reviewers in the initial submission.
D.G.R. is the president of the Society for Open, Reliable, and Transparent Ecology and Evolutionary biology (www.sortee.org), and an ambassador for the Center for Open Science (www.cos.io) and the repository Figshare (www.figshare.com). These positions are unpaid.