Rigor & Reproducibility

Scientific rigor is the strict application of the scientific method to ensure unbiased and well-controlled experimental design, methodology, analysis, interpretation and reporting of results. 

"Reproducible research practices include rigorously controlled and documented experiments using validated reagents. These practices are integral to the scientific method and they enable reliable and actionable research results." [1]
  The use of resources (materials, scientist time, energy/water, equipment, lab space, funding, admin resources, etc.) for research that is based on irreproducible research and as a result is not destined to succeed, is counter to the efficiency efforts that the laboratory sustainability community is working on.

An analysis of past studies indicates that the cumulative (total) prevalence of irreproducible preclinical research exceeds 50%, resulting in approximately US$28,000,000,000 (US$28B)/year spent on preclinical research that is not reproducible—in the United States alone.

Encouraging rigor and reproducibility in the grant process will avoid dollars wasted on the reproduction of research, freeing more money to flow towards other important, novel research projects. ​​​

The ABRF emphasizes "the critical role that shared research resources can play in achieving efficient use of research funds and broadening access to advanced skills, expertise and technologies."  Because these resources generate the majority of research data at many institutions, "their role in maintaining needed expertise and generating quality data is considerable...Therefore, the scientific shared resource community must continue to take the lead in promoting and supporting rigorous, transparent, and reproducible (R&R) research, as well as in providing critical mentoring and technical training." [1]
Check out a recent article in the Journal of Biomolecular Techniques entitled Addressing the Environmental Impact of Science Through a More Rigorous, Reproducible, and Sustainable Conduct of Research. Written in collaboration between members of ABRF and I2SL, this article discusses the intersection between reproducibility, sustainability, and shared research resources.
Many respondents noted that inadequate standardization of protocols and procedures across the research life cycle, from study planning through data analysis and reporting, contributes to variable research quality. [2]
Check out a recent article in the Journal of Biomolecular Techniques entitled Addressing the Environmental Impact of Science Through a More Rigorous, Reproducible, and Sustainable Conduct of Research. Co-written by BETR Grants and I2SL Lead Kathryn Ramirez-Aguilar, this article discusses the negative consequences of irreproducible research and the importance of shared research resources in reproducibility.

​Suggestions:

  • Core facility manager/directors as a source of support and expertise that can help lab members with rigor and reproducibility.
    • For example, in the area of experimental design and creating guidelines on sample quality to have the best results on instrumentation in their facilities.​

Seven Steps to Rigorous & Reproducible Experiments from the ABRF:
  1. If using a core facility, consult with the core staff in the planning stage. Consult with a statistician if you need help developing a Power Analysis.
  2. Design your experiment with sufficient controls (rigor) and replicates (reproducibility).
  3. Assure that ALL of your reagents (antibodies, cell lines, mice) are fully validated.
  4. Have a clear and detailed protocol (SOP) and data analysis plan. Document any deviation from the protocol. 
  5. Assure that the staff or students performing the experiment are well trained and understand each step and importance of performing them precisely (rigor again).
  6. Use only well-maintained instrumentation, preferably maintained and operated in a core facility with expert staff. 
  7. ​Document all steps, reagents, equipment and data analysis methods used in the experiment. Assure that both the data and documentation itself are properly stored in a safe data management repository. 
[1] - https://www.abrf.org/assets/docs/ccorre_2019_poster_final.pdf
[2] - ​Knudtson KL, Carnahan RH, Hegstad-Davies RL, et al. Survey on Scientific Shared Resource Rigor and Reproducibility. J Biomol Tech. 2019;30(3):36-44. doi:10.7171/jbt.19-3003-001