Reproducibility in pre-clinical life science research
Reproducibility forms the foundation of meaningful scientific research and is an issue which is causing increasing concern. In 2011 German Pharmaceutical company, Bayer, published data showing in-house target validation only reproduced 20-25% of findings from 67 pre-clinical studies1. A similar study showed 11% success rate validating pre-clinical cancer targets2 and is likely to contribute to the high failure rates translating pre-clinical drug candidates into meaningful cancer therapies. Awareness of this reproducibility problem has gained momentum with a number of projects aimed at understanding the causes underlying it; The Reproducibility Project: Cancer Biology is one such initiative which published its findings in January 20173.
The causes are complex and include lack of validation of key biological reagents and reference material (including cell lines), poor study design, inadequate data analysis and reporting, and a lack of robust laboratory protocols4. This means that the solutions will be complex and will require the concerted effort of all stakeholders involved in the pre-clinical science fields including scientists, journals, funding bodies and even mainstream media such as Times Higher Education and BBC Radio 45.
Given the low reproducibility rates reported in an increasing number of studies over the past five years it is likely that hundreds of millions of pounds is being wasted in the generation of low quality, questionable research data. In the United States this is estimated to be billions of dollars5.
What’s being done to address this issue?
The scientific community is taking action to raise standards of pre-clinical research. In 2014 publishers from 30 life science journals met to agree a common set of guidelines to improve reproducibility. This has included a requirement to make the presentation of cell line authentication data mandatory for publication. In 2015 the Academy of Medical Sciences co-organised a symposium to review this problem with major UK grant funders the Biotechnology and Biological Sciences Research Council (BBSRC), Medical Research Council (MRC) and Wellcome Trust. Last year they also published a report outlining key steps that are already being taken to help improve reproducibility. The National Institute of Health (NIH) recently added a new section to their guidelines for funding document – ‘Authentication of any key biological and/or chemical resources’6.
Other measures which have been cited in the above reports include greater scrutiny of experimental design, such as randomisation or blinding, and appropriate statistical analyses both before grants are funded or research published. Funders also pledged to ensure greater training of both students and panel members with an emphasis on experimental design and statistics. Some steps in the process have been as simple as relaxing word-limits on applications and journal methodology sections to ensure sufficient levels of detail are included. They also placed a greater emphasis on the openness and transparency of methodology, with enhanced consideration of negative or null results. Some have taken steps to develop open platforms to publish raw data from studies. Best-practice guidelines have been updated with specific mention of image handling and validation of biological reagents. In addition, UK councils are providing more information with dedicated pages about reproducibility and guidance about resources such as cell and tissue banks.
ECACC can be part of the solution
A key focus for ECACC is ensuring the authenticated biological reagents and reference materials we provide are of the highest possible quality. Our facilities operate to the quality standard ISO 9001:2015 and are accredited in accordance with International Standard ISO/IEC 17025:2017 laboratory reference standard for a number of its internal quality control assays. All of our cell lines undergo sterility testing, species identification and mycoplasma testing and are provided with a certificate of analysis.
Our human lines also undergo forensic STR profiling and selected cell lines are further characterised to independently verify their published features. ECACC offers these quality tests for customer samples and has recently launched a new authentication service where cells can simply be spotted on our FTA cards and posted at ambient temperature for analysis.
We offer cell culture training courses to help ensure the best practice within the field. As co-founding members of the International Cell Line Authentication Committee (ICLAC) we take cell line identity very seriously and are committed to helping improve reproducibility in science.
Outlook
Despite the concerning nature of the ‘reproducibility crisis’ it is something that is gaining increasing attention around the world. It is encouraging that steps are being taken across the scientific community to ensure improvements are made to drive the highest quality of research.
References
1. Prinz, F., T. Schlange, and K. Asadullah, Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov, 2011. 10(9): p. 712.
2. Begley, C.G. and L.M. Ellis, Drug development: Raise standards for preclinical cancer research. Nature, 2012. 483(7391): p. 531-3.
3. Baker, M. and E. Dolgin, Cancer reproducibility project releases first results. Nature, 2017. 541(7637): p. 269-270.
4. Freedman, L.P., I.M. Cockburn, and T.S. Simcoe, The Economics of Reproducibility in Preclinical Research. PLoS Biol, 2015. 13(6): p. e1002165.
5. Baker, M., 1,500 scientists lift the lid on reproducibility. Nature, 2016. 533(7604): p. 452-4.
6. Collins, F.S. and L.A. Tabak, Policy: NIH plans to enhance reproducibility. Nature, 2014. 505(7485): p. 612-3.