EQIPD

Poor quality data is an issue in many research fields – all too often, results carried out in one organisation cannot be replicated elsewhere, and it is not always clear why. In medical research, consequences include poor decision-making resulting in higher failure rates and longer drug development times. There is therefore an urgent need for simple, sustainable solutions to improve data quality, and that’s where the EQIPD project comes in. Their goal was to deliver simple recommendations to facilitate data quality without impacting innovation.

Addressing the reproducibility crisis

It’s no secret that scientific research is in the midst of a reproducibility crisis. A Nature survey launched in 2016 found that more than 70% of researchers had tried unsuccessfully to reproduce another scientist’s published work, and more than half had failed to replicate their own experiments. The field of preclinical cancer research, for instance, is particularly notorious for its difficulties in reproducing results, and this issue with robustness has a knock-on effect on the entire drug development pipeline. The lack of rigour leads to wasted resources and finances, both public and private, including the wastage of animal life in non-reproducible studies (both in vitro and in vivo), which raises ethical concerns. Addressing the problem would lead to a reduction in the number of animals used in experiments and would possibly lower the extremely high failure rate of drug development trials.

EQIPD was established to address this problem, with the goal of establishing a standardised quality system for preclinical research that could be rolled out across a wide range of laboratory settings. While guidelines about reporting on preclinical studies exist, there was no existing guidance for researchers regarding preclinical research design, conduct or analysis. By improving these factors, it’s hoped that the speed at which new drugs are developed could be accelerated.

The project examined the variables that were most likely to cause issues with preclinical data quality in in vivo studies by combing through the scientific literature, and then set about creating a suite of guidelines – and an entire quality system – that laboratories should adhere to in order to ensure that they produce robust and reproducible results.

It's not about bad science

Issues with reproducibility are not necessarily the fault of “bad science” per se. It could be that a laboratory uses one particular protocol or methodology which was not outlined in the peer-reviewed article when they published their results. Then, when another researcher attempted to replicate those results they might take a completely different approach and fail to reach the same conclusion. In addition, sometimes researchers don’t explain which statistical tests they performed on their data, leading to confusion when other scientists can’t match their results. For these reasons, one of EQIPD’s key recommendations is that researchers “show their workings” and explain what protocols and statistical analyses they used.  

Artificial intelligence to the rescue

Another issue is ensuring that researchers are up-to-date on relevant literature. Nowadays, the body of literature that a researcher has to be familiar with is enormous, meaning that while doing initial literature reviews before embarking on a research project, researchers can accidentally leave out a specific result that could affect their experiments.

In Alzheimer’s disease for example, there are more than 135 000 publications on the disease, and asking humans to read all of that is difficult. To tackle that, EQIPD developed a tool based on artificial intelligence (AI) that helps researchers to sweep through extensive bodies of literature and pluck relevant articles out. This is not a perfect solution, because new research is published all the time and between the start of a project and the submission of a manuscript it’s likely that additional papers will be published which were not included in the initial AI-based sweep. But it will certainly reduce the amount of relevant research that is excluded.

Rolling out the EQIPD Quality System

The first step along the road in the EQIPD Quality System is the self-assessment tool. Here, a series of targeted questions are asked to determine how robust the laboratory in question is. An important factor is that the tool is designed to evaluate a laboratory as a whole, not an individual researcher or research team. The tool is also available on scientist.com as part of the Compli platform.

The self-evaluation tool reveals what the weak points of the laboratory are. To address these, the researchers from the laboratory can then take part in a series of e-learning modules which are available on the EQIPD website. You don’t have to take the whole course – the approach is deliberately flexible so that, after having taken the self-assessment, research labs can identify the modules that are most relevant for them and follow those. The lessons taught via the e-learning are applicable to a variety of research environments, whether a university lab or an industrial lab or a SME setting. There is no set timeframe for taking the course, and a laboratory is not required to undertake the entire quality system.

When a laboratory feels ready for certification, they must do the self-assessment again. This step is followed by an interview, during which the evaluators (trained employees of PAASP, one of the members of the original EQIPD consortium) will carry out an appraisal and recommend whether the EQIPD certification can be given. Finally, GoEQIPD will decide on the awarding of certifications based on the recommendations of the evaluators. In some cases an in-person visit will be organised to assess the lab, although this is not necessary for all applications and is only done on a random basis to ensure quality control.

Restoring public faith in research

EQIPD ran from 2017 to 2021, and during the COVID-19 pandemic, the need for robust scientific data became startlingly clear. With extremely high public pressure to find a drug to treat COVID-19, medications that were already on the market – so already deemed safe for human consumption – were slotted straight into clinical trials based on extremely preliminary preclinical studies. Normally, a lot more evidence would be needed before such steps would be taken, but the urgent nature of the pandemic meant that scientific rigour was given lower priority in the name of finding a cure fast.

In the wake of the pandemic, it’s now easy for vaccine deniers and other anti-science activists to point to examples of badly done or rushed science. And when you have a few examples of poorly-executed science, the public can lose trust in science as a whole. Systems like EQIPD can push back against anti-science rhetoric, by showing that there is a robust system that ensures lack of bias and high standards in carrying out research.

The legacy of EQIPD

Although the project is over, the legacy of EQIPD lives on. The Quality System has been established and, as of 2023, 6 laboratories have been certified. 31 peer-reviewed publications have been published by the project. While the results and guidance focus specifically on preclinical in vivo studies, many of the principles can also be applied to other types of research and in vitro studies. For instance, one of the steps is predefining hypotheses and another is eliminating bias through blinding and randomisation. Although modifications would be needed, a lot of those steps could be adapted for use in other research areas.

Responsibility for the certification process, the e-learning courses, the live summer school and workshops is in the hands of a non-profit association that has been set up, called Guarantors of EQIPD (GoEQIPD). GoEQIPD produced a series of webinars relating to various topics on scientific rigor in 2023, which have been published on the GoEQIPD website. The organisation is now contributing to a Horizon Europe project that launched in 2023 called iRISE, which aims to build on the results of EQIPD to improve scientific rigour in preclinical studies using analytical and computational modelling, simulations and meta-studies.