gms | German Medical Science

Clinical trials are currently the best tool in determining the safety and therapeutic efficacy of interventions. They are a time-consuming and expensive endeavor and many factors can lead to a trial being suspended, withdrawn, or terminated. Some of them are a lack of funding, enrollment issues, lack of efficacy, and many more [1].

Learning from reasons why clinical trials have been stopped is not only important in trial design or policy. With the increasing interest in drug-repurposing the selection of suitable candidates can benefit from knowing past problems. This led, for example, to the development of the true-positive/true-negative database repoDB [2]. However, this approach only considered the overall trial status as failed and not the specific reasons. This was likely the issue of clinical trials lacking clear annotations of failure categories.

Clinical trials need to be registered with a responsible authority and nowadays are available in online databases such as ClinicalTrials.gov (https://clinicaltrials.gov/) or the EU Clinical Trials Register (https://www.clinicaltrialsregister.eu). Although this allows researchers and policy makers to inspect a trial’s information, not all information is easily machine-readable or provided in a uniform manner. As it is not realistic to request that all people who ever entered a clinical trial would go back and update their information in a more unified manner, computer-science approaches need to be employed in deriving meaning from what is available.

Here, we describe the web-based trials annotator tool which allows for stopped clinical trials to be categorized into different failure reasons. The tool is available online at https://trialsannotator.kalis-amts.de.

Different approaches to the annotation of clinical trials have been employed. As relevant information on clinical trials is often represented as plain-text, machine learning algorithms are used to extract valuable information. Kury et al. [3] used natural language processing to extract trial eligibility criteria. The prediction of low accrual by Bennette et al. [4] was done using a logistic regression model. Murali et al. [5] tried to predict the outcome of clinical trials using a random forest classifier. While many of these machine-learning approaches produce reasonable results, they are never perfect, and it is harder to justify the results, especially if they inform patient treatment in the long run.

As the results of the presented annotation tool will be used in subsequent research projects which may involve machine-learning themselves, the decision was made against a machine-learning approach to provide a higher data quality and to prevent accumulating uncertainty.

A different approach is the manual curation of information in combination with annotator consensus. Depending on the annotator’s expertise this approach often produces results of higher quality than algorithmic solutions. On the other hand, it is labor-intensive and takes longer than an algorithm. Tailored annotator tools help to mitigate some of these issues and can support annotators in preventing common mistakes. One example is the custom-built coding tool for cancer drug-repurposing trials by Pantziarka et al. [6]. It allows the annotation of trials with details such as cancer type, stage, drugs, age, and many more. While the tool itself is not available, the results are regularly updated and published on the Anticancer Fund website. Another example would be the design of an ontological metadata framework for clinical trials by the ontology group at Mayo Clinic [7] using tools such as TopBraid. Williams et al. [8] previously conducted an analysis of terminated trials. The reasons for termination were annotated in 16 categories including funding issues, product withdrawal, safety, and interim results. The analysis of trials was limited to the status of termination and all annotation was done manually by two of the authors. The results are available online as supplementary information but were not updated beyond February 2013.

From the aforementioned approaches, Williams et al. is the most related as the annotation process was done manually to ensure high data quality and the reason for stopped trials was categorized. However, only trials with the terminated status were annotated and the analysis was done only once without any tool for keeping the data updated and allowing other researchers to help in the annotation process. This also limits the ability to form an annotation consensus. The presented development of an online and open platform for the manual annotation of terminated trials to simplify this task, to form a broader consensus the more people annotate trials, and to keep up to date with newly released trial data will hopefully resolve these issues.

In the following, the preparation of clinical trial data, the definition of failure categories, and finally the web-based annotation platform are described in detail.

As a starting point for the annotation effort, ClinicalTrials.gov was chosen as the source of clinical trial information. It is easily accessible and provides a specific data field for stopping reasons. To simplify the data integration process, BioDWH2 with the AACT (Aggregated Analysis of ClinicalTrials) data source module was used [9]. AACT (https://aact.ctti-clinicaltrials.org) is a public database by the Clinical Trials Transformation Initiative (CTTI) trying to improve access to data from ClinicalTrials.gov. Although the AACT database is updated daily, the BioDWH2 data source module for AACT currently utilizes the monthly database download which is sufficient for the annotation process.

After the integration process is complete, the BioDWH2-Neo4j-Server tool is used to launch a Neo4j 3.5 (https://neo4j.com) graph database server with the “Awesome Procedures on Cypher” (APOC) 3.5 extension. This allows the use of the Cypher query language to filter and extract relevant clinical trial information.

The annotation effort focuses on reasons for failed clinical trials. The term “failed” in this scenario means that a trial did not meet the intended outcomes. A lack of efficacy outcome is still valid and important information. However, in light of using these annotations to inform future drug-repurposing analyses this is still interpreted as a failure in terms of finding a new indication. Therefore, certain criteria need to be met for a trial to be recognized as failed and included in the annotator database:

• The trial phase is one of:
  • Early Phase 1
  • Phase 1
  • Phase 1/Phase 2
  • Phase 2
  • Phase 2/Phase 3
  • Phase 3
  • Phase 4
• The trial overall status is one of:
  • Withdrawn
  • Suspended
  • Terminated
• The trial has at least one intervention MeSH (Medical Subject Headings) term.
• The trial has at least one condition (MeSH) term.

Trials with missing information on phase, overall status, interventions, or conditions are excluded. Phase 4 clinical trials would normally be excluded as well, as the efficacy has already been demonstrated in phase 3 and the post marketing surveillance phase does not fit the concept of a failed trial. However, as some repurposing trials are falsely published as phase 4 (examples from the ReDO_Trials Database [6]: NCT03645187, NCT04741204), this phase is included as well.

The exclusion of trials without Medical Subject Headings (MeSH) intervention terms is currently in place to allow for failed trial interventions to be more easily mapped in subsequent analyses. In contrast to non-MeSH conditions, which are very similar to MeSH condition terms, non-MeSH intervention information is too complex to be easily mapped. These may be included in a future release once suitable mapping is possible.

The overall status of suspension is included in the definition of a failed trial. Suspended trials may be resumed in the future and the investigator may have just set the status to suspended during interim analysis or other trial checkpoints. However, there are also trials which have last been updated in 2005 and still have a suspended status and should therefore be considered failed. If the status changes in the future, the trial will simply be excluded on subsequent data updates.

As the annotation effort is currently focused on trial failure categories, those need to be defined. From previous trial analyses certain common failure points can be derived, such as funding, recruitment, enrollment, accrual, efficacy, and toxicity. As the annotation effort has a focus on drug repurposing, drug-related categories should be expanded in order to have a clearly defined category of no clinical benefit. The following categories have been defined:

• Drug toxicity/tolerability
  • Adverse events, toxicity, patient death, or similar
• Drug supply/availability
  • Issues in study drug supply, drug expiration date reached, manufacturer stopped production, or similar
• Drug withdrawn by regulator
• Drug other
  • Any other drug related issues such as other trials showing drug related issues, or annotator is not yet sure about final drug category
• Futility/No clinical benefit
  • No clinical benefit could be shown, lack of efficacy, or similar
• Funding issues
  • Any financial issues in performing the study
• Recruitment/Accrual/Enrollment
  • Issues in patient recruitment, including enrollment, screening, and accrual
• Investigator left
  • The (principal) investigator left the study, relocated, retired, died, or similar
• Other
  • Any other reason

To build the web-based annotation tool a MySQL database is created from the failed clinical trials table. These trials are then referenced in a separate annotations table containing flags for the previously defined failure categories. Annotations are in turn connected to the users table holding the annotator information. The full database schema is visualized in Figure 1 [Fig. 1].

The tool itself is developed using PHP with the Slim framework version 4 and the laravel illuminate database object-relational mapping (ORM) library version 8. For the front-end Bootstrap version 5 is used. The tool is split into three sections: Home, Statistics, and FAQ. Home as the main landing page is also the annotation tool itself listing all clinical trials in a paginated table. The table of trials can be filtered using multiple criteria, such as trial ID, phase, status, date range, free text, and annotation. Trial information such as intervention, condition, and failure reason are provided for each trial row. An example screenshot is shown in Figure 2 [Fig. 2]. The last column of the table shows the current annotator decisions with the decided categories highlighted in color. Once logged-in, this column provides the same categories but as buttons to be used at one’s own decision for the rows trial. If the reason text is not clear enough, the annotator may click on the trial id in the first column which opens a dialogue with the trial website embedded to find further information. The decision for embedding the website instead of opening a new tab was made in order not to break the annotator’s immersion in the current annotation process. Especially while annotating many trials, the user develops a mental map of where to click and what to do on the current page. If the user needs to switch between tabs repeatedly to get more information, this mental map can be disrupted and slow down the annotation process, as was reported by some users. For each trial multiple categories can be selected as not just one reason may be responsible or relevant for a trial’s failure.

The statistics page provides basic insights into the annotation progress, failed trial composition, and annotation breakdown. Finally, the FAQ page is intended to convey the origin of information and helps new annotators with certain trial-specific language.

New annotators may register on the page which will be checked by an administrator and activated as soon as possible. The download of all annotation results is currently limited to logged-in users, but will be made available to the public once a reasonable annotation consensus has been found for a larger subset of failed trials.

As of September 23^rd 2022 the database contains 14,232 clinical trials meeting the failed trial criteria of which 7,329 (51.5%) are already annotated. These preliminary annotations do not represent a consensus yet, as is the intention of this effort. Additionally, the annotations are likely biased towards certain categories. Annotating trials with a clearly stated reason such as lack of funding or enrollment is easier than annotating those without a specific reason or complex drug-related reasons.

Figure 3 [Fig. 3] shows the distribution of failed trials and the respective number of annotated clinical trials by the year of the last update posted. Over the years, the number of trials increased steadily, which is likely due to the widespread adoption of digital trial registries and care in trial entry updates. This is also evident in that no trials for 2005 and 2006 have been annotated yet, as none of these failed trials provide a failure reason.

Nonetheless, certain trends can already be derived from the annotation effort to date. When comparing the annotation flag counts relative to the total trial count per trial phase as visualized in Figure 4 [Fig. 4], the following assumptions can be made:

1.

Getting a clinical trial funded is harder the lower the trial phase. This is reasonable as phase 3 trials already established their goal through previous trial phases and sponsors are more likely to provide funding. Phase 4 post-marketing studies are again of less interest to funders as the product is already on the market.

2.

Investigators are more likely to leave a trial with an early clinical trial phase. Aside from reasons such as retirement or investigator death, it may be of greater benefit for an investigator to leave for a better career opportunity than working on an early trial. In comparison, later phase trials could be more beneficial for an investigator’s career.

3.

Recruitment, enrollment, and accrual of patients is the most likely reason for a failed trial independent of trial phase.

Annotating all failed clinical trials is a big task and will take time and care to fulfill. Although ~51% can be annotated fairly easily due to clearly stated failure reasons, it is the rest that will take increasing amounts of effort to determine the right annotation categories if at all possible. For some trials there might just not be enough information available for a meaningful conclusion.

Understanding the reasons for failed clinical trials still merits the effort. Be it for understanding key risk factors in planning trials of certain phases or seeing trendlines if certain failure reasons could be reduced in subsequent years. But also focussing on specific failure categories can be of interest, such as futility or toxicity in finding drug repurposing candidates.

As new trials are conducted all the time, the annotation effort will be never-ending. The main goal for this effort should be to reach a consensus by as many annotators and for as many trials as possible. Once a consensus is reached, a trial could be seen as finished. This means that previous years could be finished one after the other and the ideal situation would be that someday only newly failed trails need to be annotated. For this goal it is of utmost importance to find capable people interested in this annotation effort. It is possible to register for an annotator account on the website, which needs to be approved by an administrator.

• Marcel Friedrichs: 0000-0001-9846-7212

The author declares that he has no competing interests.