Enhancing Clinical Trial Selection for Cancer Patients Using Large Language Models

The advent of Large Language Models (LLMs) in 2018¹ has significantly impacted various domains, including medicine, education² and industry. In the medical field, LLMs have demonstrated their utility in answering patient inquiries and assisting in medical education. They are well-positioned to assist researchers in analyzing text-based data, such as clinician notes.¹ This study explores the efficacy of two prominent LLMs—Gemini 2.0 and GPT-4.0—in determining the appropriateness of clinical trials for cancer patients with specific gene mutations.

Clinical trials are essential for advancing cancer treatment; however, identifying suitable trials remains a challenge. The primary source of clinical trial data, ClinicalTrials.gov, is widely used, receiving over 2 million page views per month and 90 000 unique visitors daily as of December 2024.³ However, the platform has limitations in presenting clinical studies, particularly for cancer patients.

The search functionality is largely text-based and non-specific, often retrieving extraneous or irrelevant results due to ambiguous terminology and exclusion criteria within trial descriptions. For instance, a search for the “EGFR mutation” may retrieve trials related to “eGFR,” a drug for kidney disease, due to the text-matching approach. Similarly, searching for “KRAS mutation” may return a study that explicitly excludes patients with KRAS mutations, as the trial description states that only those with a “wild-type K-Ras gene” are eligible. Such limitations necessitate the development of improved filtering mechanisms to enhance search precision.

Since the relatively recent advent of LLMs, a growing cohort of researchers has used them to connect patients with clinical trials in various stages of the clinical trial pipeline.

Hamer et al.⁷ use InstructGPT to partially automate the pre-screening process for patients by cross-referencing 10 synthesized medical profiles of candidates with eligibility criteria of trials in clinicaltrials.gov. The model was able to correctly identify criteria as screenable with an accuracy of 72%.

Jin et al. create a comprehensive LLM (TrialGPT) that has 3 modules: a module (TrialGPT-Retrieval) that predicts patient eligibility based on trial criteria, a second module (TrialGPT-Matching) that predicts patient eligibility, and a TrialGPT-Ranking module that generates scores for each trial to facilitate ranking. TrialGPT was tested on 3 cohorts of 183 synthetic patients with 75 000 trial annotations.

Peikos et al⁸ use GPT to extract patient-related information from clinical notes and use this to retrieve pertinent clinical trials. GPT was used to construct a query to search for eligible clinical trials for the patient using the extracted information. Their approach outperforms the baseline using TREC 2022 benchmarks.

Nievas et al⁹ investigated using both open source LLMs (Llama) and proprietary LLMs (GPT) for clinical trial retrieval. The use of open source LLMs mitigates concerns surrounding patient privacy and data leakage. To generate criterion-level explanations for each trial included in the TREC 2011 and 2022 benchmarks based on a patient’s summary with LLMs. They found that the performance of open source LLMs can surpass that of proprietary LLMs, given careful fine-tuning.

This article is an extension and improvement of previous work (CTMine).¹⁰ The CTMine system utilized regular expressions to verify that gene/mutation names were correctly represented in the wording of each clinical trial retrieved from ClinicalTrials.gov. To further refine the search for clinical trials by mutation, the authors created a machine learning model to exclude clinical trials not pertinent to a search for cancer mutation–focused clinical trials (such as those with double use or in exclusion circumstances documented above). The CTMine system focused on clinical trials that included patients with mutations in the ALK, BRAF, EGFR, ERBB2, KIT, and KRAS genes. Physicians (2-4 per trial) were then asked to find if clinical trials were relevant to a specific gene, and their responses were compared to the CTMine machine learning model. Results ranged between an F-score of 57.1% and 82.1%, depending on the gene mutation.

With the advent of Large Language Models, we explore whether GPT 4.0 and Gemini 2.0, when given a clinical trial and a gene mutation name via a prompt, would be able to ascertain whether a clinical trial was suited for a patient with that particular gene mutation, even with the presence of confusing information in the clinical trial, as previously mentioned. The positive results demonstrated that the 2 LLMs could successfully perform the proposed task and could be further improved with future fine-tuning of their parameters. The proposed methods are not intended to replace the clinical expertise of clinical research coordinators or physicians, but rather serve as a first step in identifying clinical trials that could be beneficial to patients with a specific cancer mutation. The system can be used for a single clinical trial or to scan a large corpus.

Tables 3 and 4 present the fivefold cross-validation results for all genes and GPT 4.0 and Gemini 2.0 using the DecisionTreeClassifier. The results for GPT show that KIT had the highest F1-score (82%, std = 0.06) but at the cost of its precision in relation to clinical trials marked as “wanted.” This same issue appeared with BRAF, where no clinical trials were chosen as “not_wanted.” Decision trees for ALK, EGFR, ERBB2, and KRAS had a more balanced approach between “wanted” and “not_wanted” tags. Overall F1-scores ranged between 49% and 82%, with the average F1-score being 64%.

Per Gemini KIT, the highest score was 83%. Overall F1-scores ranged between 57% and 83% with the average F1-score being 70%.

Table 5 compares the statistical differences between results for GPT 4.0 and Gemini 2.0 per gene. There are statistical differences between the 2 LLMs related to the ALK gene mutation, where GPT 4.0 performed better (75% vs 57%, P < .001), and the EGFR gene, where Gemini performed better (49% vs 57%, P < .001).

Tables 6 and 9 present the optimal parameters identified for both GPT-4 and Gemini 2.0. In this case, both decision trees for their respective LLMs did very well (88%-100% for GPT, 89%-100% for Gemini). Statistical significance cannot be reported, as only 1 model is trained using the top parameters against all training data. Due to this fact, the authors hesitate to use these findings as the definitive results of the system and would instead defer to the results in Tables 3 and 4. However, the decision trees generated by these ultra-fine-tuned models provide us with an interesting insight into the model’s choices from a natural language processing perspective, and we will highlight some of the more notable decisions per gene and LLM.

Starting with Figure 4, which displays the decision tree for ALK mutation and GPT 4.0, it appears that if the word “not” occurs frequently, the clinical trial is not useful for a patient with the ALK cancer mutation, which is unsurprising. This finding is reflected in the following clinical trial (NCT01838577) response from GPT 4.0, where the word “not” appears twice, indicating that a patient with an ALK gene mutation would not be eligible for the trial.

Based on the clinical trial eligibility criteria you provided, a patient with an ALK gene mutation would not be eligible for this trial. The study population specifically includes:

Gemini focused on the presence of the NSCLC (Non-Small Cell Lung Cancer) acronym in Figure 10. It ruled that the clinical trial was inappropriate for the patient with an ALK mutation if that acronym was present. NSCLC was found in the eligibility criteria for the trial NCT01838577:

"eligibilityCriteria": "* Histologically or cytologically diagnosed NSCLC, all histologies are acceptable.* Patients can be included in the study with any disease stage and at any time during the disease course.* Any type (surgery, RadioTherapy, chemotherapy, targeted agents) of previous treatment and any line of treatment are eligible. . .

According to the BRAF gene GPT 4.0, the words “not,” “wild,” “advised,” “cytological,” and “determination” indicated that the clinical trial was inappropriate for the patient, as shown in Figure 5. An example of the occurrence of these words is given in trial NCT01358812 per the eligibility criteria and the GPT 4.0 answer itself:

"eligibilityCriteria": "Inclusion Criteria:* Histologically confirmed colorectal adenocarcinoma;* Availability of formalin-fixed paraffin embedded tumor block from primary or metastasis;* KRAS and BRAF wild-type status of primary colorectal cancer or related metastasis;* :

Question:

Based on the following clinical trial information, would a patient with a BRAF gene mutation be eligible for the clinical trial?

Answer:

Based on the eligibility criteria provided for the clinical trial, a patient with a BRAF gene mutation would not be eligible to participate. The inclusion criteria specifically require "KRAS and BRAF wild-type status of primary colorectal cancer or related metastasis." This means the patient’s colorectal cancer must not have mutations in either the KRAS or BRAF genes to be eligible for the trial. Therefore, a patient with a BRAF gene mutation would not meet these criteria and would be considered ineligible for treatment.

The decision tree (Figure 11) chose different words for BRAF per Gemini’s responses to rule out a clinical trial, such as therefore and BRAF, highlighted below in Gemini’s answer for study NCT01838577:

Question:

Based on the following clinical trial information, would a patient with a BRAF gene mutation be eligible for the clinical trial?

Answer:

No. While the criteria state that patients with NSCLC (Non-Small Cell Lung Cancer) at any stage and with any prior treatment are eligible, the *studyPopulation* section specifies that this trial is for patients with proven EGFR mutations or those with wild-type EGFR (no EGFR mutations). The information provided describes a study looking at EGFR, not BRAF. Therefore, a patient with a BRAF mutation would *not* be eligible for this particular trial.

Per the ERBB2 mutation (Figure 7), the decision tree for GPT 4.0 data found that the words overexpression, eligibility, specify, and expression were keywords that determined if a clinical trial was appropriate for a patient. For example, in trial NCT01248897, the word overexpression occurs twice, marking this clinical trial as inappropriate for a patient with the ERBB2 mutation.

Question:

Based on the following clinical trial information, would a patient with an ERBB2 gene mutation be eligible for the clinical trial?

Answer:

Based on the clinical trial information provided, the eligibility for the trial is primarily based on HER2 status and prior treatment history. The trial mentions inclusion criteria for HER2-positive breast cancer patients, which are determined by either IHC3+ or FISH+ assays, indicating overexpression or amplification of the HER2 protein.

The ERBB2 gene, also known as HER2, is responsible for the production of the HER2 protein. Therefore, if a patient has an ERBB2 gene mutation that results in HER2 positivity (such as overexpression or amplification), they would meet the HER2-positive criteria part of the inclusion.

For Gemini and ERBB2 (Figure 13), the decision tree found that the appearance of words exclusion and carcinoma ruled out a clinical trial, reflected in trial NCT00863122:

"eligibilityCriteria": "Inclusion Criteria:* Concurrent or prior malignancy, other than curatively treated carcinoma-in-situ or basal cell carcinoma of the skin

Question:

Based on the following clinical trial information, would a patient with a ERBB2 gene mutation be eligible for the clinical trial?

Answer:

This trial is for patients with neurofibromatosis type 2 (NF2), indicated by the inclusion criteria requiring a diagnosis of NF2. The criteria mention nothing about ERBB2 mutations. . . . To be eligible, they would need to meet *all* other inclusion criteria and *none* of the exclusion criteria.

For the KIT gene, mutation, answer, concurrent, and factor were the most important words used to determine if a clinical trial was acceptable per GPT 4.0 (Figure 8). For example, in trial NCT01806571, the word “concurrent” was important in determining that the trial was suitable for a patient with a KIT mutation.

“eligibilityCriteria”: “Exclusion Criteria” . . . Co-morbid systemic illnesses or other severe concurrent disease which, in the judgment of the investigator, would make the patient inappropriate for entry into this study or interfere significantly with the proper assessment of safety and toxicity of the prescribed regimens

Question:

Based on the following clinical trial information, would a patient with a KIT gene mutation be eligible for the clinical trial?

Answer:

Based on the provided eligibility criteria for the clinical trial, a patient with a KIT gene mutation would not automatically be excluded from participation. The inclusion criteria specifically mention acute myeloid leukemia (AML) with Kit (CD117) expression, which can be a common feature in patients with a KIT gene mutation. The trial focuses on KIT expression rather than directly on the presence or absence of a KIT gene mutation.

However, it is important to note that there are multiple inclusion and exclusion criteria that must be met or avoided. The patient’s overall health status, previous treatments, and concurrent medications, among other factors, must comply with the trial’s requirements as listed in the other criteria.

The decision tree for Gemini 2.0 per the KIT mutation (Figure 14) found that the words answer, history, factor and needed were important for classifying a clinical trial. Per the last clinical trial we see that none of those words are present, and this would indicate that the trial NCT01806571 was appropriate for a patient with the KIT mutation.

Finally, per the KRAS mutation, the decision tree for GPT 4.0 (Figure 9) found that the words in, therefore, kras, administration, are and wild were all important in determining whether the clinical trial was appropriate. For example, in trial NCT02129257, the words indicate that the trial is appropriate.

Exclusion criteria: . . . Male patients with a partner of childbearing potential must agree to use contraception in addition to having their partner use another contraceptive method during the trial. . . .Any of the following in 3 months before inclusion: grade 3-4 gastrointestinal bleeding (unless due to resected tumor), treatment resistant peptic ulcer disease, erosive esophagitis or gastritis, infectious or inflammatory bowel disease, or diverticulitis* Other concomitant or previous malignancy, except: * adequately treated in-situ carcinoma of the uterine cervix * basal or squamous cell carcinoma of the skin * cancer in complete remission for \\>5 years*. . .

Question:

Based on the following clinical trial information, would a patient with a KRAS gene mutation be eligible for the clinical trial?

Answer:

Based on the eligibility criteria provided for the clinical trial, there is no specific mention of a KRAS gene mutation being a criterion for exclusion or inclusion. The trial does not specify genetic markers like KRAS mutations as part of the eligibility or exclusion criteria. Therefore, a patient with a KRAS gene mutation would still be eligible as long as they meet the other inclusion criteria and do not meet any of the exclusion criteria listed

Per the Gemini responses, the decision tree generated found that the words state, if, therefore, this and the were important indicators of trial appropriateness (Figure 15). For example, per the last example, the exclusion of the word state and the inclusion of the word indicates that the clinical trial is appropriate.

Per fivefold cross validation and using the DecisionTreeClassifier, overall F1-scores for GPT 4.0 ranged between 49% and 82%, with the average F1-score being 64%. Overall F1-scores for Gemini ranged between 57 and 83% with the average F1-score being 70% There are statistical differences between the 2 LLMs related to the ALK gene mutation, where GPT 4.0 performed better (75% vs 57%, P < .001) and the EGFR gene where Gemini performed better (49% vs 57%, P < .001).

In the case of Decision Tree parameter optimization both decision trees for their respective LLMs did very well (88%-100% for GPT, 89%-100% for Gemini). Statistical significance cannot be reported as only 1 model is trained via top parameters, against all training data. Due to this fact the authors hesitate to use these findings as the de facto results. However, the decision trees generated by these ultra fine-tuned models give us an interesting insight into the model’s choices from a natural language processing perspective.

Our proposed system differs and expands on previous work with respect to several points. First, the proposed system does not start from patient profiles or notes, but begins with a specific gene mutation. The resulting performance of the system demonstrates that having only this information can be beneficial, however the system could a module that extracts information from patient profiles could easily be added to the front end of our system. Secondly, this system does not use a benchmark but instead uses a collection of trials that have been hand curated by oncology residents, resulting in novel insights. Another point of impact is that the proprietary LLMs used (GPT 4.0 and Gemini 2.0) were not fine tuned per the data, and still performed well (64% GPT 4.0 and 70% Gemini 2.0). Future Work includes fine-tuning both LLMs, using a combination of proprietary and open-source LLMs. However this work demonstrates the feasibility of using proprietary LLMs “off the shelf” with limited patient information to evaluate clinical trial eligibility.