An Empirical Comparative Evaluation of BERT-Family and LLM-Family Models for Automatic CTCAE Severity Grading from Structured FDA FAERS Report Composites

Abstract

Severity grading of adverse events under the Common Terminology Criteria for Adverse Events (CTCAE) is a labor-intensive yet decision-critical task in clinical trial monitoring. While public adverse event repositories offer vast unstructured adverse event records, native CTCAE 1--5 labels are rarely available. This study presents an empirical comparative evaluation of two model families on automatic CTCAE grading: BERT-family encoders (BioBERT, ClinicalBERT, PubMedBERT) and large language models (GPT-3.5, GPT-4o, Me-LLaMA-13B). Using FDA FAERS quarterly extracts from 2018 to 2024 as the primary corpus, we constructed silver labels through a MedDRA-to-CTCAE heuristic mapping and curated a 2,000-report gold subset via dual pharmacist annotation reaching Cohen's κ = 0.78. Four supplementary corpora supported transfer evaluation. Under matched protocols, PubMedBERT-large attained the strongest in-domain performance (5-grade macro-F1 = 0.713, quadratic-weighted κ = 0.741), while LoRA-adapted Me-LLaMA-13B exhibited the most stable cross-domain behavior with transfer loss limited to 4.6 macro-F1 points. Zero-shot LLMs achieved competitive recall but lower precision and weaker calibration. The results suggest that supervised encoders remain economical for stable in-domain pipelines, whereas adapted medium-scale medical LLMs are preferable when distribution shift dominates. We release the silver-labeling rules, prompts, and evaluation harness to support reproducibility.