TEDDY

01

Overview

TEDDY—Temporal Event Decoder for Disease in Youth—is a transformer-based model for predicting the first occurrence of ICD-10 diagnoses from a patient’s longitudinal event history. The work treats clinical records as temporal sequences and focuses on learning the patterns that precede a future diagnosis.

This is model-development work rather than an application layered over a general-purpose LLM: the transformer is pretrained from scratch on the task’s clinical sequence representation.

02

Problem

Longitudinal health records are irregular, sparse, and highly variable in length. Useful forecasting requires a representation that preserves event order while remaining tractable across millions of patient events, plus an evaluation design that prevents future information from leaking into the prediction window.

03

Approach

The system uses a nanoGPT-style autoregressive transformer and a task-specific vocabulary over clinical events. More than 150 million raw events were tokenized, with roughly 70 million retained after filtering. Sequence-subsampling augmentation exposes the model to varied slices of patient history during training.

Evaluation compares disease-specific predictions against a Gemma-3-4B baseline on the same task and split, so model results are attributable to the modeling approach rather than a more favorable test set.

04

My contribution

I led the project end to end: defining the sequence formulation, building the tokenizer and data pipeline, implementing from-scratch transformer pretraining, adding sequence-subsampling augmentation, and designing the evaluation framework. The work also included large-scale GPU training and baseline comparison.

05

Evaluation & results

TEDDY reached 0.793 AUROC for asthma and 0.847 for ADHD. On the same task and evaluation split, the Gemma-3-4B baseline reached 0.627 and 0.512 respectively. These results support the value of task-specific clinical sequence pretraining for the evaluated prediction targets.

06

Constraints & lessons

Clinical prediction metrics do not by themselves establish clinical utility. Generalization, calibration, subgroup behavior, and prospective validation remain separate questions. Public descriptions intentionally omit patient-level data and internal infrastructure details.