pubmedbert-pmid-tapt

Domain-adapted PubMedBERT (a.k.a. BiomedBERT) for the kinase / small-molecule drug-discovery literature. The base encoder was continue-pretrained with masked language modelling (MLM) — i.e. task-adaptive pretraining (TAPT) — on unlabelled, ChEMBL-linked PubMed abstracts. It is intended as an encoder backbone for downstream biomedical information extraction (NER + relation extraction of potency/binding claims), not as a standalone predictor.

• Base model: microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext
• Adaptation: self-supervised MLM (15% dynamic masking), no labels
• Domain: kinase inhibitors, IC50/Ki/Kd/EC50 potency, medicinal-chemistry abstracts
• Architecture: BERT encoder (BertForMaskedLM), unchanged from the base

Intended use & limitations

Intended use. Fine-tune or feature-extract for biomedical extraction tasks — chemical/gene NER, (compound, relation, target) relation extraction, and atomic-fact-claim (AFC) extraction in a fact-verification pipeline. Usable directly for fill-mask probing.

Out of scope. Not a clinical, diagnostic, or safety tool. It does not verify facts by itself and must not be used for medical decisions. Numeric spans (values/units) are subword- tokenised, so the encoder is weak at exact numeric reasoning — bind measurements with rules downstream.

Training data

• Corpus: pmid_coverage_corpus.jsonl — PubMed title + abstract text for PMIDs linked to curated ChEMBL kinase activities (produced by NB07 of the project).
• Size: ~4,026 abstracts, packed into 1662 blocks of 512 tokens; split train / val / test.
• Labels: none — the text is its own supervision (self-supervised MLM).

Training procedure

Hyperparameter	Value
Objective	Masked LM (MLM), 15% dynamic masking
Block size	512 tokens
Epochs	3
Learning rate	2e-05
LR schedule	linear + warmup (ratio 0.06)
Weight decay	0.01
Optimizer	AdamW
Mixed precision	fp16 (GPU)
Seed	42
Selection metric	validation eval_loss (best checkpoint kept)

Evaluation

Perplexity on the held-out test split (lower is better). The meaningful signal is the drop from base → adapted on the same split.

Model	Val loss	Val ppl	Test loss	Test ppl
Base PubMedBERT	1.1065	3.02	1.0444	2.84
This model (adapted)	0.9488	2.58	0.9332	2.54

How to use

from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline

repo = "Sarbanidg/pubmedbert-pmid-tapt"
tok = AutoTokenizer.from_pretrained(repo)
mdl = AutoModelForMaskedLM.from_pretrained(repo)

fill = pipeline("fill-mask", model=mdl, tokenizer=tok)
fill("The compound inhibited EGFR with an IC50 of 12 [MASK].")

Provenance & reproducibility

Produced by notebook src/notebooks/08_pubmedbert_mlm_pretrain.ipynb (§1–§6) in the med_llm_factcheck project. CLI/local equivalent: util/pretrain_pubmedbert_mlm.py.

License

MIT, inherited from the base model microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext.