Headless Language Models: Learning without Predicting with Contrastive Weight Tying
Paper • 2309.08351 • Published • 3
# Load model directly
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("nthngdy/headless-bert-bs64-owt2")
model = AutoModelForMaskedLM.from_pretrained("nthngdy/headless-bert-bs64-owt2", device_map="auto")Quick Links
Model Sources
- Repository: https://github.com/NathanGodey/headless-lm
- Paper: https://arxiv.org/abs/2309.08351
Model Architecture and Objective
This model is a bert-base architecture trained on OpenWebText-2 using the Contrastive Weight Tying objective.
Citation
BibTeX:
@misc{godey2023headless,
title={Headless Language Models: Learning without Predicting with Contrastive Weight Tying},
author={Nathan Godey and Éric de la Clergerie and Benoît Sagot},
year={2023},
eprint={2309.08351},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
Contact
- Downloads last month
- 6
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="nthngdy/headless-bert-bs64-owt2")