Hugging Face's logo

Zoha4
/
CortX-340M

Safetensors
gla
Model card Files
xet

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

CortX-340M-1.3B-2k

A 340M parameter GLA (Gated Linear Attention) model.

Model Details

  • Architecture: GLA
  • Parameters: 340M
  • Tokens: 1.3B
  • Dataset: Fineweb-Edu
  • Context length: 2k
  • Training steps: 20480

Usage 

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Zoha4/cortx-340M-1.3B-2k")
tokenizer = AutoTokenizer.from_pretrained("Zoha4/cortx-340M-1.3B-2k")

References

  1. Yang et al. Gated Linear Attention Transformers with Hardware-Efficient Training. ICML, 2024. PDF

  2. Zellers et al. HellaSwag: Can a Machine Really Finish Your Sentence? In ACL, 2019. PDF

  3. Penedo et al. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, 2024. Blog · Fineweb-Edu

Downloads last month
10
Safetensors
Model size
0.4B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Papers for Zoha4/CortX-340M