Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,193 @@
+---
+language:
+  - cu
+  - orv
+license: apache-2.0
+library_name: transformers
+base_model: lightonai/LightOnOCR-2-1B-base
+tags:
+  - ocr
+  - old-church-slavonic
+  - ocs
+  - manuscript
+  - vision-language
+  - lightonocr
+  - document-understanding
+datasets:
+  - CATMuS/medieval
+pipeline_tag: image-text-to-text
+---
+
+# LightOnOCR-2-1B for Old Church Slavonic (Line-Level)
+
+<p align="center">
+  <img src="https://huggingface.co/lightonai/LightOnOCR-2-1B-base/resolve/main/lightonocr-banner.png" alt="LightOnOCR Banner" width="600"/>
+</p>
+
+This model is a **fine-tuned version of [lightonai/LightOnOCR-2-1B-base](https://huggingface.co/lightonai/LightOnOCR-2-1B-base)** specifically trained for **line-level OCR** of Old Church Slavonic (OCS) manuscripts.
+
+## Model Description
+
+- **Base Model:** [lightonai/LightOnOCR-2-1B-base](https://huggingface.co/lightonai/LightOnOCR-2-1B-base)
+- **Training Data:** [CATMuS/medieval](https://huggingface.co/datasets/CATMuS/medieval)
+- **Task:** Line-level text transcription from manuscript images
+- **Language:** Old Church Slavonic (cu/orv)
+- **Architecture:** Vision-Language Model (1B parameters)
+
+This is a **line-level model** - it expects cropped line images as input, not full pages. Each image should contain a single line of text.
+
+## Evaluation Results
+
+Evaluated on 50 samples from the test set:
+
+| Metric | Base Model | **Finetuned** | Improvement |
+|--------|------------|---------------|-------------|
+| CER (%) | 194.75 | **41.93** | +152.82 |
+| WER (%) | 218.38 | **70.19** | +148.19 |
+| Perfect Matches | 0 | **4** | +4 |
+
+*Lower CER/WER is better. Higher perfect matches is better.*
+
+### Example Outputs
+
+| # | Ground Truth | Base Model | **Finetuned** |
+|---|--------------|------------|---------------|
+| 1 | dos cartas anios: de un deuor Esta carta... |  |  cas casas duesa de iom esas apos fue fre... |
+| 2 | cl̃igo de yung̃ra. Et yo Maran gañz esc... |  |  T̾tore te N̾um̃m. Et coa delapto h̃oꝵ aͣ... |
+| 3 | ssegũd q̃ enella diçe. Siño q̃l qͥer ... |  |   |
+| 4 | consseiõ ⁊ con otorgamiento dela Reyna ... |  |   |
+| 5 | tos ⁊ con todas sus ꝑtenençias ⁊ con to... | তত্ত্ব ন এম সোলার নীতি যুক্তিমানতা ন এম ... |  tõt ⁊ en tõtas sus prenengas ⁊ en tõtas ... |
+
+*✓ = exact match*
+
+## Usage
+
+### Installation
+
+```bash
+# Requires transformers from source
+pip install git+https://github.com/huggingface/transformers
+pip install pillow torch
+```
+
+### Python Usage
+
+```python
+import torch
+from transformers import LightOnOcrForConditionalGeneration, LightOnOcrProcessor
+from PIL import Image
+
+# Load model and processor
+model_id = "wjbmattingly/LightOnOCR-2-1B-catmus"
+device = "cuda" if torch.cuda.is_available() else "cpu"
+dtype = torch.bfloat16 if device == "cuda" else torch.float32
+
+processor = LightOnOcrProcessor.from_pretrained(model_id)
+model = LightOnOcrForConditionalGeneration.from_pretrained(
+    model_id,
+    torch_dtype=dtype,
+).to(device)
+
+# Load your line image
+image = Image.open("your_line_image.jpg").convert("RGB")
+
+# Prepare input
+messages = [{"role": "user", "content": [{"type": "image"}]}]
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+
+inputs = processor(
+    text=[text],
+    images=[[image]],
+    return_tensors="pt",
+    padding=True,
+    size={"longest_edge": 700},
+).to(device)
+inputs["pixel_values"] = inputs["pixel_values"].to(dtype)
+
+# Generate transcription
+with torch.no_grad():
+    outputs = model.generate(**inputs, max_new_tokens=256, do_sample=False)
+
+# Decode output
+input_length = inputs["input_ids"].shape[1]
+generated_ids = outputs[0, input_length:]
+transcription = processor.decode(generated_ids, skip_special_tokens=True)
+
+print(transcription)
+# Example output: дьници въсиꙗють рєчє сл҃нцє ꙗко лоуна∙ ꙗко
+```
+
+### Batch Inference
+
+```python
+from datasets import load_dataset
+
+# Load dataset
+dataset = load_dataset("CATMuS/medieval", split="train[:10]")
+
+# Process batch
+images = [[img.convert("RGB")] for img in dataset["image"]]
+messages = [{"role": "user", "content": [{"type": "image"}]}]
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+texts = [text] * len(images)
+
+inputs = processor(
+    text=texts,
+    images=images,
+    return_tensors="pt",
+    padding=True,
+    size={"longest_edge": 700},
+).to(device)
+inputs["pixel_values"] = inputs["pixel_values"].to(dtype)
+
+outputs = model.generate(**inputs, max_new_tokens=256, do_sample=False)
+predictions = processor.batch_decode(outputs[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+
+for pred, gt in zip(predictions, dataset["text"]):
+    print(f"Prediction: {pred}")
+    print(f"Ground Truth: {gt}")
+    print()
+```
+
+## Training Details
+
+- **Base Model:** [lightonai/LightOnOCR-2-1B-base](https://huggingface.co/lightonai/LightOnOCR-2-1B-base)
+- **Training Method:** Fine-tuning with frozen language model backbone
+- **Optimizer:** AdamW (fused)
+- **Learning Rate:** 6e-5 with linear decay
+- **Precision:** bfloat16
+
+## Limitations
+
+- This model is trained on **line-level images** only. For full-page transcription, you need to first segment the page into individual lines.
+- Performance may vary on manuscript styles not represented in the training data.
+- Old Church Slavonic has many abbreviations and special characters that may require domain-specific post-processing.
+
+## Citation
+
+If you use this model, please cite:
+
+```bibtex
+@misc{lightonocr2_ocs_2026,
+  title = {LightOnOCR Fine-tuned for Old Church Slavonic},
+  author = {William Mattingly},
+  year = {2026},
+  howpublished = {\url{https://huggingface.co/wjbmattingly/LightOnOCR-2-1B-catmus}}
+}
+```
+
+And the original LightOnOCR paper:
+
+```bibtex
+@misc{lightonocr2_2026,
+  title = {LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCR},
+  author = {Said Taghadouini and Adrien Cavaill\`{e}s and Baptiste Aubertin},
+  year = {2026},
+  howpublished = {\url{https://arxiv.org/pdf/2601.14251}}
+}
+```
+
+## Acknowledgments
+
+- [LightOn AI](https://www.lighton.ai/) for the excellent LightOnOCR base model
+- The creators of the Old Church Slavonic dataset

adapter_config.json

@@ -0,0 +1,43 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "lightonai/LightOnOCR-2-1B-base",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "gate_proj",
+    "up_proj",
+    "down_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:933ec05c7e79009cbbd02c4f3e78d23680d4991fcbe3287fe28a5d3be666a235
+size 27195864

chat_template.jinja

@@ -0,0 +1,26 @@
+{% set image_count = namespace(value=0) %}
+{% for message in messages %}
+{% if loop.first and message['role'] != 'system' %}
+<|im_start|>system<|im_end|>
+{% endif %}
+<|im_start|>{{ message['role'] }}
+{% if message['content'] is string %}
+{{ message['content'] }}<|im_end|>
+{% else %}
+{% for content in message['content'] %}
+{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}
+{% set image_count.value = image_count.value + 1 %}
+{% if add_vision_id %}
+Picture {{ image_count.value }}:
+{% endif -%}
+<|image_pad|>
+{%- elif 'text' in content %}
+{{ content['text'] }}
+{% endif %}
+{% endfor %}
+<|im_end|>
+{% endif %}
+{% endfor %}
+{% if add_generation_prompt %}
+<|im_start|>assistant
+{% endif %}

optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4445e0e9a16aadae19a633cd6cde22dd5e5766c581278f1cd6c1f72efce11d4f
+size 54619587

processor_config.json

@@ -0,0 +1,42 @@
+{
+  "image_processor": {
+    "crop_size": null,
+    "data_format": "channels_first",
+    "default_to_square": true,
+    "device": null,
+    "disable_grouping": null,
+    "do_center_crop": null,
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_pad": null,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.48145466,
+      0.4578275,
+      0.40821073
+    ],
+    "image_processor_type": "PixtralImageProcessor",
+    "image_seq_length": null,
+    "image_std": [
+      0.26862954,
+      0.26130258,
+      0.27577711
+    ],
+    "input_data_format": null,
+    "pad_size": null,
+    "patch_size": {
+      "height": 14,
+      "width": 14
+    },
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_tensors": null,
+    "size": {
+      "longest_edge": 1540
+    }
+  },
+  "patch_size": 14,
+  "processor_class": "LightOnOcrProcessor",
+  "spatial_merge_size": 2
+}

rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:030da978017015da3fe26f53d72a3b732670056e0706187eb6457a22768b1863
+size 14645

scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6bf3a357417911ae5868483660e5ebab7ca7bf876831d53ab728de3c18fc0ceb
+size 1465

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f54b55fa0c3aba0c91ce09ea79ae4e62da24e2a1a630f96c4bae34aba25e234a
+size 11422822

tokenizer_config.json

@@ -0,0 +1,38 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "image_break_token": "<|vision_pad|>",
+  "image_end_token": "<|vision_end|>",
+  "image_token": "<|image_pad|>",
+  "is_local": false,
+  "max_length": null,
+  "model_max_length": 131072,
+  "model_specific_special_tokens": {},
+  "pad_to_multiple_of": null,
+  "pad_token": "<|endoftext|>",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "processor_class": "LightOnOcrProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null
+}

trainer_state.json

@@ -0,0 +1,1234 @@
+{
+  "best_global_step": 3900,
+  "best_metric": 0.6521520018577576,
+  "best_model_checkpoint": "output/LightOnOCR-ft-catmus/checkpoint-2000",
+  "epoch": 0.4399593037644018,
+  "eval_steps": 50,
+  "global_step": 4000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0054994912970550225,
+      "grad_norm": 1.7269556522369385,
+      "learning_rate": 5.9871244635193135e-05,
+      "loss": 3.2000225830078124,
+      "step": 50
+    },
+    {
+      "epoch": 0.0054994912970550225,
+      "eval_loss": 2.544747829437256,
+      "eval_runtime": 1.22,
+      "eval_samples_per_second": 81.966,
+      "eval_steps_per_second": 13.934,
+      "step": 50
+    },
+    {
+      "epoch": 0.010998982594110045,
+      "grad_norm": 2.1817049980163574,
+      "learning_rate": 5.970617365467151e-05,
+      "loss": 2.204842987060547,
+      "step": 100
+    },
+    {
+      "epoch": 0.010998982594110045,
+      "eval_loss": 1.9304633140563965,
+      "eval_runtime": 1.2142,
+      "eval_samples_per_second": 82.358,
+      "eval_steps_per_second": 14.001,
+      "step": 100
+    },
+    {
+      "epoch": 0.01649847389116507,
+      "grad_norm": 2.861886739730835,
+      "learning_rate": 5.954110267414989e-05,
+      "loss": 1.7260311889648436,
+      "step": 150
+    },
+    {
+      "epoch": 0.01649847389116507,
+      "eval_loss": 1.7684886455535889,
+      "eval_runtime": 1.2169,
+      "eval_samples_per_second": 82.178,
+      "eval_steps_per_second": 13.97,
+      "step": 150
+    },
+    {
+      "epoch": 0.02199796518822009,
+      "grad_norm": 2.9910781383514404,
+      "learning_rate": 5.937603169362826e-05,
+      "loss": 1.5795289611816405,
+      "step": 200
+    },
+    {
+      "epoch": 0.02199796518822009,
+      "eval_loss": 1.5510659217834473,
+      "eval_runtime": 1.2158,
+      "eval_samples_per_second": 82.249,
+      "eval_steps_per_second": 13.982,
+      "step": 200
+    },
+    {
+      "epoch": 0.027497456485275112,
+      "grad_norm": 3.234337329864502,
+      "learning_rate": 5.921096071310663e-05,
+      "loss": 1.4427146911621094,
+      "step": 250
+    },
+    {
+      "epoch": 0.027497456485275112,
+      "eval_loss": 1.4127395153045654,
+      "eval_runtime": 1.2374,
+      "eval_samples_per_second": 80.816,
+      "eval_steps_per_second": 13.739,
+      "step": 250
+    },
+    {
+      "epoch": 0.03299694778233014,
+      "grad_norm": 3.6628358364105225,
+      "learning_rate": 5.904588973258501e-05,
+      "loss": 1.3803778076171875,
+      "step": 300
+    },
+    {
+      "epoch": 0.03299694778233014,
+      "eval_loss": 1.3498808145523071,
+      "eval_runtime": 1.2372,
+      "eval_samples_per_second": 80.827,
+      "eval_steps_per_second": 13.741,
+      "step": 300
+    },
+    {
+      "epoch": 0.03849643907938516,
+      "grad_norm": 3.2028181552886963,
+      "learning_rate": 5.8880818752063386e-05,
+      "loss": 1.2370665740966797,
+      "step": 350
+    },
+    {
+      "epoch": 0.03849643907938516,
+      "eval_loss": 1.3298417329788208,
+      "eval_runtime": 1.2864,
+      "eval_samples_per_second": 77.736,
+      "eval_steps_per_second": 13.215,
+      "step": 350
+    },
+    {
+      "epoch": 0.04399593037644018,
+      "grad_norm": 3.344433307647705,
+      "learning_rate": 5.8715747771541766e-05,
+      "loss": 1.1885470581054687,
+      "step": 400
+    },
+    {
+      "epoch": 0.04399593037644018,
+      "eval_loss": 1.296607494354248,
+      "eval_runtime": 1.2337,
+      "eval_samples_per_second": 81.058,
+      "eval_steps_per_second": 13.78,
+      "step": 400
+    },
+    {
+      "epoch": 0.0494954216734952,
+      "grad_norm": 3.601663827896118,
+      "learning_rate": 5.855067679102014e-05,
+      "loss": 1.149856185913086,
+      "step": 450
+    },
+    {
+      "epoch": 0.0494954216734952,
+      "eval_loss": 1.264522910118103,
+      "eval_runtime": 1.2178,
+      "eval_samples_per_second": 82.114,
+      "eval_steps_per_second": 13.959,
+      "step": 450
+    },
+    {
+      "epoch": 0.054994912970550223,
+      "grad_norm": 4.889139175415039,
+      "learning_rate": 5.838560581049852e-05,
+      "loss": 1.0299188232421874,
+      "step": 500
+    },
+    {
+      "epoch": 0.054994912970550223,
+      "eval_loss": 1.2139334678649902,
+      "eval_runtime": 1.2138,
+      "eval_samples_per_second": 82.388,
+      "eval_steps_per_second": 14.006,
+      "step": 500
+    },
+    {
+      "epoch": 0.060494404267605245,
+      "grad_norm": 4.471691608428955,
+      "learning_rate": 5.822053482997689e-05,
+      "loss": 1.063873519897461,
+      "step": 550
+    },
+    {
+      "epoch": 0.060494404267605245,
+      "eval_loss": 1.1947441101074219,
+      "eval_runtime": 1.2397,
+      "eval_samples_per_second": 80.663,
+      "eval_steps_per_second": 13.713,
+      "step": 550
+    },
+    {
+      "epoch": 0.06599389556466027,
+      "grad_norm": 5.005523204803467,
+      "learning_rate": 5.805546384945527e-05,
+      "loss": 1.0306187438964844,
+      "step": 600
+    },
+    {
+      "epoch": 0.06599389556466027,
+      "eval_loss": 1.1054185628890991,
+      "eval_runtime": 1.2867,
+      "eval_samples_per_second": 77.72,
+      "eval_steps_per_second": 13.212,
+      "step": 600
+    },
+    {
+      "epoch": 0.07149338686171529,
+      "grad_norm": 5.433160781860352,
+      "learning_rate": 5.7890392868933643e-05,
+      "loss": 0.9948115539550781,
+      "step": 650
+    },
+    {
+      "epoch": 0.07149338686171529,
+      "eval_loss": 1.118335247039795,
+      "eval_runtime": 1.2905,
+      "eval_samples_per_second": 77.488,
+      "eval_steps_per_second": 13.173,
+      "step": 650
+    },
+    {
+      "epoch": 0.07699287815877032,
+      "grad_norm": 4.089204788208008,
+      "learning_rate": 5.7725321888412016e-05,
+      "loss": 0.9623870086669922,
+      "step": 700
+    },
+    {
+      "epoch": 0.07699287815877032,
+      "eval_loss": 1.1011381149291992,
+      "eval_runtime": 1.2137,
+      "eval_samples_per_second": 82.396,
+      "eval_steps_per_second": 14.007,
+      "step": 700
+    },
+    {
+      "epoch": 0.08249236945582533,
+      "grad_norm": 4.5177459716796875,
+      "learning_rate": 5.7560250907890396e-05,
+      "loss": 0.9822674560546875,
+      "step": 750
+    },
+    {
+      "epoch": 0.08249236945582533,
+      "eval_loss": 1.0355230569839478,
+      "eval_runtime": 1.2185,
+      "eval_samples_per_second": 82.066,
+      "eval_steps_per_second": 13.951,
+      "step": 750
+    },
+    {
+      "epoch": 0.08799186075288036,
+      "grad_norm": 5.314192295074463,
+      "learning_rate": 5.7395179927368776e-05,
+      "loss": 0.9478108978271484,
+      "step": 800
+    },
+    {
+      "epoch": 0.08799186075288036,
+      "eval_loss": 1.026939868927002,
+      "eval_runtime": 1.2207,
+      "eval_samples_per_second": 81.923,
+      "eval_steps_per_second": 13.927,
+      "step": 800
+    },
+    {
+      "epoch": 0.09349135204993538,
+      "grad_norm": 4.419743061065674,
+      "learning_rate": 5.723010894684714e-05,
+      "loss": 0.9230728149414062,
+      "step": 850
+    },
+    {
+      "epoch": 0.09349135204993538,
+      "eval_loss": 1.0253421068191528,
+      "eval_runtime": 1.2215,
+      "eval_samples_per_second": 81.867,
+      "eval_steps_per_second": 13.917,
+      "step": 850
+    },
+    {
+      "epoch": 0.0989908433469904,
+      "grad_norm": 4.689808368682861,
+      "learning_rate": 5.706503796632552e-05,
+      "loss": 0.9336647033691406,
+      "step": 900
+    },
+    {
+      "epoch": 0.0989908433469904,
+      "eval_loss": 0.9888076782226562,
+      "eval_runtime": 1.2198,
+      "eval_samples_per_second": 81.981,
+      "eval_steps_per_second": 13.937,
+      "step": 900
+    },
+    {
+      "epoch": 0.10449033464404543,
+      "grad_norm": 4.178891181945801,
+      "learning_rate": 5.68999669858039e-05,
+      "loss": 0.8509888458251953,
+      "step": 950
+    },
+    {
+      "epoch": 0.10449033464404543,
+      "eval_loss": 0.9283347129821777,
+      "eval_runtime": 1.2217,
+      "eval_samples_per_second": 81.854,
+      "eval_steps_per_second": 13.915,
+      "step": 950
+    },
+    {
+      "epoch": 0.10998982594110045,
+      "grad_norm": 4.549170017242432,
+      "learning_rate": 5.6734896005282274e-05,
+      "loss": 0.8530999755859375,
+      "step": 1000
+    },
+    {
+      "epoch": 0.10998982594110045,
+      "eval_loss": 0.9180505871772766,
+      "eval_runtime": 1.2178,
+      "eval_samples_per_second": 82.114,
+      "eval_steps_per_second": 13.959,
+      "step": 1000
+    },
+    {
+      "epoch": 0.11548931723815548,
+      "grad_norm": 5.8048095703125,
+      "learning_rate": 5.656982502476065e-05,
+      "loss": 0.8161511993408204,
+      "step": 1050
+    },
+    {
+      "epoch": 0.11548931723815548,
+      "eval_loss": 0.9668211936950684,
+      "eval_runtime": 1.2152,
+      "eval_samples_per_second": 82.292,
+      "eval_steps_per_second": 13.99,
+      "step": 1050
+    },
+    {
+      "epoch": 0.12098880853521049,
+      "grad_norm": 4.776002883911133,
+      "learning_rate": 5.6404754044239026e-05,
+      "loss": 0.8362539672851562,
+      "step": 1100
+    },
+    {
+      "epoch": 0.12098880853521049,
+      "eval_loss": 0.9686654806137085,
+      "eval_runtime": 1.21,
+      "eval_samples_per_second": 82.647,
+      "eval_steps_per_second": 14.05,
+      "step": 1100
+    },
+    {
+      "epoch": 0.1264882998322655,
+      "grad_norm": 4.652673721313477,
+      "learning_rate": 5.62396830637174e-05,
+      "loss": 0.8160650634765625,
+      "step": 1150
+    },
+    {
+      "epoch": 0.1264882998322655,
+      "eval_loss": 0.9779856204986572,
+      "eval_runtime": 1.2121,
+      "eval_samples_per_second": 82.5,
+      "eval_steps_per_second": 14.025,
+      "step": 1150
+    },
+    {
+      "epoch": 0.13198779112932055,
+      "grad_norm": 4.988180160522461,
+      "learning_rate": 5.607461208319577e-05,
+      "loss": 0.8510865020751953,
+      "step": 1200
+    },
+    {
+      "epoch": 0.13198779112932055,
+      "eval_loss": 0.9197137951850891,
+      "eval_runtime": 1.2363,
+      "eval_samples_per_second": 80.884,
+      "eval_steps_per_second": 13.75,
+      "step": 1200
+    },
+    {
+      "epoch": 0.13748728242637556,
+      "grad_norm": 5.33247709274292,
+      "learning_rate": 5.590954110267415e-05,
+      "loss": 0.8393131256103515,
+      "step": 1250
+    },
+    {
+      "epoch": 0.13748728242637556,
+      "eval_loss": 0.9012404084205627,
+      "eval_runtime": 1.2371,
+      "eval_samples_per_second": 80.834,
+      "eval_steps_per_second": 13.742,
+      "step": 1250
+    },
+    {
+      "epoch": 0.14298677372343058,
+      "grad_norm": 4.81198787689209,
+      "learning_rate": 5.5744470122152525e-05,
+      "loss": 0.7720973205566406,
+      "step": 1300
+    },
+    {
+      "epoch": 0.14298677372343058,
+      "eval_loss": 0.8723539710044861,
+      "eval_runtime": 1.2195,
+      "eval_samples_per_second": 81.999,
+      "eval_steps_per_second": 13.94,
+      "step": 1300
+    },
+    {
+      "epoch": 0.1484862650204856,
+      "grad_norm": 4.492185115814209,
+      "learning_rate": 5.5579399141630904e-05,
+      "loss": 0.8052091217041015,
+      "step": 1350
+    },
+    {
+      "epoch": 0.1484862650204856,
+      "eval_loss": 0.8542118668556213,
+      "eval_runtime": 1.2288,
+      "eval_samples_per_second": 81.379,
+      "eval_steps_per_second": 13.834,
+      "step": 1350
+    },
+    {
+      "epoch": 0.15398575631754063,
+      "grad_norm": 3.353337287902832,
+      "learning_rate": 5.541432816110928e-05,
+      "loss": 0.7565326690673828,
+      "step": 1400
+    },
+    {
+      "epoch": 0.15398575631754063,
+      "eval_loss": 0.8505687117576599,
+      "eval_runtime": 1.2947,
+      "eval_samples_per_second": 77.24,
+      "eval_steps_per_second": 13.131,
+      "step": 1400
+    },
+    {
+      "epoch": 0.15948524761459565,
+      "grad_norm": 4.658541202545166,
+      "learning_rate": 5.524925718058765e-05,
+      "loss": 0.7905632781982422,
+      "step": 1450
+    },
+    {
+      "epoch": 0.15948524761459565,
+      "eval_loss": 0.8416154980659485,
+      "eval_runtime": 1.3011,
+      "eval_samples_per_second": 76.857,
+      "eval_steps_per_second": 13.066,
+      "step": 1450
+    },
+    {
+      "epoch": 0.16498473891165066,
+      "grad_norm": 4.894554615020752,
+      "learning_rate": 5.508418620006603e-05,
+      "loss": 0.7522599029541016,
+      "step": 1500
+    },
+    {
+      "epoch": 0.16498473891165066,
+      "eval_loss": 0.8781094551086426,
+      "eval_runtime": 1.3077,
+      "eval_samples_per_second": 76.472,
+      "eval_steps_per_second": 13.0,
+      "step": 1500
+    },
+    {
+      "epoch": 0.1704842302087057,
+      "grad_norm": 5.099001407623291,
+      "learning_rate": 5.491911521954441e-05,
+      "loss": 0.7591278076171875,
+      "step": 1550
+    },
+    {
+      "epoch": 0.1704842302087057,
+      "eval_loss": 0.8455641269683838,
+      "eval_runtime": 1.3309,
+      "eval_samples_per_second": 75.138,
+      "eval_steps_per_second": 12.773,
+      "step": 1550
+    },
+    {
+      "epoch": 0.17598372150576072,
+      "grad_norm": 4.108503818511963,
+      "learning_rate": 5.4754044239022776e-05,
+      "loss": 0.7117195129394531,
+      "step": 1600
+    },
+    {
+      "epoch": 0.17598372150576072,
+      "eval_loss": 0.8469740152359009,
+      "eval_runtime": 1.3312,
+      "eval_samples_per_second": 75.123,
+      "eval_steps_per_second": 12.771,
+      "step": 1600
+    },
+    {
+      "epoch": 0.18148321280281574,
+      "grad_norm": 4.5479536056518555,
+      "learning_rate": 5.4588973258501155e-05,
+      "loss": 0.7624311828613282,
+      "step": 1650
+    },
+    {
+      "epoch": 0.18148321280281574,
+      "eval_loss": 0.8670312762260437,
+      "eval_runtime": 1.3395,
+      "eval_samples_per_second": 74.652,
+      "eval_steps_per_second": 12.691,
+      "step": 1650
+    },
+    {
+      "epoch": 0.18698270409987075,
+      "grad_norm": 4.809026718139648,
+      "learning_rate": 5.4423902277979535e-05,
+      "loss": 0.7137515258789062,
+      "step": 1700
+    },
+    {
+      "epoch": 0.18698270409987075,
+      "eval_loss": 0.8062080144882202,
+      "eval_runtime": 1.3269,
+      "eval_samples_per_second": 75.364,
+      "eval_steps_per_second": 12.812,
+      "step": 1700
+    },
+    {
+      "epoch": 0.1924821953969258,
+      "grad_norm": 5.400925636291504,
+      "learning_rate": 5.4258831297457914e-05,
+      "loss": 0.6972386932373047,
+      "step": 1750
+    },
+    {
+      "epoch": 0.1924821953969258,
+      "eval_loss": 0.8285601735115051,
+      "eval_runtime": 1.3236,
+      "eval_samples_per_second": 75.551,
+      "eval_steps_per_second": 12.844,
+      "step": 1750
+    },
+    {
+      "epoch": 0.1979816866939808,
+      "grad_norm": 6.0628461837768555,
+      "learning_rate": 5.409376031693628e-05,
+      "loss": 0.7547286987304688,
+      "step": 1800
+    },
+    {
+      "epoch": 0.1979816866939808,
+      "eval_loss": 0.8063403964042664,
+      "eval_runtime": 1.3254,
+      "eval_samples_per_second": 75.448,
+      "eval_steps_per_second": 12.826,
+      "step": 1800
+    },
+    {
+      "epoch": 0.20348117799103582,
+      "grad_norm": 5.176665306091309,
+      "learning_rate": 5.392868933641466e-05,
+      "loss": 0.7166018676757813,
+      "step": 1850
+    },
+    {
+      "epoch": 0.20348117799103582,
+      "eval_loss": 0.7977473735809326,
+      "eval_runtime": 1.3331,
+      "eval_samples_per_second": 75.013,
+      "eval_steps_per_second": 12.752,
+      "step": 1850
+    },
+    {
+      "epoch": 0.20898066928809086,
+      "grad_norm": 5.566205024719238,
+      "learning_rate": 5.376361835589304e-05,
+      "loss": 0.7202009582519531,
+      "step": 1900
+    },
+    {
+      "epoch": 0.20898066928809086,
+      "eval_loss": 0.7941867709159851,
+      "eval_runtime": 1.2966,
+      "eval_samples_per_second": 77.126,
+      "eval_steps_per_second": 13.112,
+      "step": 1900
+    },
+    {
+      "epoch": 0.21448016058514588,
+      "grad_norm": 6.071581840515137,
+      "learning_rate": 5.359854737537141e-05,
+      "loss": 0.718297119140625,
+      "step": 1950
+    },
+    {
+      "epoch": 0.21448016058514588,
+      "eval_loss": 0.8194384574890137,
+      "eval_runtime": 1.2884,
+      "eval_samples_per_second": 77.613,
+      "eval_steps_per_second": 13.194,
+      "step": 1950
+    },
+    {
+      "epoch": 0.2199796518822009,
+      "grad_norm": 5.566873073577881,
+      "learning_rate": 5.3433476394849786e-05,
+      "loss": 0.7216673278808594,
+      "step": 2000
+    },
+    {
+      "epoch": 0.2199796518822009,
+      "eval_loss": 0.7815269231796265,
+      "eval_runtime": 1.2875,
+      "eval_samples_per_second": 77.668,
+      "eval_steps_per_second": 13.204,
+      "step": 2000
+    },
+    {
+      "epoch": 0.2254791431792559,
+      "grad_norm": 5.567185878753662,
+      "learning_rate": 5.3268405414328165e-05,
+      "loss": 0.7164375305175781,
+      "step": 2050
+    },
+    {
+      "epoch": 0.2254791431792559,
+      "eval_loss": 0.8178561925888062,
+      "eval_runtime": 1.2891,
+      "eval_samples_per_second": 77.574,
+      "eval_steps_per_second": 13.188,
+      "step": 2050
+    },
+    {
+      "epoch": 0.23097863447631095,
+      "grad_norm": 4.171028137207031,
+      "learning_rate": 5.310333443380654e-05,
+      "loss": 0.6996173095703125,
+      "step": 2100
+    },
+    {
+      "epoch": 0.23097863447631095,
+      "eval_loss": 0.791594922542572,
+      "eval_runtime": 1.287,
+      "eval_samples_per_second": 77.698,
+      "eval_steps_per_second": 13.209,
+      "step": 2100
+    },
+    {
+      "epoch": 0.23647812577336597,
+      "grad_norm": 4.468926906585693,
+      "learning_rate": 5.293826345328491e-05,
+      "loss": 0.685142822265625,
+      "step": 2150
+    },
+    {
+      "epoch": 0.23647812577336597,
+      "eval_loss": 0.8206081986427307,
+      "eval_runtime": 1.2124,
+      "eval_samples_per_second": 82.479,
+      "eval_steps_per_second": 14.021,
+      "step": 2150
+    },
+    {
+      "epoch": 0.24197761707042098,
+      "grad_norm": 3.9659266471862793,
+      "learning_rate": 5.277319247276329e-05,
+      "loss": 0.701788330078125,
+      "step": 2200
+    },
+    {
+      "epoch": 0.24197761707042098,
+      "eval_loss": 0.7930644750595093,
+      "eval_runtime": 1.2014,
+      "eval_samples_per_second": 83.24,
+      "eval_steps_per_second": 14.151,
+      "step": 2200
+    },
+    {
+      "epoch": 0.247477108367476,
+      "grad_norm": 4.476805686950684,
+      "learning_rate": 5.2608121492241664e-05,
+      "loss": 0.710376968383789,
+      "step": 2250
+    },
+    {
+      "epoch": 0.247477108367476,
+      "eval_loss": 0.7881377339363098,
+      "eval_runtime": 1.1999,
+      "eval_samples_per_second": 83.337,
+      "eval_steps_per_second": 14.167,
+      "step": 2250
+    },
+    {
+      "epoch": 0.252976599664531,
+      "grad_norm": 3.8596489429473877,
+      "learning_rate": 5.244305051172004e-05,
+      "loss": 0.7032179260253906,
+      "step": 2300
+    },
+    {
+      "epoch": 0.252976599664531,
+      "eval_loss": 0.7591221332550049,
+      "eval_runtime": 1.2149,
+      "eval_samples_per_second": 82.31,
+      "eval_steps_per_second": 13.993,
+      "step": 2300
+    },
+    {
+      "epoch": 0.2584760909615861,
+      "grad_norm": 4.368955135345459,
+      "learning_rate": 5.2277979531198416e-05,
+      "loss": 0.6489437103271485,
+      "step": 2350
+    },
+    {
+      "epoch": 0.2584760909615861,
+      "eval_loss": 0.8142778873443604,
+      "eval_runtime": 1.2006,
+      "eval_samples_per_second": 83.294,
+      "eval_steps_per_second": 14.16,
+      "step": 2350
+    },
+    {
+      "epoch": 0.2639755822586411,
+      "grad_norm": 4.408853530883789,
+      "learning_rate": 5.211290855067679e-05,
+      "loss": 0.6344734954833985,
+      "step": 2400
+    },
+    {
+      "epoch": 0.2639755822586411,
+      "eval_loss": 0.789002001285553,
+      "eval_runtime": 1.2016,
+      "eval_samples_per_second": 83.219,
+      "eval_steps_per_second": 14.147,
+      "step": 2400
+    },
+    {
+      "epoch": 0.2694750735556961,
+      "grad_norm": 4.636422157287598,
+      "learning_rate": 5.194783757015517e-05,
+      "loss": 0.6603395080566407,
+      "step": 2450
+    },
+    {
+      "epoch": 0.2694750735556961,
+      "eval_loss": 0.739583432674408,
+      "eval_runtime": 1.2017,
+      "eval_samples_per_second": 83.216,
+      "eval_steps_per_second": 14.147,
+      "step": 2450
+    },
+    {
+      "epoch": 0.2749745648527511,
+      "grad_norm": 4.370145797729492,
+      "learning_rate": 5.178276658963355e-05,
+      "loss": 0.6568155670166016,
+      "step": 2500
+    },
+    {
+      "epoch": 0.2749745648527511,
+      "eval_loss": 0.8292286396026611,
+      "eval_runtime": 1.1996,
+      "eval_samples_per_second": 83.364,
+      "eval_steps_per_second": 14.172,
+      "step": 2500
+    },
+    {
+      "epoch": 0.28047405614980614,
+      "grad_norm": 4.47123384475708,
+      "learning_rate": 5.1617695609111914e-05,
+      "loss": 0.6639788055419922,
+      "step": 2550
+    },
+    {
+      "epoch": 0.28047405614980614,
+      "eval_loss": 0.820250391960144,
+      "eval_runtime": 1.2058,
+      "eval_samples_per_second": 82.933,
+      "eval_steps_per_second": 14.099,
+      "step": 2550
+    },
+    {
+      "epoch": 0.28597354744686115,
+      "grad_norm": 4.769968509674072,
+      "learning_rate": 5.1452624628590294e-05,
+      "loss": 0.6434815979003906,
+      "step": 2600
+    },
+    {
+      "epoch": 0.28597354744686115,
+      "eval_loss": 0.8330257534980774,
+      "eval_runtime": 1.2003,
+      "eval_samples_per_second": 83.313,
+      "eval_steps_per_second": 14.163,
+      "step": 2600
+    },
+    {
+      "epoch": 0.29147303874391617,
+      "grad_norm": 4.403727054595947,
+      "learning_rate": 5.1287553648068674e-05,
+      "loss": 0.6872652435302734,
+      "step": 2650
+    },
+    {
+      "epoch": 0.29147303874391617,
+      "eval_loss": 0.7691981792449951,
+      "eval_runtime": 1.1997,
+      "eval_samples_per_second": 83.351,
+      "eval_steps_per_second": 14.17,
+      "step": 2650
+    },
+    {
+      "epoch": 0.2969725300409712,
+      "grad_norm": 3.7627687454223633,
+      "learning_rate": 5.1122482667547047e-05,
+      "loss": 0.625857925415039,
+      "step": 2700
+    },
+    {
+      "epoch": 0.2969725300409712,
+      "eval_loss": 0.7727508544921875,
+      "eval_runtime": 1.2264,
+      "eval_samples_per_second": 81.538,
+      "eval_steps_per_second": 13.861,
+      "step": 2700
+    },
+    {
+      "epoch": 0.30247202133802625,
+      "grad_norm": 5.441561222076416,
+      "learning_rate": 5.095741168702542e-05,
+      "loss": 0.6458732604980468,
+      "step": 2750
+    },
+    {
+      "epoch": 0.30247202133802625,
+      "eval_loss": 0.7701953053474426,
+      "eval_runtime": 1.222,
+      "eval_samples_per_second": 81.833,
+      "eval_steps_per_second": 13.912,
+      "step": 2750
+    },
+    {
+      "epoch": 0.30797151263508127,
+      "grad_norm": 4.965064525604248,
+      "learning_rate": 5.07923407065038e-05,
+      "loss": 0.6677908325195312,
+      "step": 2800
+    },
+    {
+      "epoch": 0.30797151263508127,
+      "eval_loss": 0.7164813280105591,
+      "eval_runtime": 1.2202,
+      "eval_samples_per_second": 81.956,
+      "eval_steps_per_second": 13.933,
+      "step": 2800
+    },
+    {
+      "epoch": 0.3134710039321363,
+      "grad_norm": 3.931856870651245,
+      "learning_rate": 5.062726972598217e-05,
+      "loss": 0.5885476684570312,
+      "step": 2850
+    },
+    {
+      "epoch": 0.3134710039321363,
+      "eval_loss": 0.7336059808731079,
+      "eval_runtime": 1.2203,
+      "eval_samples_per_second": 81.948,
+      "eval_steps_per_second": 13.931,
+      "step": 2850
+    },
+    {
+      "epoch": 0.3189704952291913,
+      "grad_norm": 4.760177135467529,
+      "learning_rate": 5.046219874546055e-05,
+      "loss": 0.6489822387695312,
+      "step": 2900
+    },
+    {
+      "epoch": 0.3189704952291913,
+      "eval_loss": 0.7551103830337524,
+      "eval_runtime": 1.2183,
+      "eval_samples_per_second": 82.079,
+      "eval_steps_per_second": 13.953,
+      "step": 2900
+    },
+    {
+      "epoch": 0.3244699865262463,
+      "grad_norm": 4.375728130340576,
+      "learning_rate": 5.0297127764938924e-05,
+      "loss": 0.6457704162597656,
+      "step": 2950
+    },
+    {
+      "epoch": 0.3244699865262463,
+      "eval_loss": 0.7187867164611816,
+      "eval_runtime": 1.2036,
+      "eval_samples_per_second": 83.084,
+      "eval_steps_per_second": 14.124,
+      "step": 2950
+    },
+    {
+      "epoch": 0.3299694778233013,
+      "grad_norm": 4.091928005218506,
+      "learning_rate": 5.0132056784417304e-05,
+      "loss": 0.6391205596923828,
+      "step": 3000
+    },
+    {
+      "epoch": 0.3299694778233013,
+      "eval_loss": 0.7715049982070923,
+      "eval_runtime": 1.2016,
+      "eval_samples_per_second": 83.22,
+      "eval_steps_per_second": 14.147,
+      "step": 3000
+    },
+    {
+      "epoch": 0.33546896912035634,
+      "grad_norm": 4.569828510284424,
+      "learning_rate": 4.996698580389568e-05,
+      "loss": 0.6714089965820312,
+      "step": 3050
+    },
+    {
+      "epoch": 0.33546896912035634,
+      "eval_loss": 0.7370268106460571,
+      "eval_runtime": 1.2171,
+      "eval_samples_per_second": 82.162,
+      "eval_steps_per_second": 13.967,
+      "step": 3050
+    },
+    {
+      "epoch": 0.3409684604174114,
+      "grad_norm": 5.741940975189209,
+      "learning_rate": 4.980191482337405e-05,
+      "loss": 0.6890049743652343,
+      "step": 3100
+    },
+    {
+      "epoch": 0.3409684604174114,
+      "eval_loss": 0.7069912552833557,
+      "eval_runtime": 1.1974,
+      "eval_samples_per_second": 83.514,
+      "eval_steps_per_second": 14.197,
+      "step": 3100
+    },
+    {
+      "epoch": 0.3464679517144664,
+      "grad_norm": 4.857932090759277,
+      "learning_rate": 4.963684384285243e-05,
+      "loss": 0.5919662475585937,
+      "step": 3150
+    },
+    {
+      "epoch": 0.3464679517144664,
+      "eval_loss": 0.7350116968154907,
+      "eval_runtime": 1.1974,
+      "eval_samples_per_second": 83.515,
+      "eval_steps_per_second": 14.198,
+      "step": 3150
+    },
+    {
+      "epoch": 0.35196744301152144,
+      "grad_norm": 4.636399745941162,
+      "learning_rate": 4.94717728623308e-05,
+      "loss": 0.6361312866210938,
+      "step": 3200
+    },
+    {
+      "epoch": 0.35196744301152144,
+      "eval_loss": 0.7743600606918335,
+      "eval_runtime": 1.271,
+      "eval_samples_per_second": 78.677,
+      "eval_steps_per_second": 13.375,
+      "step": 3200
+    },
+    {
+      "epoch": 0.35746693430857646,
+      "grad_norm": 5.155900955200195,
+      "learning_rate": 4.930670188180918e-05,
+      "loss": 0.602264518737793,
+      "step": 3250
+    },
+    {
+      "epoch": 0.35746693430857646,
+      "eval_loss": 0.7555291652679443,
+      "eval_runtime": 1.2803,
+      "eval_samples_per_second": 78.105,
+      "eval_steps_per_second": 13.278,
+      "step": 3250
+    },
+    {
+      "epoch": 0.36296642560563147,
+      "grad_norm": 3.4023196697235107,
+      "learning_rate": 4.9141630901287555e-05,
+      "loss": 0.5788230133056641,
+      "step": 3300
+    },
+    {
+      "epoch": 0.36296642560563147,
+      "eval_loss": 0.7717245221138,
+      "eval_runtime": 1.3149,
+      "eval_samples_per_second": 76.053,
+      "eval_steps_per_second": 12.929,
+      "step": 3300
+    },
+    {
+      "epoch": 0.3684659169026865,
+      "grad_norm": 4.329448223114014,
+      "learning_rate": 4.897655992076593e-05,
+      "loss": 0.6438528442382813,
+      "step": 3350
+    },
+    {
+      "epoch": 0.3684659169026865,
+      "eval_loss": 0.6952827572822571,
+      "eval_runtime": 1.2957,
+      "eval_samples_per_second": 77.176,
+      "eval_steps_per_second": 13.12,
+      "step": 3350
+    },
+    {
+      "epoch": 0.3739654081997415,
+      "grad_norm": 3.610276222229004,
+      "learning_rate": 4.881148894024431e-05,
+      "loss": 0.6141637802124024,
+      "step": 3400
+    },
+    {
+      "epoch": 0.3739654081997415,
+      "eval_loss": 0.6926187872886658,
+      "eval_runtime": 1.3466,
+      "eval_samples_per_second": 74.259,
+      "eval_steps_per_second": 12.624,
+      "step": 3400
+    },
+    {
+      "epoch": 0.37946489949679657,
+      "grad_norm": 6.178794860839844,
+      "learning_rate": 4.864641795972269e-05,
+      "loss": 0.6356156158447266,
+      "step": 3450
+    },
+    {
+      "epoch": 0.37946489949679657,
+      "eval_loss": 0.7247772216796875,
+      "eval_runtime": 1.3462,
+      "eval_samples_per_second": 74.281,
+      "eval_steps_per_second": 12.628,
+      "step": 3450
+    },
+    {
+      "epoch": 0.3849643907938516,
+      "grad_norm": 4.9943695068359375,
+      "learning_rate": 4.848134697920105e-05,
+      "loss": 0.6148456192016601,
+      "step": 3500
+    },
+    {
+      "epoch": 0.3849643907938516,
+      "eval_loss": 0.7180845141410828,
+      "eval_runtime": 1.3322,
+      "eval_samples_per_second": 75.061,
+      "eval_steps_per_second": 12.76,
+      "step": 3500
+    },
+    {
+      "epoch": 0.3904638820909066,
+      "grad_norm": 3.8332393169403076,
+      "learning_rate": 4.831627599867943e-05,
+      "loss": 0.625141716003418,
+      "step": 3550
+    },
+    {
+      "epoch": 0.3904638820909066,
+      "eval_loss": 0.6876647472381592,
+      "eval_runtime": 1.3382,
+      "eval_samples_per_second": 74.73,
+      "eval_steps_per_second": 12.704,
+      "step": 3550
+    },
+    {
+      "epoch": 0.3959633733879616,
+      "grad_norm": 4.58771276473999,
+      "learning_rate": 4.815120501815781e-05,
+      "loss": 0.6502104187011719,
+      "step": 3600
+    },
+    {
+      "epoch": 0.3959633733879616,
+      "eval_loss": 0.690920352935791,
+      "eval_runtime": 1.343,
+      "eval_samples_per_second": 74.46,
+      "eval_steps_per_second": 12.658,
+      "step": 3600
+    },
+    {
+      "epoch": 0.40146286468501663,
+      "grad_norm": 4.435539722442627,
+      "learning_rate": 4.7986134037636185e-05,
+      "loss": 0.6129857635498047,
+      "step": 3650
+    },
+    {
+      "epoch": 0.40146286468501663,
+      "eval_loss": 0.6758005023002625,
+      "eval_runtime": 1.34,
+      "eval_samples_per_second": 74.628,
+      "eval_steps_per_second": 12.687,
+      "step": 3650
+    },
+    {
+      "epoch": 0.40696235598207164,
+      "grad_norm": 4.618966102600098,
+      "learning_rate": 4.782106305711456e-05,
+      "loss": 0.6234348678588867,
+      "step": 3700
+    },
+    {
+      "epoch": 0.40696235598207164,
+      "eval_loss": 0.7179376482963562,
+      "eval_runtime": 1.3332,
+      "eval_samples_per_second": 75.01,
+      "eval_steps_per_second": 12.752,
+      "step": 3700
+    },
+    {
+      "epoch": 0.41246184727912666,
+      "grad_norm": 4.0562052726745605,
+      "learning_rate": 4.765599207659294e-05,
+      "loss": 0.6168152999877929,
+      "step": 3750
+    },
+    {
+      "epoch": 0.41246184727912666,
+      "eval_loss": 0.7127401828765869,
+      "eval_runtime": 1.3374,
+      "eval_samples_per_second": 74.769,
+      "eval_steps_per_second": 12.711,
+      "step": 3750
+    },
+    {
+      "epoch": 0.41796133857618173,
+      "grad_norm": 5.699039459228516,
+      "learning_rate": 4.749092109607131e-05,
+      "loss": 0.5789441680908203,
+      "step": 3800
+    },
+    {
+      "epoch": 0.41796133857618173,
+      "eval_loss": 0.6780041456222534,
+      "eval_runtime": 1.3483,
+      "eval_samples_per_second": 74.167,
+      "eval_steps_per_second": 12.608,
+      "step": 3800
+    },
+    {
+      "epoch": 0.42346082987323674,
+      "grad_norm": 3.523623466491699,
+      "learning_rate": 4.7325850115549684e-05,
+      "loss": 0.5945124053955078,
+      "step": 3850
+    },
+    {
+      "epoch": 0.42346082987323674,
+      "eval_loss": 0.695915699005127,
+      "eval_runtime": 1.345,
+      "eval_samples_per_second": 74.351,
+      "eval_steps_per_second": 12.64,
+      "step": 3850
+    },
+    {
+      "epoch": 0.42896032117029176,
+      "grad_norm": 4.008049488067627,
+      "learning_rate": 4.716077913502806e-05,
+      "loss": 0.5782461929321289,
+      "step": 3900
+    },
+    {
+      "epoch": 0.42896032117029176,
+      "eval_loss": 0.6521520018577576,
+      "eval_runtime": 1.342,
+      "eval_samples_per_second": 74.517,
+      "eval_steps_per_second": 12.668,
+      "step": 3900
+    },
+    {
+      "epoch": 0.4344598124673468,
+      "grad_norm": 4.809508323669434,
+      "learning_rate": 4.6995708154506436e-05,
+      "loss": 0.5954169464111329,
+      "step": 3950
+    },
+    {
+      "epoch": 0.4344598124673468,
+      "eval_loss": 0.6668588519096375,
+      "eval_runtime": 1.347,
+      "eval_samples_per_second": 74.241,
+      "eval_steps_per_second": 12.621,
+      "step": 3950
+    },
+    {
+      "epoch": 0.4399593037644018,
+      "grad_norm": 4.741697788238525,
+      "learning_rate": 4.6830637173984816e-05,
+      "loss": 0.5780908584594726,
+      "step": 4000
+    },
+    {
+      "epoch": 0.4399593037644018,
+      "eval_loss": 0.6712077260017395,
+      "eval_runtime": 1.3399,
+      "eval_samples_per_second": 74.63,
+      "eval_steps_per_second": 12.687,
+      "step": 4000
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 18184,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 3.600928640036045e+16,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c6a9e13e2fe214895d1d166dd41dde5c9df0f04611cacb808c08c1ada3007b4
+size 5201