warenterprise commited on
Commit
41cd8ff
·
verified ·
1 Parent(s): c795f65

Add English model documentation

Browse files

Adds an English summary to the main model card and a complete English documentation file.

Files changed (2) hide show
  1. README.md +22 -1
  2. README_EN.md +258 -0
README.md CHANGED
@@ -24,6 +24,8 @@ datasets:
24
 
25
  **Prévia técnica experimental em português para pesquisa e inferência local**
26
 
 
 
27
  ![Parâmetros](https://img.shields.io/badge/parâmetros-203.263.872-19c37d)
28
  ![Pré--treino](https://img.shields.io/badge/pré--treino-1.000.013.824%20tokens-19c37d)
29
  ![Contexto](https://img.shields.io/badge/contexto-1.024%20tokens-19c37d)
@@ -38,6 +40,25 @@ datasets:
38
  <img src="assets/warmind-200m-v2-poster.png" alt="Resumo visual do WARMIND-200M V2" width="520">
39
  </p>
40
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
41
  ## Visão geral
42
 
43
  O **WARMIND-200M V2** é um modelo de linguagem causal compacto desenvolvido pela **WAR Enterprise** para pesquisa em geração de texto em português e inferência local.
@@ -141,7 +162,7 @@ python generate_local.py \
141
 
142
  ## Treinamento
143
 
144
- O checkpoint foi treinado em uma **NVIDIA H100 80 GB HBM3**, utilizando **BF16**.
145
 
146
  | Etapa | Registro |
147
  |---|---:|
 
24
 
25
  **Prévia técnica experimental em português para pesquisa e inferência local**
26
 
27
+ 🇧🇷 **Português** · 🇺🇸 [English documentation](README_EN.md)
28
+
29
  ![Parâmetros](https://img.shields.io/badge/parâmetros-203.263.872-19c37d)
30
  ![Pré--treino](https://img.shields.io/badge/pré--treino-1.000.013.824%20tokens-19c37d)
31
  ![Contexto](https://img.shields.io/badge/contexto-1.024%20tokens-19c37d)
 
40
  <img src="assets/warmind-200m-v2-poster.png" alt="Resumo visual do WARMIND-200M V2" width="520">
41
  </p>
42
 
43
+ ## English summary
44
+
45
+ **WARMIND-200M V2** is an experimental, Portuguese-first causal language model developed by **WAR Enterprise** for compact-model research and local inference.
46
+
47
+ - **203,263,872 parameters**
48
+ - **1,000,013,824 pretraining tokens**
49
+ - **23,751,277 supervised SFT tokens**
50
+ - **20 layers**, hidden size **896**
51
+ - **14 attention heads** and **2 KV heads**
52
+ - **Grouped-Query Attention, SwiGLU, RMSNorm and RoPE**
53
+ - **24,576-token SentencePiece vocabulary**
54
+ - **1,024-token operational context**
55
+ - local CPU inference; CUDA is supported by the example script
56
+ - Apache 2.0 license
57
+
58
+ This is a research checkpoint, not a production assistant. It can hallucinate, fail on simple reasoning tasks and produce incomplete or incorrect answers.
59
+
60
+ ➡️ **[Open the full English documentation](README_EN.md)**
61
+
62
  ## Visão geral
63
 
64
  O **WARMIND-200M V2** é um modelo de linguagem causal compacto desenvolvido pela **WAR Enterprise** para pesquisa em geração de texto em português e inferência local.
 
162
 
163
  ## Treinamento
164
 
165
+ O checkpoint foi treinado em uma **NVIDIA H100 80 GB HBM3**, utilizando **BF16**. A execução principal do pré-treinamento levou aproximadamente **2 horas e 30 minutos**; preparação dos dados, treinamento do tokenizer, SFT, empacotamento e testes locais foram realizados separadamente.
166
 
167
  | Etapa | Registro |
168
  |---|---:|
README_EN.md ADDED
@@ -0,0 +1,258 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # WARMIND-200M V2
2
+
3
+ **Experimental Portuguese-first technical preview for research and local inference**
4
+
5
+ 🇧🇷 [Português](README.md) · 🇺🇸 **English**
6
+
7
+ ![Parameters](https://img.shields.io/badge/parameters-203%2C263%2C872-19c37d)
8
+ ![Pretraining](https://img.shields.io/badge/pretraining-1%2C000%2C013%2C824%20tokens-19c37d)
9
+ ![Context](https://img.shields.io/badge/context-1%2C024%20tokens-19c37d)
10
+ ![License](https://img.shields.io/badge/license-Apache--2.0-blue)
11
+ ![Status](https://img.shields.io/badge/status-experimental-orange)
12
+
13
+ > ⚠️ **Usage warning:** this checkpoint may produce incorrect information, incomplete answers and hallucinations. Do not use its outputs for medical, legal, financial, safety-related or other high-impact decisions without qualified human review.
14
+
15
+ <p align="center">
16
+ <img src="assets/warmind-200m-v2-poster.png" alt="WARMIND-200M V2 visual summary" width="520">
17
+ </p>
18
+
19
+ ## Overview
20
+
21
+ **WARMIND-200M V2** is a compact causal language model developed by **WAR Enterprise** for research on Portuguese text generation and local inference.
22
+
23
+ The published version is a **research artifact**, not a production assistant. Its main purpose was to validate a complete development pipeline: data preparation, tokenizer training, pretraining, supervised fine-tuning, packaging, integrity verification and local execution.
24
+
25
+ ## At a glance
26
+
27
+ | Item | Value |
28
+ |---|---:|
29
+ | Parameters | **203,263,872** |
30
+ | Pretraining tokens | **1,000,013,824** |
31
+ | Tokens processed during SFT | **23,751,277** |
32
+ | Layers | **20** |
33
+ | Hidden size | **896** |
34
+ | Attention / KV heads | **14 / 2** |
35
+ | Vocabulary | **24,576-token SentencePiece** |
36
+ | Operational context | **1,024 tokens** |
37
+ | Published weights | **safetensors FP32** |
38
+ | Primary language | **Portuguese** |
39
+ | Execution | **Local CPU; CUDA supported by the example script** |
40
+
41
+ ## Transparent demonstration — including limitations
42
+
43
+ The video below shows the checkpoint's real behavior, including weak and incorrect answers. These limitations are part of the technical evaluation of this release.
44
+
45
+ [![Open the real WARMIND-200M V2 demonstration](assets/demo-thumb.jpg)](assets/demo-real.mp4)
46
+
47
+ The demonstration includes:
48
+
49
+ - basic Portuguese generation;
50
+ - retrieval of some facts;
51
+ - mistakes in arithmetic and factual knowledge;
52
+ - plausible but incorrect answers;
53
+ - an explicit review of observed failures.
54
+
55
+ ## Quick start
56
+
57
+ ### Requirements
58
+
59
+ - Python **3.12**
60
+ - at least **8 GB of free RAM** for a comfortable local experience
61
+
62
+ ### Installation
63
+
64
+ ```bash
65
+ python -m venv .venv
66
+ python -m pip install -r requirements.txt
67
+ ```
68
+
69
+ ### Windows PowerShell
70
+
71
+ ```powershell
72
+ .\.venv\Scripts\Activate.ps1
73
+
74
+ python generate_local.py `
75
+ --prompt "Defina fotossíntese em uma frase." `
76
+ --max-new-tokens 96
77
+ ```
78
+
79
+ ### Linux or macOS
80
+
81
+ ```bash
82
+ source .venv/bin/activate
83
+
84
+ python generate_local.py \
85
+ --prompt "Defina fotossíntese em uma frase." \
86
+ --max-new-tokens 96
87
+ ```
88
+
89
+ ### More controlled generation
90
+
91
+ ```bash
92
+ python generate_local.py \
93
+ --prompt "Explique em duas frases o que é inteligência artificial." \
94
+ --temperature 0.2 \
95
+ --top-k 20 \
96
+ --top-p 0.9 \
97
+ --repetition-penalty 1.12 \
98
+ --max-new-tokens 160
99
+ ```
100
+
101
+ > Confirm whether the published script accepts `--max-new-tokens` or `--maximum-new-tokens`, and keep only the supported form in the documentation.
102
+
103
+ ## Architecture
104
+
105
+ | Component | Value |
106
+ |---|---|
107
+ | Type | Causal decoder-only Transformer |
108
+ | Layers | 20 |
109
+ | Hidden size | 896 |
110
+ | Attention heads | 14 |
111
+ | KV heads | 2 |
112
+ | Attention | Grouped-Query Attention (GQA) |
113
+ | FFN dimension | 2,688 |
114
+ | Activation | SwiGLU |
115
+ | Normalization | RMSNorm |
116
+ | Positional encoding | RoPE, θ = 10,000 |
117
+ | Trained/operational context | 1,024 tokens |
118
+ | Extended context accepted by the code | up to 2,048 tokens, not validated as training context |
119
+
120
+ ## Training
121
+
122
+ The checkpoint was trained on a single **NVIDIA H100 80 GB HBM3** using **BF16**.
123
+
124
+ The main pretraining run took approximately **2 hours and 30 minutes**. Data preparation, tokenizer training, supervised fine-tuning, packaging and local testing were performed separately.
125
+
126
+ | Stage | Recorded value |
127
+ |---|---:|
128
+ | V2 pretraining | 1,000,013,824 tokens |
129
+ | Supervised fine-tuning | completed at step 1,200 |
130
+ | Validation loss at step 100 | 2.1085 |
131
+ | Validation loss at step 1,200 | 1.8038 |
132
+
133
+ These values describe the recorded training process and should not be interpreted on their own as proof of general model quality.
134
+
135
+ ## Data and provenance
136
+
137
+ ### Pretraining
138
+
139
+ Pretraining used a Portuguese sample from `epfml/FineWeb2-HQ` (`por_Latn`), under **ODC-By-1.0** and also subject to Common Crawl terms.
140
+
141
+ Approximate prepared-data composition:
142
+
143
+ - **93.6%** general content;
144
+ - **5.0%** legal content;
145
+ - **1.4%** technical content.
146
+
147
+ ### Supervised fine-tuning
148
+
149
+ The SFT mixture included:
150
+
151
+ - Portuguese conversations from `HuggingFaceTB/smoltalk2`;
152
+ - filtered data from `OpenAssistant/oasst2`;
153
+ - WAR Enterprise's own examples.
154
+
155
+ Attributions and the subsets used should remain documented in [`NOTICE.md`](NOTICE.md).
156
+
157
+ > Source data may contain errors, distortions, synthetic content or personal information present in the original datasets. Training does not make the model a reliable factual source.
158
+
159
+ ## Experimental local chat
160
+
161
+ `chat_calibrate.py` adds:
162
+
163
+ - short-term memory;
164
+ - search over local files;
165
+ - a deterministic calculator;
166
+ - optional Wikipedia and Wikidata lookup.
167
+
168
+ ```bash
169
+ python chat_calibrate.py
170
+ ```
171
+
172
+ This orchestrator does not change the model weights and does not provide access to the Transformer's “internal thoughts.”
173
+
174
+ When online lookup is enabled, the question text may be sent to the corresponding public APIs. Use:
175
+
176
+ ```text
177
+ /online off
178
+ ```
179
+
180
+ to keep the session offline.
181
+
182
+ ## Maturity status
183
+
184
+ **Status:** `EXPERIMENTAL_NOT_DEMO_READY` / `AWAITING_HUMAN_REVIEW`
185
+
186
+ V2 improved over the first checkpoint, but it still shows:
187
+
188
+ - hallucinated facts, names, numbers and identities;
189
+ - repetition and topic drift;
190
+ - failures to follow requested length limits;
191
+ - plausible but incorrect answers;
192
+ - sensitivity to sampling parameters;
193
+ - limitations in basic arithmetic and reasoning.
194
+
195
+ This release should therefore be treated as a **technical preview for research, learning and local reproduction**, not as a state-of-the-art benchmark.
196
+
197
+ ## Intended uses
198
+
199
+ - research on compact Portuguese models;
200
+ - study of local CPU inference;
201
+ - generation experiments with human review;
202
+ - educational prototyping;
203
+ - comparison of post-processing techniques;
204
+ - experiments with external tools.
205
+
206
+ ## Non-recommended uses
207
+
208
+ - automated or high-impact decisions;
209
+ - medical, legal or financial advice;
210
+ - factual content without external verification;
211
+ - moderation, surveillance or classification of people;
212
+ - applications requiring guaranteed safety, accuracy or availability.
213
+
214
+ ## File integrity
215
+
216
+ SHA-256 hashes are listed in [`SHA256SUMS`](SHA256SUMS).
217
+
218
+ Expected hash for `model.safetensors`:
219
+
220
+ ```text
221
+ 42218dacd46c7a3d244856e664442822e002312bf871a57181379d9026a61045
222
+ ```
223
+
224
+ ## Future research
225
+
226
+ WAR Enterprise is studying a future generation in the range of **500 million parameters**, potentially trained on a substantially larger token budget.
227
+
228
+ This is a **research objective**, not a release promise. Progress depends on infrastructure, data, evaluations and technical results.
229
+
230
+ ## Technical feedback
231
+
232
+ Useful contributions include:
233
+
234
+ - tests on different CPUs;
235
+ - RAM and tokens-per-second measurements;
236
+ - Portuguese benchmarks;
237
+ - tokenizer bug reports;
238
+ - GGUF and quantization suggestions;
239
+ - reproducible failure examples.
240
+
241
+ Use the repository's **Community** tab to open a discussion.
242
+
243
+ ## Citation
244
+
245
+ ```bibtex
246
+ @software{warmind200mv2_2026,
247
+ title = {WARMIND-200M V2},
248
+ author = {WAR Enterprise},
249
+ year = {2026},
250
+ note = {Experimental technical preview of a Portuguese language model},
251
+ url = {https://huggingface.co/warenterprise/WARMIND-200M-V2}
252
+ }
253
+ ```
254
+
255
+ ---
256
+
257
+ Developed by **WAR Enterprise**
258
+ Compact-model research, local execution and technical transparency.