iluksic/nemotron-speech-streaming-hr-0.6b

Nemotron Speech Streaming HR 0.6B

Fine-tuned Croatian automatic speech recognition model based on nvidia/nemotron-speech-streaming-en-0.6b. It is a cache-aware streaming FastConformer-RNNT model: it transcribes audio incrementally with a fixed left context and a selectable lookahead, giving a smooth accuracy/latency trade-off from ~1 s lookahead down to zero-lookahead (instant) streaming.

The model was fine-tuned on a large corpus of Croatian speech using a multi-phase strategy that balances in-domain accuracy with broader speaker and topic generalization.

Intended Use

Designed for:

• Real-time Croatian speech transcription in streaming and low-latency applications
• Formal Croatian speech (parliamentary, broadcast, dictation)
• Clean to moderately noisy audio (SNR ≥ 10 dB)
• 16 kHz mono audio input

Not recommended for:

• Heavily dialectal Croatian
• Number-heavy or date-critical content — number transcription is unreliable (see Number transcription errors)
• Audio below SNR ~10 dB — accuracy degrades
• Utterances longer than 15 s without prior segmentation

Model Details

Property	Value
Base model	nvidia/nemotron-speech-streaming-en-0.6b
Architecture	EncDecRNNTBPEModel (cache-aware FastConformer RNN-Transducer)
Framework	NVIDIA NeMo
Total parameters	622.0M
Trainable parameters	12.9M in Phase 1 (encoder frozen); the encoder is later unfrozen in Phases 2 / 2.5 (see Training Procedure)
Tokenizer	SentencePiece unigram, vocab 4,096 (retrained on Croatian text)
Trained left context	70 encoder frames (~5.6 s), fixed
Trained operating points	total buffer 1120 / 560 / 160 / 80 ms (lookahead 13 / 6 / 1 / 0 frames)
Encoder frame length	80 ms (8× subsampling)
Sample rate	16,000 Hz
Max utterance duration	15.0 s
Base checkpoint	nvidia/nemotron-speech-streaming-en-0.6b (Jan 2026 release)
Reference NeMo	NGC nemo:26.02 container / NeMo main

Evaluation

Methodology

All streaming numbers below use the model's native cache-aware inference path: the left context is held at the trained value (70 frames, ~5.6 s) and only the lookahead (right context) is varied across the four values the model was trained on. Operating points are labeled by total buffer — the new audio consumed per step — following the convention used in NVIDIA's Nemotron ASR cards; total buffer = lookahead + one 80 ms frame. The four points are 1120 / 560 / 160 / 80 ms total buffer, corresponding to lookaheads of 1040 / 480 / 80 / 0 ms. This is the configuration in which cache-aware models are designed to match offline decoding. WER/CER are computed after light normalization (lowercase, punctuation stripped, whitespace collapsed) using jiwer, with greedy decoding on an NVIDIA A40.

Streaming context. This model uses FastEmit regularization (λ=0.005), which biases toward emitting tokens earlier to reduce latency. Its main side-effect is a higher deletion rate for short function words (i, je, u, na, od). CER therefore tracks true transcription quality more closely than WER; comparisons to offline models are best made on CER.

The three datasets probe different conditions:

Dataset	N	Split	Domain / character
ParlaSpeech-HR	500	held out from training (seed 42, clips ≥ 4 s)	spontaneous parliamentary speech, clean transcripts
VoxPopuli-hr	664	full test (clips ≥ 4 s)	spontaneous EU-parliament speech; see data-quality note
FLEURS-hr	914	full test	read encyclopedic sentences, name/number dense (hardest)

Headline results (1120 ms buffer)

Dataset	WER	CER
ParlaSpeech-HR (held-out)	15.89%	6.80%
VoxPopuli-hr	24.51%	10.46%
FLEURS-hr	28.50%	10.22%

The spread reflects content difficulty, not decoding behavior: FLEURS is dominated by foreign proper nouns and numerals (its WER is high while CER stays ~10%, the signature of whole-word errors on rare tokens), whereas conversational Croatian sits around 16–24%.

Recommended operating point. For most applications, use the 560 ms buffer (att_context_size = [70, 6]): it is within 0.15–0.56 pt WER of the best (largest) buffer across the three datasets while running at ~17× real-time with a roughly half-second response floor. Move up to the 1120 ms buffer if you have latency headroom and want peak accuracy, or down to 80–160 ms for near-instant captioning at ~1–2 pts higher WER. First-token latency ≈ lookahead + ~30 ms compute (e.g. ~510 ms at the 560 ms buffer).

Accuracy vs latency (all four lookaheads)

Accuracy is nearly flat across the full latency range — the model degrades gracefully as lookahead shrinks, with no instability at any setting.

ParlaSpeech-HR (500, held-out)

Buffer	Lookahead	WER	CER	RTFx	Stability
1120 ms	1040 ms	15.89%	6.80%	33.8×	98.2%
560 ms	480 ms	16.04%	6.99%	17.8×	98.2%
160 ms	80 ms	16.67%	7.20%	5.0×	98.9%
80 ms	0 ms	17.21%	7.62%	2.6×	99.4%

VoxPopuli-hr (664)

Buffer	Lookahead	WER	CER	RTFx	Stability
1120 ms	1040 ms	24.51%	10.46%	33.0×	97.3%
560 ms	480 ms	24.89%	10.72%	16.7×	97.2%
160 ms	80 ms	26.06%	11.48%	4.7×	98.0%
80 ms	0 ms	26.81%	12.15%	2.6×	98.9%

FLEURS-hr (914)

Buffer	Lookahead	WER	CER	RTFx	Stability
1120 ms	1040 ms	28.50%	10.22%	32.2×	96.2%
560 ms	480 ms	29.06%	10.66%	16.2×	96.2%
160 ms	80 ms	29.69%	10.80%	4.8×	97.1%
80 ms	0 ms	29.95%	10.89%	2.6×	98.2%

Going from the largest buffer (1120 ms) to the smallest (80 ms, zero lookahead) costs only 1.3 pts WER on ParlaSpeech, 2.3 on VoxPopuli, 1.5 on FLEURS. Stability (fraction of partial-hypothesis words retained in the final output) actually rises as the buffer shrinks, because less future context means fewer late revisions — a useful property for live captioning.

Streaming ≈ offline

At the largest buffer the streaming model reaches parity with full-file (offline) decoding. On FLEURS, offline WER is 28.41% vs streaming-at-1120 ms-buffer 28.50% — a +0.09 pt difference. Cache-aware streaming here is effectively free: you get incremental output at no accuracy cost relative to batch transcription.

Latency

Benchmarked on NVIDIA A40, greedy decoding, ParlaSpeech.

Buffer	Lookahead	Mean ms	P95 ms	Cold ms	RTF	RTFx
1120 ms	1040 ms	31.9	36.6	32.3	0.030	33.8×
560 ms	480 ms	30.8	32.4	30.4	0.056	17.8×
160 ms	80 ms	31.7	36.0	31.1	0.199	5.0×
80 ms	0 ms	30.1	34.6	29.8	0.378	2.6×

Per-step compute latency is flat (~30–34 ms) because it is dominated by fixed encoder cost, not buffer length. What changes is throughput: a larger buffer means fewer, larger forward passes, so RTFx climbs to 33.8× at the 1120 ms buffer. Even the 80 ms buffer (zero lookahead) runs at 2.6× real-time, so every operating point is real-time-capable on this hardware. The response latency floor (how long before a token can be emitted) is set by the lookahead: 1040 / 480 / 80 / 0 ms.

Deployment

Requirement	Value
Parameters	622M
Checkpoint file	~2.5 GB .nemo (fp32 weights + tokenizer + config)
Default load precision	fp32
Peak VRAM (streaming, batch 1, 15 s clip)	5.1 GB (measured, A40)
Batch size	1 (streaming) / up to 16 (offline transcribe)
CPU inference	Possible via NeMo but far slower; GPU strongly recommended for real-time streaming
Framework	NVIDIA NeMo (NGC nemo:26.02 container or NeMo main)

The measured 5.1 GB peak is higher than the ≈2.5 GB nominal fp32 weight size (622M params × 4 bytes); the remainder is additional runtime allocation that NeMo holds at load (retained buffers, workspace, and framework overhead) rather than model weights. Streaming activations and encoder cache add only about 0.1 GB on top, so the footprint is essentially flat with utterance length — a 6 GB GPU is sufficient for single-stream inference. Casting the weights to BF16 would roughly halve the weight portion (≈2.5 GB → ≈1.2 GB) but leaves the non-weight runtime allocation largely unchanged, so the total footprint would drop modestly rather than by half. The numbers above reflect the default fp32 configuration in which the accuracy results were measured. Throughput on the A40 ranges 2.6–33.8× real-time depending on buffer size.

Transcription Examples

Real reference/hypothesis pairs from the 1120 ms-buffer runs (normalized text), illustrating typical behavior:

Clean parliamentary speech — exact match

REF: Idemo dalje, druga replika poštovani kolega Karlić, izvolite.
HYP: idemo dalje druga replika poštovani kolega karlić izvolite

Minor error — word-boundary slip (10% WER)

REF: Ovo je udar na male iznajmljivače, na kičmu hrvatskog turizma.
HYP: ovo je udar na izmale iznajmljivače na kičmu hrvatskog turizma

(na male → izmale, a boundary merge; every other word correct.)

Number handling — written-form drift

REF: ...prije otprilike 10.000 godina u planinama zagros u iranu
HYP: ...prije otprilike deset tisuća godina u planinama zagroz u iranu

(The quantity is correct but rendered as words deset tisuća instead of 10.000, so WER penalizes it — see Number transcription errors.)

Foreign names + numerals — the hardest case (38% WER)

REF: bus scotturb 403 redovito vozi za sintru a staje u cabo da roci
HYP: bus kotor 43 redovito vozi za sintru a staje u kabodiroki

(scotturb 403 → kotor 43, cabo da roca → kabodiroki: foreign proper nouns and a dropped digit — the dominant FLEURS failure mode.)

Error Analysis

Recomputed from full per-sample alignments at the 1120 ms-buffer operating point: 914 (FLEURS), 500 (ParlaSpeech), 664 (VoxPopuli) utterances, word-level Levenshtein alignment on normalized text.

WER distribution

	FLEURS	ParlaSpeech	VoxPopuli
Mean per-sample WER	28.8%	15.4%	24.6%
Median per-sample WER	26.7%	13.6%	23.1%
P90	50.0%	29.4%	42.9%
Perfect (0% WER)	33 / 914 (3.6%)	64 / 500 (12.8%)	25 / 664 (3.8%)
Samples > 50% WER	80 (8.8%)	4 (0.8%)	25 (3.8%)

The distributions are well-behaved — the corpus WER is representative of the typical case, not driven by a heavy tail. On ParlaSpeech nearly 13% of utterances are transcribed perfectly and under 1% exceed 50% WER.

Error type breakdown

Dataset	Substitutions	Deletions	Insertions	Correct
FLEURS	20.5%	5.2%	2.2%	72.1%
ParlaSpeech	10.1%	4.4%	1.2%	84.3%
VoxPopuli	16.0%	6.6%	1.6%	75.9%

Substitution classification

Each substitution is bucketed into one of five types.

Category	Definition	FLEURS	ParlaSpeech	VoxPopuli
Lexical	Unrelated words	58.5%	48.3%	52.1%
Morphological inflection	Shared stem (≥3 chars), different ending	25.0%	26.7%	35.0%
Truncation	One word is a prefix of the other	10.0%	16.5%	10.6%
Number	At least one side contains digits	6.2%	8.2%	2.2%
Diacritic-only	Identical after removing diacritics	0.2%	0.3%	0.0%

Inflectional morphology accounts for 25–35% of substitutions across all datasets — the correct stem with the wrong grammatical ending (e.g. koje → koji, neuter vs masculine; istraživanja → istraživanje, genitive vs nominative). These are semantically low-cost (the referent stays recognizable) but WER penalizes them like completely wrong words. For a heavily inflected language this is an inherent gap between WER and usable transcription quality.

Truncation on ParlaSpeech (16.5%) is largely colloquial short forms (kada → kad, sada → sad) plus a few reference-punctuation effects.

Most-deleted words

The same monosyllabic function words dominate deletions everywhere, and are the main reason WER sits above CER.

• FLEURS: i ×61 · je ×43 · u ×41 · s ×24 · od ×17 · na ×15 · za ×11
• ParlaSpeech: i ×65 · je ×56 · u ×41 · o ×18 · da ×15 · se ×10
• VoxPopuli: i ×119 · je ×54 · u ×45 · gospodine ×29 · za ×19 · na ×17

gospodine appears in VoxPopuli deletions because speakers routinely open by addressing the chair and the model sometimes drops the salutation — an in-domain parliamentary phenomenon. The English tokens two (×19) and thousand (×16) in VoxPopuli deletions are a reference artifact: some VoxPopuli-hr references contain English words (see data-quality note).

Most common substitution pairs

• FLEURS: s→sa ×14 · iz→i ×10 · s→iz ×9 · o→u ×7 · koje→koji ×6 · južnoafričkim→afričkim ×5
• ParlaSpeech: kada→kad ×10 · koje→koji ×5 · kn→kuna ×4 · sada→sad ×4 · što→šta ×4
• VoxPopuli: to→što ×36 · s→sa ×29 · takoer→također ×15 · izvjee→izvješće ×14 · izmeu→između ×14

Short prepositions (s / sa / iz / o / u) are disproportionately confused: 1–2 character words, phonetically similar, under early-emission pressure. On VoxPopuli several top "substitutions" (takoer→također, izvjee→izvješće, izmeu→između) are the model producing the correct diacritic form against a diacritic-stripped reference — an artifact, not an error (see below).

Word-length error rate (FLEURS)

Word length	Error rate
1 char	26.2%
2 chars	18.1%
3 chars	21.5%
4 chars	21.2%
5 chars	27.2%
6 chars	27.0%
7 chars	25.6%
8 chars	31.6%
9 chars	31.5%
10 chars	35.4%
11 chars	36.3%
12 chars	43.0%
13 chars	34.6%
≥14 chars	58.3%

Error rate climbs steeply beyond ~8 characters. Croatian forms long inflected compounds (e.g. južnoafričkim) that stack multiple morphological layers and fall outside common training n-grams; SentencePiece may also segment them differently at inference than in training.

Number transcription errors

Numeric content is unreliable (~6% of FLEURS substitutions involve digits) and should not be trusted without post-processing. Observed patterns, with real examples from the runs:

• Trailing-zero drop: 120 → 12, 403 → 43, 39000 → 309
• Year transposition / mangling: 2017 → 17, 1537 → 307
• Written ↔ digit oscillation: number words rendered as digits and vice-versa

Year transpositions are the most operationally dangerous — plausible and hard to catch downstream. For any date/quantity-sensitive application, add a number-normalization post-processor.

Diacritic recall

Fraction of reference diacritics preserved in the hypothesis.

Diacritic	FLEURS	ParlaSpeech	VoxPopuli†
č	89.4%	93.6%	78.7%
ć	82.2%	94.9%	82.0%
đ	86.3%	91.9%	78.1%
š	87.5%	96.1%	80.9%
ž	89.5%	96.9%	82.8%

ParlaSpeech recall is highest (92–97%) — formal register with consistent diacritic usage in training. ć is the weakest on FLEURS (82%), consistent with its acoustic similarity to č in connected speech. † VoxPopuli recall is artificially depressed by the reference issue below.

Hypothesis repetition loops

Consecutive word repetitions (e.g. igra igra igra) — a known RNNT failure mode — are rare on the native cache-aware path: 2 / 914 (FLEURS), 2 / 500 (ParlaSpeech), 0 / 664 (VoxPopuli). The worst single case is FLEURS id=1976 (igra repeated on the cricket name "Sachin Tendulkar"). A deduplication pass (collapse runs of ≥3 identical tokens) removes the few remaining cases at no cost on clean audio.

Data-quality note on VoxPopuli-hr

A substantial share of VoxPopuli-hr references are romanized ASCII that omit Croatian diacritics (takoer for također, gospoo for gospođo) and some contain English tokens (two, thousand). Where the model emits correct standard Croatian, alignment against these references scores it as an error. VoxPopuli numbers here are therefore a pessimistic estimate; ParlaSpeech and FLEURS references are orthographically standard.

Training Data

Dataset	Size (pre-filtering)	Domain
classla/ParlaSpeech-HR	1,816 h	Parliamentary speech
google/fleurs (hr_hr)	~12 h	Diverse read topics
facebook/voxpopuli (hr)	43 h	EU Parliament

Total ~1,871 h pre-filtering (effective hours lower after 0.3–15.0 s duration filtering).

Evaluation integrity. The model was trained on ParlaSpeech-HR. The 500 ParlaSpeech utterances used for evaluation above were held out from the training set (sampled with seed 42 from utterances not seen in training, clips ≥ 4 s), so the ParlaSpeech figure is a held-out in-domain result rather than a train-set score. FLEURS (full 914 test set) and VoxPopuli are reported on their standard test splits.

Training Procedure

Three-phase strategy on an NVIDIA A100-SXM4-80GB.

Phase 1 — Decoder & Joint only (encoder frozen)

Hyperparameter	Value
Epochs	5
Learning rate	5e-4 → 5e-5 (min)
Warmup steps	3,000
Early stopping	2 epochs
Optimizer	AdamW (β=(0.9, 0.98), wd=1e-3)

Adapt decoder and joint to Croatian while keeping the encoder stable.

Phase 2 — Full fine-tuning, differential learning rates

Hyperparameter	Value
Epochs	8
Encoder LR	1e-5
Decoder / Joint LR	5e-4
Min LR	1e-6
Warmup steps	3,000
Early stopping	3 epochs

Unlock the encoder at a small LR. After Phase 2, in-domain (ParlaSpeech) WER reached ~14.5% but out-of-domain FLEURS WER was ~35% — clear domain overfitting.

Phase 2.5 — Diversity fine-tuning

Hyperparameter	Value
Epochs	2
Encoder LR	5e-6
Decoder / Joint LR	2e-5
Min LR	1e-7
Warmup steps	500
Data mix	70% ParlaSpeech + 30% FLEURS + VoxPopuli

Reduce domain overfitting without sacrificing stability; brought FLEURS WER from ~35% down to the reported 28.50%.

Data augmentation & preprocessing

• Speed perturbation: 0.9×–1.1×
• SpecAugment: 3 freq masks (27 bins), 12 time masks (0.08 s)
• Text normalization: strip trailing punctuation
• Duration filtering: 0.3–15.0 s
• Tarred dataset (128 shards); bucketed batching (buckets [96, 72, 48, 32, 16])

Usage

Loading

import nemo.collections.asr as nemo_asr

model = nemo_asr.models.ASRModel.restore_from("nemotron-speech-streaming-hr-0.6b.nemo")
model.eval()

Compatibility: load with a recent NeMo (the 26.02 NGC container or NeMo main). ASRModel.restore_from auto-detects the model class. On very new PyTorch you may need weights_only=False when loading checkpoints.

Offline / batch inference

transcriptions = model.transcribe(["audio.wav"])
print(transcriptions[0])

Streaming inference (native cache-aware)

Select an operating point via the model's trained att_context_size — fixed left context (70 frames), lookahead in frames:

# lookahead frames -> total buffer:  13 -> 1120 ms (best WER)   6 -> 560 ms
#                                      1 ->  160 ms              0 ->   80 ms (instant)
LEFT, LOOKAHEAD = 70, 13
model.encoder.set_default_att_context_size([LEFT, LOOKAHEAD])
model.encoder.setup_streaming_params()

Then drive it with CacheAwareStreamingAudioBuffer + model.conformer_stream_step, carrying the cache state between chunks. A complete reference implementation is NeMo's speech_to_text_cache_aware_streaming_infer.py.

Prefer the larger buffer (1120 ms) when latency budget allows; use the 80–160 ms buffers for near-instant streaming at ~1–2 pts higher WER. Accuracy is flat across the range, so choose purely on your latency requirement.

Known Failure Modes

• Short isolated utterances. Single words / very short phrases at low lookahead can yield empty output — a structural property of streaming RNNT with limited context ("Hvala." → (empty)).
• Numeric content. Trailing-zero drops (120 → 12) and year transpositions (2017 → 17) are the most dangerous; use number post-processing for date/quantity-critical use.
• Long compound words. 10+ character words show 35–58% error rates on FLEURS; domain-adapted fine-tuning helps for technical vocabulary.
• Morphological case. Correct stems with wrong endings (istraživanja → istraživanje) score as full-word errors but carry low semantic cost; downstream lemmatization mitigates.
• Repetition loops. Rare on the cache-aware path (≤2 per ~900 utterances); a ≥3-token dedup pass removes them.

Related models

Since this model was trained, NVIDIA has released nvidia/nemotron-3.5-asr-streaming-0.6b — a multilingual (40 language-locale) cache-aware streaming model that adds native Croatian support via language-ID prompt conditioning, along with built-in punctuation and capitalization. It shares the same Cache-Aware FastConformer-RNNT architecture and the same total-buffer operating points used here, and is a strong general-purpose alternative for Croatian streaming ASR. This model remains a lightweight, Croatian-specialized fine-tune focused on the parliamentary/formal domain; a direct head-to-head on the datasets above is a natural point of comparison.

Limitations

• Optimized for 16 kHz mono audio.
• Best on clean speech; robustness degrades below ~10 dB SNR.
• Primary domain is parliamentary Croatian; strong dialectal speech may vary.
• Max recommended utterance length 15 s; segment longer audio first.
• Number transcription is unreliable without post-processing.

Training Infrastructure

• GPU: NVIDIA A100-SXM4-80GB
• Loss: RNNT (warprnnt_numba), FastEmit λ=0.005
• Batch size: 32 (per device), gradient clip 1.0
• Experiment tracking: Weights & Biases

Citation

@misc{nemotron-speech-streaming-hr-0.6b,
  title        = {Nemotron Speech Streaming HR 0.6B},
  author       = {Ivan Luksic},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/iluksic/nemotron-speech-streaming-hr-0.6b},
  note         = {Fine-tuned from nvidia/nemotron-speech-streaming-en-0.6b on Croatian speech}
}

License

This model is a derivative of nvidia/nemotron-speech-streaming-en-0.6b and is released under the NVIDIA Open Model License, inheriting the base model's license terms. Please review the base model card and the license text for the full conditions before use or redistribution.