Audio-to-Audio
MambaSSM
Safetensors
streaming speech-enhancement
speech-enhancement
universal speech enhancement
multiple input sampling rates
language-agnostic
Instructions to use nvidia/Real-time_RE-USE with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MambaSSM
How to use nvidia/Real-time_RE-USE with MambaSSM:
from mamba_ssm import MambaLMHeadModel model = MambaLMHeadModel.from_pretrained("nvidia/Real-time_RE-USE") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -8,7 +8,7 @@ tags:
|
|
| 8 |
- multiple input sampling rates
|
| 9 |
- language-agnostic
|
| 10 |
---
|
| 11 |
-
# **<span style="color:#76b900;">🤫 Real-time RE-USE: Real-time Multilingual Universal Speech Enhancement</span>**
|
| 12 |
# Model Overview
|
| 13 |
|
| 14 |
|
|
@@ -43,7 +43,7 @@ Follow the simple steps below to generate enhanced speech using our model:
|
|
| 43 |
1. Place your noisy speech files in the folder `noisy_audio/`
|
| 44 |
2. Run the following command:
|
| 45 |
```bash
|
| 46 |
-
sh offline_inference.sh
|
| 47 |
```
|
| 48 |
3. The enhanced speech files will be saved in `offline_enhanced_audio/`.
|
| 49 |
|
|
@@ -53,17 +53,12 @@ That's all !
|
|
| 53 |
|
| 54 |
a. You can enable bandwidth extension by setting the target bandwidth using the `BWE argument` in the script.
|
| 55 |
|
| 56 |
-
b. You can set `Exit_layer` between 3 and 12, and `look_ahead_frames` between 0 and 2, to achieve different quality–latency trade-offs.
|
| 57 |
|
| 58 |
---
|
| 59 |
|
| 60 |
-
We also provide the **Online Inference (one frame in, one frame out code
|
| 61 |
-
|
| 62 |
-
2. Run the following command:
|
| 63 |
-
```bash
|
| 64 |
-
sh online_inference.sh
|
| 65 |
-
```
|
| 66 |
-
3. The enhanced speech files will be saved in `enhanced_audio/`.
|
| 67 |
|
| 68 |
The output of offline inference and online inference should be almost the same!!
|
| 69 |
|
|
@@ -79,16 +74,18 @@ Global.
|
|
| 79 |
Researchers and general users can use this model to enhance the quality of their speech data.
|
| 80 |
|
| 81 |
## Release Date
|
| 82 |
-
Hugging Face 2026/
|
| 83 |
|
| 84 |
## References
|
| 85 |
[1] [Rethinking Training Targets, Architectures and Data Quality for Universal Speech Enhancement](https://arxiv.org/abs/2603.02641), 2025.
|
|
|
|
|
|
|
| 86 |
(Note: The released model checkpoint differs from the one reported in the paper. It incorporates additional degradation types (e.g., microphone response and more codecs) and is fine-tuned on a smaller, high-quality clean subset.)
|
| 87 |
|
| 88 |
## Model Architecture
|
| 89 |
**Architecture Type:** Convolutional encoder, Convolutional decoder, and Mamba for time–frequency modeling <br>
|
| 90 |
-
**Network Architecture:**
|
| 91 |
-
**Number of model parameters:**
|
| 92 |
|
| 93 |
## Input
|
| 94 |
Input Type(s): Audio <br>
|
|
@@ -117,7 +114,7 @@ Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated sys
|
|
| 117 |
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
|
| 118 |
|
| 119 |
## Model Version(s)
|
| 120 |
-
Current version:
|
| 121 |
|
| 122 |
## Training Datasets
|
| 123 |
**Data Modality:**
|
|
@@ -158,4 +155,8 @@ Please consider to cite our paper and this framework, if they are helpful in you
|
|
| 158 |
journal={arXiv preprint arXiv:2603.02641},
|
| 159 |
year={2026}
|
| 160 |
}
|
| 161 |
-
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
- multiple input sampling rates
|
| 9 |
- language-agnostic
|
| 10 |
---
|
| 11 |
+
# **<span style="color:#76b900;"> 🚀+🤫 Real-time RE-USE: Real-time Multilingual Universal Speech Enhancement</span>**
|
| 12 |
# Model Overview
|
| 13 |
|
| 14 |
|
|
|
|
| 43 |
1. Place your noisy speech files in the folder `noisy_audio/`
|
| 44 |
2. Run the following command:
|
| 45 |
```bash
|
| 46 |
+
sh offline_inference.sh
|
| 47 |
```
|
| 48 |
3. The enhanced speech files will be saved in `offline_enhanced_audio/`.
|
| 49 |
|
|
|
|
| 53 |
|
| 54 |
a. You can enable bandwidth extension by setting the target bandwidth using the `BWE argument` in the script.
|
| 55 |
|
| 56 |
+
b. You can set `Exit_layer` (between 3 and 12), and `look_ahead_frames` (between 0 and 2), to achieve different **quality–latency** trade-offs.
|
| 57 |
|
| 58 |
---
|
| 59 |
|
| 60 |
+
We also provide the **Online Inference (one frame in, one frame out) code** for streaming model:
|
| 61 |
+
Please refer to `online_inference.py` and `online_inference.py`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 62 |
|
| 63 |
The output of offline inference and online inference should be almost the same!!
|
| 64 |
|
|
|
|
| 74 |
Researchers and general users can use this model to enhance the quality of their speech data.
|
| 75 |
|
| 76 |
## Release Date
|
| 77 |
+
Hugging Face 2026/04/14 (private)
|
| 78 |
|
| 79 |
## References
|
| 80 |
[1] [Rethinking Training Targets, Architectures and Data Quality for Universal Speech Enhancement](https://arxiv.org/abs/2603.02641), 2025.
|
| 81 |
+
|
| 82 |
+
[2] One Model, Many Latencies: Universal Speech Enhancement for Diverse Real-Time Applications
|
| 83 |
(Note: The released model checkpoint differs from the one reported in the paper. It incorporates additional degradation types (e.g., microphone response and more codecs) and is fine-tuned on a smaller, high-quality clean subset.)
|
| 84 |
|
| 85 |
## Model Architecture
|
| 86 |
**Architecture Type:** Convolutional encoder, Convolutional decoder, and Mamba for time–frequency modeling <br>
|
| 87 |
+
**Network Architecture:** Mamba with up to 12 layers <br>
|
| 88 |
+
**Number of model parameters:** 3.7M <br>
|
| 89 |
|
| 90 |
## Input
|
| 91 |
Input Type(s): Audio <br>
|
|
|
|
| 114 |
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
|
| 115 |
|
| 116 |
## Model Version(s)
|
| 117 |
+
Current version: 12_random_layer_ahead_sep_conv2_1010k
|
| 118 |
|
| 119 |
## Training Datasets
|
| 120 |
**Data Modality:**
|
|
|
|
| 155 |
journal={arXiv preprint arXiv:2603.02641},
|
| 156 |
year={2026}
|
| 157 |
}
|
| 158 |
+
```
|
| 159 |
+
|
| 160 |
+
and
|
| 161 |
+
|
| 162 |
+
One Model, Many Latencies: Universal Speech Enhancement for Diverse Real-Time Applications
|