---
license: cc-by-nc-4.0
library_name: wav2taste
pipeline_tag: audio-classification
datasets:
- csc-unipd/sonic-seasoning
language:
- en
tags:
- audio
- music-information-retrieval
- crossmodal
- taste
- flavor
- multimodal
metrics:
- pearsonr
- rmse
---

# wav2taste — trained taste-prediction heads

[![arXiv](https://img.shields.io/badge/arXiv-2607.03296-b31b1b?logo=arxiv&logoColor=white)](https://arxiv.org/abs/2607.03296)

Predict the five basic **tastes** — `sweet`, `bitter`, `salty`, `sour`, `spicy`, each in `[0, 1]` — that a piece of audio evokes. These are the trained heads behind the paper *Taste-aware music retrieval from audio embeddings* (Matteo Spanio, Antonio Rodà — University of Padua, CBMI 2026).

Each head is a small MLP (single encoder) or a gated late-fusion module (multiple encoders) sitting on top of **frozen** audio encoders. The heads are tiny and downloaded from this repo; the frozen encoders they sit on are pulled from their own upstream repos at load time. Inference needs **no access to the training dataset**.

## Usage

```bash
pip install git+https://github.com/CSCPadova/wav2taste
```

```python
from wav2taste import load_taste_model

model = load_taste_model("vggish+mule")     # strongest configuration
model.predict("song.wav")
# {'sweet': 0.34, 'bitter': 0.71, 'salty': 0.12, 'sour': 0.58, 'spicy': 0.09}
```

`load_taste_model(name)` downloads `heads/<name>.pt` from this repo and rebuilds the encoder(s) + head. Audio is decoded to mono, resampled to each encoder's rate, split into 15-second windows that the encoder pools, then averaged across windows — exactly as at training time. Outputs are a `sigmoid`, so every value is in `[0, 1]`.

## Available heads

**Single encoders (10):** `mfcc`, `clap`, `mert`, `ast`, `encodec`, `vggish`, `hubert`, `panns`, `omar-rq`, `mule`.

**Gated fusions (7):** `ast+vggish`, `ast+mule`, `vggish+mule`, `clap+mule`, `clap+vggish`, `ast+vggish+mule`, `clap+ast+vggish+mule`.

`mfcc` is the untrained-DSP floor; `mule` uses the [`matteospanio/mule`](https://huggingface.co/matteospanio/mule) weights (CC-BY-NC). The full list is also exposed as `wav2taste.PRETRAINED`.

## Performance

On the 40-item held-out test set (five-seed means):

| model | macro Pearson r | macro RMSE |
|-------|----------------:|-----------:|
| `vggish+mule` (best fusion) | **0.724 ± 0.020** | **0.134** |
| best single encoder (~`vggish`/`ast`) | ≈ 0.666 | ≈ 0.134 |
| prior per-taste-AST baseline | 0.556 | 0.219 |

- On **absolute error** the encoders are statistically flat — a single `vggish` matches the best fusion (macro RMSE ≈ 0.134); the fusion advantage is confined to **rank correlation** (0.724 vs 0.666), and the lift over strong single encoders is within bootstrap confidence intervals.
- On held-out **real music**, model error is less than half an average human rater's deviation from the group consensus (RMSE 0.13 vs 0.28) — the model tracks the consensus more closely than a typical individual rater.
- Each head was selected on validation loss from a single seed; the headline numbers above are five-seed means. See the paper for per-target, per-source, and CI tables.

## Training data

Trained on [`csc-unipd/sonic-seasoning`](https://huggingface.co/datasets/csc-unipd/sonic-seasoning) (269 train / 68 val / 40 test clips) with **masked multi-task MSE** over the five tastes — one model, five outputs, and each row contributes loss only on the tastes it was actually rated for (no imputation). The 40-item test split is a real-music + generated-music generalization probe held out entirely from training.

## Intended use & limitations

- **Intended use:** research on crossmodal sound↔taste correspondences, content-based music retrieval by taste profile, and as baselines/reference heads for the wav2taste benchmark. Non-commercial only.
- **Taste ratings are perceptual consensus values, not ground-truth chemistry** — they encode how listeners *associate* a sound with a taste. Cross-source comparability is approximate (different elicitation procedures; see the dataset card).
- `spicy` is the sparsest target (fewest training rows); its predictions are the least reliable. Report confidence intervals when comparing models on it.
- Trained on short music/sound clips at mono/44.1 kHz; behavior on speech, very long recordings, or out-of-distribution audio is unvalidated.

## License

**CC-BY-NC-4.0** (non-commercial): the heads derive from a non-commercial training corpus and from MULE's CC-BY-NC weights. The training/inference **code** is Apache-2.0 at [`github.com/CSCPadova/wav2taste`](https://github.com/CSCPadova/wav2taste). The frozen upstream encoders keep their own licenses.

## Citation

```bibtex
@inproceedings{spanio2026taste,
  title     = {Taste-aware music retrieval from audio embeddings},
  author    = {Spanio, Matteo and Rod{\`a}, Antonio},
  booktitle = {Proceedings of the International Conference on Content-Based Multimedia Indexing (CBMI)},
  year      = {2026},
}
```