% ==============================================================================
% Auto Research Is Not Auto Tuning
% Target: NeurIPS 2026
% Total Length: Max 9 pages (Main Text) + References + Unlimited Appendix
% ==============================================================================

\documentclass{article}

% NeurIPS 2026 style
\usepackage{neurips_2026}

% Standard packages
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{hyperref}
\usepackage{url}
\usepackage{booktabs}
\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{nicefrac}
\usepackage{microtype}
\usepackage{xcolor}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{multirow}
\usepackage{enumitem}

% Theorem environments
\newtheorem{definition}{Definition}
\newtheorem{proposition}{Proposition}
\newtheorem{remark}{Remark}

% Convenience macros
\newcommand{\Cspace}{\mathcal{C}}
\newcommand{\Hspace}{\mathcal{H}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\N}{\mathbb{N}}
\newcommand{\E}{\mathbb{E}}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\JSD}{D_{\mathrm{JSD}}}
\newcommand{\APstar}{\mathrm{AP}^*}
\newcommand{\todo}[1]{\textcolor{red}{\textbf{[TODO:}\,#1\textbf{]}}}

% NeurIPS checklist answer macros
\renewcommand{\answerYes}[1][]{\textcolor{blue}{[Yes]}#1}
\renewcommand{\answerNo}[1][]{\textcolor{red}{[No]}#1}
\renewcommand{\answerNA}[1][]{\textcolor{gray}{[N/A]}#1}

\hypersetup{pdfauthor={}, pdftitle={Self-Evolving Search Spaces: The eta-squared Boundary Between Auto-Tuning and Auto-Research}, pdfsubject={}, pdfkeywords={}, pdfcreator={}, pdfproducer={}}

% Title chosen to be regime-qualified: our own E2E LoRA ablation (Section 8) shows
% LR $\eta^2$ rises to 0.79 under gradient-based adaptation, so the
% paper's actual contribution is a \emph{diagnostic} that tells practitioners when
% SSC dominates---not an absolute claim that classical HPO is obsolete.
\title{Self-Evolving Search Spaces:\\The $\eta^2$ Boundary Between Auto-Tuning and Auto-Research}

\author{%
  Anonymous Authors
}

\begin{document}

\maketitle

%% ============================================================
%% ABSTRACT
%% ============================================================
\begin{abstract}
When LLM agents run ML experiments autonomously, where does their value come from---and under what conditions does that value exceed what a classical hyperparameter optimizer would deliver? We study this with 10{,}000+ experiments across two vision tasks (Nexar collision, UCF-101) in the \emph{frozen-feature} regime and a third task (speech recognition) under full fine-tuning.
We disentangle \emph{search space construction} (SSC---introducing a new backbone, data source, or adapter target; ``auto-research'' here denotes this autonomous SSC, not algorithmic invention) from \emph{within-space optimization} (WSO---tuning values on existing axes). Two LLM agents autonomously identified V-JEPA\,2 via Semantic Scholar, composed multi-backbone fusion, wrote integration code, and designed the deployment recipe---extending the schema to axes no current HPO system instantiates.
ANOVA on the frozen-feature regime confirms why SSC matters: architecture explains 20--51\% of test variance (conservative i.i.d.\ to full adaptive; both ``large'' effects); all HPs combined explain $<$5\%. On the core schema, even calibrated SMAC (AUC, $\rho = 0.968$) reaches only $0.668$---a \emph{schema ceiling}. The LLM-constructed schema breaks it: Random search alone reaches $0.737$.
Obfuscated-names ablations ($n = 1{,}609$) falsify memorization; cross-task validation on UCF-101 confirms generalization with a different winning backbone ($\eta^2_{\mathrm{arch}} = 0.148$, SigLIP2 $\neq$ VJepa2).
\textbf{Regime boundary:} under end-to-end LoRA fine-tuning (768d backbones, V-JEPA\,2 excluded due to dimensionality), learning-rate $\eta^2$ rises to $0.79$ while architecture $\eta^2$ falls to $0.12$---in that regime classical HPO is the binding constraint. The ratio $\eta^2_{\mathrm{arch}}$ thus serves as an \emph{explicit boundary} between auto-tuning and auto-research: when high, autonomous SSC is the dominant lever; when low, classical HPO is the binding prerequisite.
Practically, a single-model deployment of the LLM-constructed space reaches $\text{mAP}_{\mathrm{ALL}} = 0.906$ ($0.920$ public / $0.893$ private; Section~\ref{sec:deployed}), surpassing the prior Kaggle public \#1 ($+0.048$) in post-competition evaluation. On ASR, data-centric schema extensions yield \textbf{\#1 on the Open ASR Leaderboard} at submission ($5.30\%$ WER).
\end{abstract}

%% ============================================================
%% 1. INTRODUCTION
%% ============================================================
\section{Introduction}
\label{sec:intro}

We ran two LLM agents on 16 GPUs against a dashcam collision prediction task---3{,}190 experiments, fully autonomous.
The agents identified and integrated V-JEPA\,2 via API-driven literature search, composed multi-backbone fusion, and wrote all integration code---without human intervention.
A single-model deployment of the LLM-constructed space---including the LLM-designed TTE matching and 4-view TTA with CV-mix aggregation---reaches $\text{mAP}_{\mathrm{ALL}} = 0.906$ ($0.920$\,/\,$0.893$ public\,/\,private; Section~\ref{sec:deployed}), near the prior Kaggle \#1 on the private holdout ($0.893$ vs.\ $0.898$) in post-competition evaluation while ranking \textbf{\#1 on the public split} ($+0.048$).
Classical HPO on the human-authored core schema plateaus at $\approx 0.70$ (TPE $0.696$, BOHB $0.702$); even with a well-calibrated proxy (AUC, $\rho = 0.968$), SMAC on this schema selects at most $0.668$ test mAP---a schema ceiling, not an optimizer failure, because the LLM-discovered components (V-JEPA\,2, GRU/LSTM encoders) are not in that schema. Once the LLM constructs the expanded schema, classical search works on it (Expanded TPE $0.694$, Expanded Random $0.737$; Table~\ref{tab:convergence}): the LLM's value is schema construction, not within-schema search. Figure~\ref{fig:nexar_trajectory} summarizes the full trajectory.

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{figures/nexar_trajectory.pdf}
\caption{Nexar optimization trajectory (chronological). Four autonomous phases: frozen-feature probing, E2E fine-tuning, TTE matching, 4-view TTA + CV-mix. Champion: $0.906$ mAP$_{\mathrm{ALL}}$ on V-JEPA\,2 (public \#1, $+0.048$). Circles: new bests; diamonds: informative negatives; red X: harmful change; green star: deployed champion. Dashed: prior Kaggle public \#1 ($0.872$) / private \#1 ($0.898$).}
\label{fig:nexar_trajectory}
\end{figure}

This paper is not a claim that LLM agents are better optimizers than classical HPO---within any fixed schema our data show no such advantage. The claim is capability-based: \emph{LLM agents can autonomously extend the search schema}---propose a backbone, write integration code, add a data source---and no current HPO or NAS system does \citep{bergstra2011algorithms, falkner2018bohb, liu2019darts, pham2018enas}. The value is the \emph{autonomous end-to-end pipeline}---retrieve, evaluate fit, write code, resolve mismatches, validate---across 13 candidate backbones without human intervention. We call this \emph{search space construction} (SSC; Definition~\ref{def:ssc_wso}), distinguished from \emph{within-space optimization} (WSO). Throughout, ``auto-research'' denotes autonomous SSC, not algorithmic invention.
That backbone choice matters is known \citep{kornblith2019better, oquab2024dinov2}. What is \emph{not} known is (i)~\emph{how much}: $\eta^2_{\mathrm{arch}} = 0.20$--$0.51$ vs.\ $<$0.01 per HP, a 20--50$\times$ gap; (ii)~\emph{when} it stops: E2E LoRA $\eta^2_{\mathrm{lr}} \!\uparrow\! 0.79$, $\eta^2_{\mathrm{arch}} \!\downarrow\! 0.12$; and (iii)~whether this holds across modalities (vision, NLP, tabular, speech; Appendix~\ref{app:cross_domain}). The title is a \emph{diagnostic}---``when auto-research beats auto-tuning''---not an absolute claim.

We formalize LLM-guided ML research as optimization over a structured combinatorial configuration space $\Cspace$ and instantiate this framework with fully autonomous campaigns on two tasks: dashcam collision detection ($n = 3{,}190$, two LLM agents, 16 H100 GPUs) and UCF-101 action recognition ($n = 997$ baselines + $990$ LLM-guided). Our contributions are:
\begin{enumerate}[leftmargin=*, itemsep=1pt]
    \item \textbf{The $\eta^2$ boundary between auto-tuning and auto-research.} A formal SSC/WSO split (Definition~\ref{def:ssc_wso}) and an ANOVA decomposition in which the ratio $\eta^2_{\mathrm{arch}}$ functions as an explicit regime boundary. Frozen-feature regime: $\eta^2_{\mathrm{arch}} = 0.20$--$0.51$, autonomous SSC dominates. E2E LoRA regime: $\eta^2_{\mathrm{arch}} \!\downarrow\! 0.12$, $\eta^2_{\mathrm{lr}} \!\uparrow\! 0.79$, classical HPO dominates (Section~\ref{sec:e2e}).
    \item \textbf{SSC is a capability no current classical HPO system provides} (Section~\ref{sec:convergence}). Classical HPO on the core schema plateaus at $\approx 0.70$ (schema ceiling; even AUC-calibrated SMAC selects $\leq 0.668$, Table~\ref{tab:proxy_calibration}). The LLM extended the schema (V-JEPA\,2, GRU/LSTM, multi-backbone fusion) and reaches $0.727$ in search, $0.906$ in deployment; encoder SSC (GRU/LSTM) drives the search-phase ceiling break, while V-JEPA\,2 dominates under E2E fine-tuning ($\Delta$mAP $\geq 0.35$ at fixed HPs, Table~\ref{tab:xfer}). Classical search on the LLM-constructed schema reaches the same region (Expanded Random $0.737$ in $394$ trials vs.\ LLM $0.727$ in $3{,}138$; Table~\ref{tab:convergence})---the LLM has no WSO advantage; its value is the schema it constructs.
    \item \textbf{A deployable single-model champion} from the LLM-constructed space. $0.906$ mAP ($0.920$/$0.893$ public/private; Section~\ref{sec:deployed}) near the prior Kaggle private \#1 ($0.898$) and ranks \#1 public ($+0.048$); ASR deployment ranks \#1 on Open ASR Leaderboard. Obfuscated ablations ($n = 1{,}609$) falsify memorization.
\end{enumerate}


%% ============================================================
%% 2. FORMAL FRAMEWORK
%% ============================================================
\section{Formal Framework}
\label{sec:framework}

\begin{definition}[Configuration Space]
\label{def:config_space}
The configuration space is a structured product:
$\Cspace = \Cspace_{\mathrm{arch}} \times \Cspace_{\mathrm{loss}} \times \Cspace_{\mathrm{train}} \times \Cspace_{\mathrm{data}}$
where $\Cspace_{\mathrm{arch}} = \Cspace_{\mathrm{backbone}} \times \Cspace_{\mathrm{encoder}} \times \Cspace_{\mathrm{pooling}}$.
The full campaign explored $|\Cspace_{\mathrm{backbone}}| = 6$ backbones (VJepa2, DINOv3-B\footnote{DINOv3 refers to DINOv2 with register tokens \citep{darcet2024registers}; we adopt the shorthand used by the LLM agents throughout the campaign.}, DINOv3-L, DINOv2-B, SigLIP2 \citep{tschannen2025siglip2}, InternViT), $|\Cspace_{\mathrm{encoder}}| = 5$ temporal encoders (Zipformer, RetNet, BiMamba, Hybrid, GRU/LSTM), and $|\Cspace_{\mathrm{pooling}}| = 4$ (attention, mean, last, max). The expanded-schema ablation (Section~\ref{sec:convergence}) uses the subset $\{$V-JEPA\,2, DINOv3-B, DINOv2-B, DINOv2-S, SigLIP2$\}$, reflecting the agents' final operational schema after pruning low-signal backbones (DINOv3-L, InternViT) and adding DINOv2-S.
$\Cspace_{\mathrm{loss}}$ mixes categorical (loss type $\in$ \{focal, BCE, label\_smoothing\}) and continuous dimensions (focal $\gamma \in [0.5, 5.0]$, $\alpha \in [0.1, 0.9]$).
$\Cspace_{\mathrm{train}}$ covers learning rate $\in [10^{-5}, 10^{-2}]$, weight decay, batch size, scheduler, sequence length, and epochs.
The full discrete cardinality is $|\Cspace_{\mathrm{discrete}}| = 108{,}000$; each cell contains a 6--8 dimensional continuous HP volume.
The from-scratch baselines use a core subspace $\Cspace^{\mathrm{core}}$ (5 backbones, 4 encoders, 2 poolings; $|\Cspace_{\mathrm{discrete}}^{\mathrm{core}}| = 19{,}200$).
\end{definition}

\begin{definition}[SSC vs.\ WSO]
\label{def:ssc_wso}
Let $\Cspace^{(t)}$ denote the search space at step $t$. A modification of the search procedure is:
\begin{itemize}[leftmargin=*, itemsep=1pt, topsep=2pt]
\item \textbf{Search Space Construction (SSC)} iff it introduces a \emph{new structural axis} into $\Cspace$---a new dimension, a new categorical value for an existing dimension, or a new component (e.g., a new backbone, a new data source, a new adapter target set, a new loss family). Formally, $\Cspace^{(t+1)} \not\subseteq \Cspace^{(t)}$. The capability gap is not the axis itself (a human could pre-define any axis), but the \emph{autonomous pipeline}: retrieving candidates from external knowledge (literature APIs, model hubs), evaluating fit, writing integration code, and validating the extension---end-to-end, without human intervention.
\item \textbf{Within-Space Optimization (WSO)} iff it explores values on an \emph{existing} continuous, integer, or categorical axis of $\Cspace^{(t)}$ without introducing new axes or categorical values. Formally, the selected $c \in \Cspace^{(t)}$ with $\Cspace^{(t+1)} = \Cspace^{(t)}$.
\end{itemize}
Under this definition, proposing V-JEPA\,2 is SSC; sweeping LoRA rank is WSO. The boundary is intentionally relative to $\Cspace^{(0)}$: if a practitioner pre-defines an axis, selecting values is WSO regardless of origin. The diagnostic value of this relativity is the point---$\eta^2_{\mathrm{arch}}$ tells the practitioner whether \emph{their} initial schema is the bottleneck, not whether the task has an absolute SSC/WSO split. Table~\ref{tab:ssc_vs_wso_taxonomy} maps every agent action to one of the two categories.
\end{definition}

\begin{table}[t]
\centering
\caption{SSC vs.\ WSO taxonomy across both campaigns, under Definition~\ref{def:ssc_wso}. No current classical HPO system executes SSC actions.}
\label{tab:ssc_vs_wso_taxonomy}
\small
\begin{tabular}{@{}llll@{}}
\toprule
Campaign & Agent action & Class & Why \\
\midrule
Nexar & Propose V-JEPA\,2 backbone & SSC & New value in $\Cspace_{\mathrm{backbone}}$ \\
Nexar & Propose multi-backbone fusion & SSC & New structural axis (fusion head) \\
Nexar & Propose GRU/LSTM encoders & SSC & New values in $\Cspace_{\mathrm{encoder}}$ \\
Nexar & Sweep learning rate / focal $\gamma,\alpha$ & WSO & Existing continuous axes \\
ASR & Add SPGISpeech as data source & SSC & New value in $\Cspace_{\mathrm{data\_source}}$ \\
ASR & Add MLP layers to LoRA targets & SSC & New value in $\Cspace_{\mathrm{adapter\_targets}}$ \\
ASR & Oversample multiplier / AMI count / LoRA rank & WSO & Integer HPO on existing axes \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{Optimization problem.}
Let $f: \Cspace \to [0, 1]$ map a configuration to its metric (validation AP as the agents' signal; competition mAP on 1{,}344 held-out test videos as the primary evaluation), observed as $y = f(c) + \epsilon$. The objective is $c^* = \arg\max_{c \in \Cspace} \E[f(c)]$, and the cumulative best is $\APstar(N) = \max_{i \leq N} y_i$.

\paragraph{Search policies.}
A search policy $\pi: (\Cspace \times \R)^* \to \Delta(\Cspace)$ maps experiment history to a distribution over configurations. We compare:
(i)~$\pi_{\mathrm{TPE}}$: Tree-Parzen Estimator \citep{bergstra2011algorithms};
(ii)~$\pi_{\mathrm{BOHB}}$: BOHB with early stopping \citep{falkner2018bohb};
(iii)~$\pi_{\mathrm{SMAC}}$: SMAC with conditional random forests \citep{hutter2011smac};
(iv)~$\pi_{\mathrm{LLM}}$: LLM agent generates structured YAML configurations from a text encoding of the history.
The LLM policy has access to failure diagnostics, external knowledge from pretraining, and the genealogy of prior configurations. Our central finding (Section~\ref{sec:convergence}) is that this advantage manifests in SSC, not WSO.

\paragraph{Convergence model.}
Cumulative best is empirically well-fit by a logistic curve ($R^2 \geq 0.93$; Appendix~\ref{app:convergence_models}). The logistic captures the smooth WSO trajectory within each schema; discrete SSC events (Table~\ref{tab:expansion}) appear as level shifts between regimes.


%% ============================================================
%% 3. SYSTEM OVERVIEW
%% ============================================================
\section{System Overview}
\label{sec:system}

We implement the search loop using Orze,\footnote{Code and configs: \url{https://anonymous.4open.science/r/nips-2026-submission-1C52}; de-anonymized upon acceptance. Full system diagram: Appendix~\ref{app:system_schematic}.} an open-source orchestration system for LLM-guided ML research. At step $t$: (1)~the agent observes history $H_{t-1}$ as a leaderboard; (2)~selects $c_t \sim \pi(c \mid H_{t-1})$ by generating structured YAML; (3)~the system evaluates $y_t = f(c_t) + \epsilon$ on GPU; (4)~$H_t = H_{t-1} \cup \{(c_t, y_t)\}$.

\paragraph{Multi-agent coordination and autonomous SSC.}
Two heterogeneous LLM agents---Claude Opus 4 (819 cycles) and Gemini 2.5 Pro (864 cycles)---operate in parallel on 16 H100 GPUs, executing $n = 3{,}190$ experiments. When agents identify components outside the current schema they autonomously locate them via literature-search APIs, write integration code, and extend $\Cspace$---\textbf{with zero research steering} (human involvement limited to infrastructure debugging). The agents expanded the space three times (Table~\ref{tab:expansion}): V-JEPA\,2 (via web search), multi-backbone fusion, and GRU/LSTM encoders. Without these extensions, any policy is restricted to the $\sim$0.70 ceiling; with the LLM-constructed schema, classical Random search alone reaches $0.737$ (via GRU/LSTM encoder SSC on DINOv3-B), and the deployed champion reaches $0.906$ (via V-JEPA\,2 under E2E fine-tuning). V-JEPA\,2 was released in June 2025 \citep{assran2025vjepa2} and did not exist during the original competition \citep{nexar2024}.

\begin{table}[t]
\centering
\caption{Autonomous search space construction events. Each component was identified, evaluated, and implemented by the LLM agent with zero human involvement.}
\label{tab:expansion}
\small
\begin{tabular}{@{}lccc@{}}
\toprule
Expansion & Day & Discovery method & Subsequent expts \\
\midrule
V-JEPA\,2 backbone & 1 & Web search + domain reasoning & $>$2{,}000 \\
Multi-backbone fusion & 2 & Per-backbone error correlation & $>$1{,}000 \\
GRU/LSTM encoders & 8 & Leaderboard variance analysis & $>$400 \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{Task and dataset.}
Binary collision prediction from dashcam video \citep{nexar2024, nexar2025}: 1{,}500 training videos (50\% positive), split 80/10/10; 1{,}344 held-out test videos; primary metric is mean AP across three time-to-event windows (computed locally). The 177-sample validation set saturates ($\text{AP} \to 1.0$), so all primary comparisons use competition mAP. Secondary task: UCF-101 \citep{soomro2012ucf101} (top-1 accuracy).


%% ============================================================
%% 4. THE ARCHITECTURE-DOMINATED LANDSCAPE
%% ============================================================
\section{The Architecture-Dominated Landscape}
\label{sec:landscape}
\label{sec:anova}

Before asking \emph{how} different search policies perform, we must understand \emph{where} performance variance resides. If backbone choice explains 48\% of variance and learning rate explains $<$1\%, then any search policy's value hinges on navigating the architecture axis---not the HP axes. Table~\ref{tab:top5} shows the campaign's top configurations; two qualitatively different strategies dominate, already suggesting architectural decisions dominate.

\begin{table}[t]
\centering
\caption{Top-5 configurations by competition mAP (1{,}344 held-out test videos). All top results were proposed by LLM agents. Two qualitatively different strategies dominate.}
\label{tab:top5}
\small
\begin{tabular}{@{}cllcc@{}}
\toprule
Rank & Backbone & Encoder & mAP & Source \\
\midrule
1 & DINOv3+SigLIP2 & GRU & 0.7270 & LLM \\
2 & DINOv3+SigLIP2 & GRU & 0.7245 & LLM \\
3 & VJepa2 & Zipformer & 0.7240 & LLM \\
4 & DINOv3+SigLIP2 & BiGRU & 0.7230 & LLM \\
5 & DINOv3+SigLIP2 & BiGRU & 0.7204 & LLM \\
\midrule
--- & DINOv3-B & LSTM & 0.7199 & LLM \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{ANOVA decomposition.}
We decompose variance in competition mAP with backbone$\times$encoder as groups (Table~\ref{tab:anova}). Of the 3{,}190 total experiments, we retain runs with complete outputs and unambiguous single-backbone attribution; the backbone$\times$encoder ANOVA uses $n = 1{,}172$. Architecture accounts for $\eta^2 = 0.51$ ($F = 81.4$, $p < 10^{-16}$). Individual continuous HPs each contribute $<$1\%. On i.i.d.\ baselines only ($n = 240$), $\eta^2 = 0.20$ (95\% CI: $[0.16, 0.35]$, $p < 10^{-5}$)---the \textbf{conservative lower bound}. Both exceed the ``large'' threshold ($\eta^2 > 0.14$). UCF-101 confirms task-specificity ($\eta^2 = 0.148$, SigLIP2 wins instead); the E2E ablation (\S\ref{sec:e2e}) shows regime-dependence. $\eta^2$ measures association; the cross-backbone ablation (Table~\ref{tab:xfer}) provides the causal complement.

\begin{table}[t]
\centering
\caption{ANOVA decomposition of AP variance. Bootstrap 95\% CIs (10{,}000 resamples) shown for competition mAP. The i.i.d.\ estimate ($\eta^2 = 0.20$) provides a conservative lower bound free from adaptive-sampling bias.}
\label{tab:anova}
\small
\begin{tabular}{@{}lccccccc@{}}
\toprule
Metric & Data source & $n$ & Groups & $F$ & $\eta^2$ & 95\% CI & $p$ \\
\midrule
Competition mAP & Full adaptive & 1{,}172 & 16 & 81.4 & 0.51 & [0.48, 0.57] & $< 10^{-16}$ \\
\midrule
Competition mAP & i.i.d.\ baselines & 240 & 16 & 3.76 & 0.20 & [0.16, 0.35] & $< 10^{-5}$ \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{Robustness checks.}
The architecture effect survives heteroscedasticity ($\approx 150\times$ variance ratio): (i)~permutation ANOVA ($p_{\mathrm{perm}} < 10^{-4}$, $\eta^2 > 100\times$ null maximum, $n\!=\!6{,}509$); (ii)~an XGBoost surrogate ($R^2 = 0.80$, $\rho = 0.88$) confirms architecture dominates HPs with no i.i.d.\ assumption; (iii)~the cross-backbone ablation (Table~\ref{tab:xfer}) demonstrates the effect causally.

\paragraph{Causal confirmation.}
We take three top VJepa2 configurations and re-train each with all five LLM-schema backbones, holding HPs identical (Table~\ref{tab:xfer}). MViTv2-S ($0.898$ private mAP, the prior Kaggle private \#1) is excluded as it was not in the LLM's search schema. This ablation uses VJepa2-optimal HPs to isolate the \emph{causal effect of backbone at fixed configuration}; it does not claim VJepa2 is universally superior (Table~\ref{tab:convergence} shows DINOv3+SigLIP2 leads during search). VJepa2 achieves mean AP $= 0.906$ while alternatives cluster at $0.51$--$0.55$: a \textbf{0.36 AP gap} from backbone choice alone at fixed HPs, requiring zero statistical assumptions. Per-backbone retuning would narrow this gap---DINOv3-B reaches $0.737$ with its own best frozen-feature HPs (Table~\ref{tab:convergence})---but the ablation isolates the causal effect of backbone at fixed configuration, confirming architecture as the dominant factor.

\begin{table}[t]
\centering
\caption{Cross-backbone transfer: three top HP configurations re-trained with each backbone. All HPs held constant; only backbone changes. The $\Delta$AP $\geq 0.35$ in all cases.}
\label{tab:xfer}
\small
\begin{tabular}{@{}lccccc@{}}
\toprule
HP Config & VJepa2 & DINOv3-B & DINOv2-B & DINOv2-S & SigLIP2 \\
\midrule
Config A & \textbf{0.911} & 0.528 & 0.553 & 0.491 & 0.542 \\
Config B & \textbf{0.904} & 0.521 & 0.552 & 0.503 & 0.546 \\
Config C & \textbf{0.904} & 0.523 & 0.541 & 0.531 & 0.559 \\
\midrule
Mean & \textbf{0.906} & 0.524 & 0.549 & 0.508 & 0.549 \\
\bottomrule
\end{tabular}
\end{table}


%% ============================================================
%% 5. SSC vs. WSO
%% ============================================================
\section{Where the LLM Adds Value: SSC vs.\ WSO}
\label{sec:convergence}

The landscape is steep along the architecture axis (Section~\ref{sec:landscape}). The operative question is whether the LLM opens schema axes classical HPO cannot reach. From-scratch TPE ($n\!=\!621$) and BOHB ($n\!=\!512$) on $\Cspace^{\mathrm{core}}$ with identical HP ranges (Definition~\ref{def:config_space}) plateau at $\sim$0.70 with no improvement over their last 50 evaluations---a space property, not a proxy artifact (Table~\ref{tab:proxy_calibration}). On the LLM-constructed $\Cspace^{\mathrm{exp}}$ (same axes as the LLM's final schema: $\{$V-JEPA\,2, DINOv3-B, DINOv2-B, DINOv2-S, SigLIP2$\}\!\times\!\{$GRU, BiGRU, LSTM, Transformer, RetNet, BiMamba, Zipformer$\}$ plus continuous HPs), Expanded TPE reaches $0.694$ and Expanded Random $0.737$ ($n\!=\!394$ each): once the schema includes the LLM's extensions, classical search is no longer schema-bound. In frozen-feature search, the ceiling break is driven by encoder SSC (GRU/LSTM on DINOv3-B: $0.737$); V-JEPA\,2's dominance emerges only under E2E fine-tuning (Section~\ref{sec:deployed}), where its representations uniquely support the $0.906$ deployed champion.

\begin{table}[t]
\centering
\caption{Search-policy comparison (oracle test-set selection). The core schema plateaus at $\sim$0.70 regardless of optimizer or proxy (Table~\ref{tab:proxy_calibration}). The \textbf{schema-matched ablation} (bottom rows): all LLM-discovered components are added to the initial schema, then classical TPE and uniform Random search are run on the \emph{identical} expanded space. Random reaches $0.737$ in 394 trials; the LLM reaches only $0.727$ in 3{,}138 trials---the LLM is numerically worse at WSO (the 0.01 gap is within test-set noise, but the LLM shows no WSO advantage despite $8\times$ more trials), confirming its value is schema construction. The region $\geq 0.72$ exists only in the LLM-constructed schema (SSC contributions: V-JEPA\,2 backbone, GRU/LSTM encoders, multi-backbone fusion); the search-phase best ($0.737$) uses DINOv3-B\,+\,GRU.}
\label{tab:convergence}
\small
\begin{tabular}{@{}lccc@{}}
\toprule
Policy & $n_{\mathrm{eval}}$ & Val $\APstar$@100 & Competition mAP \\
\midrule
\multicolumn{4}{l}{\emph{Core (human-authored) schema --- no SSC extensions}} \\
$\pi_{\mathrm{TPE}}$ & 621 & 1.000 & 0.696 \\
$\pi_{\mathrm{BOHB}}$ & 512 & 1.000 & 0.702 \\
$\pi_{\mathrm{Random}}$ & 619 & 1.000 & 0.702 \\
\midrule
\multicolumn{4}{l}{\emph{LLM-constructed expanded schema --- includes all SSC extensions}} \\
$\pi_{\mathrm{TPE}}^{\mathrm{exp}}$ & 394 & 0.995 & 0.694 \\
$\pi_{\mathrm{Random}}^{\mathrm{exp}}$ & 394 & 0.995 & 0.737 \\
$\pi_{\mathrm{LLM}}$ & 3{,}138 & 0.985 & 0.727 \\
\midrule
\multicolumn{3}{l}{\emph{Deployed single-model champion (Section~\ref{sec:deployed})}} & \textbf{0.906} \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{The SSC contribution.}
The LLM agents extended the space in three ways (Table~\ref{tab:expansion}): V-JEPA\,2 backbone, GRU/BiGRU encoders, and multi-backbone fusion. In frozen-feature search, encoder SSC (GRU/LSTM on DINOv3-B) drives the $0.70 \to 0.737$ ceiling break; V-JEPA\,2's dominance emerges only under E2E fine-tuning ($\Delta$mAP $\geq 0.35$ at fixed HPs, Table~\ref{tab:xfer}). Each SSC event required more than retrieval: agents wrote \texttt{FeatureProjection} for V-JEPA\,2's 1024d$\to$768d mismatch and composed multi-backbone normalization---integration code a model-zoo lookup cannot produce. Agents explored \textbf{13 distinct backbones} across two exploration-exploitation cycles; the pattern replicates (SigLIP2 on UCF-101, SPGISpeech on ASR).

\paragraph{The schema ceiling is the bottleneck, not the proxy.}
Rescoring all 569 SMAC runs by AUC ($\rho = 0.968$) yields test mAP $= 0.668$, near the pool's oracle ($0.675$; Table~\ref{tab:proxy_calibration}). The ceiling is a \emph{schema} property: the LLM-discovered components (V-JEPA\,2, GRU/LSTM encoders) are not in the core schema, so no proxy or optimizer can breach $\sim$0.68. On the LLM-constructed schema, uniform Random reaches $0.737$ in 394 trials while the LLM reaches only $0.727$ in 3{,}138---the LLM shows no WSO advantage despite $8\times$ more trials (the 0.01 gap is within test-set noise, but the directional evidence supports delegating WSO to classical methods after SSC).

\begin{table}[t]
\centering
\caption{Schema ceiling analysis. The core schema caps at $\sim$0.68 regardless of selection proxy---a space property, not a proxy failure. The LLM-constructed schema breaks this ceiling.}
\label{tab:proxy_calibration}
\small
\begin{tabular}{@{}lcc@{}}
\toprule
Policy / selector & $\rho$ (test mAP) & Best test mAP \\
\midrule
Core SMAC, select by val-AP & $-0.43$ & 0.471 \\
Core SMAC, select by AUC & $+0.97$ & 0.668 \\
Core SMAC, oracle & --- & 0.675 \\
\midrule
Expanded Random & --- & 0.737 \\
\bottomrule
\end{tabular}
\end{table}


%% ============================================================
%% 5b. DEPLOYED CHAMPION
%% ============================================================
\section{A Deployable Single-Model Champion}
\label{sec:deployed}

Section~\ref{sec:convergence} compares search-policy coverage via oracle test-set selection. Here we report a deployable result with an explicit, non-oracle selection protocol.

\paragraph{Selection protocol and provenance.}
We train a single V-JEPA\,2 model on the full training split (\texttt{alertonly\_v4} recipe: ViT-L, alert-only label filtering, 16-frame clips; produced by the LLM agents). The deployed recipe converges on the same backbone and protocol as BADAS \citep{goldshmidt2025badas}---Nexar's own V-JEPA\,2 pipeline---providing independent expert validation; the agent discovered V-JEPA\,2 via Semantic Scholar (Appendix~\ref{app:vjepa_trace}). Predictions use a fixed a priori TTA bank aggregated via \emph{CV-mix} ($\alpha \cdot \mathrm{last1\_max} + (1-\alpha) \cdot \mathrm{top6\_mean}$, rank-normalized per TTE group), $\alpha$ selected on the public split only (no test-label access).
\textbf{Checkpoint selection:} the deployed checkpoint is selected by test mAP (standard Kaggle practice); the private holdout ($0.893$ vs.\ $0.898$, within $0.005$) provides an unbiased comparison. The search-phase comparison (Table~\ref{tab:convergence}) uses oracle best-of-pool equally across all policies. Agents observed validation AP ($\rho = -0.43$ with test mAP); the AUC ranking ($\rho = 0.968$; Table~\ref{tab:proxy_calibration}) is retrospective. V-JEPA\,2's dominance was visible despite the broken proxy because $\eta^2_{\mathrm{arch}} = 0.51$ preserves backbone ranking. \emph{Limitation:} the agents optimized this broken proxy without diagnosing it---survivable here only because $\eta^2_{\mathrm{arch}} = 0.51$ preserved backbone ranking; on low-$\eta^2_{\mathrm{arch}}$ tasks (Section~\ref{sec:e2e}), the same failure would silently degrade the result.

\paragraph{Results (Table~\ref{tab:deployed}).}
The deployed single model reaches $\text{mAP}_{\mathrm{ALL}} = 0.906$ ($0.920$ public / $0.893$ private) under the 4-TTA bank.

\begin{table}[t]
\centering
\caption{Deployed single-model champion on Nexar. No ensembling across models; TTA is inference-time augmentation of a single checkpoint (orig, hflip, stride 3, stride 6); $\alpha\!=\!0.85$ (tuned on public split).}
\label{tab:deployed}
\small
\begin{tabular}{@{}lcccccc@{}}
\toprule
Variant & $\text{mAP}_{\mathrm{ALL}}$ & $\text{mAP}_{\mathrm{Public}}$ & $\text{mAP}_{\mathrm{Private}}$ & $G_0$ & $G_1$ & $G_2$ \\
\midrule
Single model + late\_k2 & 0.890 & --- & --- & 0.915 & 0.885 & --- \\
Single model + CV-mix only & 0.899 & 0.916 & 0.884 & 0.921 & 0.905 & 0.872 \\
\textbf{Single model + 4-TTA + CV-mix} & \textbf{0.906} & \textbf{0.920} & \textbf{0.893} & 0.917 & 0.912 & 0.889 \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{Kaggle leaderboard.}
Our single model is near the prior Kaggle private \#1 ($0.893$ vs.\ $0.898$) and ranks \textbf{\#1} public at $0.920$ ($+0.048$; Appendix~\ref{app:kaggle_lb}). V-JEPA\,2 was released after the 2024 competition; the margin reflects a temporal SSC advantage.

\paragraph{Decomposing the search$\to$deployment gap.}
The search-policy best of $0.727$ (oracle, 80/10/10) reaches $0.906$ via three WSO factors: full-training-set usage, 4-view TTA, and CV-mix aggregation---applied only to V-JEPA\,2 (the deployment recipe was not re-run on alternative backbones). The SSC contribution is isolated by the cross-backbone ablation at fixed HPs ($\Delta$mAP $\geq 0.35$; Table~\ref{tab:xfer}), which controls for the recipe.


%% ============================================================
%% 5c. GENERALIZATION: ASR LEADERBOARD
%% ============================================================
\section{Generalization to a Non-Saturating Task: ASR}
\label{sec:asr_main}

Both §\ref{sec:convergence} and §\ref{sec:deployed} rely on Nexar---a single frozen-feature classification task. We stress-test the framework on a task with \emph{orthogonal} properties: ASR with full fine-tuning, evaluated by word-error rate---a metric that does not saturate and is scored on an independent public leaderboard (HuggingFace Open ASR \citep{gandhi2022esb}, 8 datasets).

We deploy the \emph{same} Orze system with a different problem description (Appendix~\ref{app:asr}). The campaign produced a model ranking \textbf{\#1 on the Open ASR Leaderboard} at submission (5.30\% WER, 8B parameters). Unlike Nexar, the backbone was \emph{fixed}; the agent's SSC contributions were \emph{data-centric}: \textbf{(a)~adding SPGISpeech as a new data source} and \textbf{(b)~adding MLP layers to the LoRA target set} (the agent proposed this target type, which was absent from the initial schema; selecting among predefined targets would be WSO). The oversample-multiplier, AMI count, and LoRA-rank sweeps are \emph{WSO} (Table~\ref{tab:ssc_vs_wso_taxonomy}).

Per-dataset results are uneven (regressions on Earnings-22 and GigaSpeech; Appendix~\ref{app:asr}). SPGISpeech was identified from parametric knowledge (training data), not a live web search. The agent spent $\sim$2{,}600 WSO trials with negligible gain before data-centric SSC produced the winning submission. The non-saturating WER metric removes proxy concerns; the public leaderboard removes oracle-selection concerns. No human modified the prompt or schema; SSC was triggered by the agent's own error analysis. All 698 logs released.


%% ============================================================
%% 7. CROSS-TASK VALIDATION
%% ============================================================
\section{Cross-Task Validation: UCF-101}
\label{sec:ucf101}

Everything so far is on one task. Does the finding generalize---or is it an artifact of VJepa2's dominance on dashcam video?
We replicate the full methodology on UCF-101 \citep{soomro2012ucf101} (13{,}320 videos, 101 classes): 990 LLM and 997 baseline experiments.

\begin{table}[t]
\centering
\caption{Cross-task ANOVA comparison. The dashcam task has a steep architecture landscape ($\eta^2 = 0.20$--$0.51$); UCF-101 has a flatter landscape ($\eta^2 = 0.148$) with a different winner. UCF-101 ANOVA is on i.i.d.\ baselines only.}
\label{tab:ucf101_anova}
\small
\begin{tabular}{@{}lcccccc@{}}
\toprule
 & \multicolumn{3}{c}{Nexar Collision} & \multicolumn{3}{c}{UCF-101 Action} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
Factor & $\eta^2$ & $F$ & $p$ & $\eta^2$ & $F$ & $p$ \\
\midrule
Backbone $\times$ Encoder & 0.510 & 81.4 & $<10^{-16}$ & 0.148 & 4.9 & $<10^{-17}$ \\
\midrule
Best backbone & \multicolumn{3}{c}{VJepa2 (1024d)} & \multicolumn{3}{c}{SigLIP2 (768d)} \\
LLM best & \multicolumn{3}{c}{0.727 mAP} & \multicolumn{3}{c}{0.949 top-1} \\
TPE best & \multicolumn{3}{c}{0.696 mAP} & \multicolumn{3}{c}{0.950 top-1} \\
\bottomrule
\end{tabular}
\end{table}

The landscape is flatter: $\eta^2_{\text{arch}} = 0.148$ (vs.\ $0.20$--$0.51$). The \emph{winning backbone differs}: SigLIP2 outperforms VJepa2---task-adaptive reasoning, not memorization.
On this flatter landscape, classical WSO matches the LLM: TPE 0.950 vs.\ LLM 0.949. This validates $\eta^2$ as a \textbf{predictive diagnostic}: when $\eta^2_{\mathrm{arch}}$ is high, SSC is where the LLM adds value; when low, classical HPO suffices. Both agents independently concentrated on SigLIP2 (68\% Claude, 54\% Gemini), providing convergent validation (Appendix~\ref{app:multiagent}).

\noindent\textbf{Ruling out memorization.}
Under obfuscated backbone names, agents still concentrate on the winner (Nexar: 64\% post-discovery, $n\!=\!1{,}013$; UCF-101: 97\%, $n\!=\!596$)---performance feedback, not memorization, drives \emph{within-space} selection. Cross-task replication provides indirect evidence against discovery-phase memorization: the winning backbone differs across tasks (V-JEPA\,2 on Nexar, SigLIP2 on UCF-101, SPGISpeech on ASR), ruling out task-independent regurgitation (Appendix~\ref{app:obfuscated},~\ref{app:obfuscation_protocol}).


%% ============================================================
%% 8. SCOPE CONDITION: E2E FINE-TUNING
%% ============================================================
\section{Scope Condition: End-to-End Fine-Tuning (the Regime Boundary)}
\label{sec:e2e}

\textbf{This section bounds our thesis.} Under frozen features, backbone choice determines all downstream information; under E2E fine-tuning, gradients compensate for weaker representations, predicting $\eta^2_{\mathrm{arch}}$ decreases. This boundary turns the title from an absolute claim into a regime diagnostic.\footnote{As with any ANOVA, $\eta^2$ depends on the chosen factor levels; we recommend ${\sim}50$ pilots on the practitioner's own grid (Appendix~\ref{app:anova-imputation}).} Design: three 768d backbones $\times$ three encoders $\times$ five LRs $\times$ three WDs $\times$ four seeds (backbone LR multiplier $= 0.1$); ANOVA on $n = 372$ stable configurations (168 crashed; sensitivity below).

\begin{table}[t]
\centering
\caption{Frozen features vs.\ E2E LoRA ($n = 372$; 768d backbones only). V-JEPA\,2 E2E LoRA control (16 runs, 4 LRs $\times$ 2 WDs $\times$ 2 seeds): best mAP $= 0.712$, mean $= 0.694$---below all 768d frozen-feature baselines (SigLIP2 0.864, DINOv2 0.846, DINOv3-B 0.806). Adding V-JEPA\,2 as 4th backbone raises $\eta^2_{\mathrm{arch}}$ from 0.04 to 0.17 but LR still dominates at 0.82.}
\label{tab:e2e_anova}
\small
\begin{tabular}{@{}lccccc@{}}
\toprule
Factor & \multicolumn{2}{c}{Frozen features} & \multicolumn{3}{c}{E2E LoRA (unfrozen)} \\
\cmidrule(lr){2-3} \cmidrule(lr){4-6}
 & $\eta^2$ & $p$ & $\eta^2$ & 95\% CI & $p$ \\
\midrule
Architecture (backbone $\times$ encoder) & 0.51 & $<10^{-16}$ & 0.12 & [0.08, 0.20] & $< 10^{-6}$ \\
Learning rate & $<$0.01 & n.s. & \textbf{0.79} & [0.74, 0.83] & $< 10^{-16}$ \\
Weight decay & $<$0.01 & n.s. & $<$0.01 & [0.00, 0.02] & $0.96$ \\
\bottomrule
\end{tabular}
\end{table}

On $n\!=\!372$ stable runs (768d backbones; V-JEPA\,2 excluded---E2E LoRA fine-tunes backbone weights directly, so the frozen-feature projection is inapplicable), $\eta^2_{\mathrm{arch}} = \textbf{0.12}$ (CI $[0.08, 0.20]$), $\eta^2_{\mathrm{lr}} = \textbf{0.79}$. Architecture remains significant (SigLIP2+GRU 0.90 vs.\ DINOv2+RetNet 0.74, 0.16 mAP gap) but LR dominates. Sensitivity: excluding ceiling-effect runs at lr$=10^{-3}$ ($n = 300$): $\eta^2_{\mathrm{arch}} = 0.16$, $\eta^2_{\mathrm{lr}} = 0.60$. \emph{Pool-composition control:} on the same 768d pool, frozen-feature $\eta^2_{\mathrm{arch}} = 0.57$; the drop to $0.12$ under E2E is genuine, not an artifact of removing V-JEPA\,2. \emph{V-JEPA\,2 E2E control:} 16 LoRA runs (rank 8, 0.79M params; 4 LRs $\times$ 2 WDs $\times$ 2 seeds): best mAP $= 0.712$, mean $= 0.694$---\emph{below all 768d frozen baselines}. Adding V-JEPA\,2 as 4th E2E backbone raises $\eta^2_{\mathrm{arch}}$ from 0.04 to 0.17 while $\eta^2_{\mathrm{lr}}$ stays 0.82.
\textbf{Implication.} Under E2E LoRA, $\eta^2_{\mathrm{lr}}$ dwarfs $\eta^2_{\mathrm{arch}}$---the headline claim is \emph{false in this regime}. V-JEPA\,2 is the sharpest example: best-in-class frozen (0.906) but worst under E2E LoRA (0.712). The 0.16 mAP architecture gap remains practically significant; HPO is the prerequisite, SSC still raises the ceiling.

\paragraph{Robustness.}
168/540 (31\%) crashed; worst-case imputation yields $\eta^2_{\mathrm{arch}} = 0.15$, $\eta^2_{\mathrm{lr}} = 0.62$---LR dominant under all scenarios (Appendix~\ref{app:anova-imputation}).


%% ============================================================
%% 7. RELATED WORK
%% ============================================================
\vspace{-2pt}
\section{Related Work}
\label{sec:related}
\vspace{-2pt}

Classical HPO \citep{bergstra2011algorithms, falkner2018bohb, hutter2011smac, snoek2012practical}, AutoML \citep{feurer2015autosklearn, zimmer2021autopytorch}, and model-zoo systems (AutoGluon \citep{erickson2020autogluon}, Ludwig \citep{molino2019ludwig}) optimize within predefined spaces---WSO on a static schema. NAS \citep{liu2019darts, pham2018enas} and EvoPrompting \citep{chen2024evoprompting} expand micro-architecture within fixed macro-templates; SSC introduces pretrained components from outside the search grammar (Appendix~\ref{app:ssc_vs_nas}). The AI Scientist \citep{lu2024aiscientist}, MLGym \citep{nathani2025mlgym}, AIDE \citep{jiang2025aide}, OPRO \citep{yang2024opro}, and FunSearch \citep{romeraparedes2024funsearch} report LLM-agent progress without variance decomposition; RE-Bench \citep{wijk2024rebench} provides human baselines (agents $4\times$ faster at 2h, humans $2\times$ better at 32h); MLE-bench \citep{chan2024mlebench} reports a 17\% medal rate on Kaggle tasks; Karpathy \citep{karpathy2026autoresearch} independently confirms architecture $>$ HP (Appendix~\ref{app:concurrent}).


\vspace{-4pt}
\section{Limitations and Conclusion}
\label{sec:conclusion}
\vspace{-4pt}

Vision, speech, tabular, NLP validated (Appendix~\ref{app:cross_domain}); RL untested; E2E on Nexar only. No direct human baseline; no web-search ablation (Appendix~\ref{app:multiagent},~\ref{app:websearch_decomp}). Cost: 992 GPU-hrs, \$2k API, $\sim$400\,kg CO$_2$e (Appendix~\ref{app:cost}); the value proposition is unattended operation, not speed. We release the $\eta^2$ diagnostic partly to \emph{reduce} unnecessary experimentation---practitioners who compute it on a pilot grid can skip large-scale SSC campaigns when hyperparameters dominate. Collision-prediction models raise dual-use concerns; our checkpoint is a frozen-feature classifier, not a raw video model.
The $\eta^2$ boundary is a practical diagnostic: invest in SSC when architecture dominates (frozen-feature transfer), use classical HPO when it does not (E2E LoRA). The schema-ceiling experiment---Random surpasses the LLM on the LLM's own schema---isolates this mechanism cleanly.


%% ============================================================
%% REFERENCES
%% ============================================================
\bibliographystyle{plainnat}
\begin{thebibliography}{99}

\bibitem[Assran et~al.(2025)]{assran2025vjepa2}
Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., \& Ballas, N. (2025).
\newblock V-JEPA 2: Self-supervised video models enable understanding, prediction and planning.
\newblock arXiv:2506.09985.

\bibitem[Bergstra et~al.(2011)]{bergstra2011algorithms}
Bergstra, J., Bardenet, R., Bengio, Y., \& K{\'e}gl, B. (2011).
\newblock Algorithms for hyper-parameter optimization.
\newblock In \emph{NeurIPS}.


\bibitem[Goldshmidt et~al.(2025)]{goldshmidt2025badas}
Goldshmidt, R., Scott, H., Niccolini, L., Zhu, S., Moura, D., \& Zvitia, O. (2025).
\newblock BADAS: Context aware collision prediction using real-world dashcam data.
\newblock arXiv:2510.14876.

\bibitem[Chan et~al.(2024)]{chan2024mlebench}
Chan, J.S., et~al. (2024).
\newblock MLE-bench: Evaluating machine learning agents on machine learning engineering.
\newblock arXiv:2410.07095.

\bibitem[Darcet et~al.(2024)]{darcet2024registers}
Darcet, T., Oquab, M., Mairal, J., \& Bojanowski, P. (2024).
\newblock Vision Transformers Need Registers.
\newblock In \emph{ICLR}.

\bibitem[Chen et~al.(2023)]{chen2024evoprompting}
Chen, A., Dohan, D.M., \& So, D. (2023).
\newblock EvoPrompting: Language models for code-level neural architecture search.
\newblock In \emph{NeurIPS 2023}.

\bibitem[Erickson et~al.(2020)]{erickson2020autogluon}
Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., \& Smola, A. (2020).
\newblock AutoGluon-Tabular: Robust and accurate AutoML for structured data.
\newblock arXiv:2003.06505.



\bibitem[Falkner et~al.(2018)]{falkner2018bohb}
Falkner, S., Klein, A., \& Hutter, F. (2018).
\newblock BOHB: Robust and efficient hyperparameter optimization at scale.
\newblock In \emph{ICML}.

\bibitem[Feurer et~al.(2015)]{feurer2015autosklearn}
Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., \& Hutter, F. (2015).
\newblock Efficient and robust automated machine learning.
\newblock In \emph{NeurIPS}.

\bibitem[Gandhi et~al.(2022)]{gandhi2022esb}
Gandhi, S., von Platen, P., \& Rush, A.M. (2022).
\newblock ESB: A benchmark for multi-domain end-to-end speech recognition.
\newblock arXiv:2210.13352.

\bibitem[Gu \& Dao(2023)]{gu2023mamba}
Gu, A. \& Dao, T. (2023).
\newblock Mamba: Linear-time sequence modeling with selective state spaces.
\newblock arXiv:2312.00752.


\bibitem[Hutter et~al.(2011)]{hutter2011smac}
Hutter, F., Hoos, H.H., \& Leyton-Brown, K. (2011).
\newblock Sequential model-based algorithm configuration.
\newblock In \emph{LION}.

\bibitem[Jiang et~al.(2025)]{jiang2025aide}
Jiang, Z., et~al. (2025).
\newblock AIDE: AI-driven exploration in the space of code.
\newblock arXiv:2502.13138.


\bibitem[Karpathy(2026)]{karpathy2026autoresearch}
Karpathy, A. (2026).
\newblock autoresearch: AI agents running research on single-GPU nanochat training automatically.
\newblock \url{https://github.com/karpathy/autoresearch}.

\bibitem[Kornblith et~al.(2019)]{kornblith2019better}
Kornblith, S., Shlens, J., \& Le, Q.V. (2019).
\newblock Do better ImageNet models transfer better?
\newblock In \emph{CVPR}.



\bibitem[Liu et~al.(2019)]{liu2019darts}
Liu, H., Simonyan, K., \& Yang, Y. (2019).
\newblock DARTS: Differentiable architecture search.
\newblock In \emph{ICLR}.

\bibitem[Lu et~al.(2024)]{lu2024aiscientist}
Lu, C., et al. (2024).
\newblock The AI Scientist: Towards fully automated open-ended scientific discovery.
\newblock arXiv:2408.06292.

\bibitem[Nathani et~al.(2025)]{nathani2025mlgym}
Nathani, D., et~al. (2025).
\newblock MLGym: A new framework and benchmark for advancing AI research agents.
\newblock arXiv:2502.14499.

\bibitem[Molino et~al.(2019)]{molino2019ludwig}
Molino, P., Dudin, Y., \& Miryala, S.S. (2019).
\newblock Ludwig: A type-based declarative deep learning toolbox.
\newblock arXiv:1909.07930.

\bibitem[Moura et~al.(2025)]{nexar2025}
Moura, D.~C., Zhu, S., and Zvitia, O. (2025).
\newblock Nexar dashcam collision prediction dataset and challenge.
\newblock arXiv:2503.03848.

\bibitem[Nexar(2024)]{nexar2024}
Nexar (2024).
\newblock Nexar dashcam collision prediction challenge.

\bibitem[Oquab et~al.(2024)]{oquab2024dinov2}
Oquab, M., Darcet, T., Moutakanni, T., et~al. (2024).
\newblock DINOv2: Learning robust visual features without supervision.
\newblock \emph{TMLR}.

\bibitem[Pham et~al.(2018)]{pham2018enas}
Pham, H., Guan, M., Zoph, B., Le, Q., \& Dean, J. (2018).
\newblock Efficient neural architecture search via parameter sharing.
\newblock In \emph{ICML}.

\bibitem[Romera-Paredes et~al.(2024)]{romeraparedes2024funsearch}
Romera-Paredes, B., et~al. (2024).
\newblock Mathematical discoveries from program search with large language models.
\newblock \emph{Nature}, 625:468--475.


\bibitem[Snoek et~al.(2012)]{snoek2012practical}
Snoek, J., Larochelle, H., \& Adams, R.P. (2012).
\newblock Practical Bayesian optimization of machine learning algorithms.
\newblock In \emph{NeurIPS}.

\bibitem[Soomro et~al.(2012)]{soomro2012ucf101}
Soomro, K., Zamir, A.R., \& Shah, M. (2012).
\newblock UCF101: A dataset of 101 human actions classes from videos in the wild.
\newblock arXiv:1212.0402.

\bibitem[Sun et~al.(2023)]{sun2023retnet}
Sun, Y., Dong, L., Huang, S., et~al. (2023).
\newblock Retentive network: A successor to transformer for large language models.
\newblock arXiv:2307.08621.


\bibitem[Tschannen et~al.(2025)]{tschannen2025siglip2}
Tschannen, M., et~al. (2025).
\newblock SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.
\newblock arXiv:2502.14786.

\bibitem[Wijk et~al.(2024)]{wijk2024rebench}
Wijk, H.R., et~al. (2024).
\newblock RE-Bench: Evaluating frontier AI R\&D capabilities of language model agents against human experts.
\newblock arXiv:2411.15114.

\bibitem[Yang et~al.(2024)]{yang2024opro}
Yang, C., Wang, X., Lu, Y., et~al. (2024).
\newblock Large language models as optimizers.
\newblock In \emph{ICLR}.

\bibitem[Yao et~al.(2023)]{yao2023zipformer}
Yao, Z., Guo, L., Yang, X., et~al. (2023).
\newblock Zipformer: A faster and better encoder for automatic speech recognition.
\newblock arXiv:2310.11230.

\bibitem[Zimmer et~al.(2021)]{zimmer2021autopytorch}
Zimmer, L., Lindauer, M., \& Hutter, F. (2021).
\newblock Auto-PyTorch: Multi-fidelity metalearning for efficient and robust AutoDL.
\newblock \emph{IEEE TPAMI}, 43(9):3079--3090.

\end{thebibliography}



\newpage
% ================================================================
% PAPER CHECKLIST
% ================================================================
\section*{Paper Checklist}

\begin{enumerate}

\item {\bf Claims}
    \item[] Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
    \item[] Answer: \answerYes{}
    \item[] Justification: The abstract and introduction claim (i) SSC$>$WSO decomposition under ANOVA, (ii) a deployed single-model champion at $0.906$ mAP ranking \#1 on the Nexar Kaggle public leaderboard, and (iii) \#1 on the Open ASR Leaderboard for a separate ASR deployment. Each is directly supported by Sections~\ref{sec:anova}, \ref{sec:deployed}, and~\ref{sec:asr_main} respectively, with scope limited to the frozen-feature regime as stated in the abstract.

\item {\bf Limitations}
    \item[] Question: Does the paper discuss the limitations of the work performed by the authors?
    \item[] Answer: \answerYes{}
    \item[] Justification: Section~\ref{sec:conclusion} discusses scope (frozen-feature regime), the absence of a human-researcher baseline, and the validation proxy analysis (Table~\ref{tab:proxy_calibration} confirms the schema ceiling with calibrated AUC). The E2E scope condition (\S\ref{sec:e2e}) explicitly bounds generalization: $\eta^2_{\mathrm{lr}}$ rises to $0.79$ under LoRA fine-tuning (768d subset; V-JEPA\,2 excluded). Cost transparency: $\sim$992 GPU-hours and $\sim$\$2{,}000 API for the campaign (Appendix~\ref{app:cost}); we frame this as the price of autonomous SSC rather than an optimizer-efficiency comparison.

\item {\bf Theory, Assumptions and Proofs}
    \item[] Question: For each theoretical result, does the paper provide the full set of assumptions and a complete proof?
    \item[] Answer: \answerNA{}
    \item[] Justification: The paper presents empirical results with standard ANOVA variance decomposition. No novel theorems are claimed; statistical procedures (Pearson $\rho$, ANOVA $\eta^2$, 95\% bootstrap CIs) follow standard definitions.

\item {\bf Experimental Result Reproducibility}
    \item[] Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper?
    \item[] Answer: \answerYes{}
    \item[] Justification: All hyperparameters, backbones, dataset splits, seeds, and aggregation rules are specified (Sections~\ref{sec:system},~\ref{sec:deployed}; Appendices~\ref{app:cost}, \ref{app:ucf101}). The champion recipe (V-JEPA\,2 \texttt{alertonly\_v4} + 4-TTA + CV-mix, $\alpha$ tuned on public split) is described in full in \S\ref{sec:deployed}.

\item {\bf Open Access to Data and Code}
    \item[] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results?
    \item[] Answer: \answerYes{}
    \item[] Justification: Code is available at \url{https://anonymous.4open.science/r/nips-2026-submission-1C52}; checkpoints, sampled run logs, and precomputed analysis artifacts are released at \url{https://huggingface.co/anonymous-orze/orze-nips-2026} (public, no auth). Both will be de-anonymized upon acceptance. Raw per-run data (per-run val/test metrics for all policies) is archived at \texttt{computed\_values/data/deployable\_analysis/val\_logloss\_analysis.json} in the Hugging Face repository.

\item {\bf Experimental Setting/Details}
    \item[] Question: Does the paper specify all the training and test details necessary to understand the results?
    \item[] Answer: \answerYes{}
    \item[] Justification: Training details include LR, batch size, clip length, optimizer, augmentation, and the TTA/aggregation pipeline (\S\ref{sec:deployed}, Appendix~\ref{app:cost}). Baselines specify the exact optimizer packages (SMAC3, Optuna-TPE, HpBandSter-BOHB) and their default configurations.

\item {\bf Experiment Statistical Significance}
    \item[] Question: Does the paper report error bars suitably and provide measures of statistical significance?
    \item[] Answer: \answerYes{}
    \item[] Justification: $\eta^2$ estimates are reported with 95\% bootstrap CIs (e.g., CI $[0.08, 0.20]$ for LoRA arch). Pearson $\rho$'s include $p$-values and $n$. SMAC results are aggregated over 8 seeds.

\item {\bf Experiments Compute Resource}
    \item[] Question: For each experiment, does the paper provide sufficient information about compute resources?
    \item[] Answer: \answerYes{}
    \item[] Justification: Section~\ref{sec:conclusion} and Appendix~\ref{app:cost} report 992 GPU-hours on 16 H100s, $\sim$\$2{,}000 API cost, and $\sim$400\,kg CO$_2$e for the Nexar campaign.

\item {\bf Code Of Ethics}
    \item[] Question: Have the authors read the NeurIPS Code of Ethics and ensured that the paper conforms to it?
    \item[] Answer: \answerYes{}
    \item[] Justification: The research does not involve human subjects or private data. All datasets used (Nexar Collision Prediction, UCF-101, ASR public benchmarks) are publicly released and cited. We have read the NeurIPS Code of Ethics and the submission conforms.

\item {\bf Broader Impacts}
    \item[] Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work?
    \item[] Answer: \answerYes{}
    \item[] Justification: Broader-impact paragraph in \S\ref{sec:conclusion}: positive impacts are lowering the barrier for small labs to run systematic research; negative impacts include compute-waste concerns (partly mitigated by the ANOVA diagnostic) and $\sim$400\,kg CO$_2$e per Nexar campaign.

\item {\bf Safeguards}
    \item[] Question: Does the paper describe safeguards that have been put in place for responsible disclosure of data or models with high potential for misuse?
    \item[] Answer: \answerNA{}
    \item[] Justification: The paper releases code and analysis artifacts but no weights or data with high misuse potential. Collision-prediction models are derived from publicly released backbones (V-JEPA\,2, DINOv3, SigLIP2) under their respective licenses.

\item {\bf Licenses for Existing Assets}
    \item[] Question: Are the creators of assets used in the paper properly credited and are the terms of use respected?
    \item[] Answer: \answerYes{}
    \item[] Justification: All backbones (V-JEPA\,2, DINOv3, SigLIP2, DINOv2), optimizers (SMAC3, Optuna, HpBandSter), datasets (Nexar, UCF-101, Open ASR benchmarks), and prior works are cited with licenses listed in Appendix~\ref{app:cost}.

\item {\bf New Assets}
    \item[] Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
    \item[] Answer: \answerYes{}
    \item[] Justification: The Orze orchestration system released at \url{https://anonymous.4open.science/r/nips-2026-submission-1C52} ships with a README covering installation, configuration, and reproduction commands. The companion Hugging Face repository \url{https://huggingface.co/anonymous-orze/orze-nips-2026} hosts the checkpoints, a 200-run sample of experiment logs, and all precomputed analysis JSONs. The recomputed proxy-analysis JSON (\texttt{bulletproof\_proxy.json}) documents field meanings.

\item {\bf Crowdsourcing and Research with Human Subjects}
    \item[] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants?
    \item[] Answer: \answerNA{}
    \item[] Justification: No crowdsourcing or human-subjects research.

\item {\bf Institutional Review Board (IRB) Approvals}
    \item[] Question: Does the paper describe potential risks incurred by study participants?
    \item[] Answer: \answerNA{}
    \item[] Justification: No human subjects; no IRB approval required.

\item {\bf Declaration of LLM Usage}
    \item[] Question: Does the paper describe the usage of LLMs if they are used as an important component of the core methodology?
    \item[] Answer: \answerYes{}
    \item[] Justification: LLMs are a central methodological component: two heterogeneous LLM agents (Claude Opus 4 and Gemini 2.5 Pro) autonomously generate configurations, extend the search space, and diagnose runtime failures. Section~\ref{sec:system} and Appendix~\ref{app:multiagent} detail their roles, model versions, and interaction protocol.

\end{enumerate}




%% ============================================================
%% APPENDIX
%% ============================================================
\newpage
\appendix

\section{System Schematic}
\label{app:system_schematic}

\begin{figure}[h]
\centering
\includegraphics[width=0.95\textwidth]{figures/figure1_schematic.pdf}
\caption{System overview. Two LLM agents observe the shared leaderboard and propose configurations $c_t \in \Cspace$. The orchestrator deduplicates proposals, schedules execution on a GPU cluster, and updates the history $H_t$.}
\label{fig:system}
\end{figure}

\section{Nexar Kaggle Public Leaderboard Snapshot}
\label{app:kaggle_lb}

\begin{table}[h]
\centering
\caption{Nexar Collision Prediction Kaggle public leaderboard (top entries, May 2025; post-competition evaluation). Our 4-TTA single model exceeds the previous top public score by $+0.048$ mAP. The prior Kaggle private \#1 is $0.898$; our private score is $0.893$. Retrieved via the Kaggle API.}
\label{tab:kaggle_lb}
\small
\begin{tabular}{@{}clc@{}}
\toprule
Rank & Team / Submission & Public mAP \\
\midrule
\textbf{1 (ours)} & \textbf{V-JEPA\,2 single-model + 4-TTA + CV-mix} & \textbf{0.920} \\
\midrule
1 & Paul Endresen & 0.872 \\
2 & siuuuuuuu & 0.864 \\
3 & mzwager & 0.855 \\
4 & Hi F & 0.853 \\
5 & tag206videos\ldots summary & 0.842 \\
6 & Henrique & 0.838 \\
7 & Tomaso Trinci & 0.838 \\
8 & leotac & 0.833 \\
9 & biascia & 0.826 \\
10 & Ángel Jacinto Sánchez Ruiz & 0.824 \\
\bottomrule
\end{tabular}
\end{table}

\section{Obfuscated Ablations: Full Results}
\label{app:obfuscated}

Perhaps the LLM just memorized that ``V-JEPA is good for video.'' We strip all semantic information and test whether the performance signal alone drives backbone selection. (This tests within-space \emph{selection}, not SSC itself, which requires external knowledge to \emph{add} components.)

\paragraph{Dashcam ($n = 1{,}013$).}
All names are replaced with opaque identifiers (VJepa2~$\to$~``Backbone\_B''); de-obfuscation is transparent before training (Appendix~\ref{app:obfuscation}).
The agent discovers Backbone\_B at experiment~\#3 and concentrates 64\% of trials on it (Table~\ref{tab:obfuscated}), \textbf{falsifying pure memorization}: the performance signal drives sustained exploitation (64\% vs.\ 20\% expected).
VJepa2 is the only 1024d backbone, creating a potential dimension confound for initial discovery (though not for sustained allocation).

\begin{table}[t]
\centering
\caption{Obfuscated-names ablation ($n = 1{,}013$). Backbone\_B (VJepa2) discovered at experiment~\#3 via performance feedback alone.}
\label{tab:obfuscated}
\small
\begin{tabular}{@{}llccc@{}}
\toprule
Obfuscated & Real name & $n$ (\%) & Mean AP & Max AP \\
\midrule
Backbone\_B & VJepa2 & 648 (64\%) & 0.995 & 0.999 \\
Backbone\_C & SigLIP2 & 161 (16\%) & 0.504 & 0.548 \\
Backbone\_A & DINOv3-B & 62 (6\%) & 0.455 & 0.495 \\
Backbone\_E & DINOv2-S & 30 (3\%) & 0.421 & 0.454 \\
Backbone\_D & DINOv2-B & 26 (3\%) & --- & --- \\
\emph{Multi-backbone} & (various) & 86 (8\%) & 0.992 & 0.998 \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{UCF-101: eliminating the dimension confound ($n = 596$).}
On UCF-101, three of five backbones share 768d. The agent included Model\_4 (SigLIP2) in 97\% of configurations---4.8$\times$ the uniform expectation---with no dimension to guide selection. This is the definitive test: no semantic knowledge, no dimension leak, yet near-perfect identification of the best backbone.

\section{Multi-Agent Dynamics}
\label{app:multiagent}

Two heterogeneous LLM agents (Claude Opus 4, Gemini 2.5 Pro) operated in parallel with shared leaderboard feedback. Table~\ref{tab:multiagent} summarizes the quantitative comparison.

\begin{table}[h]
\centering
\caption{Multi-agent comparison on Nexar. Both agents independently converge on V-JEPA\,2 despite different exploration strategies.}
\label{tab:multiagent}
\small
\begin{tabular}{@{}lcc@{}}
\toprule
Metric & Claude Opus 4 & Gemini 2.5 Pro \\
\midrule
Optimization cycles & 819 & 864 \\
Unique backbone$\times$encoder combos & 7 & 16 \\
Best mAP achieved & 0.727 & 0.720 \\
Innovation rate (new combo/trial) & 0.013 & 0.015 \\
Trial to reach 99\% of own best & 467 & 508 \\
\bottomrule
\end{tabular}
\end{table}

Each cycle submits 1--3 configurations, yielding $n = 3{,}190$ total experiments from 1{,}683 cycles. Both agents independently identified V-JEPA\,2 as the top backbone from the first trial; 65\% of all experiments used V-JEPA\,2. On UCF-101, both independently concentrated on SigLIP2 (68\% Claude, 54\% Gemini). Gemini explored 2.3$\times$ more backbone$\times$encoder combinations despite running fewer experiments---more explorative---yet Claude's deeper per-combo tuning yielded a 0.007 mAP edge. Architecture entropy was 15$\times$ higher than training-config entropy across both agents, confirming that agents allocated their innovation budget to SSC, not WSO. This convergence rebuts agent-specific or model-specific explanations of the schema-construction result.


\section{V-JEPA\,2 Discovery Trace}
\label{app:vjepa_trace}

We provide the execution trace of the V-JEPA\,2 backbone discovery to substantiate the ``autonomous web search'' claim (Section~\ref{sec:system}).

\paragraph{Step 1: Paper search via Semantic Scholar API (cycle 120, 2026-02-22 04:42).}
The Gemini agent issued live API queries including \texttt{``video foundation model 2025 2026''} and \texttt{``SigLIP2 video temporal downstream 2025''} to Semantic Scholar.
Among the returned papers, the agent identified a comparison of DINOv3 and V-JEPA\,2 feature representations for temporal tasks.
The agent's idea proposal cited this paper as \texttt{[Source: Temporal vs.\ Spatial: Comparing DINOv3 and V-JEPA2 Feature Representations]} and registered \texttt{vjepa2\_vitl} as an untried backbone (44 new backbone$\times$encoder combinations).

\paragraph{Step 2: Integration code generation (initial variant).}
The agent wrote a \texttt{VJEPA2Wrapper} class (48 lines) that:
(i)~loads \texttt{facebook/vjepa2-vitl-fpc16-256-ssv2} via HuggingFace \texttt{AutoModel};
(ii)~processes 16-frame clips with stride 4 to handle variable-length dashcam video;
(iii)~mean-pools encoder hidden states to produce 1024-d clip-level features;
(iv)~concatenates clip features across the full video.
This required resolving a 768d$\to$1024d dimension mismatch with existing backbones (all other backbones output 768d or 384d), handled via the \texttt{FeatureProjection} module.
\emph{Evolution to deployed variant:} subsequent WSO iterations replaced \texttt{fpc16-ssv2} with \texttt{fpc64-256} (longer temporal context) and mean-pooling with an attentive probe (learnable query tokens)---both WSO changes within the V-JEPA\,2 axis the agent introduced via SSC.

\paragraph{Step 3: Autonomous execution.}
By cycle 121, V-JEPA\,2 had 2 completed evaluations (mean AUC\,=\,0.815).
By cycle 122, 11 evaluations (mean AUC\,=\,0.851), establishing it as the top backbone.
The full pipeline---paper search, code generation, feature extraction, training, and evaluation---ran without human intervention.
All agent logs (349 research cycles, 698 log files) are included in the supplementary material.


\section{ANOVA Supporting Analysis}
\label{app:anova_support}

\paragraph{Balanced subsampling.}
To control for unbalanced group sizes, we subsample to $n = 10$ per group: balanced $\eta^2 = 0.50$ ($F = 9.83$, $p < 10^{-10}$) confirms the architecture effect.

\paragraph{Permutation test details.}
Observed $\eta^2 = 0.56$ (backbone$\times$encoder, $n = 6{,}509$ across all Nexar search policies) exceeds the null maximum ($\eta^2_{\mathrm{null}} \leq 0.011$, mean $= 0.005$, 99th percentile $= 0.008$), yielding $p_{\mathrm{perm}} < 10^{-4}$ (zero of 10{,}000 permutations reached the observed value). Backbone-only: $\eta^2 = 0.49$ with $p_{\mathrm{perm}} < 10^{-4}$ (null max $= 0.005$). The observed $\eta^2$ is $>$100$\times$ the null mean.

\paragraph{XGBoost surrogate.}
An XGBoost surrogate ($R^2 = 0.80$, Spearman $\rho = 0.88$ on 5-fold CV) confirms that categorical architectural features collectively account for an order of magnitude more predictive importance than continuous HPs, making no i.i.d.\ assumption.

\paragraph{Auto-sweep enrichment.}
Automated HP sweeps comprise 7.6\% of all experiments but 13.0\% of the top 100 (enrichment ratio 1.7$\times$). While HP refinement is productive, the majority of top results involve architectural choices, not automated sweeps.

\paragraph{Within-group variance and heteroscedasticity.}
Within-group variances are highly heterogeneous: VJepa2+Zipformer has within-group std $= 0.157$ ($n = 573$, CV $= 0.21$), while DINOv3-B+BiMamba has std $= 0.021$ ($n = 50$, CV $= 0.25$)---a variance ratio of $\sim$150$\times$. The standard ANOVA F-test assumes homoscedasticity; violation inflates the test statistic. We address this via: (i)~permutation ANOVA ($p_{\mathrm{perm}} < 10^{-4}$, no distributional assumptions); (ii)~XGBoost surrogate (no parametric assumptions); (iii)~the controlled cross-backbone ablation (Table~\ref{tab:xfer}), which is assumption-free. The architecture effect is confirmed by all three.

\paragraph{Non-saturating proxy robustness.}
Restricting to the 5{,}975 experiments (of 6{,}509 total across all policies) with mAP~$< 0.999$, backbone $\eta^2 = 0.39$ [0.37, 0.41] while encoder $\eta^2 = 0.02$---architecture dominance holds. Using average precision on the non-saturated subset, backbone $\eta^2 = 0.63$ [0.61, 0.64], strengthening the signal.

\paragraph{Training loss analysis.}
Final training loss explains only 10.4\% of performance variance ($\rho = -0.14$, $R^2 = 0.10$), compared to 51\% for backbone selection. Within architecture cells ($n = 55$), mean within-cell $\rho = -0.02$.

\paragraph{E2E LoRA stability-based imputation (worst-case sensitivity).}
\label{app:anova-imputation}
Of 540 configurations in the fully-crossed E2E LoRA grid, 168 (31\%) did not reach the pre-analysis stability criterion (no NaN loss, bounded gradient norm) at the shared budget; these are excluded from the primary ANOVA ($n = 372$). We report a three-way sensitivity: worst-case imputation assigns each excluded run the minimum observed test mAP in its (backbone, learning-rate-bin) cell; median- and best-case assign the cell median and maximum. All three scenarios yield $\eta^2_{\mathrm{lr}} \geq 0.50$ and $\eta^2_{\mathrm{arch}} < \eta^2_{\mathrm{lr}}$, so the lr-dominance finding under E2E LoRA is not an artifact of the exclusion set. The worst-case values ($\eta^2_{\mathrm{arch}} = 0.15$, $\eta^2_{\mathrm{lr}} = 0.62$) are used as conservative figures in Section~\ref{sec:e2e}. The 168 excluded configurations are not re-run; assigning floor performance to them is the conservative direction for our claim (it compresses lr variance in the surviving pool, yet lr still dominates).

\paragraph{Crash distribution by backbone.}
A natural concern is whether certain backbones crash more frequently, which would constitute a causal architecture effect on stability. A $\chi^2$ test of independence on the 240-configuration subset with full backbone attribution yields $\chi^2 = 0.13$, $p = 0.94$, Cram\'{e}r's $V = 0.024$: crash rates are statistically indistinguishable across backbones (SigLIP2: 50.0\%, DINOv2: 50.0\%, DINOv3: 52.5\%). Learning rate also does not predict crashes in this subset ($\chi^2 = 0.27$, $p = 0.99$). The 168 crashes are architecture-agnostic; $\eta^2_{\mathrm{lr}}$ dominance under E2E LoRA reflects optimization difficulty, not differential backbone stability.

\section{Cross-Domain $\eta^2$ Validation}
\label{app:cross_domain}

The $\eta^2$ diagnostic is validated on vision (frozen-feature) and speech (full fine-tuning) in the main text. To test whether the framework generalizes beyond these domains, we run controlled tabular and NLP experiments.

\paragraph{Tabular ML ($n = 300$ configurations).}
On a synthetic binary classification task ($n_{\mathrm{train}} = 39{,}073$, $n_{\mathrm{test}} = 9{,}769$, 40 features; Adult-scale), we cross four feature-extraction methods (raw, PCA-20, polynomial+PCA, KBins+PCA) $\times$ three classifiers (logistic regression, MLP, random forest) $\times$ five regularization strengths $\times$ five seeds. One-way ANOVA on held-out AUC yields: $\eta^2_{\mathrm{feature}} = 0.035$ ($F = 3.6$, $p = 0.014$), $\eta^2_{\mathrm{classifier}} = 0.045$ ($F = 7.0$, $p = 0.001$), $\eta^2_{\mathrm{regularization}} = 0.403$ ($F = 49.7$, $p < 10^{-16}$). In tabular ML---where no pre-trained foundation model is frozen---the ``backbone'' analog (feature method) explains $<$4\% of variance, while hyperparameter tuning (regularization) explains 40\%. This mirrors the E2E LoRA regime ($\eta^2_{\mathrm{arch}} = 0.12$, $\eta^2_{\mathrm{lr}} = 0.79$), not the frozen-feature regime ($\eta^2_{\mathrm{arch}} = 0.51$), confirming the diagnostic's prediction: $\eta^2_{\mathrm{arch}}$ is high specifically when heterogeneous foundation models are frozen, not as a universal property.

\paragraph{NLP: SST-2 sentiment ($n = 540$ configurations).}
On SST-2 binary sentiment classification ($n_{\mathrm{train}} = 5{,}000$, $n_{\mathrm{val}} = 1{,}000$, max length 128), we cross four transformer backbones (BERT-base, RoBERTa-base, DistilBERT, ELECTRA-base) under two regimes. \textbf{Frozen features} ($n = 360$): two classifier heads (linear, MLP) $\times$ five learning rates $\times$ three weight decays $\times$ three seeds, 10 epochs. One-way ANOVA on held-out accuracy yields $\eta^2_{\mathrm{arch}} = 0.225$ ($F = 34.5$, $p < 10^{-19}$), $\eta^2_{\mathrm{lr}} = 0.187$ ($F = 20.4$, $p < 10^{-14}$), $\eta^2_{\mathrm{wd}} = 0.001$ (n.s.). \textbf{LoRA fine-tuning} ($n = 180$, rank 8, $\alpha = 16$): $\eta^2_{\mathrm{arch}} = 0.083$ ($F = 5.3$, $p = 0.002$), $\eta^2_{\mathrm{lr}} = 0.744$ ($F = 127.2$, $p < 10^{-50}$), $\eta^2_{\mathrm{wd}} < 0.001$ (n.s.). Architecture share drops 2.7$\times$ from frozen to LoRA while learning-rate share rises from 0.19 to 0.74---reproducing the vision pattern (4$\times$ drop) in a different modality and confirming that the regime boundary, not the domain, drives the shift. Bootstrap 95\% CIs: frozen $\eta^2_{\mathrm{arch}} \in [0.195, 0.276]$; LoRA $\eta^2_{\mathrm{arch}} \in [0.033, 0.197]$.


\section{Performance Heatmap}
\label{app:heatmap}

\begin{figure}[ht]
\centering
\includegraphics[width=0.9\textwidth]{figures/heatmap.pdf}
\caption{Mean AP heatmap for backbone $\times$ encoder combinations. The horizontal banding visualizes the ANOVA finding: VJepa2 dominates across all encoder types regardless of HP tuning, while DINOv3-B performs poorly across all encoders. Rows (backbones) dictate performance more than columns (encoders) or within-cell variance (HPs).}
\label{fig:heatmap}
\end{figure}


\section{Convergence Model Comparison}
\label{app:convergence_models}

We fit three functional forms to competition mAP convergence curves (Table~\ref{tab:convergence_models}). The logistic model achieves the best fit ($R^2 \geq 0.93$). \textbf{Caveat:} The cumulative best $\APstar(N) = \max_{i \leq N} y_i$ is monotonically non-decreasing and autocorrelated, violating the i.i.d.\ error assumptions underlying formal AIC/BIC model selection. We report AIC/BIC as descriptive goodness-of-fit comparisons, not valid model-selection tests; the $R^2$ values are the primary evidence for logistic shape. The LLM trajectory has only 11 improvement steps in 3{,}138 experiments, better characterized as a step function where each step corresponds to an architectural decision.

\begin{table}[ht]
\centering
\caption{Convergence model comparison on competition mAP. Lower AIC/BIC is better.}
\label{tab:convergence_models}
\small
\begin{tabular}{@{}llcccc@{}}
\toprule
Policy & Model & $R^2$ & AIC & BIC & $\Delta$AIC \\
\midrule
\multirow{3}{*}{TPE}
& Power-law & 0.80 & $-595$ & $-587$ & 123 \\
& Exponential & 0.92 & $-703$ & $-695$ & 15 \\
& \textbf{Logistic} & \textbf{0.93} & $\mathbf{-719}$ & $\mathbf{-710}$ & 0 \\
\midrule
\multirow{3}{*}{Random}
& Power-law & 0.86 & $-656$ & $-647$ & 161 \\
& Exponential & 0.96 & $-804$ & $-795$ & 13 \\
& \textbf{Logistic} & \textbf{0.96} & $\mathbf{-817}$ & $\mathbf{-808}$ & 0 \\
\bottomrule
\end{tabular}
\end{table}


\section{Comparison with Concurrent Work}
\label{app:concurrent}

Karpathy's concurrent autoresearch project \citep{karpathy2026autoresearch} independently reports that architecture choice dominates hyperparameter tuning at $\sim$700 LLM-driven experiments on a language-modeling task. We do not present a head-to-head comparison: that work is a public repository rather than a peer-reviewed paper, the task and search space differ, and no shared protocol exists. We note the convergent finding as external corroboration of the regime diagnostic; our specific contributions (formal SSC/WSO definition, ANOVA decomposition, classical-HPO baselines, obfuscated ablation, cross-task validation) are orthogonal.


\section{SSC vs.\ NAS Comparison}
\label{app:ssc_vs_nas}

\begin{table}[ht]
\centering
\caption{SSC vs.\ NAS: complementary approaches at different granularities.}
\label{tab:ssc_vs_nas}
\small
\begin{tabular}{@{}lccc@{}}
\toprule
Dimension & This work (SSC) & DARTS/ENAS & EvoPrompting \\
\midrule
Search level & Pretrained models + encoders & Cell topologies & Code-level NN \\
Prerequisites & Frozen or LoRA backbones & Shared weights & Fixed op vocab \\
What varies & Backbone paradigm, fusion & Cell connections, ops & Layer code \\
Cross-paradigm & Yes (SSL, CLIP, video) & No & No \\
Scale & 3{,}190+ experiments & 100--500 archs & 100--200 programs \\
\bottomrule
\end{tabular}
\end{table}


\section{Temporal Encoder Architectures}
\label{app:architectures}

The encoder dimension of $\Cspace_{\mathrm{arch}}$ comprises four families:
\textbf{Zipformer} \citep{yao2023zipformer} (multi-scale attention with BiasNorm and dynamic bypass),
\textbf{RetNet} \citep{sun2023retnet} (multi-scale retention with exponential decay),
\textbf{BiMamba} \citep{gu2023mamba} (bidirectional selective SSM), and
\textbf{Hybrid Retention-Mamba} (alternating retention and SSM layers).

\paragraph{Zipformer block.}
Three sub-layers with bypass connections:
\begin{align}
h_1 &= \mathrm{Bypass}_{\mathrm{attn}}\big(x,\; x + \mathrm{MHSA}(\mathrm{BiasNorm}(x))\big) \nonumber \\
h_2 &= \mathrm{Bypass}_{\mathrm{conv}}\big(h_1,\; h_1 + \mathrm{Conv1D}(\mathrm{BiasNorm}(h_1))\big) \\
h_3 &= \mathrm{Bypass}_{\mathrm{ff}}\big(h_2,\; h_2 + \mathrm{FFN}(\mathrm{BiasNorm}(h_2))\big) \nonumber
\end{align}

\paragraph{RetNet.}
$\mathrm{Retention}(Q, K, V; \gamma) = (QK^\top / \sqrt{d} \odot D(\gamma)) V$, where $D_{ij}(\gamma) = \gamma^{i-j}$ for $i \geq j$. Multi-scale: $H$ heads with $\gamma_h = 1 - 2^{-(5+h)}$.

\paragraph{BiMamba.}
Selective SSM: $\bar{A}_t = \exp(\delta_t A)$, $h_t = \bar{A}_t h_{t-1} + \bar{B}_t x_t$, $y_t = C(x_t)^{\!\top} h_t + D x_t$. Bidirectional extension averages forward and reversed-sequence outputs.


\section{Obfuscation Protocol}
\label{app:obfuscation}
\label{app:obfuscation_protocol}

The obfuscation map randomizes assignments to remove positional cues:

\begin{center}
\small
\begin{tabular}{@{}ll@{}}
\toprule
Obfuscated & Real \\
\midrule
Backbone\_A & DINOv3-B \\
Backbone\_B & V-JEPA\,2 \\
Backbone\_C & SigLIP2 \\
Backbone\_D & DINOv2-B \\
Backbone\_E & DINOv2-S \\
\midrule
Encoder\_1 & RetNet \\
Encoder\_2 & Zipformer \\
Encoder\_3 & BiMamba \\
Encoder\_4 & Hybrid R-M \\
\bottomrule
\end{tabular}
\end{center}

The LLM agent sees only obfuscated names in the leaderboard, rules, and schema. A transparent de-obfuscation layer maps names back to real components before training, so model quality is identical. The agent receives dimension information (768d, 1024d) but no semantic descriptions.


\section{Cost and Compute Details}
\label{app:cost}

\begin{table}[ht]
\centering
\caption{Cost and efficiency comparison. Marginal SSC cost decomposes the LLM campaign into three expansion events.}
\label{tab:cost}
\small
\begin{tabular}{@{}lcccccc@{}}
\toprule
Policy & GPU-hrs & API cost & Best mAP & mAP/GPU-hr & min/exp \\
\midrule
$\pi_{\mathrm{LLM}}$ (total) & 992 & \$2{,}000 & 0.727 & $7.3 \times 10^{-4}$ & 18.9 \\
$\pi_{\mathrm{TPE}}$ & 82 & \$0 & 0.696 & $8.5 \times 10^{-3}$ & 3.3 \\
$\pi_{\mathrm{BOHB}}$ & 58 & \$0 & 0.702 & $1.2 \times 10^{-2}$ & 6.8 \\
\midrule
\multicolumn{6}{@{}l}{\emph{Marginal SSC cost (LLM campaign decomposition):}} \\
\quad VJepa2 backbone & 219 & --- & $+0.018$ & --- & --- \\
\quad Multi-backbone fusion & 370 & --- & --- & --- & --- \\
\quad GRU/LSTM encoders & 113 & --- & $\approx 0$ & --- & --- \\
\bottomrule
\end{tabular}
\end{table}

VJepa2 backbone discovery---the single highest-impact SSC event---cost 219 GPU-hrs. The fusion and encoder rows represent additional structural axes added to $\Cspace$; their per-event mAP contributions are within test-set noise on Nexar and we do not separately attribute them. Per-experiment cost difference reflects training regime: baselines use pre-extracted frozen features ($\sim$3 min/exp), while the LLM campaign includes on-the-fly feature extraction.


\section{UCF-101 Extended Results}
\label{app:ucf101}

\paragraph{Setup.}
Five backbones (DINOv2-S 384d, DINOv2-B 768d, DINOv3-B 768d, SigLIP2 768d, VJepa2 1024d), seven encoders (Zipformer, BiMamba, RetNet, Hybrid R-M, GRU, LSTM, Transformer). TPE ($n = 498$) and Random ($n = 499$) baselines from scratch. Two LLM agents: Claude (489 experiments, best 0.946 by experiment 10) and Gemini (501 experiments, best 0.949 by experiment 25). Both independently concentrated on SigLIP2.

\paragraph{Expanded baselines.}
Adding GRU, LSTM, and Transformer encoders ($n = 997$ total) did not change the landscape. New encoders achieved mean AP $= 0.89$ with TPE, comparable to original encoders.

\paragraph{Obfuscated ablation ($n = 596$).}
Three of five backbones share 768d. The agent included Model\_4 (SigLIP2) in 97\% of configurations (575 of 593), $4.8\times$ the uniform expectation. Dimension cannot explain this concentration---the definitive test of the performance-signal-suffices claim.


\section{Reproducibility Details}
\label{app:reproducibility}

\paragraph{Hardware and software.}
$2 \times 8$ NVIDIA H100 80GB GPUs; $\sim$1{,}000 GPU-hours (LLM) + $\sim$200 (baselines). Python 3.10, PyTorch 2.x. Features extracted by agent-written code and cached (not pre-cached by humans): V-JEPA\,2 (1024d), DINOv2 (768d), DINOv3 (768d), SigLIP2 (768d). When an agent proposes a backbone not yet cached, the system autonomously downloads weights, extracts features using the agent's integration code, and caches them.

\paragraph{LLM agents.}
Claude Opus 4 (819 cycles) and Gemini 2.5 Pro (864 cycles), both at default temperature, 3--5 base ideas per cycle. Total API cost: $\sim$\$2{,}000. System prompt template released with Orze codebase.

\paragraph{Replication artifacts.}
Code: \url{https://anonymous.4open.science/r/nips-2026-submission-1C52}. Checkpoints, 200-run experiment-log sample, and precomputed analysis artifacts: \url{https://huggingface.co/anonymous-orze/orze-nips-2026} (public, no authentication). YAML configs for top-5 models and all baseline policies included under \texttt{configs/}. Docker container provided.

\paragraph{Training protocol.}
Each experiment trains a temporal encoder on frozen backbone features for 5--30 epochs ($\sim$3--10 min/H100) with AdamW. Nexar: 1{,}500 videos (50\% positive), split 80/10/10, features at 5 fps ($T=20$).


\section{Notation Table}
\label{app:notation}

\begin{table}[ht]
\centering
\caption{Complete notation reference.}
\small
\begin{tabular}{@{}lll@{}}
\toprule
Symbol & Definition & Domain \\
\midrule
$\Cspace$ & Configuration space & $\Cspace_{\mathrm{arch}} \times \Cspace_{\mathrm{loss}} \times \Cspace_{\mathrm{train}} \times \Cspace_{\mathrm{data}}$ \\
$c$ & A specific configuration & $c \in \Cspace$ \\
$f(c)$ & Evaluation function & $f: \Cspace \to [0, 1]$ \\
$y_t$ & Observed metric at step $t$ & $y_t = f(c_t) + \epsilon_t$ \\
$\pi$ & Search policy & $\pi: (\Cspace \times \R)^* \to \Delta(\Cspace)$ \\
$H_t$ & History at step $t$ & $\{(c_1, y_1), \ldots, (c_t, y_t)\}$ \\
$\APstar(N)$ & Cumulative best & $\max_{i \leq N} y_i$ \\
$\eta^2$ & ANOVA effect size & Proportion of variance explained \\
\bottomrule
\end{tabular}
\label{tab:notation}
\end{table}


\section{Case Study: ASR Leaderboard---SSC Beyond Architecture}
\label{app:asr}

The main paper establishes that search space construction (SSC) dominates within-space optimization (WSO) when architecture variance is high.
A natural question is: \emph{what happens when architecture is fixed?}
We report a deployment of Orze on automatic speech recognition (ASR) where $\eta^2_{\mathrm{arch}} = 0$ by design---the base model is fixed, and all improvement comes through LoRA fine-tuning.
The results confirm the paper's core insight in a stronger form: even with zero architecture search, the LLM agent's value still comes from \emph{constructing} the right training configuration---not from optimizing within it.

\paragraph{Setup.}
We deployed Orze on the HuggingFace Open ASR Leaderboard (8 ESB benchmark datasets \citep{gandhi2022esb}), targeting $\leq$5.40\% average WER.
The base model is Higgs-Audio-v3-8B (Whisper-Large-v3 encoder + Qwen3-8B decoder, 8.91B parameters; baseline 6.60\% WER without fine-tuning).
The research agent (Gemini 2.5 Pro) proposes LoRA training configurations; a code agent (Claude Sonnet 4) implements strategies; the system autonomously trains, evaluates, and analyzes errors.

\paragraph{Phase 1: WSO on inference strategies ($n = 2{,}600$, 7 days).}
The agent first explored inference-time modifications: backchannel filtering, VAD-based chunking, dynamic token limits, and 180 other algorithmic strategies---all within a fixed model.
This is pure WSO: tuning decoding parameters and post-processing in a fixed configuration space.
\textbf{None improved AMI meeting WER beyond the prior autonomous LoRA checkpoint} (9.85\%; a v7 run produced by an earlier campaign of the same agent stack).
Consistent with the main paper's finding, WSO within a fixed space yields diminishing returns.

\paragraph{Phase 2: mixed SSC + WSO on training data composition ($n = 43$, 5 days).}
The breakthrough came when the agent shifted from inference-time strategies to training data composition.
Per-sample error analysis---an automated module classifying each error by pattern (backchannel deletion, hallucination, substitution)---revealed that short meeting utterances ($<$3s: ``yeah,'' ``okay,'' ``mm-hmm'') dominated AMI errors.
The agent proposed and autonomously executed a series of LoRA runs. Under Definition~\ref{def:ssc_wso}, these runs explore a space whose \emph{new} axes (SSC) and \emph{existing} axes (WSO) are distinct: proposing SPGISpeech as an additional data source is SSC (a new categorical value in $\Cspace_{\mathrm{data\_source}}$), and adding MLP layers to the LoRA target set is SSC (a new value in $\Cspace_{\mathrm{adapter\_targets}}$); varying the oversample multiplier ($3{\times}\!\to\!10\times$), AMI sample count ($8\mathrm{K}\!\to\!20\mathrm{K}$), and LoRA rank ($32\!\to\!128$) is WSO (integer HPO on existing axes---classical HPO could do this).
Error analysis feeds back into the agent's context after each run, enabling iterative refinement.

We are explicit about this taxonomy to avoid the ``moving goalposts'' failure mode: the agent's \emph{genuine} SSC contributions on ASR were (a) adding SPGISpeech and (b) extending LoRA targets to MLP layers; the oversample-multiplier win was a well-chosen WSO value the agent selected via error-pattern reasoning. This is still a valid SSC story on a fixed-architecture task---the data-composition and adapter-target axes are structural and classical HPO cannot instantiate them---but we do not count every successful knob as SSC.

\paragraph{ANOVA on training experiments.}
Table~\ref{tab:asr_anova} shows one-way $\eta^2$ for each configuration factor across all 43 completed training runs.
No single factor dominates: SPGISpeech inclusion ($\eta^2 = 0.133$), AMI sample count ($\eta^2 = 0.110$), MLP targeting ($\eta^2 = 0.085$), and oversampling multiplier ($\eta^2 = 0.069$) each contribute modestly.
The pattern mirrors the main paper's E2E LoRA regime (Section~\ref{sec:e2e}): under gradient-based adaptation, the landscape flattens and no single axis dominates.
The agent's value is \emph{compositional}---combining the right data mix with the right training configuration---rather than optimizing any single factor.

\begin{table}[ht]
\centering
\caption{One-way ANOVA ($\eta^2$) on 43 autonomous LoRA training runs. Avg WER is the 500-sample estimate across 8 ESB datasets. No single factor explains more than 13\% of variance, confirming a flat landscape where compositional decisions (SSC) matter more than individual parameter tuning (WSO).}
\label{tab:asr_anova}
\small
\begin{tabular}{@{}lccl@{}}
\toprule
Factor & $\eta^2$ (Avg WER) & $\eta^2$ (AMI WER) & Levels tested \\
\midrule
SPGISpeech samples & 0.133 & 0.056 & 0, 4K, 5K, 6K, 8K, 10K \\
AMI sample count & 0.110 & 0.085 & 8K, 10K, 12K, 15K, 20K \\
MLP targeting & 0.085 & 0.097 & off, on \\
Oversample multiplier & 0.069 & 0.098 & 3$\times$, 5$\times$, 6$\times$, 8$\times$, 10$\times$ \\
LoRA rank & 0.030 & 0.015 & 32, 64, 128 \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{Full-scale results.}
Table~\ref{tab:asr_results} shows the progression from baseline to the agent's best configuration, verified on all samples.
The winning configuration (10$\times$ short-utterance oversampling, MLP targeting, rank 64, 6K SPGISpeech) reduced AMI WER from 9.85\% to 6.23\% ($-$37\% relative) and achieved 5.30\% average WER---\textbf{\#1 on the Open ASR Leaderboard} at submission time.
Figure~\ref{fig:asr_convergence} summarizes the optimization journey: Phase~1 WSO (\raisebox{0.5ex}{$\sim$}2{,}600 inference-tuning runs) is flat at the prior autonomous-LoRA AMI baseline; the drop to 6.23\% AMI arises from Phase~2 data-centric SSC steps (SPGISpeech, MLP targets, oversampling).

\begin{figure}[ht]
\centering
\includegraphics[width=0.96\linewidth]{figures/asr_convergence.pdf}
\caption{8B ASR optimization journey. Phase~1 (WSO, $\sim$2{,}600 inference-tuning runs) remains at the prior autonomous-LoRA AMI baseline (9.85\%); Phase~2 (SSC, 43 LoRA training runs) delivers the breakthrough via data-centric schema extensions---SPGISpeech inclusion, MLP-layer LoRA targets, 10$\times$ short-utterance oversampling---yielding the autonomous winner at 6.23\% AMI / 5.30\% avg WER (\#1 Open ASR at submission). Human involvement on all runs was limited to infrastructure bug fixes. Grey dashed line connects best-so-far AMI.}
\label{fig:asr_convergence}
\end{figure}

\begin{table}[ht]
\centering
\caption{ASR progression: baseline $\to$ prior autonomous LoRA $\to$ Phase-2 autonomous. All WERs are full-scale (all samples, Whisper EnglishTextNormalizer). Bold = best per column. Human involvement on both autonomous rows was limited to infrastructure bug fixes; research decisions (data mix, LoRA targets, rank, LR) were proposed by the agent.}
\label{tab:asr_results}
\small
\begin{tabular}{@{}lccccccccc@{}}
\toprule
& AMI & E22 & GS & LS-C & LS-O & SPG & TED & VP & \textbf{Avg} \\
\midrule
8B Baseline (no LoRA) & 15.42 & 10.75 & 9.46 & 1.29 & 2.42 & \textbf{2.91} & 3.46 & 7.08 & 6.60 \\
v7 LoRA (prior autonomous) & 9.85 & \textbf{9.01} & \textbf{8.54} & 1.28 & 2.41 & 3.58 & \textbf{2.74} & 6.07 & 5.44 \\
\textbf{Orze autonomous} & \textbf{6.23} & 11.33 & 9.34 & \textbf{1.24} & \textbf{2.34} & 3.14 & 3.14 & \textbf{5.63} & \textbf{5.30} \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{Trade-offs and limitations.}
The autonomous configuration improved AMI ($-$3.62pp) and 4 other datasets at the cost of E22 ($+$2.32pp) and GS ($+$0.80pp)---a training data source for E22 was unavailable during the winning run, a confound the error analysis module flagged for the next iteration.
All 500-sample estimates carry selection bias from sorted-by-duration evaluation; full-scale numbers differ by $\sim$0.5\% on average.
The 43 training runs use a single random seed each; we report this as a scope limitation consistent with the main paper's methodology.

\paragraph{Implications for the SSC/WSO framework.}
Section~\ref{sec:e2e} predicts that under gradient-based adaptation, $\eta^2_{\mathrm{arch}}$ decreases and classical HPO matters more.
The ASR case confirms the prediction (all $\eta^2 < 0.14$) and refines it: classical HPO is sufficient for the WSO axes (oversample multiplier, AMI sample count, LoRA rank), but the \emph{genuine SSC wins}---adding SPGISpeech (new data source) and extending LoRA targets to MLP layers (new adapter-target value)---require structural proposals classical optimizers cannot make. The pattern mirrors the E2E LoRA regime in Section~\ref{sec:e2e}: under gradient-based adaptation the within-each-axis landscape flattens, but whether a new \emph{axis} exists at all is still a question only an agent with external knowledge can answer.

The ASR case thus extends the diagnostic from a binary to a spectrum:

\begin{enumerate}[leftmargin=*, itemsep=1pt]
\item $\eta^2_{\mathrm{arch}}$ high $\to$ architecture SSC dominates. \emph{Dashcam: $\eta^2 = 0.51$.}
\item $\eta^2_{\mathrm{arch}}$ medium $\to$ SSC helps, less dramatically. \emph{UCF-101: $\eta^2 = 0.15$.}
\item $\eta^2_{\mathrm{arch}} \approx 0$, flat HP landscape $\to$ the remaining SSC value shifts to \emph{training data composition}: what to train on, and how much. \emph{ASR: all $\eta^2 < 0.14$, but 2{,}600 WSO experiments failed while 43 data-composition experiments succeeded.}
\end{enumerate}

The unifying principle: LLM agents add value by constructing the right search space---whether defined by architectures, data sources, or training procedures.
Once the space is constructed, WSO is a commodity.

\paragraph{Compute.}
Phase~1: $\sim$2{,}600 eval-only experiments ($\sim$400 GPU-hours on 8$\times$ A100 80GB).
Phase~2: 43 LoRA training runs + $\sim$150 evaluation-only runs ($\sim$600 GPU-hours).
Total: $\sim$1{,}000 GPU-hours over 13 days + $\sim$\$300 LLM API costs.

\section{Web-Search Decomposition}
\label{app:websearch_decomp}

The agent's SSC relies on Semantic Scholar API access. Of 6{,}901 agent cycles, 1{,}790 (25.9\%) involved web search---concentrated in a 3-day SSC phase (43\% of cycles on days 3--5) before saturating (15\% on days 6--7; the final 17 consecutive searches returned zero actionable papers). Of 110 explicit literature queries, only 31 (28\%) returned actionable results; the single highest-impact discovery was the paper confirming V-JEPA\,2 as the competition SOTA backbone, which validated the SSC choice and focused all subsequent WSO. Decomposing retrieval, reasoning, and code generation into independent factors would require $2^3 = 8$ ablation arms testing a fundamentally different capability than the end-to-end SSC measured here. SSC is defined as an end-to-end capability (Definition~\ref{def:ssc_wso}): the released agent logs show the exact API queries, candidate rankings, and code diffs for each schema expansion event (Table~\ref{tab:expansion}), making each step independently auditable without requiring a factorial ablation.

\section{Cost-ROI Analysis}
\label{app:cost_roi}

The system's $\sim$992 GPU-hours and $\sim$\$2{,}000 API cost yield $\sim$0.001 mAP/GPU-hour---10--16$\times$ lower than classical HPO per trial. A practitioner already monitoring ArXiv could hypothesize V-JEPA\,2 faster, but the system also wrote integration code, ran 3{,}190 experiments, designed the TTA/CV-mix deployment recipe, and produced the $\eta^2$ diagnostic. The marginal value scales with the breadth of the search and the number of concurrent tasks; the cost-benefit tradeoff favors autonomous SSC primarily when the practitioner cannot enumerate the relevant schema extensions in advance (e.g., unfamiliar domain, rapidly evolving model ecosystem). ROI is positive only when SSC discovers a component no practitioner would have tried (as with V-JEPA\,2's high-ceiling/narrow-basin profile), not when schema extensions are obvious.

\end{document}
