Harness Hub
All papers

A Taxonomy and Taxonomy-Guided Benchmark Reduction for Spreadsheet Tasks

Bogdan Vasilescu, Emerson Murphy-Hill, Anikesh Parashar, Aayush Kumar, Gustavo Soares, Mar Ginés Marín, Sumit Gulwani

Verdict: Incremental

The framing (CI logs as training data for task selection) and the taxonomy-based coverage audit are practical ideas. The selector is a simple greedy Ridge tie-break, and its evidence is weak: honest R^2 is near zero or negative, epsilon was picked on outer-fold results, the data are small, correlated, single-source and confidential, and there are no external baselines or measured cost savings.

  • Novelty

    2 out of 5
  • Rigor

    2 out of 5
  • Reproducibility

    1 out of 5
  • Relevance to harnesses

    3 out of 5

TL;DR

A spreadsheet-task taxonomy plus greedy Ridge-based task selection that breaks near-ties in inner-CV R^2 by taxonomic diversity. On confidential Microsoft logs (n=67 configurations), coverage rises at little predictive cost. Honest out-of-fold R^2 is low, often negative under walk-forward validation, and the reduced benchmark does not transfer across time.

Adversarial review by Harness Hub · Nov 18, 2026 · read full text (truncated) · Suggest a correction

Original abstract (authors’ text, select to expand)

Spreadsheets let millions of end users without programming training do computational work, and teams are increasingly building AI agents to help them. Evaluating each candidate agent configuration across many tasks is costly. Benchmark reduction can lower this cost by selecting a small subset of tasks whose outcomes predict full-benchmark scores, but selecting tasks only for predictive accuracy can leave whole categories of spreadsheet work untested. We contribute a taxonomy of spreadsheet tasks, grounded in a review of benchmarks and refined with the Excel Agent product team at Microsoft, and a benchmark-reduction method that favors taxonomic diversity among similarly predictive choices. On benchmark logs from Microsoft, taxonomy-guided selection broadens coverage at little or no cost in predictive accuracy and often improves prediction on configurations excluded from selection. Broader coverage can therefore support, rather than compromise, predictive accuracy. Generalization remains limited across time: subsets selected using little recent history predict poorly for configurations much later in time. Beyond benchmark reduction, the taxonomy provides a reusable framework for auditing gaps in task coverage, as teams at Microsoft have done.

Filed underevaluationagents

Summary

The authors build a taxonomy of spreadsheet tasks using the iterative method of Nickerson et al. It covers table properties (size, vocabulary, header type, structure, data types, cleanliness, language), workbook properties (size, structural complexity, domain) and instruction or task properties (clarity, operations, task type, difficulty, LLM capabilities). It is grounded in SpreadsheetBench and an internal Microsoft benchmark. Subjective dimensions are labeled by GPT-4.1 and spot-checked by hand. Benchmark reduction is framed as feature selection. Tasks are features, configurations are datapoints, and the target is each configuration's mean outcome over the tasks observed for it. Greedy forward selection adds the task that most improves inner 5-fold CV R^2 under Ridge (alpha=0.01). Among candidates within epsilon of the best inner-CV R^2, it picks the one that most increases a mean per-dimension diversity score. The evaluation data are Microsoft benchmark logs filtered to one production-candidate configuration family: - Principal sweep: n=67 configurations (126 execution IDs) and p=869 tasks, with missing cells mean-imputed. - Complete-case matrix: 99 execution IDs by 715 tasks, corresponding to only 46 distinct treatments. There are three outer validation schemes: nested K-fold, walk-forward expanding and walk-forward sliding. Five matched-budget controls are run at k=10. The claim is that moderate epsilon (0.05 to 0.1) raises coverage and often slightly improves honest R^2, acting as a regularizer. The authors concede that absolute honest R^2 is weak, and that walk-forward results are often negative.

Key findings

  • Motivating cost: one full-benchmark run with a new frontier model in Excel Agent took about four hours and cost roughly $1,000 in tokens at retail prices (Section I). No savings were measured; the reduction is in task count only (Section VI-A).
  • Table II, k=10, Nested K-Fold at 1% min completion: honest R^2 goes from -0.157 at epsilon=0 to +0.059 at epsilon=0.1.
  • Table II, Nested K-Fold at 5%: +0.050 to +0.159.
  • Table II, Walk-Forward Sliding at 5%: -1.242 to -0.277, still negative.
  • Table II, Walk-Forward Expanding at 5%: -0.253 to -0.316, worse.
  • Complete-case matrix, k=10 (Section V-I): moving epsilon from 0 to 0.05 raised diversity from 0.649 to 0.815 and improved honest R^2 under all three validation strategies (values not printed in the main text).
  • Across six budgets on the complete-case matrix, epsilon 0 to 0.1 increased diversity in all 18 validation-by-budget comparisons and honest R^2 in 13 of the 18 (Section V-I).
  • On the imputed logs at epsilon=0.1 under Nested K-Fold, epsilon-band random beats taxonomy guidance (0.401 vs. 0.396) (Section V-I).
  • Diversity-only selection maximizes coverage but predicts poorly (Section V-I).
  • Validation ordering is Nested K-Fold > Walk-Forward Expanding > Walk-Forward Sliding; Sliding often turns negative, so subsets do not transfer across time (Section V-H, Fig. 2).
  • SpreadsheetBench: Formulas, Logic & Automation accounts for over 400 tasks, a count the LLM's catch-all tendency may inflate. Nearly all instructions are Well- or Very Well-Specified (Section IV-C, Fig. 1).

The paper at a glance

01Label tasks alongtaxonomy dimensions(programmatic +GPT-4.1)02Buildconfiguration-by-taskpass/fail matrixfrom logs,mean-impute missing03Greedy forwardselection scored byinner 5-fold CV R^2with Ridge04At each step, pickthe candidatewithin epsilon ofthe best R^2 thatmaximizes taxonomy…05Fit Ridge weightson the k selectedtasks to predictfull-benchmarkscore06Evaluate with outernested K-fold /walk-forwardexpanding /walk-forward…
  1. Label tasks along taxonomy dimensions (programmatic + GPT-4.1)
  2. Build configuration-by-task pass/fail matrix from logs, mean-impute missing
  3. Greedy forward selection scored by inner 5-fold CV R^2 with Ridge
  4. At each step, pick the candidate within epsilon of the best R^2 that maximizes taxonomy diversity
  5. Fit Ridge weights on the k selected tasks to predict full-benchmark score
  6. Evaluate with outer nested K-fold / walk-forward expanding / walk-forward sliding
Fig. 1 — How the paper works, step by step (arXiv:2611.21452).
Pure greedy (epsilon=0)0.05 R^2Taxonomy-guided (epsilon=0.1) (this paper)0.159 R^2
Fig. 2 — Honest R^2, Nested K-Fold, k=10, min_completion_rate=5%, higher is better. Source: Table II, arXiv:2611.21452.
Data table
Honest R^2, Nested K-Fold, k=10, min_completion_rate=5%
MethodR^2
Pure greedy (epsilon=0)0.05 R^2
Taxonomy-guided (epsilon=0.1) (this paper)0.159 R^2

Why it matters

Harness teams face the problem this paper addresses. Every change to a prompt, tool schema, guardrail or model needs a benchmark score, and a full run here reportedly cost hours and about $1,000. The transferable idea is to treat historical CI pass/fail logs as training data for choosing a small task subset. A second idea is to label both benchmark tasks and production traffic on shared dimensions to find what the eval suite never exercises. The selector itself is simple: greedy Ridge with a diversity tie-break. Its out-of-sample predictive power is weak. Honest R^2 at k=10 is near zero or negative in most reported cells, and it degrades over time. The data are one company's confidential logs, and no token or runtime savings were measured. Harness builders should take away the coverage-audit practice and the warning that reduced benchmarks must be reselected as configurations drift, not a working method for predicting full scores. The spreadsheet taxonomy itself matters only to teams building spreadsheet agents.

Issues (10)

  • High severity

    The abstract claims coverage comes at 'little or no cost in predictive accuracy' and 'often improves prediction'. Both rest on honest R^2 values that are mostly near zero or negative. The best Table II cell is 0.159. Two cells stay negative after the change (-0.277, -0.316), and one gets worse. With R^2 in this range there is little accuracy to lose, and the reduced benchmark does not usefully predict scores for new configurations. The authors concede this in Section V-F; the abstract does not.

  • High severity

    Epsilon is chosen after the fact by looking at honest R^2 across the sweep. The paper recommends 0.05 to 0.1, notes that 0.1 was best on the principal sweep and 0.05 on the complete-case matrix, and says epsilon should be tuned to noise. Choosing epsilon on outer-fold results reintroduces selection bias that the nested design was meant to remove. The reported gains at the chosen epsilon are therefore optimistic. There is no outer loop over epsilon.

  • High severity

    The data are small and correlated. The principal sweep has n=67 configurations. The complete-case matrix has 99 executions from only 46 treatments. The paper says repeated executions weaken the i.i.d. assumption behind Nested K-Fold. It does not say folds were grouped by treatment, so repeats may sit on both sides of a split and inflate contemporaneous honest R^2. No confidence intervals are given, and randomized controls use three seeds.

  • Medium severity

    Table II shows four 'representative' cells from a 6 (k) x 7 (min_completion_rate) x 3 (strategy) grid. The rest is in appendix figures with no summary count of how often epsilon=0.1 beats epsilon=0 in the principal sweep. Readers cannot tell how typical the displayed gains are.

  • Medium severity

    No external reduction method is run. IRT, coreset methods and matrix completion are excluded on framing grounds, even though the 99 x 715 complete-case matrix is dense. The only predictive baseline is the authors' own epsilon=0 greedy selector.

  • Medium severity

    The stated use case is ranking and comparing configurations. Only R^2 on pass rate is reported, with no rank correlation. The authors themselves note R^2 is unstable when Var(y) is small.

  • Medium severity

    Cost is motivated in dollars but measured only in task count. The logs lack token and wall-clock telemetry, and tasks differ substantially in cost.

  • Medium severity

    In the principal analysis, the target y_c averages over a configuration-specific observed task set, with mean imputation. Differences in y_c partly reflect which tasks were run. On the imputed logs, taxonomy guidance does not beat epsilon-band random at every epsilon (0.401 vs. 0.396).

  • Medium severity

    The logs are confidential and the code is only promised, with no URL. No reported number can currently be replicated externally.

  • Low severity

    Subjective taxonomy labels come from GPT-4.1, with only spot checks and no agreement statistic. There is a known catch-all bias toward Formulas, Logic & Automation, and the coverage figures inherit this noise.

Questions for the authors

  • Across the full k x min_completion_rate x strategy grid, in what fraction of cells does epsilon=0.1 beat epsilon=0 on honest R^2?
  • Are outer folds grouped by treatment (flight)? If not, how does honest R^2 change when they are?
  • What is honest R^2 when epsilon is selected inside the outer loop rather than read off the sweep?
  • How does the method compare with IRT or matrix-completion approaches on the dense 99 x 715 complete-case matrix?
  • Do rank correlations between predicted and true configuration orderings agree with the R^2 results?
  • What are the actual token and wall-clock costs of the selected k=10 subsets?
  • How stable is the selected task set across outer folds and over time?

Related in the directory

Open-source projects

Cited as an agent benchmark that offers no subset-selection or coverage framework, the gap the paper targets.
Cited alongside SWE-bench as an agent benchmark without representative-subset selection.

Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.

Yiyan Ji, Qixun Wang +11

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.

Mengxiao Wang, Mahbub Alam +2

Embodied intelligence aims to let robots perform complex physical tasks with human-like adaptability and reliability, which requires three complementary capabilities: history-based cognitive reasoning, precise and timely physical execution, and the composition of reasoning and execution through environmental feedback. Yet existing benchmarks either evaluate reasoning and execution within the same tasks or assess them separately without their composition, so they reveal little about whether a failure comes from reasoning, execution, or their composition. We introduce Mind2Act World, a diagnostic benchmark of 30 simulated manipulation tasks from five real-world application domains. Each task has three difficulty levels and two scene settings (Clean and Random), yielding 180 configurations. The tasks form three views: Mind tests decisions based on past observations, Act tests precise and timely execution, and Mind2Act combines both in closed-loop tasks where action outcomes guide later decisions. Comparing the three views shows whether a system lacks a capability or fails to compose capabilities it already has. A diagnosis harness further attributes the failed episodes of three representative systems to Mind, Act, or their composition and summarizes each system in a profile card. Our evaluation of multimodal agents, vision-language-action models (VLAs), and world-action models (WAMs) shows that competence in a single view does not carry over to Mind2Act tasks. Even the strongest agent succeeds in 52.7% of Mind and 39.0% of Act episodes but in only 12.0% of Mind2Act episodes. The strongest agent retains task state but struggles with sustained contact, whereas end-to-end policies execute practiced motions but retain little task state. These findings point to composing reasoning and execution through environmental feedback as the main open problem.

Yuzhi Huang, Jie Wu +11

The Harness Hub newsletter

New harness companies, projects, releases and the monthly State of Harness report, in your inbox.