PaperarXiv:2611.14952 · cs.SE · Nov 11, 2026
JoyFrontBench: Evaluating Front-End Coding Agents through Goal-Oriented Acceptance Contracts
Xiaoyi Qiao, Yicheng Wang, Jianan Lin, Xiaoyang Ji, Dongyi Lv, Yidan Huang, Shi Suo, Shiying Fan, Nan Duan, Jiaqi Wang
Review pending. Harness Hub reviews new papers adversarially as they arrive; until then, here is the authors’ abstract.
Original abstract (authors’ text, shown)
Front-end coding agents are expected to produce applications that fulfill user goals, yet high task scores can coexist with failures in essential workflows. Evaluating these agents therefore requires an explicit distinction between partial implementation and application success. We introduce JoyFrontBench, a benchmark that makes this distinction through goal-oriented acceptance contracts. Each contract identifies core tasks and mandatory criteria before generation: a core task passes only when all mandatory criteria are satisfied and a shared completion threshold is met, while application success requires every core task to pass on the same operational artifact within a single run. Derived from practitioner requirements, JoyFrontBench comprises 100 front-end development problems, 764 tasks, and 2,596 atomic criteria, with reviewed annotations linking user goals to core tasks and evidence requirements. The evaluation framework combines evidence-grounded judgments with deterministic record auditing and fixed scoring rules, preserving graded task assessment alongside explicit core-task and application-level acceptance. We evaluate five coding models under two agent harnesses, OpenHands and OpenCode, on the 70-problem test set; the remaining 30 problems form a public set for validation and diagnostic analyses. Under OpenHands, the mean Average Task Score (ATS) across models is 87.69%, while the mean Application Success Rate (ASR) is 51.52%. Further analyses reveal changes in model rankings across harnesses and variation in application success across repeated runs. These findings highlight the limitations of aggregate task scores in characterizing application success and motivate explicit assessment of core-goal fulfillment alongside graded task performance.
Authors’ note
20 pages, 8 figures, 9 tables
Filed underharnessevaluationcoding-agentsagents
Related papers
arXiv:2611.26595 · cs.CV · Nov 19, 2026
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Yiyan Ji, Qixun Wang +11
arXiv:2611.26553 · cs.CR · Nov 19, 2026
Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
Mengxiao Wang, Mahbub Alam +2
arXiv:2611.26444 · cs.RO · Nov 19, 2026
Embodied intelligence aims to let robots perform complex physical tasks with human-like adaptability and reliability, which requires three complementary capabilities: history-based cognitive reasoning, precise and timely physical execution, and the composition of reasoning and execution through environmental feedback. Yet existing benchmarks either evaluate reasoning and execution within the same tasks or assess them separately without their composition, so they reveal little about whether a failure comes from reasoning, execution, or their composition. We introduce Mind2Act World, a diagnostic benchmark of 30 simulated manipulation tasks from five real-world application domains. Each task has three difficulty levels and two scene settings (Clean and Random), yielding 180 configurations. The tasks form three views: Mind tests decisions based on past observations, Act tests precise and timely execution, and Mind2Act combines both in closed-loop tasks where action outcomes guide later decisions. Comparing the three views shows whether a system lacks a capability or fails to compose capabilities it already has. A diagnosis harness further attributes the failed episodes of three representative systems to Mind, Act, or their composition and summarizes each system in a profile card. Our evaluation of multimodal agents, vision-language-action models (VLAs), and world-action models (WAMs) shows that competence in a single view does not carry over to Mind2Act tasks. Even the strongest agent succeeds in 52.7% of Mind and 39.0% of Act episodes but in only 12.0% of Mind2Act episodes. The strongest agent retains task state but struggles with sustained contact, whereas end-to-end policies execute practiced motions but retain little task state. These findings point to composing reasoning and execution through environmental feedback as the main open problem.
Yuzhi Huang, Jie Wu +11