PaperarXiv:2611.21579 · cs.CR · Nov 16, 2026
ScenarioForge: Resource-Grounded Generation and Execution Validation of Cyber Exercises
Jaime C. Acosta, Anantaa Kotal, Aritran Piplai
The engineering is substantial and the artifact release is extensive. However, the headline claims rest on three weak foundations: - an author-built dataset run once, with readiness rather than solvability verified; - a comparison in which only one arm has a validation and repair path; - a lenient definition of build success. Direct relevance to agent harness design is limited.
Novelty
2 out of 5Rigor
2 out of 5Reproducibility
4 out of 5Relevance to harnesses
2 out of 5
TL;DR
ScenarioForge is a cyber-range authoring system. An LLM proposes scenario XML, deterministic code binds every element to a cataloged Docker or CORE resource, and the deployed range is checked live. The headline 100% results come from two places: an author-built dataset run once, and a benchmark in which only the framework arms get a compiler, catalog and repair path. No agent or human ever solved the challenges.
Adversarial review by Harness Hub · Nov 18, 2026 · read full text · Suggest a correction
Original abstract (authors’ text, select to expand)
Cyber-range and capture-the-flag scenario construction remains labor-intensive because authors must coordinate infrastructure, vulnerable services, attack paths, artifacts, and success conditions. LLMs can propose detailed exercises, but plausible descriptions do not guarantee deployable software or verifiable objectives. We present ScenarioForge, which turns high-level requirements into resource-grounded topologies, services, traffic, segmentation, and challenge chains, then instantiates and validates them in the Common Open Research Emulator (CORE). The ScenarioForge Execution Dataset contains 226 materialized scenarios covering 294 Vulhub images and all 85 custom-built challenge-resource generators developed with our AI-assisted scaffolding. All 226 passed construction, preview, sequencing, deployment, strict runtime validation, and live artifact checks; their 867 steps, each an individually verifiable challenge objective, comprise 457 vulnerability-backed and 410 generated-resource-backed instances. In a separate 50-prompt matched benchmark, ScenarioForge+Qwen3.8 and ScenarioForge+GPT5.6 each achieved 50/50 (100%) topology/build success and 160/160 intent fields; direct frontier generation reached 41/50 builds and 147/160 fields only with the full cumulative scaffold: structure, vocabulary, policy, and container-image guidance. The test still omitted ScenarioForge's downstream challenge-resource generation, dependency/flag sequencing, deployment, and live validation. The artifacts support operationally rich, reproducible exercises for training and evaluation.
Filed underscaffoldevaluationruntime
Summary
ScenarioForge treats LLM output as a proposal over a persistent XML representation. Deterministic resolution binds each component to a catalog entry: Vulhub images or one of 85 AI-scaffolded challenge-resource generators. It rejects cyclic or unresolved challenge dependencies and compiles to CORE 9.2. It then deploys and runs runtime validation and live artifact checks. A Solutions Script traverses the challenge chain, but it automatically checks only supported HTTP, SSH and TCP challenge-node strategies. Vulnerability-host and artifact-only steps are left for participants or an exploit harness. The paper reports two studies. EQ1 is the Execution Dataset: 226 scenarios with 867 steps, expanded from 30 author-written family specifications. All 226 passed every phase in a single run per case. The run used one Apple M4 Max host with an 8 GB VMware VM and the authors' customized CORE 9.2 fork. EQ2 is a 50-prompt benchmark. gpt-5.6-sol generated the prompts and their expected fields. The benchmark compares ScenarioForge using Qwen3.8-27B-NVFP4 or gpt-5.6-sol against direct gpt-5.6-sol under five cumulative prompt scaffolds (R1-R5). - Both framework arms reach 50/50 builds and 160/160 intent fields. - Direct generation peaks at 41/50 builds and 147/160 fields. - Buildability means CORE parses and instantiates a nonempty session with nodes left inactive. A separate runtime audit of the framework outputs, with Flow disabled, passed 46/50 (Qwen) and 47/50 (GPT) on first pass. Each reached 50/50 only after replays and resource substitutions.
Key findings
- All 226 Execution Dataset scenarios passed construction, preview, sequencing, deployment, strict runtime validation and live artifact checks. They contain 867 steps: 457 vulnerability-backed and 410 generated-resource-backed (Table II, Sec. V-B).
- Matched benchmark results (Table III): - ScenarioForge+Qwen3.8 and ScenarioForge+GPT5.6 each reached 50/50 (100%) build success and 100% (160/160) intent accuracy. - Direct frontier generation went from 4/49 (8.2%) build and 55.0% (88/160) intent at R1 to 41/50 (82%) and 91.9% (147/160) at R5.
- Cost inside the framework (Table III): - Qwen3.8: 89,680 mean total tokens and 94.4 s median AI time. - GPT5.6: 41,437 tokens and 37.4 s. - Direct rungs: 2,808-5,812 tokens.
- The runtime audit of the 100 framework outputs (Flow disabled) passed 46/50 for Qwen and 47/50 for GPT on first pass. Reaching 50/50 required one unchanged replay of a GPT case after a transient CORE failure, plus two GPT and four Qwen documented resource substitutions (Sec. V-D).
- Total per-scenario time ranged from 73.7 to 727.0 s with pre-cached images (n=194) and from 87.7 to 550.0 s when image pulls were observed (n=32). The execution phase alone ranged from 56.4 to 657.2 s and from 70.8 to 486.8 s (Sec. V-B).
The paper at a glance
- High-level request or partial spec
- LLM fills scenario XML (proposal only)
- Deterministic grounding against Vulhub and generator catalogs
- Dependency graph check and topology compile
- Preflight and persisted preview
- Deploy in CORE 9.2 via gRPC/SSH
- Runtime validation and positive/negative reachability probes
- Solutions Script checks supported steps; evidence bundle exported
Data table
| Method | % |
|---|---|
| ScenarioForge+Qwen3.8 (this paper) | 100% |
| ScenarioForge+GPT5.6 (this paper) | 100% |
| Frontier R1 | 8.2% |
| Frontier R2 | 56% |
| Frontier R3 | 76% |
| Frontier R4 | 76% |
| Frontier R5 | 82% |
Why it matters
For most harness builders this paper is peripheral. It is a cyber-range construction tool, not an agent loop, memory system or general eval method. The transferable idea is the trust boundary. The model only proposes a typed artifact, and deterministic code decides whether each element maps to a real resource. Deployment and observation-based checks then decide success, and failures carry typed locations. Tool-use harnesses that validate structured output against schemas and registries already follow this pattern, and the paper is a worked example of it. The dataset could in principle serve security-agent evaluation: 226 replayable multi-step CTF-style environments. However, the paper never runs an agent or a human against them, and most vulnerability-backed steps are not automatically solved, so their solvability as benchmarks is unverified. The direct-generation comparison mainly confirms that a compiler, catalog and repair path beats unguided one-shot XML. That is expected and offers little new guidance on harness design.
Issues (7)
- High severityWhere in paper: Abstract; Sec. III-F; Sec. III-J; Sec. VII
The claim that every step is an 'individually verifiable challenge objective' outruns what was verified. The Solutions Script automatically checks only supported HTTP, SSH and TCP challenge-node strategies. Vulnerability-host and artifact-only steps only 'preserve flags, hints, solutions, and interfaces for participant or exploit-harness interaction'. Strict validation establishes deployment and artifact readiness: containers running, injects present, generator outputs present. It does not show that the 457 vulnerability-backed steps can be exploited, or that the chains can be completed. No agent or human solved any scenario. The abstract's 'operationally rich ... exercises for training and evaluation' is therefore asserted, not measured.
- High severityWhere in paper: Abstract; Sec. III-C; Sec. IV-A; Sec. V-B
The 226/226 pass rate comes from a dataset the authors built with their own tooling, so it is not evidence of success on new requests. - The 85 generators were contract-tested and repaired before installation. - Ten catalog entries needing online services were excluded. - The 226 cases are iteration and seed expansions of 30 author-written family specs. - Each case was run once. The paper does not report whether any scenarios or generators were dropped or revised before the reported batch.
- High severityWhere in paper: Table III; Sec. V-C
The benchmark arms are not matched in tooling or compute. The framework arms get the schema, intent compiler, catalogs, validation and a repair path. The direct arm receives no catalog lookup, compiler, validation tool or repair loop. The framework arms also spend far more tokens: 89,680 and 41,437 mean total tokens, against 2,808-5,812 for the direct rungs. A fair baseline would give direct generation a CORE-load-and-repair loop at a comparable budget. As run, the gap largely measures the presence of a feedback loop rather than resource grounding.
- Medium severityWhere in paper: Sec. V-C (buildability definition); Sec. V-D (robustness audit); Abstract
The abstract's '100% topology/build success' rests on a weak definition: open_xml(start=False), which parses and instantiates a session while leaving nodes, containers and interfaces inactive. The paper's own runtime audit of the same outputs passed only 46/50 (Qwen) and 47/50 (GPT) on first pass, and reached 50/50 only after replays and documented resource substitutions. The abstract does not report these first-pass numbers.
- Medium severityWhere in paper: Sec. V-C; Table III; Sec. V-E
The benchmark design weakens both the tie between models and the baseline comparison. - gpt-5.6-sol generated the 50 prompts and their expected fields, and the same model is both the direct baseline and one framework arm. - Intent covers only six field types: router count, host count, services, segmentation, vulnerabilities and traffic. - Both framework arms hit the ceiling, so 'Qwen3.8 matched the frontier model' cannot be separated from a ceiling effect. - No repeated sampling or variance is reported.
- Low severityWhere in paper: Table III; Sec. V-C
The R1 build denominator is 49, not 50 (4/49, 8.2%). This appears to conflict with the stated rule that missing or unreadable output stays in the denominator, and the paper does not explain it.
- Low severityWhere in paper: Sec. V-B; Sec. V-D
All execution results come from one host: a 16-core, 48 GB Apple M4 Max running CORE 9.2 in an 8 GB VMware VM, using the authors' customized fork. Some first-pass failures were architecture-incompatible containers on the ARM endpoint. Portability to other hosts or to upstream CORE is not demonstrated.
Questions for the authors
- What happens when the direct gpt-5.6-sol arm gets a CORE load-and-repair loop at a token budget matching the framework arms?
- How many of the 867 steps, especially the 457 vulnerability-backed ones, were actually solved end to end by an exploit harness, agent or human?
- How many scenarios or generators were generated, failed, revised or discarded before the 226-case batch was frozen?
- Why is the R1 denominator 49 rather than 50?
- What is the first-pass end-to-end success rate, with Flow enabled and including runtime and artifact checks, for framework outputs from fresh user prompts rather than curated family specs?
- Do the 50/50 results hold with prompts and expected fields written by humans, or by a model other than gpt-5.6-sol, and with repeated samples?
Related papers
arXiv:2611.26595 · cs.CV · Nov 19, 2026
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Yiyan Ji, Qixun Wang +11
arXiv:2611.26553 · cs.CR · Nov 19, 2026
Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
Mengxiao Wang, Mahbub Alam +2
arXiv:2611.26450 · cs.AI · Nov 19, 2026
Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).
Tanmay Sah