Harness Hub
All papers

BLIP-Live: Balancing Timeliness & Firing Precision in Proactive Streaming Video Agents

Kanchana Ranasinghe, Honglu Zhou, Dominick Reilly, Le Xue, Shusheng Yang, Juan Carlos Niebles, Silvio Savarese, Ran Xu

Verdict: Incremental

Two contributions are worth borrowing outside video: the precision-aware metric, and evidence that a small searched harness lifts frozen VLMs. The headline gains, however, are inflated in three ways: - the baseline chosen for the 44% figure - an unequal harness tuning budget - reliance on the authors' own metric On official external metrics the gains are marginal, and on ProactiveVideoQA's official PAUC the model trails closed models. Relevance to general harness builders is limited.

  • Novelty

    3 out of 5
  • Rigor

    2 out of 5
  • Reproducibility

    2 out of 5
  • Relevance to harnesses

    2 out of 5

TL;DR

Salesforce presents four pieces for proactive streaming video: a human-verified dataset (784,944 responses over 177,789 videos), a precision-aware metric (PAUC-F), an agent-searched inference harness (sliding frame window, interaction memory, structured silence/response output) and SFT on Qwen3.5 and Gemma4. The headline 44% gain is measured on the authors' own metric against a weaker streaming-native baseline. The untrained Qwen backbone in the same harness already covers most of that gap. On the ProactiveVideoQA official metric (PAUC), the trained model trails several closed models.

Adversarial review by Harness Hub · Nov 18, 2026 · read full text (truncated) · Suggest a correction

Original abstract (authors’ text, select to expand)

Streaming video understanding is becoming increasingly important for real-world interactive systems. Such systems must be proactive for natural and useful human interaction: they should respond as soon as sufficient evidence becomes available while avoiding premature, redundant, or incorrect responses. We present BLIP-Live, a framework unifying data, metrics, harness, and training around a shared formulation of proactivity as effective causal response decisions balancing the two competing objectives of responding vs silence. First, we construct BLIP-Live-780K dataset containing human annotations of proactive response content and timing, which defines this desired behavior. We then propose PAUC-F, a metric to effectively measure our desired proactive behavior. PAUC-F combines existing response timeliness metrics with firing precision to reward early useful responses without rewarding excessive firing. With our desired behavior defined and measurable, we finally investigate model-agnostic inference harnesses that draw out proactive streaming capabilities of offline vision language models (VLMs), together with training pipelines that learn to further improve their proactive responsiveness. Across both settings, we find that combinations of sliding visual windows, online interaction memory, and structured response outputs consistently yield strongest proactive performance. Our experiments result in two models, with our BLIP-Live-Qwen model achieving state-of-the-art performance across four proactive streaming benchmarks, with gains up to 44% over baselines and outperforming multiple closed API models. Our code and data will be released publicly.

Filed underharnessmemoryevaluationagentsllm

Summary

The paper treats proactive streaming video QA as a choice made at every timestep: respond or stay silent, using only past frames. It has four parts: 1. **Dataset (BLIP-Live-780K).** A VLM (Gemini) generates query and response/silence timelines at 1 FPS. Humans then correct timing and content, and PhD-level reviewers check each batch. 2. **Metric (PAUC-F).** The harmonic mean of PAUC, used as recall, and a firing-precision term counted over all emitted responses. This penalizes out-of-window and repeated fires, which PAUC ignores and ESTP-F1 only partly catches. 3. **Harness.** An agent-assisted propose-evaluate-promote search over four harness components (system prompt, frame window, memory, output parsing) for frozen offline VLMs. It ran 10 rounds of 10 candidates each on Gemma4-26B-A4B and on Qwen3.5-35B-A3B. Every other offline model got only a 5-candidate prompt/schema calibration on 20 samples. 4. **Training.** Teacher-forced SFT with Silence/Response/Hold state tokens, 32-frame chunks, last-response memory and silence/hold loss masking. The authors claim state of the art on ProactiveVideoQA (PAUC-F), ESTP-Bench (PAUC-F and ESTP-F1), StreamingBench PO and OVO-Bench CRR. On the benchmarks scored with official metrics and published baselines, the margins are small. On ProactiveVideoQA's official PAUC metric (Table A.1), BLIP-Live-Qwen trails most proprietary models.

Key findings

  • Headline: BLIP-Live-Qwen scores 49.08 mean PAUC-F (ω=0.5) on ProactiveVideoQA, versus 34.01 for MMDuet2 (Section 6.1, Table 6). This comparison is the source of the '44%' figure.
  • The untrained Qwen3.5-A3B in the authors' harness already reaches 42.94 in the same column (Table 6). Finetuning on BLIP-Live-780K adds 6.14 points (Table 14a).
  • Harness ablation on Gemma4-A4B, PAUC-F WEB/EGO (Table 12): - full harness: 43.4/36.0 - without sliding window: 33.6/7.0 - without online memory: 36.8/14.1 - without structured output: 42.2/34.6 - VideoChat3 harness: 26.3/2.4
  • Training ablation for BLIP-Live-Gemma, WEB/EGO (Table 13): - full pipeline: 56.5/43.8 - without the Hold state: 32.9/24.4 - without loss masking: 0.0/0.0 (the model collapses to silence)
  • Removing Hold on WEB (Table 15): - fires: 978 rise to 2,840 - precision: 75.1% falls to 25.8% - PAUC: 45.26 vs 45.39, essentially unchanged PAUC alone does not register the over-firing.
  • PAUC-optimized harness v1 vs PAUC-F-optimized v2 on WEB (Table 4): - PAUC: 64.82 vs 57.30 - repeat responses: 14.30 vs 2.20 - human preference: 6% vs 94%
  • Human verification gives a modest gain over raw Gemini-generated labels: WEB/EGO goes from 54.8/42.4 to 56.5/43.8 (Table 11).
  • On the official ProactiveVideoQA metric, mean PAUC at ω=0.5 (Table A.1), BLIP-Live-Qwen trails several closed models: - BLIP-Live-Qwen: 45.07 - GPT-4.1: 58.89 - Gemini-2.5-Flash: 64.51 The authors attribute the closed models' scores to over-firing.
  • On official external metrics the margins are thin: - ESTP-F1 overall: 35.1 vs EyeWO 34.7 (Table 8) - StreamingBench PO: 47.60 vs Gemini 1.5 Pro 45.10 (Table 9) - OVO-Bench CRR: 62.50 vs Gemini 1.5 Pro 61.67 (Table 10)

The paper at a glance

01VLM (Gemini)generates query anda 1 FPSresponse/silencetimeline02Human correction oftiming and content,PhD batch QC03PAUC-F metric:harmonic mean ofPAUC recall andfiring precision04Agent-assistedharness search (10rounds x 10candidates) on afrozen VLM05Per-modelprompt/schemacalibration06Teacher-forced SFTwithSilence/Response/Holdtokens and lossmasking07Causal 1 Hzstreamingevaluation on fourbenchmarks
  1. VLM (Gemini) generates query and a 1 FPS response/silence timeline
  2. Human correction of timing and content, PhD batch QC
  3. PAUC-F metric: harmonic mean of PAUC recall and firing precision
  4. Agent-assisted harness search (10 rounds x 10 candidates) on a frozen VLM
  5. Per-model prompt/schema calibration
  6. Teacher-forced SFT with Silence/Response/Hold tokens and loss masking
  7. Causal 1 Hz streaming evaluation on four benchmarks
Fig. 1 — How the paper works, step by step (arXiv:2611.21553).
BLIP-Live-Qwen (this paper)49.08 PAUC-FQwen3.5-A3B (frozen, authors' harness)42.94 PAUC-FGPT-5.6-Sol (authors' harness)42.03 PAUC-FBLIP-Live-Gemma (this paper)41.38 PAUC-FGPT-4.1 (authors' harness)38.97 PAUC-FMMDuet2 (default harness)34.01 PAUC-FVideoChat3 (default harness)31.8 PAUC-F
Fig. 2 — ProactiveVideoQA mean PAUC-F (omega=0.5, GPT-4.1 judge), higher is better. Source: Table 6, arXiv:2611.21553.
Data table
ProactiveVideoQA mean PAUC-F (omega=0.5, GPT-4.1 judge)
MethodPAUC-F
BLIP-Live-Qwen (this paper)49.08 PAUC-F
Qwen3.5-A3B (frozen, authors' harness)42.94 PAUC-F
GPT-5.6-Sol (authors' harness)42.03 PAUC-F
BLIP-Live-Gemma (this paper)41.38 PAUC-F
GPT-4.1 (authors' harness)38.97 PAUC-F
MMDuet2 (default harness)34.01 PAUC-F
VideoChat3 (default harness)31.8 PAUC-F

Why it matters

For most harness builders this is a video-VLM paper and peripheral. Two ideas transfer to anyone building agents that must decide when to act without being asked, such as monitoring agents or ambient assistants. First, the metric argument: scoring only timeliness rewards spamming. Table 15 shows fires roughly tripling while PAUC stays flat, and the fix is a precision term counted over every emitted action. Second, the harness result: a frozen offline model with a short sliding observation window, memory of the query and last response, and a structured silence/response output gets close to or beyond purpose-trained streaming models on the authors' metric. That harness came from an agent-driven propose-evaluate-promote search over separated components, which adds a data point that automated harness search against a validation metric can work. Table 4 is the caution: the metric you search on shapes the behavior you get. The training pieces (the Hold state, silence loss masking) matter only if you finetune streaming models.

Issues (10)

  • High severity

    The '44% over baselines' claim compares BLIP-Live-Qwen (49.08) with MMDuet2 (34.01), a streaming-native model run with its default harness. MMDuet2 is not the strongest baseline in that column. The frozen Qwen3.5-A3B in the authors' harness scores 42.94, and GPT-5.6-Sol scores 42.03. Against the same backbone in the same harness, finetuning adds 6.14 points (Table 14a). The headline mixes harness gains with training gains and uses a weaker reference.

  • High severity

    Harness tuning budgets are unequal across models: - Gemma4 and Qwen3.5, the backbones of the proposed models, get the full agent search (100 candidate evaluations on BLIP-Live validation). - All other offline and proprietary models get 5 calibration candidates on a 20-sample subset. - Streaming-native baselines run with default inference and no tuning. - Gemini-Live in the authors' harness (ours-v) gets no calibration. The claim of 'outperforming multiple closed API models' therefore partly reflects tuning budget.

  • High severity

    The SOTA claim depends on the authors' own metric. On ProactiveVideoQA's official metric, PAUC (Table A.1, mean ω=0.5), BLIP-Live-Qwen scores 45.07, below GPT-4.1 (58.89), GPT-5.6-Sol (55.19), Gemini-2.5-Pro (55.07) and Gemini-2.5-Flash (64.51). The authors argue PAUC rewards over-firing, which is plausible. Still, the abstract's 'state-of-the-art across four benchmarks' rests on swapping in PAUC-F for this benchmark.

  • Medium severity

    The SOTA claim is not uniform even under PAUC-F: - On the WEB split, GPT-5.6-Sol (60.35) and GPT-4.1 (60.19) beat BLIP-Live-Qwen (57.48) at ω=0.5 (Table 6). - On ESTP-Bench PAUC-F, the untrained Qwen3.5-A3B (26.20) beats the trained BLIP-Live-Gemma (23.80) (Table 7). That table's closed-model rows are limited to GPT-4.1, GPT-4.1-mini and two Gemini-2.5 models. - On OVO-Bench the evaluation covers only the single CRR task.

  • Medium severity

    Most headline numbers use PAUC-F, the authors' metric, scored by a single GPT-4.1 judge. Harnesses and checkpoints were selected by validation PAUC-F. No seeds, variance or confidence intervals are reported. On the official external metrics the margins are small enough to fall within plausible noise: - ESTP-F1: 35.1 vs 34.7 - StreamingBench PO: 47.60 vs 45.10 - OVO-Bench CRR: 62.50 vs 61.67 On ESTP Implicit, EyeWO scores 52.5 against BLIP-Live-Qwen's 32.1. The authors attribute this to a weakness in the metric.

  • Medium severity

    Baseline numbers in Tables 8 and 10, and most of Table 9, are copied from prior papers and leaderboards. Those numbers came from different inference protocols, such as polling and response-in-last. The proposed models run under the authors' causal harness. The closed models in those tables (Gemini 1.5 Pro, GPT-4o, Claude 3.5 Sonnet) are older generations.

  • Medium severity

    The controlled training comparisons rest on re-implementations that look broken: - Training Qwen3.5-A3B on MMDuet2 data drops mean PAUC-F from 42.94 to 18.92. - MMDuet2's own recipe yields 24.39. Both are far below the untrained base. External datasets were converted to the authors' Silence/Response/Hold targets using a fixed 2-second Hold rule. These results say little about the original methods.

  • Low severity

    Validation of PAUC-F is thin. The human preference study uses 10 participants on 20 samples and compares only two of the authors' own harness variants. The Figure 2 simulation is built to show exactly the failure modes PAUC-F was designed to catch.

  • Low severity

    Contamination handling is only referenced. Training videos come from LLaVA-Video-178K, and overlap filtering against the test benchmarks is deferred to Appendix C, which is not shown. Data, code and checkpoints are promised but not released, and no repository URL is given.

  • Low severity

    The frozen-model harness uses 3 frames and binary output. The finetuned models use 32 frames and three-state output. Base-vs-finetuned comparisons (Table 14a) therefore change the context window and the output space as well as the weights.

Questions for the authors

  • What does each closed model score on PAUC-F with the same 100-candidate harness search that Gemma4 and Qwen3.5 received?
  • What is the run-to-run variance of PAUC-F across seeds, and across judge models other than GPT-4.1?
  • How many source videos from ProactiveVideoQA, ESTP-Bench, StreamingBench and OVO-Bench overlap with LLaVA-Video-178K, before and after the Appendix C filtering?
  • Does running the frozen Qwen3.5-A3B harness with 32 frames instead of 3 close more of the gap to the finetuned model?
  • Which hypotheses did the harness-search agent propose and reject, and how sensitive is the final design to the 20-sample calibration subset?
  • Why does training on MMDuet2 data push the backbone well below its untrained score, and does the 2-second Hold conversion explain it?

Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.

Yiyan Ji, Qixun Wang +11

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.

Mengxiao Wang, Mahbub Alam +2

Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).

Tanmay Sah

The Harness Hub newsletter

New harness companies, projects, releases and the monthly State of Harness report, in your inbox.