🤝🗣️ Speech-Hands

🤝 Agent integration with OpenClaw 🦞

Speech-Hands plugs into OpenClaw. The user speaks, OpenClaw hands it the audio, and it picks one of three action tokens: trust itself, defer to a perception tool, or re-derive the answer from scratch. Four real samples below — two AudioQA and two ASR.

Draft PR openclaw/openclaw#69073 · upstreaming Speech-Hands as a MediaUnderstandingProvider

Abstract

We introduce a voice-agentic framework that learns one critical omni-understanding skill: knowing when to trust itself versus when to consult external audio perception. Our work is motivated by a crucial yet counterintuitive finding: naively fine-tuning an omni-model on both speech recognition and external sound understanding tasks often degrades performance, as the model can be easily misled by noisy hypotheses. To address this, our framework, Speech-Hands, recasts the problem as an explicit self-reflection decision. This learnable reflection primitive proves effective in preventing the model from being derailed by flawed external candidates. We show that this agentic action mechanism generalizes naturally from speech recognition to complex, multiple-choice audio reasoning. Across the OpenASR leaderboard, Speech-Hands consistently outperforms strong baselines by 12.1% WER on seven benchmarks. The model also achieves 77.37% accuracy and high F1 on audio QA decisions, showing robust generalization and reliability across diverse audio question answering datasets.

Method

Self-reflection primitive

For every input, the model first picks one of three learnable tokens, then writes the answer. Each token is a different stance on the same trade-off:

Results

ASR — WER (%) across 7 OpenASR benchmarks

Speech-Hands beats both single-system baselines (ASR / Omni-LMs) and cascaded GER. Paired with Parakeet, it gets the lowest average WER.

AudioQA — Accuracy (%) on DCASE 2025

Evaluated on bioacoustic QA, multi-sound-object soundscapes, and MMAU-style complex audio reasoning. Majority-sampled Speech-Hands gets the highest average accuracy.

Action-token routing quality

How often each action token appears in training vs test, plus per-token test F1 on each OpenASR benchmark. <rewrite> is rare by design — it fires only when both internal and external are wrong.

Case Gallery

Audio QA

Seven DCASE 2025 AudioQA dev samples, grouped by the action token Speech-Hands picked. Play the audio; compare what Qwen2.5-Omni alone and Audio Flamingo 3 alone would say; see Speech-Hands' final answer.

Speech Recognition

Three OpenASR cases, one per action token. WER is normalized (Whisper-style), so "tomorrow" vs "to morrow" or missing apostrophes don't count. Each panel shows the reference, Qwen's first pass, the external ASR 5-best, and Speech-Hands' final.

BibTeX

@inproceedings{anonymous2026speechhands,
  title     = {Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech
               Recognition and Audio Reasoning with Omni Perception},
  author    = {Anonymous Authors},
  booktitle = {Proceedings of the Annual Meeting of the Association for
               Computational Linguistics (ACL)},
  year      = {2026}
}