feat(orchestrator): speak into every text field of the hub (requirement 013)

Requirement 013 and its implementation: page-mic.js puts a microphone on every textarea and text input, records with MediaRecorder and inserts the transcript at the cursor; Escape cancels. /api/stt transcribes with the local whisper.cpp model ggml-small-q8_0 (pinned to commit 5359861c and its sha256) in a per-request temp dir, capped at 60 s and 5 MiB, and returns German or English only, retranscribing a clip detected as another language in both and keeping the surer one. Without a model the button is disabled and names the install row. Tests run once on unify/skills-20261114 after the merge (operator order 2026-11-14).

🎯 Fix: 72 %
💥 Break: 12 %
💸 Tokens: –
⏱️ Velocity: –
🛡️ Security: 100 %
🏆 Score: 63
🚦 Gate: ⛔ fix 72 % < 90, break 12 % > 5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kevin Veen-BirkenbachandClaude Opus 5.5 committed 2026-11-14 21:37:26 +00:00
1 parent a2a03995cb
commit a438a2f54e
27 files changed
+861 -149

No files matched your search

+1 -1
View File
@@ -464,7 +464,7 @@ make project TARGET=/path/to/repo
Both commands also patch the target's `.claude/settings.local.json` — the machine-local, untracked file, so a project's reviewed `.claude/settings.json` is never rewritten by an install: the caveman and ponytail plugins are enabled so their modes auto-activate on session start, and two reminder hooks are registered — `kvb-dream` points each agent at `/kvb-dream` at most once every twelve hours (stamped per session in `~/.claude/.dream-stamps/`), and `kvb-coach` runs its short form at the first prompt of every full hour, once across all agents (stamped in `~/.claude/.coach-hours/`); the global install also enables the `kvb-dream-notify.timer` systemd user timer, which recommends `/kvb-dream` in a GNOME notification daily at 17:00 local time and catches up once after a missed run, and the `kvb-org-queue-cleanup.timer` and `kvb-org-queue-cleanup.path` user units, which start a `kvb-org-queue-cleanup` round at 00:00 and 12:00 local time without catch-up and whenever an agent runs `cleanup.py request`. A PreToolUse gate from the `kvb-org-organize` skill blocks edits to files in a git worktree until the session holds a claim (`skills/kvb-org-organize/scripts/council.py`), and its resource watch (`skills/kvb-org-organize/scripts/watch.py`) measures disk, Docker and caches, cleans up low-risk data and blocks new heavy jobs before the disk runs out; it stays inactive until `~/.claude/agents-council/config.json` exists, and because its hooks run that config's commands outside the sandbox, the install adds `Edit(~/.claude/agents-council/**)` to `permissions.deny`, which also covers Write in every permission mode; `Edit(~/.local/state/agents-council/**)` joins it, so the state the skills' scripts keep there changes only through those scripts. The global install writes both rules into `~/.claude/settings.json` too (`scripts/council-deny.js`), the only user-wide settings file, so they bind sessions in every repository. The `kvb-org-radio` skill registers five hooks: its prompt hook and a PostToolBatch hook deliver the immediate radio (`call`, `broadcast`, `security`, `pan-pan`, `mayday`) mid-turn, a Stop hook delivers the post (`post`, `newsletter` and call receipts) when the turn ends, an `asyncRewake` Stop hook wakes an idle session for the radio, and a PreToolUse hook refuses every writing action while a mayday holds, except those of its sender and of the agents that join it with `radio.py assist`. The PreToolUse hook of the `kvb-org-queue` skill sends heavy commands through `queue.py hold` once the config sets `queue_enforce`, and holds a held command to the settings' Bash deny and ask rules, which see only `queue.py hold`. The global install also installs the MEDUSA security scanner with pipx for `kvb-check-security`, unless `medusa` is already on the PATH. Two hooks of the `kvb-org-perform` skill (`skills/kvb-org-perform/scripts/log.py`) log every Agent spawn with its task class, model and effort, and store the quality feedback a sandboxed session prints, in `~/.local/state/agents-council/subagent-model.jsonl`. Every hook runs `/usr/bin/python3 -I`, so neither the `python3` of a sandbox-writable virtualenv on the `PATH` nor `PYTHONPATH` reaches the hooks, which run outside the sandbox; older bare `python3` hook entries are rewritten. The reminder hooks run `/usr/bin/bash` and pin `PATH=/usr/bin:/bin`, and the council scripts keep only the `PATH` directories this uid cannot write (`skills/kvb-org-organize/scripts/hookpath.py`), so no binary an agent plants on the inherited `PATH` runs outside the sandbox. Existing settings are preserved.
The global install also runs the Agent Orchestrator, a local review page in which the operator approves or rejects the agents' staged commits (the `kvb-git-diff` skill): it copies `orchestrator/` to `~/.local/share/kvb-orchestrator`, starts it there with `docker compose up -d --build` (restart `unless-stopped`, bound to `127.0.0.1` only, state in `~/.local/state/kvb-orchestrator`) and registers its MCP server for Claude Code at user scope. Its port is `KVB_ORCHESTRATOR_PORT` in `orchestrator/.env`; the hub page is `http://localhost:<port>/`, the MCP endpoint `/mcp`. Without docker the hub is skipped and the commit skills ask in chat. The hub reads new radio traffic aloud in maritime phraseology and takes push-to-talk input; its image builds both engines from pinned, sha256-verified sources and runs each as a separate process: Piper (`piper-tts`, GPL-3.0-or-later) with the `en_US-libritts-high` voice (904 speakers, trained on LibriTTS, CC BY 4.0, from `rhasspy/piper-voices`) and whisper.cpp (MIT) with the multilingual `ggml-base-q8_0` model (OpenAI Whisper weights, MIT, 82 MB). The operator's own voice is `KVB_ORCHESTRATOR_OPERATOR_VOICE` in `orchestrator/.env`. Its Tree reads only the file `pkgmgr branch tree` writes: the install clones `PKGMGR_REPO` (default the GitHub repository, never the sandbox-writable local checkout), builds the newest `v*` tag a fully trusted GPG key signed into `~/.local/share/kvb-orchestrator/pkgmgr`, and enables the `kvb-orchestrator-tree.timer` user timer, which writes `~/.local/state/pkgmgr/tree/branch-tree.json` a minute after its previous run while the hub page polls, on at most half a core; the hub mounts that directory read-only and starts no git. A signed tag without `branch tree` leaves the timer off.
The global install also runs the Agent Orchestrator, a local review page in which the operator approves or rejects the agents' staged commits (the `kvb-git-diff` skill): it copies `orchestrator/` to `~/.local/share/kvb-orchestrator`, starts it there with `docker compose up -d --build` (restart `unless-stopped`, bound to `127.0.0.1` only, state in `~/.local/state/kvb-orchestrator`) and registers its MCP server for Claude Code at user scope. Its port is `KVB_ORCHESTRATOR_PORT` in `orchestrator/.env`; the hub page is `http://localhost:<port>/`, the MCP endpoint `/mcp`. Without docker the hub is skipped and the commit skills ask in chat. The hub reads new radio traffic aloud in maritime phraseology and takes push-to-talk input; its image builds both engines from pinned, sha256-verified sources and runs each as a separate process: Piper (`piper-tts`, GPL-3.0-or-later) with the `en_US-libritts-high` voice (904 speakers, trained on LibriTTS, CC BY 4.0, from `rhasspy/piper-voices`) and whisper.cpp (MIT) with the multilingual `ggml-small-q8_0` model (OpenAI Whisper weights, MIT, 264 MB). The operator's own voice is `KVB_ORCHESTRATOR_OPERATOR_VOICE` in `orchestrator/.env`. Its Tree reads only the file `pkgmgr branch tree` writes: the install clones `PKGMGR_REPO` (default the GitHub repository, never the sandbox-writable local checkout), builds the newest `v*` tag a fully trusted GPG key signed into `~/.local/share/kvb-orchestrator/pkgmgr`, and enables the `kvb-orchestrator-tree.timer` user timer, which writes `~/.local/state/pkgmgr/tree/branch-tree.json` a minute after its previous run while the hub page polls, on at most half a core; the hub mounts that directory read-only and starts no git. A signed tag without `branch tree` leaves the timer off.
Stream → Radio (`#stream/radio`) is the hub's radio interface. Its station list shows one line per station, the full name in the tooltip, with its state and a play or stop button; when the browser blocks autoplay, the stream fails or the CSP refuses its origin, the row and a toast say so, and one click starts the station. New stations play daily around the clock. While the hub reads a message aloud, the speaking agent pulses there and in the Orbit and Chord views, under the caption "<Agent> speaking" with the message's first line; under `prefers-reduced-motion` the ring stays still. The speech bubble icon in the header (a hub setting, kept across reloads) puts a bubble at each speaking agent in the Agents graph, 3D, Hierarchy, Radio and Stream views, or in a caption bar at the bottom when the view does not show that agent: the agent being read gets the largest bubble and its words appear in time with the audio, waiting messages show a smaller `…` bubble per agent, and without speech output the same rows run at 180 words a minute, at least 1.5 s each. The composer sends on all seven channels: post and call to one agent, newsletter, broadcast, pan-pan, security and mayday to all. "Record" records with MediaRecorder; the hub transcribes the recording with whisper.cpp off the request thread (at most 120 s of audio, a 120 s timeout, four at a time), keeps it for replay in `~/.local/state/kvb-orchestrator/voice` (the newest 50) and puts the transcript into the field to edit; the message goes out as text, with the recording's host path and sha256 only when "Attach recording" is ticked. The hub transcribes at most six recordings a minute and 600 s of audio an hour. Every operator message is passkey-signed, and the hub verifies the signature against the pinned keys, each once, before it relays; `operator_message(id, text, to)` answers `match` only for the signed text and recipient; for a denial the signature covers the recipient the inbox resolved at signing, and the answer's text is only the signed reason, never the agent's own words the radio row quotes. "Cancel mayday", Evening, Sunset, Sunrise and a settings save that adds a stream URL take the passkey the same way.
+90
View File
@@ -0,0 +1,90 @@
# 013 - Voice Input
## User Story
As the operator, I want a microphone on every text field of the hub that
turns my speech into text on this machine, so that I can work without a
keyboard on a desktop and on a small screen.
## Decisions (operator-confirmed 2026-11-14; do not re-litigate)
1. **Everywhere**: one shared microphone component sits on every `textarea`
and text input of the hub (composer, radio, feedback popup, marks,
filters, settings), present by default. Rejected: an opt-in per field.
2. **Local Whisper**: the browser records with MediaRecorder and sends the
audio to the hub on this machine, which transcribes it with a local
Whisper model; the audio never leaves the machine and is deleted right
after the transcription. Rejected: the browser's speech recognition
(Chrome sends audio to Google, Firefox has none); a browser fallback.
3. **Insert**: the text lands at the cursor; German and English are detected
automatically; a second tap stops the recording, Escape cancels it, and a
visible indicator shows while it records.
4. **Model**: the one-time model download is an operator row; without a
model the button is disabled and its tooltip names that row.
5. **Limits**: a recording is capped at 60 s and 5 MB; the transcription
endpoint accepts requests only from the hub page, with the same origin
and CSRF checks as every other POST of the hub.
6. **Responsive**: touch targets of at least 44 px at 390 px; the component
works with mouse, touch and keyboard.
7. **Order**: built in parallel with 012 in its own sidetree; 012's
feedback popup and marks get the microphone through the shared component.
## Approach
```mermaid
flowchart TD
subgraph P["🧭 0 · Preparation"]
P1["📋 Write requirement"] --> P2{"✅ Operator approves?"}
P2 -- "✏️ corrections" --> P1
end
P2 -- "🤖 yes" --> S1
subgraph S["🗣️ A · Local transcription"]
S1["📥 Operator row: model download"] --> S2["🧩 /api/stt: caps, origin, CSRF, temp cleanup"]
S2 --> S3["🧠 Local Whisper, German and English"]
S3 --> ST{"🧪 Unit and fixture audio green?"}
ST -- "🔴 no: fix" --> S2
end
ST -- "🟢 yes" --> M1
subgraph M["🎙️ B · Microphone everywhere"]
M1["🧩 Shared component on every text field"] --> M2["⏺️ Record, stop, cancel, indicator"]
M2 --> M3["✍️ Insert at the cursor"]
M3 --> MT{"🧪 E2E with a fake microphone green?"}
MT -- "🔴 no: fix" --> M1
MT -- "🟢 yes" --> MF{"💬 Operator feedback on screenshots?"}
MF -- "✏️ changes" --> M1
end
MF -- "👍 fits" --> E1["🏁 Review, commit with operator OK, hand-over"]
```
## Acceptance Criteria
### Transcription
- [ ] A POST of the German fixture WAV to `/api/stt` returns "hallo welt",
the English fixture returns "hello world".
- [ ] The transcription's temp directory is empty after each request,
including a failed one.
- [ ] A request from another origin, without the CSRF token, over 60 s or
over 5 MB is refused with 4xx and is not transcribed.
- [ ] Without an installed model, `/api/stt` answers with the operator row
that installs it, and the button is disabled with that row in its
tooltip.
### Microphone
- [ ] A DOM scan of every hub view finds no `textarea` or text input without
the microphone button.
- [ ] Pressing the button records, a second press stops and the transcript
appears at the cursor; Escape cancels without inserting text; an
indicator shows while it records.
- [ ] At 390 px the button's touch target is at least 44 by 44 px.
### Tests
- [ ] Unit: the endpoint caps, the origin and CSRF checks, the temp cleanup
and the language selection.
- [ ] Integration: the real model transcribes both fixture files, run
through `queue.py hold`.
- [ ] E2E: Chromium and Firefox with a fake microphone stream fed from a
WAV file, at 1920 and 390 px; the transcript appears in the composer;
negative control: with the model missing the button is disabled.
+1
View File
@@ -12,3 +12,4 @@ skill defines the format.
- [006 - Score Metric Overlays](006-score-metric-overlays.md)
- [007 - Resources View](007-resources-view.md)
- [008 - Home Dashboard](008-home-dashboard.md)
- [013 - Voice Input](013-voice-input.md)
+3 -3
View File
@@ -50,9 +50,9 @@ ADD --checksum=sha256:2efdc6d7f954588b8180132cbd9b8001933fdd00932c92bc92fd0d2028
${INFINITO_ARTEFACT_BASE:-https:/}/huggingface.co/rhasspy/piper-voices/resolve/e7229f7d33a10e145e109b886d661984201ac3c4/en/en_US/libritts/high/en_US-libritts-high.onnx.json \
/opt/voices/en_US-libritts-high.onnx.json
# hadolint ignore=DL3020
ADD --checksum=sha256:c577b9a86e7e048a0b7eada054f4dd79a56bbfa911fbdacf900ac5b567cbb7d9 --chmod=0644 \
${INFINITO_ARTEFACT_BASE:-https:/}/huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-base-q8_0.bin \
/opt/whisper/ggml-base-q8_0.bin
ADD --checksum=sha256:49c8fb02b65e6049d5fa6c04f81f53b867b5ec9540406812c643f177317f779f --chmod=0644 \
${INFINITO_ARTEFACT_BASE:-https:/}/huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-small-q8_0.bin \
/opt/whisper/ggml-small-q8_0.bin
COPY --from=whisper-cli /out/whisper-cli /opt/whisper/whisper-cli
# ADD --chmod=0644 also gives the directories it creates mode 0644; without the search bit the non-root hub finds no voice and no whisper.cpp
RUN chmod 0755 /opt/voices /opt/whisper
+1 -1
View File
@@ -76,7 +76,7 @@ services:
- --whisper
- /opt/whisper/whisper-cli
- --whisper-model
- /opt/whisper/ggml-base-q8_0.bin
- /opt/whisper/ggml-small-q8_0.bin
- --scratch
- /scratch
- --speech-url
+74 -17
View File
@@ -1,11 +1,13 @@
from __future__ import annotations
import io
import json
import os
import re
import secrets
import struct
import subprocess
import tempfile
import threading
import time
import wave
@@ -30,6 +32,12 @@ STT_RATE = 16000
STT_SECONDS = 120
STT_LIMIT = 44 + STT_RATE * 2 * STT_SECONDS
STT_TIMEOUT = 120
INPUT_SECONDS = 60
INPUT_LIMIT = 5 << 20
MODEL_ROW = "make orchestrator-install"
HEARING_DIR = "stt-"
LANGUAGES = ("de", "en")
SPECIAL = "[_"
HEARING_WINDOW = 60
HEARING_RATE = 6
HEARING_HOUR = 3600
@@ -100,7 +108,15 @@ def voice_of(key: str, pool: list[str], operator: str) -> str:
return pool[int(digest(key), 16) % len(pool)]
def wav_seconds(body: bytes) -> float:
def wav_seconds(body: bytes, seconds: int) -> float:
"""Args:
body: the recording.
seconds: the longest recording accepted.
Returns:
Its length in seconds once it is 16 kHz mono 16-bit PCM WAV lasting
more than 0 and at most seconds; otherwise it raises ToolError.
"""
try:
with wave.open(io.BytesIO(body), "rb") as reader:
shape = (reader.getnchannels(), reader.getsampwidth(), reader.getframerate(), reader.getcomptype())
@@ -109,8 +125,8 @@ def wav_seconds(body: bytes) -> float:
raise ToolError("the recording must be a WAV file") from None
if shape != (1, 2, STT_RATE, "NONE"):
raise ToolError(f"the recording must be {STT_RATE} Hz mono 16-bit PCM")
if not 0 < frames <= STT_RATE * STT_SECONDS:
raise ToolError(f"the recording must last more than 0 and at most {STT_SECONDS} seconds")
if not 0 < frames <= STT_RATE * seconds:
raise ToolError(f"the recording must last more than 0 and at most {seconds} seconds")
return frames / STT_RATE
@@ -156,6 +172,27 @@ def heard(output: str) -> str:
return " ".join(NOISE.sub(" ", output).split())[:TEXT_LIMIT]
def transcript(path: Path) -> tuple[str, str, float]:
"""Args:
path: the JSON whisper.cpp writes with --output-json-full.
Returns:
(the language it names, its segment texts joined, the mean
probability of its text tokens, 0.0 without any); a file that is
missing or not of that shape raises EngineError.
"""
try:
data = json.loads(path.read_bytes().decode("utf-8", errors="replace"))
language, segments = data["result"]["language"], data["transcription"]
texts = [segment["text"] for segment in segments]
sure = [token["p"] for segment in segments for token in segment.get("tokens", []) if not str(token["text"]).startswith(SPECIAL)]
except (OSError, ValueError, RecursionError, KeyError, IndexError, TypeError, AttributeError):
raise EngineError("whisper.cpp wrote no readable JSON transcript") from None
if not isinstance(language, str) or not all(isinstance(text, str) for text in texts) or not all(isinstance(p, int | float) and not isinstance(p, bool) for p in sure):
raise EngineError("whisper.cpp wrote no readable JSON transcript")
return language, " ".join(texts), sum(sure) / len(sure) if sure else 0.0
@dataclass(frozen=True)
class Engines:
piper: Path
@@ -207,14 +244,15 @@ class Speech:
def input_gap(self) -> str:
"""Returns:
"" when speech input works, otherwise why not, naming the
whisper.cpp executable or model with its path; always "" with the
speech service.
whisper.cpp executable or model with its path and the operator row
MODEL_ROW that installs both; always "" with the speech service.
"""
if self.engines.service.enabled():
return ""
model = self.engines.whisper_model
gap = unreachable(self.engines.whisper, "the whisper.cpp executable", os.X_OK, "executable") or unreachable(model, "the whisper.cpp model", os.R_OK, "readable")
return gap or ("" if os.path.isfile(model) else f"the whisper.cpp model {model} is not a file")
gap = gap or ("" if os.path.isfile(model) else f"the whisper.cpp model {model} is not a file")
return f"{gap}; the operator row {MODEL_ROW} installs it" if gap else ""
def voice_of(self, key: str) -> str:
"""Args:
@@ -233,7 +271,7 @@ class Speech:
silent, deaf = self.output_gap(), self.input_gap()
return {
"tts": not silent, "stt": not deaf, "tts_off": silent, "stt_off": deaf, "voices": len(self.voices), "operator": operator, "sample": sample,
"limit": SPEECH_LIMIT, "rate": STT_RATE, "seconds": STT_SECONDS,
"limit": SPEECH_LIMIT, "rate": STT_RATE, "seconds": STT_SECONDS, "input": INPUT_SECONDS,
}
def synthesize(self, body: bytes, admit: Callable[[str], None]) -> bytes:
@@ -333,21 +371,40 @@ class Speech:
gap = self.input_gap()
if gap:
raise Unavailable(f"speech input is off: {gap}")
self.admit(wav_seconds(body))
self.admit(wav_seconds(body, INPUT_SECONDS))
return self.whisper(body)
def whisper(self, body: bytes) -> dict:
"""Returns:
{"text"} from the speech service, otherwise {"text", "language"}
in one of LANGUAGES.
"""
if self.engines.service.enabled():
with self.hearing:
return {"text": heard(self.engines.service.hear(body))}
command = [str(self.engines.whisper), "--model", str(self.engines.whisper_model), "--file", "-", "--output-txt", "--language", "auto", "--no-prints", "--suppress-nst"]
with self.hearing:
try:
done = subprocess.run(command, input=body, capture_output=True, timeout=STT_TIMEOUT, env=self.env(), cwd=self.engines.scratch, check=False)
except subprocess.TimeoutExpired:
raise EngineError(f"whisper.cpp took longer than {STT_TIMEOUT} seconds") from None
except OSError as error:
raise EngineError(f"whisper.cpp did not start: {error.strerror}") from None
with self.hearing, tempfile.TemporaryDirectory(prefix=HEARING_DIR, dir=self.engines.scratch) as folder:
language, text, _ = self.listen(body, Path(folder), "auto")
if language not in LANGUAGES:
language, text, _ = max((self.listen(body, Path(folder), choice) for choice in LANGUAGES), key=lambda found: found[2])
return {"text": heard(text), "language": language}
def listen(self, body: bytes, folder: Path, language: str) -> tuple[str, str, float]:
"""Args:
body: the recording.
folder: the request's own directory, cwd and TMPDIR of the run.
language: "auto" to detect it, otherwise the language to hear.
Returns:
transcript() of the JSON whisper.cpp writes into folder.
"""
target = folder / language
command = [str(self.engines.whisper), "--model", str(self.engines.whisper_model), "--file", "-", "--output-json", "--output-json-full", "--output-file", str(target), "--language", language, "--no-prints", "--suppress-nst"]
try:
done = subprocess.run(command, input=body, capture_output=True, timeout=STT_TIMEOUT, env={**self.env(), "TMPDIR": str(folder)}, cwd=folder, check=False)
except subprocess.TimeoutExpired:
raise EngineError(f"whisper.cpp took longer than {STT_TIMEOUT} seconds") from None
except OSError as error:
raise EngineError(f"whisper.cpp did not start: {error.strerror}") from None
if done.returncode != 0:
raise EngineError(f"whisper.cpp failed with exit code {done.returncode}: {done.stderr.decode('utf-8', errors='replace').strip()[-300:]}")
return {"text": heard(done.stdout.decode("utf-8", errors="replace"))}
return transcript(target.with_name(f"{language}.json"))
+2 -2
View File
@@ -8,7 +8,7 @@ from http import HTTPStatus
from urllib.parse import urlsplit
from .diffs import DIFF_BODY_LIMIT
from .engines import STT_LIMIT
from .engines import INPUT_LIMIT, STT_LIMIT
from .exchange import PLAIN_CSP, Exchange
from .resources import RESOURCES, sampled
from .scores import scored
@@ -285,7 +285,7 @@ class Handler(Exchange):
def do_POST(self) -> None:
path = urlsplit(self.path).path
length = self.body_length({UPLOADS: UPLOAD_LIMIT, STT_ROUTE: STT_LIMIT, VOICE_ROUTE: STT_LIMIT, "/mcp": DIFF_BODY_LIMIT, "/api/asks": DIFF_BODY_LIMIT}.get(path, BODY_LIMIT))
length = self.body_length({UPLOADS: UPLOAD_LIMIT, STT_ROUTE: INPUT_LIMIT, VOICE_ROUTE: STT_LIMIT, "/mcp": DIFF_BODY_LIMIT, "/api/asks": DIFF_BODY_LIMIT}.get(path, BODY_LIMIT))
if length is None:
return
host = self.host()
+3 -2
View File
@@ -10,7 +10,8 @@ from collections import Counter
from collections.abc import Callable, Iterator
from pathlib import Path
from .engines import Speech, wav_seconds
from . import engines
from .engines import Speech
from .errors import EngineError, Missing, ToolError, Unavailable
from .records import write_atomic
@@ -44,7 +45,7 @@ class VoiceNotes:
gap = self.speech.input_gap()
if gap:
raise Unavailable(f"speech input is off: {gap}")
seconds = wav_seconds(body)
seconds = engines.wav_seconds(body, engines.STT_SECONDS)
with self.lock:
if sum(1 for note in self.notes.values() if note["status"] == "pending") >= HEARING_LIMIT:
raise Unavailable(f"{HEARING_LIMIT} recordings are already waiting for their transcript")
+12 -1
View File
@@ -319,7 +319,16 @@ textarea { width: 100%; min-height: 84px; padding: 8px 10px; border: 1px solid v
.palette-box input { width: 100%; padding: 14px 16px; border: 0; background: transparent; font-size: var(--fs-lg); outline: none; }
.palette-row { display: flex; align-items: center; gap: 6px; padding-right: 10px; border-bottom: 1px solid var(--line); }
.palette-row input { flex: 1; min-width: 0; }
.mic[aria-pressed="true"] { background: var(--bad-soft); color: var(--bad); }
.mic { flex: none; }
.mic[aria-pressed="true"] { background: var(--bad-soft); color: var(--bad); animation: mic-pulse 1.2s ease-in-out infinite; }
.mic-auto { margin-left: 4px; vertical-align: middle; }
textarea + .mic-auto { display: flex; margin: 4px 0 0 auto; }
.mic-live { position: fixed; left: 50%; bottom: calc(18px + env(safe-area-inset-bottom, 0px)); z-index: 60; display: flex; align-items: center; gap: 8px; max-width: calc(100vw - 32px); padding: 8px 14px; transform: translateX(-50%); border-radius: 999px; background: var(--bad); color: #fff; font-weight: 650; box-shadow: var(--shadow-lift); }
.mic-live[hidden] { display: none; }
.mic-live::before { content: ""; flex: none; width: 10px; height: 10px; border-radius: 50%; background: #fff; animation: mic-pulse 1s ease-in-out infinite; }
@keyframes mic-pulse { 50% { opacity: .45; } }
@media (max-width: 640px) { .mic { min-width: 44px; min-height: 44px; } }
@media (prefers-reduced-motion: reduce) { .mic[aria-pressed="true"], .mic-live::before { animation: none; } }
.palette-list { max-height: 55vh; margin: 0; padding: 6px; overflow: auto; list-style: none; }
.palette-list li { display: flex; align-items: center; gap: 10px; padding: 8px 10px; border-radius: 8px; cursor: pointer; }
.palette-list li[aria-selected="true"] { background: var(--accent-soft); color: var(--accent); }
@@ -605,6 +614,8 @@ pre.notes { margin: 0; padding: 8px 10px; background: var(--sunken); border-radi
{% include "page-speech.js" %}
{% include "page-mic.js" %}
{% include "page-bubbles.js" %}
{% include "page-alerts.js" %}
+1 -1
View File
@@ -94,7 +94,7 @@
const classes = choice("to-one", TO_ONE, "post");
const form = h("form", {class: "popup-form"}, text, file, classes,
h("p", {class: "muted"}, "Post arrives at the end of the turn, Call at once with a read receipt; your passkey signs, verifiable with operator_message."),
h("div", {class: "actions"}, h("span", {class: "spacer"}), micButton(text), button));
h("div", {class: "actions"}, h("span", {class: "spacer"}), micButton(text, false), button));
form.addEventListener("submit", async (event) => {
event.preventDefault();
if (!text.value.trim()) {
+1
View File
@@ -37,6 +37,7 @@
window.addEventListener("resize", refresh3d);
window.addEventListener("resize", placePopup);
skyPanel();
micWatch();
radioSpot();
radio.element = radioElement();
drawSky();
+1 -1
View File
@@ -63,7 +63,7 @@
throw new Error("no recording");
}
parts.status.textContent = "Converting …";
const stored = await send("/api/voice", await wavOf(blob, sound), "audio/wav");
const stored = await send("/api/voice", await wavOf(blob, sound, state.speech.seconds), "audio/wav");
if (!stored.ok || !stored.data) {
throw new Error(`${stored.status} ${stored.text}`.trim());
}
+185
View File
@@ -0,0 +1,185 @@
const MIC_HINT = "Voice input: tap to record, tap again to insert the text at the cursor; Escape cancels";
const MIC_NODES = `${Voice.FIELDS}, button.mic`;
const micButtons = new WeakMap();
const micFields = new WeakMap();
let micCount = 0;
function micLive() {
let live = document.getElementById("mic-live");
if (!live) {
live = h("div", {id: "mic-live", class: "mic-live", role: "status", "aria-live": "polite", hidden: true});
document.body.append(live);
}
return live;
}
function micState(button) {
const reason = micGap();
button.disabled = Boolean(reason);
button.title = reason || MIC_HINT;
}
function micLook(button, look) {
const recording = look === "recording";
button.setAttribute("aria-pressed", String(recording));
button.setAttribute("aria-label", recording ? "Stop voice input" : "Voice input");
button.replaceChildren(icon(recording ? "bi-stop-fill" : look === "busy" ? "bi-hourglass-split" : "bi-mic"));
micLive().hidden = !recording;
if (look === "busy") {
button.disabled = true;
} else {
micState(button);
}
}
/**
* @param {HTMLInputElement|HTMLTextAreaElement} field the text field the transcript lands in.
* @param {boolean} auto whether the shared component placed the button right after the field and keeps it there.
* @returns {HTMLButtonElement} the microphone button of the field.
*/
function micButton(field, auto) {
if (!field.id) {
micCount += 1;
field.id = `mic-field-${micCount}`;
}
const button = h("button", {type: "button", class: auto ? "btn small ghost icon-only mic mic-auto" : "btn small ghost icon-only mic", "aria-controls": field.id}, icon("bi-mic"));
micButtons.set(field, button);
micFields.set(button, field);
field.dataset.mic = "on";
micLook(button, "idle");
button.addEventListener("click", () => toggleMic(field, button));
return button;
}
function micSweep() {
for (const field of document.querySelectorAll(Voice.FIELDS)) {
const button = micButtons.get(field);
if (field.readOnly || (button && !button.classList.contains("mic-auto"))) {
continue;
}
if (!button) {
field.after(micButton(field, true));
} else if (button.previousElementSibling !== field) {
field.after(button);
}
}
for (const button of document.querySelectorAll("button.mic-auto")) {
if (!micFields.get(button).isConnected) {
button.remove();
}
}
}
function micTouched(records) {
return records.some((record) => [...record.addedNodes, ...record.removedNodes].some((node) => node instanceof Element && (node.matches(MIC_NODES) || Boolean(node.querySelector(MIC_NODES)))));
}
function micWatch() {
micSweep();
new MutationObserver((records) => micTouched(records) && micSweep()).observe(document.body, {childList: true, subtree: true});
window.addEventListener("keydown", (event) => {
if (event.key === "Escape" && mic.button && !mic.busy) {
event.preventDefault();
event.stopImmediatePropagation();
stopMic(true);
}
}, true);
}
function stopMic(cancel) {
mic.cancel = cancel;
window.clearInterval(mic.timer);
if (mic.recorder && mic.recorder.state === "recording") {
mic.recorder.stop();
}
}
function micTick() {
const seconds = Math.floor((Date.now() - mic.started) / 1000);
micLive().textContent = `Recording ${seconds} s of at most ${state.speech.input} s: tap the microphone again to insert, Escape cancels`;
if (seconds >= state.speech.input) {
stopMic(false);
}
}
function micDone() {
Object.assign(mic, {recorder: null, button: null, field: null, busy: false, cancel: false});
}
async function toggleMic(field, button) {
if (mic.button) {
if (mic.button === button && mic.recorder) {
stopMic(false);
}
return;
}
Object.assign(mic, {button, field, cancel: false});
let media = null;
try {
media = await navigator.mediaDevices.getUserMedia({audio: {channelCount: 1, echoCancellation: true, noiseSuppression: true}});
} catch (error) {
micDone();
toast("Microphone not available", [String(error.message || error)], false);
return;
}
if (mic.cancel) {
media.getTracks().forEach((track) => track.stop());
micDone();
return;
}
const recorder = new MediaRecorder(media);
const chunks = [];
recorder.addEventListener("dataavailable", (event) => {
if (event.data.size) {
chunks.push(event.data);
}
});
recorder.addEventListener("stop", () => {
media.getTracks().forEach((track) => track.stop());
window.clearInterval(mic.timer);
const cancelled = mic.cancel;
micLook(button, "idle");
if (cancelled) {
micDone();
return;
}
mic.recorder = null;
micTranscribe(new Blob(chunks, {type: recorder.mimeType}), field, button);
}, {once: true});
Object.assign(mic, {recorder, started: Date.now()});
recorder.start();
micLook(button, "recording");
micTick();
mic.timer = window.setInterval(micTick, 250);
}
function micInsert(field, text) {
const start = field.selectionStart ?? field.value.length;
const spliced = Voice.spliced(field.value, start, field.selectionEnd ?? start, text);
field.value = spliced.value;
field.focus();
field.setSelectionRange(spliced.caret, spliced.caret);
field.dispatchEvent(new Event("input", {bubbles: true}));
}
async function micTranscribe(blob, field, button) {
const sound = context();
if (!sound || !blob.size) {
micDone();
return;
}
mic.busy = true;
micLook(button, "busy");
try {
const result = await send("/api/stt", await wavOf(blob, sound, state.speech.input), "audio/wav");
if (!result.ok || !result.data) {
throw new Error(`${result.status} ${result.text}`.trim());
}
micInsert(field, result.data.text || "");
} catch (error) {
toast("Voice input failed", [String(error.message || error)], false);
} finally {
micDone();
micLook(button, "idle");
}
}
+1 -1
View File
@@ -142,7 +142,7 @@
runPalette(palette.index);
}
});
box.replaceChildren(h("div", {class: "palette-box", role: "dialog", "aria-label": "Commands"}, h("div", {class: "palette-row"}, palette.input, micButton(palette.input)), palette.list));
box.replaceChildren(h("div", {class: "palette-box", role: "dialog", "aria-label": "Commands"}, h("div", {class: "palette-row"}, palette.input, micButton(palette.input, false)), palette.list));
box.hidden = false;
drawPalette();
palette.input.focus();
+2 -101
View File
@@ -259,84 +259,10 @@
: !window.MediaRecorder ? "This browser cannot record" : !state.speech.stt ? `Voice input is off: ${state.speech.stt_off}` : "";
}
function micButton(field) {
const button = h("button", {type: "button", class: "btn small ghost icon-only mic", "aria-pressed": "false", ...named("Voice input", "Voice input: hold and speak; the text lands in the field and is not sent")}, icon("bi-mic"));
const reason = micGap();
if (reason) {
button.disabled = true;
button.title = `Voice input: ${reason}`;
return button;
}
const press = (event) => {
event.preventDefault();
record(field, button);
};
button.addEventListener("pointerdown", press);
for (const name of ["pointerup", "pointerleave", "pointercancel"]) {
button.addEventListener(name, release);
}
button.addEventListener("keydown", (event) => {
if ((event.key === " " || event.key === "Enter") && !event.repeat) {
press(event);
}
});
button.addEventListener("keyup", (event) => {
if (event.key === " " || event.key === "Enter") {
event.preventDefault();
release();
}
});
return button;
}
async function record(field, button) {
if (mic.held || mic.recorder || mic.busy) {
return;
}
mic.held = true;
let media = null;
try {
media = await navigator.mediaDevices.getUserMedia({audio: {channelCount: 1, echoCancellation: true, noiseSuppression: true}});
} catch (error) {
mic.held = false;
toast("Microphone not available", [String(error.message || error)], false);
return;
}
if (!mic.held) {
media.getTracks().forEach((track) => track.stop());
return;
}
const recorder = new MediaRecorder(media);
const chunks = [];
recorder.addEventListener("dataavailable", (event) => {
if (event.data.size) {
chunks.push(event.data);
}
});
recorder.addEventListener("stop", () => {
media.getTracks().forEach((track) => track.stop());
mic.recorder = null;
button.setAttribute("aria-pressed", "false");
transcribe(new Blob(chunks, {type: recorder.mimeType}), field, button);
}, {once: true});
mic.recorder = recorder;
recorder.start();
button.setAttribute("aria-pressed", "true");
mic.timer = window.setTimeout(release, state.speech.seconds * 1000);
}
function release() {
mic.held = false;
window.clearTimeout(mic.timer);
if (mic.recorder && mic.recorder.state === "recording") {
mic.recorder.stop();
}
}
async function wavOf(blob, sound) {
async function wavOf(blob, sound, seconds) {
const decoded = await sound.decodeAudioData(await blob.arrayBuffer());
const rate = state.speech.rate;
const length = Voice.frames(decoded.duration, rate, state.speech.seconds);
const length = Voice.frames(decoded.duration, rate, seconds);
if (length < rate / 4) {
throw new Error("too short: speak for at least a quarter second");
}
@@ -348,28 +274,3 @@
const rendered = await offline.startRendering();
return Voice.wav(rendered.getChannelData(0), rate);
}
async function transcribe(blob, field, button) {
const sound = context();
if (!sound || !blob.size) {
return;
}
mic.busy = true;
button.disabled = true;
try {
const result = await send("/api/stt", await wavOf(blob, sound), "audio/wav");
if (!result.ok || !result.data) {
throw new Error(`${result.status} ${result.text}`.trim());
}
if (result.data.text) {
field.value = [field.value.trim(), result.data.text].filter(Boolean).join(" ");
field.dispatchEvent(new Event("input", {bubbles: true}));
}
field.focus();
} catch (error) {
toast("Voice input failed", [String(error.message || error)], false);
} finally {
mic.busy = false;
button.disabled = false;
}
}
+1 -1
View File
@@ -47,7 +47,7 @@
const radio = {element: null, url: "", level: 0, timer: 0, playing: false, waiting: false, index: -1, pick: null, sig: "", state: "", short: "", note: "", told: null};
const speech = {pending: [], busy: false, warned: 0, current: null, source: null, loading: null, timer: 0, item: null};
let radioSet = RadioSet.valid(stored(RadioSet.KEY, null));
const mic = {recorder: null, held: false, busy: false, timer: 0};
const mic = {recorder: null, button: null, field: null, busy: false, cancel: false, timer: 0, started: 0};
const space = {
views: {graph: {...state.settings.space.graph}, sky: {...state.settings.space.sky}}, glides: {graph: null, sky: null}, edits: 0, saved: 0, timer: 0, look: false, compass: null,
};
+21 -1
View File
@@ -3,6 +3,7 @@ const Voice = (() => {
const RANKS = {mayday: 0, "pan-pan": 1, security: 2};
const ALARMS = new Set(["mayday", "pan-pan"]);
const ROUTINE = 3;
const FIELDS = 'textarea, input:not([type]), input[type="text"], input[type="search"]';
function toneOf(row, human) {
if (ALARMS.has(row.class)) {
@@ -59,7 +60,26 @@ const Voice = (() => {
return buffer;
}
return {toneOf, loudness, beep, enqueue, frames, wav};
/**
* @param {string} value the text of the field.
* @param {number} start the start of its selection.
* @param {number} end the end of its selection.
* @param {string} text the transcript.
* @returns {{value: string, caret: number}} value with the trimmed transcript in place of the selection, one space on each side where a neighbour is no whitespace, and the caret right after the transcript.
*/
function spliced(value, start, end, text) {
const words = String(text).trim();
if (!words) {
return {value, caret: end};
}
const before = value.slice(0, start);
const after = value.slice(end);
const lead = before && !/\s$/.test(before) ? " " : "";
const tail = after && !/^\s/.test(after) ? " " : "";
return {value: `${before}${lead}${words}${tail}${after}`, caret: before.length + lead.length + words.length};
}
return {FIELDS, toneOf, loudness, beep, enqueue, frames, wav, spliced};
})();
if (typeof module === "object" && module.exports) {
module.exports = Voice;
+238
View File
@@ -0,0 +1,238 @@
from __future__ import annotations
import base64
import dataclasses
import json
import os
import shutil
import sys
import unittest
from pathlib import Path
from unittest import mock
HERE = Path(__file__).resolve().parent
sys.path[:0] = [str(HERE.parent), str(HERE)]
import test_orchestrator_e2e as base
from hub_case import HubCase
from kvb_orchestrator import engines
OUT = Path(os.environ.get("E2E_OUT", "/tmp/voice-input-shots"))
WIDTHS = ((1920, 1080), (390, 844))
FIXTURE = HERE.parent / "fixtures" / "voice" / "hello-world.wav"
HEARD = "hello from the mic second line"
FAKE_MIC = """() => {
const wav = Uint8Array.from(atob("@WAV@"), (char) => char.charCodeAt(0));
const posted = [];
window.__voice = {posted, opened: 0};
const fetched = window.fetch.bind(window);
window.fetch = (path, init) => {
if (String(path) === "/api/stt") {
posted.push(Date.now());
}
return fetched(path, init);
};
if (navigator.mediaDevices) {
navigator.mediaDevices.getUserMedia = async () => {
window.__voice.opened += 1;
const sound = new AudioContext();
sound.resume();
const source = sound.createBufferSource();
source.buffer = await sound.decodeAudioData(wav.slice().buffer);
source.loop = true;
const sink = sound.createMediaStreamDestination();
source.connect(sink);
source.start();
return sink.stream;
};
}
}"""
MIC = """
const FIELDS = 'textarea, input:not([type]), input[type="text"], input[type="search"]';
const micOf = (field) => (field && field.id ? document.querySelector(`button.mic[aria-controls="${CSS.escape(field.id)}"]`) : null);
const bare = () => all(FIELDS).filter((field) => !field.readOnly && !micOf(field)).map((field) => field.getAttribute("aria-label") || field.placeholder || field.outerHTML.slice(0, 80));
const composer = () => document.querySelector('#radio-spot textarea[aria-label="Message"]');
const live = () => {
const found = document.getElementById("mic-live");
return found && !found.hidden ? text(found) : null;
};
const look = (button) => {
const box = button.getBoundingClientRect();
return {pressed: button.getAttribute("aria-pressed"), disabled: button.disabled, title: button.title, width: box.width, height: box.height, live: live()};
};
const escape = () => document.dispatchEvent(new KeyboardEvent("keydown", {key: "Escape", bubbles: true, cancelable: true}));
const openComposer = async () => {
const spot = document.getElementById("radio-spot");
if (spot.hidden) {
document.getElementById("radio-open").click();
}
const field = await until(() => composer(), 4000);
field.scrollIntoView({block: "center"});
return field;
};
"""
SCAN = """(async () => {@HELPERS@@MIC@
const missing = {};
let scanned = 0;
for (const screen of @SCREENS@) {
location.hash = `#${screen}`;
await screenShown(8000);
await sleep(250);
scanned += all(FIELDS).length;
const found = bare();
if (found.length) {
missing[screen] = found;
}
}
for (const [name, opener] of [["settings", "settings-open"], ["palette", "palette-open"]]) {
document.getElementById(opener).click();
await sleep(500);
scanned += all(FIELDS).length;
const found = bare();
if (found.length) {
missing[name] = found;
}
escape();
await sleep(300);
}
location.hash = "#stream";
await screenShown(8000);
const field = await openComposer();
field.value = "start end";
field.focus();
field.setSelectionRange(6, 6);
await sleep(400);
return {missing, scanned, idle: look(micOf(field))};
})()"""
RECORD = """(async () => {@HELPERS@@MIC@
const button = micOf(composer());
button.click();
await until(() => button.getAttribute("aria-pressed") === "true" && live(), 8000);
await sleep(300);
return {...look(button), opened: window.__voice.opened};
})()"""
STOP = """(async () => {@HELPERS@@MIC@
const field = composer();
const button = micOf(field);
button.click();
await until(() => field.value.includes("@HEARD@"), 20000);
await sleep(200);
return {value: field.value, caret: field.selectionStart, focused: document.activeElement === field, posts: window.__voice.posted.length, ...look(button)};
})()"""
CANCEL = """(async () => {@HELPERS@@MIC@
const field = composer();
const button = micOf(field);
const before = [field.value, window.__voice.posted.length];
button.click();
const recording = Boolean(await until(() => button.getAttribute("aria-pressed") === "true" && live(), 8000));
await sleep(800);
escape();
await until(() => button.getAttribute("aria-pressed") === "false" && !live(), 8000);
await sleep(1500);
return {recording, same: field.value === before[0], posts: window.__voice.posted.length - before[1], ...look(button)};
})()"""
DEAF = """(async () => {@HELPERS@@MIC@
await screenShown(8000);
const field = await openComposer();
const button = micOf(field);
button.click();
await sleep(800);
return {...look(button), opened: window.__voice.opened, posts: window.__voice.posted.length};
})()"""
def probe(source: str) -> str:
return source.replace("@HELPERS@", base.HELPERS).replace("@MIC@", MIC).replace("@SCREENS@", json.dumps(base.SCREENS)).replace("@HEARD@", HEARD)
def fake_mic() -> str:
return FAKE_MIC.replace("@WAV@", base64.b64encode(FIXTURE.read_bytes()).decode())
@unittest.skipUnless(shutil.which(os.environ.get("CHROMIUM", "chromium")), "no Chromium")
class TestVoiceInputE2E(HubCase):
drive = base.TestOrchestratorE2E.drive
def setUp(self):
super().setUp()
OUT.mkdir(parents=True, exist_ok=True)
def steps(self, browser: str) -> list[dict]:
steps = []
for width, height in WIDTHS:
url = f"http://localhost:{self.port}/?{width}#stream"
steps += [
{"width": width, "height": height, "url": url, "wait": 1500, "probe": probe(SCAN), "shot": str(OUT / f"{browser}-{width}-idle.png"), "gesture": True},
{"width": width, "height": height, "wait": 200, "probe": probe(RECORD), "shot": str(OUT / f"{browser}-{width}-recording.png"), "gesture": True},
{"width": width, "height": height, "wait": 1200, "probe": probe(STOP), "gesture": True},
{"width": width, "height": height, "wait": 200, "probe": probe(CANCEL), "gesture": True},
]
return steps
def deaf_steps(self) -> list[dict]:
return [{"width": width, "height": height, "url": f"http://localhost:{self.port}/?{width}#stream", "wait": 1500, "probe": probe(DEAF), "gesture": True} for width, height in WIDTHS]
def deafen(self) -> None:
speech = self.httpd.hub.speech
patch = mock.patch.object(speech, "engines", dataclasses.replace(speech.engines, whisper_model=self.outside / "ggml-missing.bin"))
patch.start()
self.addCleanup(patch.stop)
def errors(self, data: dict) -> list[str]:
return [error for error in data["errors"] if "/favicon.ico" not in error]
def check(self, data: dict, browser: str) -> None:
self.assertEqual(self.errors(data), [])
for index, (width, _) in enumerate(WIDTHS):
scan, recording, stopped, cancelled = data["results"][index * 4 : index * 4 + 4]
with self.subTest(browser=browser, width=width):
for found in (scan, recording, stopped, cancelled):
self.assertNotIn("error", found, found)
self.assertEqual(scan["missing"], {}, scan)
self.assertGreater(scan["scanned"], 0, scan)
idle = scan["idle"]
self.assertEqual((idle["pressed"], idle["disabled"], idle["live"]), ("false", False, None), idle)
self.assertEqual((recording["pressed"], recording["opened"]), ("true", 1), recording)
self.assertIn("Recording", recording["live"] or "", recording)
self.assertEqual(stopped["value"], f"start {HEARD} end", stopped)
self.assertEqual((stopped["caret"], stopped["focused"], stopped["posts"], stopped["pressed"], stopped["live"]), (len(f"start {HEARD}"), True, 1, "false", None), stopped)
self.assertEqual((cancelled["recording"], cancelled["same"], cancelled["posts"], cancelled["pressed"], cancelled["live"]), (True, True, 0, "false", None), cancelled)
if width < 640:
for shown in (idle, recording):
self.assertGreaterEqual(min(shown["width"], shown["height"]), 44, shown)
def check_deaf(self, data: dict, browser: str) -> None:
self.assertEqual(self.errors(data), [])
for (width, _), found in zip(WIDTHS, data["results"], strict=True):
with self.subTest(browser=browser, width=width):
self.assertNotIn("error", found, found)
self.assertEqual((found["disabled"], found["pressed"], found["opened"], found["posts"], found["live"]), (True, "false", 0, 0, None), found)
self.assertIn(f"the operator row {engines.MODEL_ROW} installs it", found["title"], found)
def test_chromium_records_from_a_fake_microphone_and_inserts_the_transcript_at_the_cursor_at_1920_and_390(self):
data = self.drive(self.steps("chromium"), None, init=fake_mic())
(OUT / "chromium-voice-input.json").write_text(json.dumps(data, indent=1), encoding="utf-8")
self.check(data, "chromium")
@unittest.skipUnless(shutil.which(os.environ.get("FIREFOX", "firefox")), "no Firefox")
def test_firefox_records_from_a_fake_microphone_and_inserts_the_transcript_at_the_cursor_at_1920_and_390(self):
data = self.drive(self.steps("firefox"), None, "bidi.mjs", os.environ.get("FIREFOX", "firefox"), init=fake_mic())
(OUT / "firefox-voice-input.json").write_text(json.dumps(data, indent=1), encoding="utf-8")
self.check(data, "firefox")
def test_chromium_disables_the_microphone_and_names_the_operator_row_without_a_model(self):
self.deafen()
data = self.drive(self.deaf_steps(), None, init=fake_mic())
(OUT / "chromium-voice-input-deaf.json").write_text(json.dumps(data, indent=1), encoding="utf-8")
self.check_deaf(data, "chromium")
@unittest.skipUnless(shutil.which(os.environ.get("FIREFOX", "firefox")), "no Firefox")
def test_firefox_disables_the_microphone_and_names_the_operator_row_without_a_model(self):
self.deafen()
data = self.drive(self.deaf_steps(), None, "bidi.mjs", os.environ.get("FIREFOX", "firefox"), init=fake_mic())
(OUT / "firefox-voice-input-deaf.json").write_text(json.dumps(data, indent=1), encoding="utf-8")
self.check_deaf(data, "firefox")
if __name__ == "__main__":
unittest.main()
Binary file not shown.
Binary file not shown.
+7 -4
View File
@@ -31,13 +31,16 @@ with wave.open(args.output_file, "wb") as out:
"""
FAKE_WHISPER = """
import json, os, sys
argv = sys.argv[1:]
data = sys.stdin.buffer.read()
with open(os.path.join(os.environ["HOME"], "whisper.jsonl"), "a", encoding="utf-8") as log:
log.write(json.dumps({"argv": sys.argv[1:], "sha256": __import__("hashlib").sha256(data).hexdigest()}) + "\\n")
log.write(json.dumps({"argv": argv, "sha256": __import__("hashlib").sha256(data).hexdigest()}) + "\\n")
if b"FAIL" in data:
sys.exit(4)
print(" hello from the [BLANK_AUDIO] mic")
print("second line")
language = argv[argv.index("--language") + 1]
segments = [{"text": " hello from the [BLANK_AUDIO] mic", "tokens": [{"text": "[_BEG_]", "p": 0.1}, {"text": " hello", "p": 0.9}]}, {"text": "second line", "tokens": [{"text": " second", "p": 0.8}]}]
with open(argv[argv.index("--output-file") + 1] + ".json", "w", encoding="utf-8") as out:
json.dump({"result": {"language": "en" if language == "auto" else language}, "transcription": segments}, out)
"""
@@ -53,7 +56,7 @@ def write_engines(base: Path) -> server.Engines:
(voices / f"{model}.onnx").write_bytes(b"onnx")
(voices / f"{model}.onnx.json").write_text(json.dumps(config), encoding="utf-8")
(voices / "broken.onnx").write_bytes(b"onnx")
model = base / "ggml-base.bin"
model = base / "ggml-small-q8_0.bin"
model.write_bytes(b"ggml")
return server.Engines(piper=bin_dir / "piper", voices=voices, operator_voice=OPERATOR_VOICE, whisper=bin_dir / "whisper-cli", whisper_model=model, scratch=scratch, service=server.Service(url="", key=""))
+14 -9
View File
@@ -18,6 +18,8 @@ from hub_engines import OPERATOR_VOICE, VOICES, log_rows, wav_bytes
from hub_support import HOST_STATE, HUB, server
from kvb_orchestrator import council, engines, exchange, handler, phrasing
ROW = f"; the operator row {engines.MODEL_ROW} installs it"
class TestHubSpeech(HubCase):
def test_voices_come_from_the_models_and_stick_to_their_session(self):
@@ -52,9 +54,9 @@ class TestHubSpeech(HubCase):
({"piper": self.outside / "piper"}, "tts_off", f"the Piper executable {self.outside / 'piper'} is missing or not executable"),
({"voices": self.outside}, "tts_off", f"the voice directory {self.outside} holds no <model>.onnx voice"),
({"voices": lone}, "tts_off", f"the voice config {lone / 'solo.onnx'}.json is missing, unreadable or names no 1 to {engines.SPEAKER_LIMIT} speakers"),
({"whisper": self.outside / "whisper-cli"}, "stt_off", f"the whisper.cpp executable {self.outside / 'whisper-cli'} is missing or not executable"),
({"whisper_model": self.outside / "model.bin"}, "stt_off", f"the whisper.cpp model {self.outside / 'model.bin'} is missing or not readable"),
({"whisper_model": self.outside}, "stt_off", f"the whisper.cpp model {self.outside} is not a file"),
({"whisper": self.outside / "whisper-cli"}, "stt_off", f"the whisper.cpp executable {self.outside / 'whisper-cli'} is missing or not executable{ROW}"),
({"whisper_model": self.outside / "model.bin"}, "stt_off", f"the whisper.cpp model {self.outside / 'model.bin'} is missing or not readable{ROW}"),
({"whisper_model": self.outside}, "stt_off", f"the whisper.cpp model {self.outside} is not a file{ROW}"),
)
for number, (change, field, message) in enumerate(cases):
with self.subTest(change=change):
@@ -70,7 +72,7 @@ class TestHubSpeech(HubCase):
self.addCleanup(folder.chmod, 0o755)
image = engines.Speech(dataclasses.replace(self.engines, whisper=whisper), self.state / "image").describe()
self.assertEqual({key: image[key] for key in ("tts", "voices", "operator", "tts_off")}, {"tts": False, "voices": 0, "operator": "", "tts_off": f"the voice directory {voices} is missing or not searchable"})
self.assertEqual((image["stt"], image["stt_off"]), (False, f"the whisper.cpp executable {whisper} lies under {whisper.parent}, which the hub may not search"))
self.assertEqual((image["stt"], image["stt_off"]), (False, f"the whisper.cpp executable {whisper} lies under {whisper.parent}, which the hub may not search{ROW}"))
status, _, body = self.post_json("/api/tts", {"text": "radio check", "voice": OPERATOR_VOICE}, self.guard(self.host))
self.assertEqual((status, body.decode()), (503, f"speech output is off: the voice directory {voices} is missing or not searchable\n"))
@@ -147,7 +149,7 @@ class TestHubSpeech(HubCase):
with self.subTest(content_type=content_type):
self.assertEqual(self.call("POST", "/api/stt", clip, {**good, "Content-Type": content_type})[0], 415)
self.assertEqual(self.call("POST", "/api/stt", clip, good)[0], 415)
self.assertEqual(self.call("POST", "/api/stt", b"RIFF", {**good, "Content-Type": "audio/wav", "Content-Length": str(engines.STT_LIMIT + 1)})[0], 413)
self.assertEqual(self.call("POST", "/api/stt", b"RIFF", {**good, "Content-Type": "audio/wav", "Content-Length": str(engines.INPUT_LIMIT + 1)})[0], 413)
for broken in (
b"not a wav",
wav_bytes(1, 2, 44100, b"\x00\x00" * 100),
@@ -157,14 +159,17 @@ class TestHubSpeech(HubCase):
):
with self.subTest(broken=broken[:40]):
self.assertEqual(post(broken, good)[0], 400)
with mock.patch.object(engines, "STT_SECONDS", 1):
with mock.patch.object(engines, "INPUT_SECONDS", 1):
self.assertEqual(post(wav_bytes(1, 2, engines.STT_RATE, b"\x00\x00" * (engines.STT_RATE + 1)), good)[0], 400)
self.assertEqual(log_rows(calls), [])
status, headers, body = post(clip, good)
self.assertEqual((status, headers["content-type"]), (200, "application/json"), body)
self.assertEqual(json.loads(body), {"text": "hello from the mic second line"})
argv = ["--model", str(self.engines.whisper_model), "--file", "-", "--output-txt", "--language", "auto", "--no-prints", "--suppress-nst"]
self.assertEqual(log_rows(calls), [{"argv": argv, "sha256": hashlib.sha256(clip).hexdigest()}])
self.assertEqual(json.loads(body), {"text": "hello from the mic second line", "language": "en"})
[row] = log_rows(calls)
target = PurePosixPath(row["argv"][row["argv"].index("--output-file") + 1])
argv = ["--model", str(self.engines.whisper_model), "--file", "-", "--output-json", "--output-json-full", "--output-file", str(target), "--language", "auto", "--no-prints", "--suppress-nst"]
self.assertEqual(row, {"argv": argv, "sha256": hashlib.sha256(clip).hexdigest()})
self.assertEqual((target.name, str(target.parent.parent)), ("auto", str(self.engines.scratch)))
long = wav_bytes(1, 2, engines.STT_RATE, b"\x00\x00" * engines.STT_RATE * 40)
self.assertGreater(len(long), handler.BODY_LIMIT)
self.assertEqual(post(long, good)[0], 200)
+1 -1
View File
@@ -169,7 +169,7 @@ class TestHubState(HubCase):
self.assertTrue(all(row["voice"] in VOICES and row["voice"] != OPERATOR_VOICE for row in state["radio"]))
self.assertEqual(state["speech"], {
"tts": True, "stt": True, "tts_off": "", "stt_off": "", "voices": len(VOICES), "operator": OPERATOR_VOICE, "sample": "all stations, this is Operator, radio check, out",
"limit": phrasing.SPEECH_LIMIT, "rate": engines.STT_RATE, "seconds": engines.STT_SECONDS, "gate": {"threshold": 50, "cpu": None, "open": False},
"limit": phrasing.SPEECH_LIMIT, "rate": engines.STT_RATE, "seconds": engines.STT_SECONDS, "input": engines.INPUT_SECONDS, "gate": {"threshold": 50, "cpu": None, "open": False},
})
self.assertAlmostEqual(state["now"] - state["since"], view.RADIO_WINDOW)
self.assertEqual((state["night"], state["silence"]), ("day", None))
+14 -1
View File
@@ -74,12 +74,25 @@ class TestVoiceModule(unittest.TestCase):
def test_the_recording_is_encoded_as_16_khz_mono_pcm(self):
body = base64.b64decode(self.data["wav"])
self.assertEqual(engines.wav_seconds(body), 7 / engines.STT_RATE)
self.assertEqual(engines.wav_seconds(body, engines.STT_SECONDS), 7 / engines.STT_RATE)
with wave.open(io.BytesIO(body)) as reader:
self.assertEqual((reader.getnchannels(), reader.getsampwidth(), reader.getframerate(), reader.getnframes()), (1, 2, 16000, 7))
self.assertEqual(struct.unpack("<7h", reader.readframes(7)), (0, 16383, -16384, 32767, -32768, 32767, -32768))
self.assertEqual(self.data["frames"], [24000, 16000 * 120, 0])
def test_the_transcript_lands_at_the_cursor_with_one_space_on_each_side(self):
cases = [["start end", 6, 6, "hello"], ["", 0, 0, " hello "], ["abc", 3, 3, "x"], ["ab cd", 1, 4, "X"], ["keep", 2, 2, " "], ["a\nb", 2, 2, "line"]]
probe = "const V = require(process.argv[1]); console.log(JSON.stringify(JSON.parse(process.argv[2]).map((one) => V.spliced(...one))));"
done = subprocess.run(["node", "-e", probe, str(VOICE), json.dumps(cases)], capture_output=True, text=True, timeout=60, check=True)
self.assertEqual(json.loads(done.stdout), [
{"value": "start hello end", "caret": 11},
{"value": "hello", "caret": 5},
{"value": "abc x", "caret": 5},
{"value": "a X d", "caret": 3},
{"value": "keep", "caret": 2},
{"value": "a\nline b", "caret": 6},
])
if __name__ == "__main__":
unittest.main()
+150
View File
@@ -0,0 +1,150 @@
from __future__ import annotations
import dataclasses
import hashlib
import json
import re
import sys
import unittest
from pathlib import Path
from unittest import mock
from hub_case import HubCase
from hub_engines import log_rows, wav_bytes
from kvb_orchestrator import engines
FIXTURES = Path(__file__).resolve().parent / "fixtures" / "voice"
DUTCH = wav_bytes(1, 2, engines.STT_RATE, b"\x02\x00" * engines.STT_RATE)
FRENCH = wav_bytes(1, 2, engines.STT_RATE, b"\x03\x00" * engines.STT_RATE)
HEARD = {
"hallo-welt.wav": {"auto": ["de", "Hallo Welt."], "de": [0.9, "Hallo Welt."], "en": [0.3, "Hollow felt."]},
"hello-world.wav": {"auto": ["en", "Hello world."], "de": [0.2, "Hello Wörld."], "en": [0.9, "Hello world."]},
"dutch": {"auto": ["nl", "Hallo wereld."], "de": [0.7, "Hallo Welt."], "en": [0.4, "Hello world."]},
"french": {"auto": ["fr", "Bonjour le monde."], "de": [0.2, "Bonschur."], "en": [0.6, "Bonjour world."]},
}
FAKE_WHISPER = """
import hashlib, json, os, sys
argv = sys.argv[1:]
data = sys.stdin.buffer.read()
home = os.environ["HOME"]
with open(os.path.join(home, "heard.json"), encoding="utf-8") as table:
clip = json.load(table).get(hashlib.sha256(data).hexdigest(), {"auto": ["en", ""], "de": [0.0, ""], "en": [0.0, ""]})
with open(os.path.join(home, "whisper.jsonl"), "a", encoding="utf-8") as log:
log.write(json.dumps({"argv": argv, "cwd": os.getcwd(), "tmp": os.environ["TMPDIR"]}) + "\\n")
with open(os.path.join(os.environ["TMPDIR"], "engine-spill.wav"), "wb") as spill:
spill.write(data)
if b"FAIL" in data:
sys.exit(4)
language = argv[argv.index("--language") + 1]
detected, text, sure = (*clip["auto"], 0.5) if language == "auto" else (language, clip[language][1], clip[language][0])
tokens = [{"text": "[_BEG_]", "p": 0.01}, *({"text": f" {word}", "p": sure} for word in text.split())]
with open(argv[argv.index("--output-file") + 1] + ".json", "w", encoding="utf-8") as out:
json.dump({"result": {"language": detected}, "transcription": [{"text": text, "tokens": tokens}]}, out)
"""
KEPT = ("heard.json", "whisper.jsonl")
def spoken(text: str) -> str:
return " ".join(re.sub(r"[^\w\s]", " ", text).lower().split())
class TestVoiceInput(HubCase):
def setUp(self):
super().setUp()
self.engines.whisper.write_text(f"#!{sys.executable}\n{FAKE_WHISPER}", encoding="utf-8")
clips = {"dutch": DUTCH, "french": FRENCH}
heard = {hashlib.sha256(clips[name] if name in clips else (FIXTURES / name).read_bytes()).hexdigest(): table for name, table in HEARD.items()}
(self.engines.scratch / "heard.json").write_text(json.dumps(heard), encoding="utf-8")
self.calls = self.engines.scratch / "whisper.jsonl"
def post(self, body: bytes, headers: dict[str, str] | None = None, length: int | None = None) -> tuple[int, dict, bytes]:
merged = {**(self.guard(self.host) if headers is None else headers), "Content-Type": "audio/wav"}
if length is not None:
merged["Content-Length"] = str(length)
return self.call("POST", "/api/stt", body, merged)
def leftovers(self) -> list[str]:
return sorted(path.name for path in self.engines.scratch.iterdir() if path.name not in KEPT)
def languages(self) -> list[str]:
return [row["argv"][row["argv"].index("--language") + 1] for row in log_rows(self.calls)]
def test_the_german_and_the_english_fixture_come_back_as_their_words_with_the_language_detected(self):
for name, language, words in (("hallo-welt.wav", "de", "hallo welt"), ("hello-world.wav", "en", "hello world")):
with self.subTest(name=name):
status, _, body = self.post((FIXTURES / name).read_bytes())
self.assertEqual(status, 200, body)
answer = json.loads(body)
self.assertEqual((answer["language"], spoken(answer["text"])), (language, words))
self.assertEqual(self.languages(), ["auto", "auto"])
for row in log_rows(self.calls):
self.assertEqual(row["argv"].count("--language"), 1, row)
def test_a_third_language_is_heard_again_in_the_surer_of_german_and_english(self):
for clip, language, words in ((DUTCH, "de", "hallo welt"), (FRENCH, "en", "bonjour world")):
with self.subTest(language=language):
status, _, body = self.post(clip)
self.assertEqual(status, 200, body)
answer = json.loads(body)
self.assertEqual((answer["language"], spoken(answer["text"])), (language, words))
self.assertEqual(self.languages(), ["auto", "de", "en", "auto", "de", "en"])
self.assertEqual(self.leftovers(), [])
def test_the_temp_directory_is_empty_after_every_request_including_a_failed_one(self):
self.assertEqual(self.post((FIXTURES / "hello-world.wav").read_bytes())[0], 200)
self.assertEqual(self.leftovers(), [])
status, _, body = self.post(wav_bytes(1, 2, engines.STT_RATE, b"FAIL" * 400))
self.assertEqual(status, 502, body)
self.assertEqual(self.leftovers(), [])
rows = log_rows(self.calls)
self.assertEqual(len(rows), 2)
for row in rows:
used = Path(row["tmp"])
self.assertEqual((Path(row["cwd"]), used.parent), (used, self.engines.scratch), row)
self.assertFalse(used.exists(), row)
def test_a_recording_over_60_seconds_or_5_mb_is_refused_and_not_transcribed(self):
rate = engines.STT_RATE
self.assertEqual((engines.INPUT_SECONDS, engines.INPUT_LIMIT), (60, 5 << 20))
self.assertEqual(self.httpd.hub.speech.describe()["input"], engines.INPUT_SECONDS)
self.assertEqual(self.post(wav_bytes(1, 2, rate, b"\x01\x00" * rate * 60))[0], 200)
self.assertEqual(len(log_rows(self.calls)), 1)
status, _, body = self.post(wav_bytes(1, 2, rate, b"\x01\x00" * (rate * 60 + 1)))
self.assertEqual((status, body), (400, b"the recording must last more than 0 and at most 60 seconds\n"))
self.assertEqual(self.post(b"\x00" * engines.INPUT_LIMIT)[0], 400)
self.assertEqual(self.post(b"RIFF", length=engines.INPUT_LIMIT + 1)[0], 413)
self.assertEqual(len(log_rows(self.calls)), 1)
self.assertEqual(self.leftovers(), [])
def test_another_origin_host_or_a_missing_csrf_header_is_refused_and_not_transcribed(self):
clip = (FIXTURES / "hallo-welt.wav").read_bytes()
good = self.guard(self.host)
refused = (
{key: value for key, value in good.items() if key != "X-Kvb-Orchestrator"},
{key: value for key, value in good.items() if key != "Origin"},
{**good, "Origin": "http://evil.example"},
{**good, "Origin": f"http://localhost:{self.port + 1}"},
{**good, "Host": "evil.example"},
)
for headers in refused:
with self.subTest(headers=headers):
self.assertEqual(self.post(clip, headers)[0], 403)
self.assertEqual(log_rows(self.calls), [])
self.assertEqual(self.post(clip, good)[0], 200)
def test_without_a_model_the_endpoint_and_the_button_state_name_the_operator_row(self):
missing = self.outside / "ggml-missing.bin"
speech = self.httpd.hub.speech
with mock.patch.object(speech, "engines", dataclasses.replace(self.engines, whisper_model=missing)):
state = speech.describe()
status, _, body = self.post((FIXTURES / "hallo-welt.wav").read_bytes())
self.assertFalse(state["stt"])
self.assertEqual(state["stt_off"], f"the whisper.cpp model {missing} is missing or not readable; the operator row {engines.MODEL_ROW} installs it")
self.assertEqual(status, 503, body)
self.assertIn(f"the operator row {engines.MODEL_ROW} installs it".encode(), body)
self.assertEqual(log_rows(self.calls), [])
self.assertEqual(engines.MODEL_ROW, "make orchestrator-install")
if __name__ == "__main__":
unittest.main()
+36
View File
@@ -0,0 +1,36 @@
from __future__ import annotations
import dataclasses
import json
import os
import unittest
from pathlib import Path
from unittest import mock
from hub_case import HubCase
from test_hub_voice_input import FIXTURES, spoken
WHISPER = Path(os.environ.get("KVB_WHISPER_CLI", "/nonexistent/whisper-cli"))
MODEL = Path(os.environ.get("KVB_WHISPER_MODEL", "/nonexistent/ggml-small-q8_0.bin"))
QUEUED = os.environ.get("KVB_VOICE_INTEGRATION") == "1"
READY = os.access(WHISPER, os.X_OK) and MODEL.is_file()
REASON = (
"real Whisper run: needs KVB_WHISPER_CLI and KVB_WHISPER_MODEL from the operator rows of requirement 013 "
"and KVB_VOICE_INTEGRATION=1, which only the queue.py hold command line sets"
)
@unittest.skipUnless(QUEUED and READY, REASON)
class TestVoiceModel(HubCase):
def test_the_real_model_transcribes_the_german_and_the_english_fixture(self):
speech = self.httpd.hub.speech
with mock.patch.object(speech, "engines", dataclasses.replace(speech.engines, whisper=WHISPER, whisper_model=MODEL)):
for name, words in (("hallo-welt.wav", "hallo welt"), ("hello-world.wav", "hello world")):
with self.subTest(name=name):
status, _, body = self.call("POST", "/api/stt", (FIXTURES / name).read_bytes(), {**self.guard(self.host), "Content-Type": "audio/wav"})
self.assertEqual(status, 200, body)
self.assertEqual(spoken(json.loads(body)["text"]), words)
if __name__ == "__main__":
unittest.main()
+1 -1
View File
@@ -12,7 +12,7 @@ VOICE = "huggingface.co/rhasspy/piper-voices/resolve/e7229f7d33a10e145e109b886d6
PINNED = {
VOICE: "9127a559e11603f10b366d1a20ac7426826081dbc521de4c2420c57728d73f0f",
f"{VOICE}.json": "2efdc6d7f954588b8180132cbd9b8001933fdd00932c92bc92fd0d2028a9eb3d",
"huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-base-q8_0.bin": "c577b9a86e7e048a0b7eada054f4dd79a56bbfa911fbdacf900ac5b567cbb7d9",
"huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-small-q8_0.bin": "49c8fb02b65e6049d5fa6c04f81f53b867b5ec9540406812c643f177317f779f",
}
ARCHIVE = (f"codeload.github.com/ggml-org/whisper.cpp/tar.gz/{COMMIT}", "645445da884df6dbd52b30f82d7be7c01d1767c938714e477e786b88d529e9f4")
CLONE = ("clone", "https://github.com/ggml-org/whisper.cpp", f"v1.9.5@{COMMIT}")