Merge branch 'feature/hub-voice-input' into unify/skills-20261114

# Conflicts:
#	README.md
#	docs/requirements/README.md
This commit is contained in:
Kevin Veen-Birkenbach committed 2026-11-14 22:03:42 +00:00
commit b95e305db0
27 files changed
+856 -144

No files matched your search

+1 -1
View File
@@ -468,7 +468,7 @@ Both commands also patch the target's `.claude/settings.local.json` — the mach
The council root is the one directory every `kvb-org` script, hook and the hub share: the register, radio, queue, claims, operator rows, score journal, gate receipts, security cache and night state, plus `pages/` (the `kvb-git-diff` review pages the hub reads), `approvals/` (the replay stores of `approval.py verify`) and `gates/` (gate logs). `skills/kvb-org-organize/scripts/councilroot.py` alone resolves it: `$AGENTS_COUNCIL_ROOT`, else `$XDG_STATE_HOME/agents-shared`, else `~/.local/state/agents-shared`, so it survives a reboot; `councilroot.py root` prints it and `councilroot.py dir <part>` creates and prints a part. It and `agents-release` below are the only state paths a sandbox's `allowWrite` may name. Its sibling `~/.local/state/agents-council` (`councilroot.py durable`) is host-only and never belongs in `allowWrite`: only hooks, the runner and the systemd units write it (`gates.db`, the watch hook's copies of the receipt, parent, acceptance and queue-history journals, `queue-cleanup.json`, `resources/`, `subagent-model.jsonl`), and the hub mounts it read-only; no directory is both agent-writable and host-written except the council root, where every host writer opens files through `nofollow.py`. The `kvb-release` drafts lie in `agents-release/<version>` beside both (`councilroot.py release <version>`). The global install first runs `councilroot.py preflight` and stops before it copies anything unless `~/.claude/settings.json` lists both `agents-shared` and `agents-release` in `sandbox.filesystem.allowWrite`, no `allowWrite` entry reaches `agents-council` in the managed file, its `managed-settings.d` drop-ins, the user or user-local file (the list `councilroot.py` shares with the queue's permission check), in a project `.claude/settings.json` or `.claude/settings.local.json` below the pkgmgr repositories and worktrees directories (directory and file links followed), or in the start directory of any session listed in `~/.claude/projects`, and the move below would succeed; it warns while the agents are not suspended. Fixing those entries is the operator's step. It then builds the skills, stops the hub, the runner, the offices, resources and queue-cleanup units, runs `councilroot.py migrate`, which moves an earlier council root in `/tmp/agents-council` into the council root, copies the skills, runs `migrate --catch-up` for what the old skills wrote meanwhile, and starts the units again; the hub image is built before the container is replaced, and a failed build leaves the hub down and fails the install until `bash scripts/orchestrator.sh install` succeeds. Migrate refuses while a lock there is held or a queue job waits, and, checked under those locks, moves nothing (exit 4) while a file cannot be moved or a whole file both roots hold differs; the install then stops with the earlier root intact. Each file is renamed to a `.migrating` name before it is copied, so a writer that opens the old name meanwhile makes a new file the catch-up moves. An identical file leaves one copy, a `.jsonl` journal gets the rows it lacks (by id, else each exact line as often as the earlier root holds it more often, in `ts` order), links are skipped, and a second run moves nothing and duplicates no row. The catch-up keeps a differing whole file beside its namesake as `<name>.legacy-<hash>` (exit 3); a file that fails after the check stays behind (exit 5). After exit 3 or 5 the install goes on so the new skills read the new root, names every such file and exits 1 at the end. `~/.local/state/agents-council/migrate.started` exists from the first move to the end of a run; when a crash leaves it, the install copies the new skills anyway, names it and exits 1, and the next completed migrate removes it. Because the council root survives reboots, each `kvb-org-queue-cleanup` round first runs `councilroot.py retire`, which moves gate logs and review pages older than 30 days into `archive/<date>/`, and the radio appends the rows it drops past its last 500 to `archive/news-<date>.jsonl`; no reader opens `archive/`, which `kvb-cleanup` offers for deletion. Run the install with every agent suspended (`kvb-org-suspend`) and the queue empty, so nothing writes either root during the move. The council root is the one directory every `kvb-org` script, hook and the hub share: the register, radio, queue, claims, operator rows, score journal, gate receipts, security cache and night state, plus `pages/` (the `kvb-git-diff` review pages the hub reads), `approvals/` (the replay stores of `approval.py verify`) and `gates/` (gate logs). `skills/kvb-org-organize/scripts/councilroot.py` alone resolves it: `$AGENTS_COUNCIL_ROOT`, else `$XDG_STATE_HOME/agents-shared`, else `~/.local/state/agents-shared`, so it survives a reboot; `councilroot.py root` prints it and `councilroot.py dir <part>` creates and prints a part. It and `agents-release` below are the only state paths a sandbox's `allowWrite` may name. Its sibling `~/.local/state/agents-council` (`councilroot.py durable`) is host-only and never belongs in `allowWrite`: only hooks, the runner and the systemd units write it (`gates.db`, the watch hook's copies of the receipt, parent, acceptance and queue-history journals, `queue-cleanup.json`, `resources/`, `subagent-model.jsonl`), and the hub mounts it read-only; no directory is both agent-writable and host-written except the council root, where every host writer opens files through `nofollow.py`. The `kvb-release` drafts lie in `agents-release/<version>` beside both (`councilroot.py release <version>`). The global install first runs `councilroot.py preflight` and stops before it copies anything unless `~/.claude/settings.json` lists both `agents-shared` and `agents-release` in `sandbox.filesystem.allowWrite`, no `allowWrite` entry reaches `agents-council` in the managed file, its `managed-settings.d` drop-ins, the user or user-local file (the list `councilroot.py` shares with the queue's permission check), in a project `.claude/settings.json` or `.claude/settings.local.json` below the pkgmgr repositories and worktrees directories (directory and file links followed), or in the start directory of any session listed in `~/.claude/projects`, and the move below would succeed; it warns while the agents are not suspended. Fixing those entries is the operator's step. It then builds the skills, stops the hub, the runner, the offices, resources and queue-cleanup units, runs `councilroot.py migrate`, which moves an earlier council root in `/tmp/agents-council` into the council root, copies the skills, runs `migrate --catch-up` for what the old skills wrote meanwhile, and starts the units again; the hub image is built before the container is replaced, and a failed build leaves the hub down and fails the install until `bash scripts/orchestrator.sh install` succeeds. Migrate refuses while a lock there is held or a queue job waits, and, checked under those locks, moves nothing (exit 4) while a file cannot be moved or a whole file both roots hold differs; the install then stops with the earlier root intact. Each file is renamed to a `.migrating` name before it is copied, so a writer that opens the old name meanwhile makes a new file the catch-up moves. An identical file leaves one copy, a `.jsonl` journal gets the rows it lacks (by id, else each exact line as often as the earlier root holds it more often, in `ts` order), links are skipped, and a second run moves nothing and duplicates no row. The catch-up keeps a differing whole file beside its namesake as `<name>.legacy-<hash>` (exit 3); a file that fails after the check stays behind (exit 5). After exit 3 or 5 the install goes on so the new skills read the new root, names every such file and exits 1 at the end. `~/.local/state/agents-council/migrate.started` exists from the first move to the end of a run; when a crash leaves it, the install copies the new skills anyway, names it and exits 1, and the next completed migrate removes it. Because the council root survives reboots, each `kvb-org-queue-cleanup` round first runs `councilroot.py retire`, which moves gate logs and review pages older than 30 days into `archive/<date>/`, and the radio appends the rows it drops past its last 500 to `archive/news-<date>.jsonl`; no reader opens `archive/`, which `kvb-cleanup` offers for deletion. Run the install with every agent suspended (`kvb-org-suspend`) and the queue empty, so nothing writes either root during the move.
The global install also runs the Agent Orchestrator, a local review page in which the operator approves or rejects the agents' staged commits (the `kvb-git-diff` skill): it copies `orchestrator/` to `~/.local/share/kvb-orchestrator`, starts it there with `docker compose up -d --build` (restart `unless-stopped`, bound to `127.0.0.1` only, state in `~/.local/state/kvb-orchestrator`) and registers its MCP server for Claude Code at user scope. Its port is `KVB_ORCHESTRATOR_PORT` in `orchestrator/.env`; the hub page is `http://localhost:<port>/`, the MCP endpoint `/mcp`. Without docker the hub is skipped and the commit skills ask in chat. The hub reads new radio traffic aloud in maritime phraseology and takes push-to-talk input; its image builds both engines from pinned, sha256-verified sources and runs each as a separate process: Piper (`piper-tts`, GPL-3.0-or-later) with the `en_US-libritts-high` voice (904 speakers, trained on LibriTTS, CC BY 4.0, from `rhasspy/piper-voices`) and whisper.cpp (MIT) with the multilingual `ggml-base-q8_0` model (OpenAI Whisper weights, MIT, 82 MB). The operator's own voice is `KVB_ORCHESTRATOR_OPERATOR_VOICE` in `orchestrator/.env`. Its Tree reads only the file `pkgmgr branch tree` writes: the install clones `PKGMGR_REPO` (default the GitHub repository, never the sandbox-writable local checkout), builds the newest `v*` tag a fully trusted GPG key signed into `~/.local/share/kvb-orchestrator/pkgmgr`, and enables the `kvb-orchestrator-tree.timer` user timer, which writes `~/.local/state/pkgmgr/tree/branch-tree.json` a minute after its previous run while the hub page polls, on at most half a core; the hub mounts that directory read-only and starts no git. A signed tag without `branch tree` leaves the timer off. The global install also runs the Agent Orchestrator, a local review page in which the operator approves or rejects the agents' staged commits (the `kvb-git-diff` skill): it copies `orchestrator/` to `~/.local/share/kvb-orchestrator`, starts it there with `docker compose up -d --build` (restart `unless-stopped`, bound to `127.0.0.1` only, state in `~/.local/state/kvb-orchestrator`) and registers its MCP server for Claude Code at user scope. Its port is `KVB_ORCHESTRATOR_PORT` in `orchestrator/.env`; the hub page is `http://localhost:<port>/`, the MCP endpoint `/mcp`. Without docker the hub is skipped and the commit skills ask in chat. The hub reads new radio traffic aloud in maritime phraseology and takes push-to-talk input; its image builds both engines from pinned, sha256-verified sources and runs each as a separate process: Piper (`piper-tts`, GPL-3.0-or-later) with the `en_US-libritts-high` voice (904 speakers, trained on LibriTTS, CC BY 4.0, from `rhasspy/piper-voices`) and whisper.cpp (MIT) with the multilingual `ggml-small-q8_0` model (OpenAI Whisper weights, MIT, 264 MB). The operator's own voice is `KVB_ORCHESTRATOR_OPERATOR_VOICE` in `orchestrator/.env`. Its Tree reads only the file `pkgmgr branch tree` writes: the install clones `PKGMGR_REPO` (default the GitHub repository, never the sandbox-writable local checkout), builds the newest `v*` tag a fully trusted GPG key signed into `~/.local/share/kvb-orchestrator/pkgmgr`, and enables the `kvb-orchestrator-tree.timer` user timer, which writes `~/.local/state/pkgmgr/tree/branch-tree.json` a minute after its previous run while the hub page polls, on at most half a core; the hub mounts that directory read-only and starts no git. A signed tag without `branch tree` leaves the timer off.
Stream → Radio (`#stream/radio`) is the hub's radio interface. Its station list shows one line per station, the full name in the tooltip, with its state and a play or stop button; when the browser blocks autoplay, the stream fails or the CSP refuses its origin, the row and a toast say so, and one click starts the station. New stations play daily around the clock. While the hub reads a message aloud, the speaking agent pulses there and in the Orbit and Chord views, under the caption "<Agent> speaking" with the message's first line; under `prefers-reduced-motion` the ring stays still. The speech bubble icon in the header (a hub setting, kept across reloads) puts a bubble at each speaking agent in the Agents graph, 3D, Hierarchy, Radio and Stream views, or in a caption bar at the bottom when the view does not show that agent: the agent being read gets the largest bubble and its words appear in time with the audio, waiting messages show a smaller `…` bubble per agent, and without speech output the same rows run at 180 words a minute, at least 1.5 s each. The composer sends on all seven channels: post and call to one agent, newsletter, broadcast, pan-pan, security and mayday to all. "Record" records with MediaRecorder; the hub transcribes the recording with whisper.cpp off the request thread (at most 120 s of audio, a 120 s timeout, four at a time), keeps it for replay in `~/.local/state/kvb-orchestrator/voice` (the newest 50) and puts the transcript into the field to edit; the message goes out as text, with the recording's host path and sha256 only when "Attach recording" is ticked. The hub transcribes at most six recordings a minute and 600 s of audio an hour. Every operator message is passkey-signed, and the hub verifies the signature against the pinned keys, each once, before it relays; `operator_message(id, text, to)` answers `match` only for the signed text and recipient; for a denial the signature covers the recipient the inbox resolved at signing, and the answer's text is only the signed reason, never the agent's own words the radio row quotes. "Cancel mayday", Evening, Sunset, Sunrise and a settings save that adds a stream URL take the passkey the same way. Stream → Radio (`#stream/radio`) is the hub's radio interface. Its station list shows one line per station, the full name in the tooltip, with its state and a play or stop button; when the browser blocks autoplay, the stream fails or the CSP refuses its origin, the row and a toast say so, and one click starts the station. New stations play daily around the clock. While the hub reads a message aloud, the speaking agent pulses there and in the Orbit and Chord views, under the caption "<Agent> speaking" with the message's first line; under `prefers-reduced-motion` the ring stays still. The speech bubble icon in the header (a hub setting, kept across reloads) puts a bubble at each speaking agent in the Agents graph, 3D, Hierarchy, Radio and Stream views, or in a caption bar at the bottom when the view does not show that agent: the agent being read gets the largest bubble and its words appear in time with the audio, waiting messages show a smaller `…` bubble per agent, and without speech output the same rows run at 180 words a minute, at least 1.5 s each. The composer sends on all seven channels: post and call to one agent, newsletter, broadcast, pan-pan, security and mayday to all. "Record" records with MediaRecorder; the hub transcribes the recording with whisper.cpp off the request thread (at most 120 s of audio, a 120 s timeout, four at a time), keeps it for replay in `~/.local/state/kvb-orchestrator/voice` (the newest 50) and puts the transcript into the field to edit; the message goes out as text, with the recording's host path and sha256 only when "Attach recording" is ticked. The hub transcribes at most six recordings a minute and 600 s of audio an hour. Every operator message is passkey-signed, and the hub verifies the signature against the pinned keys, each once, before it relays; `operator_message(id, text, to)` answers `match` only for the signed text and recipient; for a denial the signature covers the recipient the inbox resolved at signing, and the answer's text is only the signed reason, never the agent's own words the radio row quotes. "Cancel mayday", Evening, Sunset, Sunrise and a settings save that adds a stream URL take the passkey the same way.
+90
View File
@@ -0,0 +1,90 @@
# 013 - Voice Input
## User Story
As the operator, I want a microphone on every text field of the hub that
turns my speech into text on this machine, so that I can work without a
keyboard on a desktop and on a small screen.
## Decisions (operator-confirmed 2026-11-14; do not re-litigate)
1. **Everywhere**: one shared microphone component sits on every `textarea`
and text input of the hub (composer, radio, feedback popup, marks,
filters, settings), present by default. Rejected: an opt-in per field.
2. **Local Whisper**: the browser records with MediaRecorder and sends the
audio to the hub on this machine, which transcribes it with a local
Whisper model; the audio never leaves the machine and is deleted right
after the transcription. Rejected: the browser's speech recognition
(Chrome sends audio to Google, Firefox has none); a browser fallback.
3. **Insert**: the text lands at the cursor; German and English are detected
automatically; a second tap stops the recording, Escape cancels it, and a
visible indicator shows while it records.
4. **Model**: the one-time model download is an operator row; without a
model the button is disabled and its tooltip names that row.
5. **Limits**: a recording is capped at 60 s and 5 MB; the transcription
endpoint accepts requests only from the hub page, with the same origin
and CSRF checks as every other POST of the hub.
6. **Responsive**: touch targets of at least 44 px at 390 px; the component
works with mouse, touch and keyboard.
7. **Order**: built in parallel with 012 in its own sidetree; 012's
feedback popup and marks get the microphone through the shared component.
## Approach
```mermaid
flowchart TD
subgraph P["🧭 0 · Preparation"]
P1["📋 Write requirement"] --> P2{"✅ Operator approves?"}
P2 -- "✏️ corrections" --> P1
end
P2 -- "🤖 yes" --> S1
subgraph S["🗣️ A · Local transcription"]
S1["📥 Operator row: model download"] --> S2["🧩 /api/stt: caps, origin, CSRF, temp cleanup"]
S2 --> S3["🧠 Local Whisper, German and English"]
S3 --> ST{"🧪 Unit and fixture audio green?"}
ST -- "🔴 no: fix" --> S2
end
ST -- "🟢 yes" --> M1
subgraph M["🎙️ B · Microphone everywhere"]
M1["🧩 Shared component on every text field"] --> M2["⏺️ Record, stop, cancel, indicator"]
M2 --> M3["✍️ Insert at the cursor"]
M3 --> MT{"🧪 E2E with a fake microphone green?"}
MT -- "🔴 no: fix" --> M1
MT -- "🟢 yes" --> MF{"💬 Operator feedback on screenshots?"}
MF -- "✏️ changes" --> M1
end
MF -- "👍 fits" --> E1["🏁 Review, commit with operator OK, hand-over"]
```
## Acceptance Criteria
### Transcription
- [ ] A POST of the German fixture WAV to `/api/stt` returns "hallo welt",
the English fixture returns "hello world".
- [ ] The transcription's temp directory is empty after each request,
including a failed one.
- [ ] A request from another origin, without the CSRF token, over 60 s or
over 5 MB is refused with 4xx and is not transcribed.
- [ ] Without an installed model, `/api/stt` answers with the operator row
that installs it, and the button is disabled with that row in its
tooltip.
### Microphone
- [ ] A DOM scan of every hub view finds no `textarea` or text input without
the microphone button.
- [ ] Pressing the button records, a second press stops and the transcript
appears at the cursor; Escape cancels without inserting text; an
indicator shows while it records.
- [ ] At 390 px the button's touch target is at least 44 by 44 px.
### Tests
- [ ] Unit: the endpoint caps, the origin and CSRF checks, the temp cleanup
and the language selection.
- [ ] Integration: the real model transcribes both fixture files, run
through `queue.py hold`.
- [ ] E2E: Chromium and Firefox with a fake microphone stream fed from a
WAV file, at 1920 and 390 px; the transcript appears in the composer;
negative control: with the model missing the button is disabled.
+1
View File
@@ -16,4 +16,5 @@ skill defines the format.
- [010 - Security Scan Cache per File](010-security-scan-cache-per-file.md) - [010 - Security Scan Cache per File](010-security-scan-cache-per-file.md)
- [011 - Test Result Cache](011-test-result-cache.md) - [011 - Test Result Cache](011-test-result-cache.md)
- [012 - Match Review](012-match-review.md) - [012 - Match Review](012-match-review.md)
- [013 - Voice Input](013-voice-input.md)
- [014 - Header Rates](014-header-rates.md) - [014 - Header Rates](014-header-rates.md)
+3 -3
View File
@@ -50,9 +50,9 @@ ADD --checksum=sha256:2efdc6d7f954588b8180132cbd9b8001933fdd00932c92bc92fd0d2028
${INFINITO_ARTEFACT_BASE:-https:/}/huggingface.co/rhasspy/piper-voices/resolve/e7229f7d33a10e145e109b886d661984201ac3c4/en/en_US/libritts/high/en_US-libritts-high.onnx.json \ ${INFINITO_ARTEFACT_BASE:-https:/}/huggingface.co/rhasspy/piper-voices/resolve/e7229f7d33a10e145e109b886d661984201ac3c4/en/en_US/libritts/high/en_US-libritts-high.onnx.json \
/opt/voices/en_US-libritts-high.onnx.json /opt/voices/en_US-libritts-high.onnx.json
# hadolint ignore=DL3020 # hadolint ignore=DL3020
ADD --checksum=sha256:c577b9a86e7e048a0b7eada054f4dd79a56bbfa911fbdacf900ac5b567cbb7d9 --chmod=0644 \ ADD --checksum=sha256:49c8fb02b65e6049d5fa6c04f81f53b867b5ec9540406812c643f177317f779f --chmod=0644 \
${INFINITO_ARTEFACT_BASE:-https:/}/huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-base-q8_0.bin \ ${INFINITO_ARTEFACT_BASE:-https:/}/huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-small-q8_0.bin \
/opt/whisper/ggml-base-q8_0.bin /opt/whisper/ggml-small-q8_0.bin
COPY --from=whisper-cli /out/whisper-cli /opt/whisper/whisper-cli COPY --from=whisper-cli /out/whisper-cli /opt/whisper/whisper-cli
# ADD --chmod=0644 also gives the directories it creates mode 0644; without the search bit the non-root hub finds no voice and no whisper.cpp # ADD --chmod=0644 also gives the directories it creates mode 0644; without the search bit the non-root hub finds no voice and no whisper.cpp
RUN chmod 0755 /opt/voices /opt/whisper RUN chmod 0755 /opt/voices /opt/whisper
+1 -1
View File
@@ -77,7 +77,7 @@ services:
- --whisper - --whisper
- /opt/whisper/whisper-cli - /opt/whisper/whisper-cli
- --whisper-model - --whisper-model
- /opt/whisper/ggml-base-q8_0.bin - /opt/whisper/ggml-small-q8_0.bin
- --scratch - --scratch
- /scratch - /scratch
- --speech-url - --speech-url
+69 -12
View File
@@ -1,11 +1,13 @@
from __future__ import annotations from __future__ import annotations
import io import io
import json
import os import os
import re import re
import secrets import secrets
import struct import struct
import subprocess import subprocess
import tempfile
import threading import threading
import time import time
import wave import wave
@@ -30,6 +32,12 @@ STT_RATE = 16000
STT_SECONDS = 120 STT_SECONDS = 120
STT_LIMIT = 44 + STT_RATE * 2 * STT_SECONDS STT_LIMIT = 44 + STT_RATE * 2 * STT_SECONDS
STT_TIMEOUT = 120 STT_TIMEOUT = 120
INPUT_SECONDS = 60
INPUT_LIMIT = 5 << 20
MODEL_ROW = "make orchestrator-install"
HEARING_DIR = "stt-"
LANGUAGES = ("de", "en")
SPECIAL = "[_"
HEARING_WINDOW = 60 HEARING_WINDOW = 60
HEARING_RATE = 6 HEARING_RATE = 6
HEARING_HOUR = 3600 HEARING_HOUR = 3600
@@ -100,7 +108,15 @@ def voice_of(key: str, pool: list[str], operator: str) -> str:
return pool[int(digest(key), 16) % len(pool)] return pool[int(digest(key), 16) % len(pool)]
def wav_seconds(body: bytes) -> float: def wav_seconds(body: bytes, seconds: int) -> float:
"""Args:
body: the recording.
seconds: the longest recording accepted.
Returns:
Its length in seconds once it is 16 kHz mono 16-bit PCM WAV lasting
more than 0 and at most seconds; otherwise it raises ToolError.
"""
try: try:
with wave.open(io.BytesIO(body), "rb") as reader: with wave.open(io.BytesIO(body), "rb") as reader:
shape = (reader.getnchannels(), reader.getsampwidth(), reader.getframerate(), reader.getcomptype()) shape = (reader.getnchannels(), reader.getsampwidth(), reader.getframerate(), reader.getcomptype())
@@ -109,8 +125,8 @@ def wav_seconds(body: bytes) -> float:
raise ToolError("the recording must be a WAV file") from None raise ToolError("the recording must be a WAV file") from None
if shape != (1, 2, STT_RATE, "NONE"): if shape != (1, 2, STT_RATE, "NONE"):
raise ToolError(f"the recording must be {STT_RATE} Hz mono 16-bit PCM") raise ToolError(f"the recording must be {STT_RATE} Hz mono 16-bit PCM")
if not 0 < frames <= STT_RATE * STT_SECONDS: if not 0 < frames <= STT_RATE * seconds:
raise ToolError(f"the recording must last more than 0 and at most {STT_SECONDS} seconds") raise ToolError(f"the recording must last more than 0 and at most {seconds} seconds")
return frames / STT_RATE return frames / STT_RATE
@@ -156,6 +172,27 @@ def heard(output: str) -> str:
return " ".join(NOISE.sub(" ", output).split())[:TEXT_LIMIT] return " ".join(NOISE.sub(" ", output).split())[:TEXT_LIMIT]
def transcript(path: Path) -> tuple[str, str, float]:
"""Args:
path: the JSON whisper.cpp writes with --output-json-full.
Returns:
(the language it names, its segment texts joined, the mean
probability of its text tokens, 0.0 without any); a file that is
missing or not of that shape raises EngineError.
"""
try:
data = json.loads(path.read_bytes().decode("utf-8", errors="replace"))
language, segments = data["result"]["language"], data["transcription"]
texts = [segment["text"] for segment in segments]
sure = [token["p"] for segment in segments for token in segment.get("tokens", []) if not str(token["text"]).startswith(SPECIAL)]
except (OSError, ValueError, RecursionError, KeyError, IndexError, TypeError, AttributeError):
raise EngineError("whisper.cpp wrote no readable JSON transcript") from None
if not isinstance(language, str) or not all(isinstance(text, str) for text in texts) or not all(isinstance(p, int | float) and not isinstance(p, bool) for p in sure):
raise EngineError("whisper.cpp wrote no readable JSON transcript")
return language, " ".join(texts), sum(sure) / len(sure) if sure else 0.0
@dataclass(frozen=True) @dataclass(frozen=True)
class Engines: class Engines:
piper: Path piper: Path
@@ -207,14 +244,15 @@ class Speech:
def input_gap(self) -> str: def input_gap(self) -> str:
"""Returns: """Returns:
"" when speech input works, otherwise why not, naming the "" when speech input works, otherwise why not, naming the
whisper.cpp executable or model with its path; always "" with the whisper.cpp executable or model with its path and the operator row
speech service. MODEL_ROW that installs both; always "" with the speech service.
""" """
if self.engines.service.enabled(): if self.engines.service.enabled():
return "" return ""
model = self.engines.whisper_model model = self.engines.whisper_model
gap = unreachable(self.engines.whisper, "the whisper.cpp executable", os.X_OK, "executable") or unreachable(model, "the whisper.cpp model", os.R_OK, "readable") gap = unreachable(self.engines.whisper, "the whisper.cpp executable", os.X_OK, "executable") or unreachable(model, "the whisper.cpp model", os.R_OK, "readable")
return gap or ("" if os.path.isfile(model) else f"the whisper.cpp model {model} is not a file") gap = gap or ("" if os.path.isfile(model) else f"the whisper.cpp model {model} is not a file")
return f"{gap}; the operator row {MODEL_ROW} installs it" if gap else ""
def voice_of(self, key: str) -> str: def voice_of(self, key: str) -> str:
"""Args: """Args:
@@ -233,7 +271,7 @@ class Speech:
silent, deaf = self.output_gap(), self.input_gap() silent, deaf = self.output_gap(), self.input_gap()
return { return {
"tts": not silent, "stt": not deaf, "tts_off": silent, "stt_off": deaf, "voices": len(self.voices), "operator": operator, "sample": sample, "tts": not silent, "stt": not deaf, "tts_off": silent, "stt_off": deaf, "voices": len(self.voices), "operator": operator, "sample": sample,
"limit": SPEECH_LIMIT, "rate": STT_RATE, "seconds": STT_SECONDS, "limit": SPEECH_LIMIT, "rate": STT_RATE, "seconds": STT_SECONDS, "input": INPUT_SECONDS,
} }
def synthesize(self, body: bytes, admit: Callable[[str], None]) -> bytes: def synthesize(self, body: bytes, admit: Callable[[str], None]) -> bytes:
@@ -333,21 +371,40 @@ class Speech:
gap = self.input_gap() gap = self.input_gap()
if gap: if gap:
raise Unavailable(f"speech input is off: {gap}") raise Unavailable(f"speech input is off: {gap}")
self.admit(wav_seconds(body)) self.admit(wav_seconds(body, INPUT_SECONDS))
return self.whisper(body) return self.whisper(body)
def whisper(self, body: bytes) -> dict: def whisper(self, body: bytes) -> dict:
"""Returns:
{"text"} from the speech service, otherwise {"text", "language"}
in one of LANGUAGES.
"""
if self.engines.service.enabled(): if self.engines.service.enabled():
with self.hearing: with self.hearing:
return {"text": heard(self.engines.service.hear(body))} return {"text": heard(self.engines.service.hear(body))}
command = [str(self.engines.whisper), "--model", str(self.engines.whisper_model), "--file", "-", "--output-txt", "--language", "auto", "--no-prints", "--suppress-nst"] with self.hearing, tempfile.TemporaryDirectory(prefix=HEARING_DIR, dir=self.engines.scratch) as folder:
with self.hearing: language, text, _ = self.listen(body, Path(folder), "auto")
if language not in LANGUAGES:
language, text, _ = max((self.listen(body, Path(folder), choice) for choice in LANGUAGES), key=lambda found: found[2])
return {"text": heard(text), "language": language}
def listen(self, body: bytes, folder: Path, language: str) -> tuple[str, str, float]:
"""Args:
body: the recording.
folder: the request's own directory, cwd and TMPDIR of the run.
language: "auto" to detect it, otherwise the language to hear.
Returns:
transcript() of the JSON whisper.cpp writes into folder.
"""
target = folder / language
command = [str(self.engines.whisper), "--model", str(self.engines.whisper_model), "--file", "-", "--output-json", "--output-json-full", "--output-file", str(target), "--language", language, "--no-prints", "--suppress-nst"]
try: try:
done = subprocess.run(command, input=body, capture_output=True, timeout=STT_TIMEOUT, env=self.env(), cwd=self.engines.scratch, check=False) done = subprocess.run(command, input=body, capture_output=True, timeout=STT_TIMEOUT, env={**self.env(), "TMPDIR": str(folder)}, cwd=folder, check=False)
except subprocess.TimeoutExpired: except subprocess.TimeoutExpired:
raise EngineError(f"whisper.cpp took longer than {STT_TIMEOUT} seconds") from None raise EngineError(f"whisper.cpp took longer than {STT_TIMEOUT} seconds") from None
except OSError as error: except OSError as error:
raise EngineError(f"whisper.cpp did not start: {error.strerror}") from None raise EngineError(f"whisper.cpp did not start: {error.strerror}") from None
if done.returncode != 0: if done.returncode != 0:
raise EngineError(f"whisper.cpp failed with exit code {done.returncode}: {done.stderr.decode('utf-8', errors='replace').strip()[-300:]}") raise EngineError(f"whisper.cpp failed with exit code {done.returncode}: {done.stderr.decode('utf-8', errors='replace').strip()[-300:]}")
return {"text": heard(done.stdout.decode("utf-8", errors="replace"))} return transcript(target.with_name(f"{language}.json"))
+2 -2
View File
@@ -8,7 +8,7 @@ from http import HTTPStatus
from urllib.parse import urlsplit from urllib.parse import urlsplit
from .diffs import DIFF_BODY_LIMIT from .diffs import DIFF_BODY_LIMIT
from .engines import STT_LIMIT from .engines import INPUT_LIMIT, STT_LIMIT
from .exchange import PLAIN_CSP, Exchange from .exchange import PLAIN_CSP, Exchange
from .rates import rated from .rates import rated
from .resources import RESOURCES, sampled from .resources import RESOURCES, sampled
@@ -294,7 +294,7 @@ class Handler(Exchange):
def do_POST(self) -> None: def do_POST(self) -> None:
path = urlsplit(self.path).path path = urlsplit(self.path).path
length = self.body_length({UPLOADS: UPLOAD_LIMIT, STT_ROUTE: STT_LIMIT, VOICE_ROUTE: STT_LIMIT, "/mcp": DIFF_BODY_LIMIT, "/api/asks": DIFF_BODY_LIMIT}.get(path, BODY_LIMIT)) length = self.body_length({UPLOADS: UPLOAD_LIMIT, STT_ROUTE: INPUT_LIMIT, VOICE_ROUTE: STT_LIMIT, "/mcp": DIFF_BODY_LIMIT, "/api/asks": DIFF_BODY_LIMIT}.get(path, BODY_LIMIT))
if length is None: if length is None:
return return
host = self.host() host = self.host()
+3 -2
View File
@@ -10,7 +10,8 @@ from collections import Counter
from collections.abc import Callable, Iterator from collections.abc import Callable, Iterator
from pathlib import Path from pathlib import Path
from .engines import Speech, wav_seconds from . import engines
from .engines import Speech
from .errors import EngineError, Missing, ToolError, Unavailable from .errors import EngineError, Missing, ToolError, Unavailable
from .records import write_atomic from .records import write_atomic
@@ -44,7 +45,7 @@ class VoiceNotes:
gap = self.speech.input_gap() gap = self.speech.input_gap()
if gap: if gap:
raise Unavailable(f"speech input is off: {gap}") raise Unavailable(f"speech input is off: {gap}")
seconds = wav_seconds(body) seconds = engines.wav_seconds(body, engines.STT_SECONDS)
with self.lock: with self.lock:
if sum(1 for note in self.notes.values() if note["status"] == "pending") >= HEARING_LIMIT: if sum(1 for note in self.notes.values() if note["status"] == "pending") >= HEARING_LIMIT:
raise Unavailable(f"{HEARING_LIMIT} recordings are already waiting for their transcript") raise Unavailable(f"{HEARING_LIMIT} recordings are already waiting for their transcript")
+12 -1
View File
@@ -320,7 +320,16 @@ textarea { width: 100%; min-height: 84px; padding: 8px 10px; border: 1px solid v
.palette-box input { width: 100%; padding: 14px 16px; border: 0; background: transparent; font-size: var(--fs-lg); outline: none; } .palette-box input { width: 100%; padding: 14px 16px; border: 0; background: transparent; font-size: var(--fs-lg); outline: none; }
.palette-row { display: flex; align-items: center; gap: 6px; padding-right: 10px; border-bottom: 1px solid var(--line); } .palette-row { display: flex; align-items: center; gap: 6px; padding-right: 10px; border-bottom: 1px solid var(--line); }
.palette-row input { flex: 1; min-width: 0; } .palette-row input { flex: 1; min-width: 0; }
.mic[aria-pressed="true"] { background: var(--bad-soft); color: var(--bad); } .mic { flex: none; }
.mic[aria-pressed="true"] { background: var(--bad-soft); color: var(--bad); animation: mic-pulse 1.2s ease-in-out infinite; }
.mic-auto { margin-left: 4px; vertical-align: middle; }
textarea + .mic-auto { display: flex; margin: 4px 0 0 auto; }
.mic-live { position: fixed; left: 50%; bottom: calc(18px + env(safe-area-inset-bottom, 0px)); z-index: 60; display: flex; align-items: center; gap: 8px; max-width: calc(100vw - 32px); padding: 8px 14px; transform: translateX(-50%); border-radius: 999px; background: var(--bad); color: #fff; font-weight: 650; box-shadow: var(--shadow-lift); }
.mic-live[hidden] { display: none; }
.mic-live::before { content: ""; flex: none; width: 10px; height: 10px; border-radius: 50%; background: #fff; animation: mic-pulse 1s ease-in-out infinite; }
@keyframes mic-pulse { 50% { opacity: .45; } }
@media (max-width: 640px) { .mic { min-width: 44px; min-height: 44px; } }
@media (prefers-reduced-motion: reduce) { .mic[aria-pressed="true"], .mic-live::before { animation: none; } }
.palette-list { max-height: 55vh; margin: 0; padding: 6px; overflow: auto; list-style: none; } .palette-list { max-height: 55vh; margin: 0; padding: 6px; overflow: auto; list-style: none; }
.palette-list li { display: flex; align-items: center; gap: 10px; padding: 8px 10px; border-radius: 8px; cursor: pointer; } .palette-list li { display: flex; align-items: center; gap: 10px; padding: 8px 10px; border-radius: 8px; cursor: pointer; }
.palette-list li[aria-selected="true"] { background: var(--accent-soft); color: var(--accent); } .palette-list li[aria-selected="true"] { background: var(--accent-soft); color: var(--accent); }
@@ -617,6 +626,8 @@ pre.notes { margin: 0; padding: 8px 10px; background: var(--sunken); border-radi
{% include "page-speech.js" %} {% include "page-speech.js" %}
{% include "page-mic.js" %}
{% include "page-bubbles.js" %} {% include "page-bubbles.js" %}
{% include "page-alerts.js" %} {% include "page-alerts.js" %}
+1 -1
View File
@@ -94,7 +94,7 @@
const classes = choice("to-one", TO_ONE, "post"); const classes = choice("to-one", TO_ONE, "post");
const form = h("form", {class: "popup-form"}, text, file, classes, const form = h("form", {class: "popup-form"}, text, file, classes,
h("p", {class: "muted"}, "Post arrives at the end of the turn, Call at once with a read receipt; your passkey signs, verifiable with operator_message."), h("p", {class: "muted"}, "Post arrives at the end of the turn, Call at once with a read receipt; your passkey signs, verifiable with operator_message."),
h("div", {class: "actions"}, h("span", {class: "spacer"}), micButton(text), button)); h("div", {class: "actions"}, h("span", {class: "spacer"}), micButton(text, false), button));
form.addEventListener("submit", async (event) => { form.addEventListener("submit", async (event) => {
event.preventDefault(); event.preventDefault();
if (!text.value.trim()) { if (!text.value.trim()) {
+1
View File
@@ -39,6 +39,7 @@
window.addEventListener("resize", placePopup); window.addEventListener("resize", placePopup);
window.addEventListener("resize", placeStations); window.addEventListener("resize", placeStations);
skyPanel(); skyPanel();
micWatch();
radioSpot(); radioSpot();
radio.element = radioElement(); radio.element = radioElement();
drawSky(); drawSky();
+1 -1
View File
@@ -78,7 +78,7 @@
throw new Error("no recording"); throw new Error("no recording");
} }
parts.status.textContent = "Converting …"; parts.status.textContent = "Converting …";
const stored = await send("/api/voice", await wavOf(blob, sound), "audio/wav"); const stored = await send("/api/voice", await wavOf(blob, sound, state.speech.seconds), "audio/wav");
if (!stored.ok || !stored.data) { if (!stored.ok || !stored.data) {
throw new Error(`${stored.status} ${stored.text}`.trim()); throw new Error(`${stored.status} ${stored.text}`.trim());
} }
+185
View File
@@ -0,0 +1,185 @@
const MIC_HINT = "Voice input: tap to record, tap again to insert the text at the cursor; Escape cancels";
const MIC_NODES = `${Voice.FIELDS}, button.mic`;
const micButtons = new WeakMap();
const micFields = new WeakMap();
let micCount = 0;
function micLive() {
let live = document.getElementById("mic-live");
if (!live) {
live = h("div", {id: "mic-live", class: "mic-live", role: "status", "aria-live": "polite", hidden: true});
document.body.append(live);
}
return live;
}
function micState(button) {
const reason = micGap();
button.disabled = Boolean(reason);
button.title = reason || MIC_HINT;
}
function micLook(button, look) {
const recording = look === "recording";
button.setAttribute("aria-pressed", String(recording));
button.setAttribute("aria-label", recording ? "Stop voice input" : "Voice input");
button.replaceChildren(icon(recording ? "bi-stop-fill" : look === "busy" ? "bi-hourglass-split" : "bi-mic"));
micLive().hidden = !recording;
if (look === "busy") {
button.disabled = true;
} else {
micState(button);
}
}
/**
* @param {HTMLInputElement|HTMLTextAreaElement} field the text field the transcript lands in.
* @param {boolean} auto whether the shared component placed the button right after the field and keeps it there.
* @returns {HTMLButtonElement} the microphone button of the field.
*/
function micButton(field, auto) {
if (!field.id) {
micCount += 1;
field.id = `mic-field-${micCount}`;
}
const button = h("button", {type: "button", class: auto ? "btn small ghost icon-only mic mic-auto" : "btn small ghost icon-only mic", "aria-controls": field.id}, icon("bi-mic"));
micButtons.set(field, button);
micFields.set(button, field);
field.dataset.mic = "on";
micLook(button, "idle");
button.addEventListener("click", () => toggleMic(field, button));
return button;
}
function micSweep() {
for (const field of document.querySelectorAll(Voice.FIELDS)) {
const button = micButtons.get(field);
if (field.readOnly || (button && !button.classList.contains("mic-auto"))) {
continue;
}
if (!button) {
field.after(micButton(field, true));
} else if (button.previousElementSibling !== field) {
field.after(button);
}
}
for (const button of document.querySelectorAll("button.mic-auto")) {
if (!micFields.get(button).isConnected) {
button.remove();
}
}
}
function micTouched(records) {
return records.some((record) => [...record.addedNodes, ...record.removedNodes].some((node) => node instanceof Element && (node.matches(MIC_NODES) || Boolean(node.querySelector(MIC_NODES)))));
}
function micWatch() {
micSweep();
new MutationObserver((records) => micTouched(records) && micSweep()).observe(document.body, {childList: true, subtree: true});
window.addEventListener("keydown", (event) => {
if (event.key === "Escape" && mic.button && !mic.busy) {
event.preventDefault();
event.stopImmediatePropagation();
stopMic(true);
}
}, true);
}
function stopMic(cancel) {
mic.cancel = cancel;
window.clearInterval(mic.timer);
if (mic.recorder && mic.recorder.state === "recording") {
mic.recorder.stop();
}
}
function micTick() {
const seconds = Math.floor((Date.now() - mic.started) / 1000);
micLive().textContent = `Recording ${seconds} s of at most ${state.speech.input} s: tap the microphone again to insert, Escape cancels`;
if (seconds >= state.speech.input) {
stopMic(false);
}
}
function micDone() {
Object.assign(mic, {recorder: null, button: null, field: null, busy: false, cancel: false});
}
async function toggleMic(field, button) {
if (mic.button) {
if (mic.button === button && mic.recorder) {
stopMic(false);
}
return;
}
Object.assign(mic, {button, field, cancel: false});
let media = null;
try {
media = await navigator.mediaDevices.getUserMedia({audio: {channelCount: 1, echoCancellation: true, noiseSuppression: true}});
} catch (error) {
micDone();
toast("Microphone not available", [String(error.message || error)], false);
return;
}
if (mic.cancel) {
media.getTracks().forEach((track) => track.stop());
micDone();
return;
}
const recorder = new MediaRecorder(media);
const chunks = [];
recorder.addEventListener("dataavailable", (event) => {
if (event.data.size) {
chunks.push(event.data);
}
});
recorder.addEventListener("stop", () => {
media.getTracks().forEach((track) => track.stop());
window.clearInterval(mic.timer);
const cancelled = mic.cancel;
micLook(button, "idle");
if (cancelled) {
micDone();
return;
}
mic.recorder = null;
micTranscribe(new Blob(chunks, {type: recorder.mimeType}), field, button);
}, {once: true});
Object.assign(mic, {recorder, started: Date.now()});
recorder.start();
micLook(button, "recording");
micTick();
mic.timer = window.setInterval(micTick, 250);
}
function micInsert(field, text) {
const start = field.selectionStart ?? field.value.length;
const spliced = Voice.spliced(field.value, start, field.selectionEnd ?? start, text);
field.value = spliced.value;
field.focus();
field.setSelectionRange(spliced.caret, spliced.caret);
field.dispatchEvent(new Event("input", {bubbles: true}));
}
async function micTranscribe(blob, field, button) {
const sound = context();
if (!sound || !blob.size) {
micDone();
return;
}
mic.busy = true;
micLook(button, "busy");
try {
const result = await send("/api/stt", await wavOf(blob, sound, state.speech.input), "audio/wav");
if (!result.ok || !result.data) {
throw new Error(`${result.status} ${result.text}`.trim());
}
micInsert(field, result.data.text || "");
} catch (error) {
toast("Voice input failed", [String(error.message || error)], false);
} finally {
micDone();
micLook(button, "idle");
}
}
+1 -1
View File
@@ -142,7 +142,7 @@
runPalette(palette.index); runPalette(palette.index);
} }
}); });
box.replaceChildren(h("div", {class: "palette-box", role: "dialog", "aria-label": "Commands"}, h("div", {class: "palette-row"}, palette.input, micButton(palette.input)), palette.list)); box.replaceChildren(h("div", {class: "palette-box", role: "dialog", "aria-label": "Commands"}, h("div", {class: "palette-row"}, palette.input, micButton(palette.input, false)), palette.list));
box.hidden = false; box.hidden = false;
drawPalette(); drawPalette();
palette.input.focus(); palette.input.focus();
+2 -101
View File
@@ -309,84 +309,10 @@
: !window.MediaRecorder ? "This browser cannot record" : !state.speech.stt ? `Voice input is off: ${state.speech.stt_off}` : ""; : !window.MediaRecorder ? "This browser cannot record" : !state.speech.stt ? `Voice input is off: ${state.speech.stt_off}` : "";
} }
function micButton(field) { async function wavOf(blob, sound, seconds) {
const button = h("button", {type: "button", class: "btn small ghost icon-only mic", "aria-pressed": "false", ...named("Voice input", "Voice input: hold and speak; the text lands in the field and is not sent")}, icon("bi-mic"));
const reason = micGap();
if (reason) {
button.disabled = true;
button.title = `Voice input: ${reason}`;
return button;
}
const press = (event) => {
event.preventDefault();
record(field, button);
};
button.addEventListener("pointerdown", press);
for (const name of ["pointerup", "pointerleave", "pointercancel"]) {
button.addEventListener(name, release);
}
button.addEventListener("keydown", (event) => {
if ((event.key === " " || event.key === "Enter") && !event.repeat) {
press(event);
}
});
button.addEventListener("keyup", (event) => {
if (event.key === " " || event.key === "Enter") {
event.preventDefault();
release();
}
});
return button;
}
async function record(field, button) {
if (mic.held || mic.recorder || mic.busy) {
return;
}
mic.held = true;
let media = null;
try {
media = await navigator.mediaDevices.getUserMedia({audio: {channelCount: 1, echoCancellation: true, noiseSuppression: true}});
} catch (error) {
mic.held = false;
toast("Microphone not available", [String(error.message || error)], false);
return;
}
if (!mic.held) {
media.getTracks().forEach((track) => track.stop());
return;
}
const recorder = new MediaRecorder(media);
const chunks = [];
recorder.addEventListener("dataavailable", (event) => {
if (event.data.size) {
chunks.push(event.data);
}
});
recorder.addEventListener("stop", () => {
media.getTracks().forEach((track) => track.stop());
mic.recorder = null;
button.setAttribute("aria-pressed", "false");
transcribe(new Blob(chunks, {type: recorder.mimeType}), field, button);
}, {once: true});
mic.recorder = recorder;
recorder.start();
button.setAttribute("aria-pressed", "true");
mic.timer = window.setTimeout(release, state.speech.seconds * 1000);
}
function release() {
mic.held = false;
window.clearTimeout(mic.timer);
if (mic.recorder && mic.recorder.state === "recording") {
mic.recorder.stop();
}
}
async function wavOf(blob, sound) {
const decoded = await sound.decodeAudioData(await blob.arrayBuffer()); const decoded = await sound.decodeAudioData(await blob.arrayBuffer());
const rate = state.speech.rate; const rate = state.speech.rate;
const length = Voice.frames(decoded.duration, rate, state.speech.seconds); const length = Voice.frames(decoded.duration, rate, seconds);
if (length < rate / 4) { if (length < rate / 4) {
throw new Error("too short: speak for at least a quarter second"); throw new Error("too short: speak for at least a quarter second");
} }
@@ -398,28 +324,3 @@
const rendered = await offline.startRendering(); const rendered = await offline.startRendering();
return Voice.wav(rendered.getChannelData(0), rate); return Voice.wav(rendered.getChannelData(0), rate);
} }
async function transcribe(blob, field, button) {
const sound = context();
if (!sound || !blob.size) {
return;
}
mic.busy = true;
button.disabled = true;
try {
const result = await send("/api/stt", await wavOf(blob, sound), "audio/wav");
if (!result.ok || !result.data) {
throw new Error(`${result.status} ${result.text}`.trim());
}
if (result.data.text) {
field.value = [field.value.trim(), result.data.text].filter(Boolean).join(" ");
field.dispatchEvent(new Event("input", {bubbles: true}));
}
field.focus();
} catch (error) {
toast("Voice input failed", [String(error.message || error)], false);
} finally {
mic.busy = false;
button.disabled = false;
}
}
+1 -1
View File
@@ -47,7 +47,7 @@
const radio = {element: null, url: "", level: 0, timer: 0, playing: false, waiting: false, index: -1, pick: null, sig: "", state: "", short: "", note: "", told: null}; const radio = {element: null, url: "", level: 0, timer: 0, playing: false, waiting: false, index: -1, pick: null, sig: "", state: "", short: "", note: "", told: null};
const speech = {pending: [], busy: false, warned: 0, current: null, source: null, loading: null, timer: 0, item: null}; const speech = {pending: [], busy: false, warned: 0, current: null, source: null, loading: null, timer: 0, item: null};
let radioSet = RadioSet.valid(stored(RadioSet.KEY, null)); let radioSet = RadioSet.valid(stored(RadioSet.KEY, null));
const mic = {recorder: null, held: false, busy: false, timer: 0}; const mic = {recorder: null, button: null, field: null, busy: false, cancel: false, timer: 0, started: 0};
const space = { const space = {
views: {graph: {...state.settings.space.graph}, sky: {...state.settings.space.sky}}, glides: {graph: null, sky: null}, edits: 0, saved: 0, timer: 0, look: false, compass: null, views: {graph: {...state.settings.space.graph}, sky: {...state.settings.space.sky}}, glides: {graph: null, sky: null}, edits: 0, saved: 0, timer: 0, look: false, compass: null,
}; };
+21 -1
View File
@@ -3,6 +3,7 @@ const Voice = (() => {
const RANKS = {mayday: 0, "pan-pan": 1, security: 2}; const RANKS = {mayday: 0, "pan-pan": 1, security: 2};
const ALARMS = new Set(["mayday", "pan-pan"]); const ALARMS = new Set(["mayday", "pan-pan"]);
const ROUTINE = 3; const ROUTINE = 3;
const FIELDS = 'textarea, input:not([type]), input[type="text"], input[type="search"]';
function toneOf(row, human) { function toneOf(row, human) {
if (ALARMS.has(row.class)) { if (ALARMS.has(row.class)) {
@@ -59,7 +60,26 @@ const Voice = (() => {
return buffer; return buffer;
} }
return {toneOf, loudness, beep, enqueue, frames, wav}; /**
* @param {string} value the text of the field.
* @param {number} start the start of its selection.
* @param {number} end the end of its selection.
* @param {string} text the transcript.
* @returns {{value: string, caret: number}} value with the trimmed transcript in place of the selection, one space on each side where a neighbour is no whitespace, and the caret right after the transcript.
*/
function spliced(value, start, end, text) {
const words = String(text).trim();
if (!words) {
return {value, caret: end};
}
const before = value.slice(0, start);
const after = value.slice(end);
const lead = before && !/\s$/.test(before) ? " " : "";
const tail = after && !/^\s/.test(after) ? " " : "";
return {value: `${before}${lead}${words}${tail}${after}`, caret: before.length + lead.length + words.length};
}
return {FIELDS, toneOf, loudness, beep, enqueue, frames, wav, spliced};
})(); })();
if (typeof module === "object" && module.exports) { if (typeof module === "object" && module.exports) {
module.exports = Voice; module.exports = Voice;
+238
View File
@@ -0,0 +1,238 @@
from __future__ import annotations
import base64
import dataclasses
import json
import os
import shutil
import sys
import unittest
from pathlib import Path
from unittest import mock
HERE = Path(__file__).resolve().parent
sys.path[:0] = [str(HERE.parent), str(HERE)]
import test_orchestrator_e2e as base
from hub_case import HubCase
from kvb_orchestrator import engines
OUT = Path(os.environ.get("E2E_OUT", "/tmp/voice-input-shots"))
WIDTHS = ((1920, 1080), (390, 844))
FIXTURE = HERE.parent / "fixtures" / "voice" / "hello-world.wav"
HEARD = "hello from the mic second line"
FAKE_MIC = """() => {
const wav = Uint8Array.from(atob("@WAV@"), (char) => char.charCodeAt(0));
const posted = [];
window.__voice = {posted, opened: 0};
const fetched = window.fetch.bind(window);
window.fetch = (path, init) => {
if (String(path) === "/api/stt") {
posted.push(Date.now());
}
return fetched(path, init);
};
if (navigator.mediaDevices) {
navigator.mediaDevices.getUserMedia = async () => {
window.__voice.opened += 1;
const sound = new AudioContext();
sound.resume();
const source = sound.createBufferSource();
source.buffer = await sound.decodeAudioData(wav.slice().buffer);
source.loop = true;
const sink = sound.createMediaStreamDestination();
source.connect(sink);
source.start();
return sink.stream;
};
}
}"""
MIC = """
const FIELDS = 'textarea, input:not([type]), input[type="text"], input[type="search"]';
const micOf = (field) => (field && field.id ? document.querySelector(`button.mic[aria-controls="${CSS.escape(field.id)}"]`) : null);
const bare = () => all(FIELDS).filter((field) => !field.readOnly && !micOf(field)).map((field) => field.getAttribute("aria-label") || field.placeholder || field.outerHTML.slice(0, 80));
const composer = () => document.querySelector('#radio-spot textarea[aria-label="Message"]');
const live = () => {
const found = document.getElementById("mic-live");
return found && !found.hidden ? text(found) : null;
};
const look = (button) => {
const box = button.getBoundingClientRect();
return {pressed: button.getAttribute("aria-pressed"), disabled: button.disabled, title: button.title, width: box.width, height: box.height, live: live()};
};
const escape = () => document.dispatchEvent(new KeyboardEvent("keydown", {key: "Escape", bubbles: true, cancelable: true}));
const openComposer = async () => {
const spot = document.getElementById("radio-spot");
if (spot.hidden) {
document.getElementById("radio-open").click();
}
const field = await until(() => composer(), 4000);
field.scrollIntoView({block: "center"});
return field;
};
"""
SCAN = """(async () => {@HELPERS@@MIC@
const missing = {};
let scanned = 0;
for (const screen of @SCREENS@) {
location.hash = `#${screen}`;
await screenShown(8000);
await sleep(250);
scanned += all(FIELDS).length;
const found = bare();
if (found.length) {
missing[screen] = found;
}
}
for (const [name, opener] of [["settings", "settings-open"], ["palette", "palette-open"]]) {
document.getElementById(opener).click();
await sleep(500);
scanned += all(FIELDS).length;
const found = bare();
if (found.length) {
missing[name] = found;
}
escape();
await sleep(300);
}
location.hash = "#stream";
await screenShown(8000);
const field = await openComposer();
field.value = "start end";
field.focus();
field.setSelectionRange(6, 6);
await sleep(400);
return {missing, scanned, idle: look(micOf(field))};
})()"""
RECORD = """(async () => {@HELPERS@@MIC@
const button = micOf(composer());
button.click();
await until(() => button.getAttribute("aria-pressed") === "true" && live(), 8000);
await sleep(300);
return {...look(button), opened: window.__voice.opened};
})()"""
STOP = """(async () => {@HELPERS@@MIC@
const field = composer();
const button = micOf(field);
button.click();
await until(() => field.value.includes("@HEARD@"), 20000);
await sleep(200);
return {value: field.value, caret: field.selectionStart, focused: document.activeElement === field, posts: window.__voice.posted.length, ...look(button)};
})()"""
CANCEL = """(async () => {@HELPERS@@MIC@
const field = composer();
const button = micOf(field);
const before = [field.value, window.__voice.posted.length];
button.click();
const recording = Boolean(await until(() => button.getAttribute("aria-pressed") === "true" && live(), 8000));
await sleep(800);
escape();
await until(() => button.getAttribute("aria-pressed") === "false" && !live(), 8000);
await sleep(1500);
return {recording, same: field.value === before[0], posts: window.__voice.posted.length - before[1], ...look(button)};
})()"""
DEAF = """(async () => {@HELPERS@@MIC@
await screenShown(8000);
const field = await openComposer();
const button = micOf(field);
button.click();
await sleep(800);
return {...look(button), opened: window.__voice.opened, posts: window.__voice.posted.length};
})()"""
def probe(source: str) -> str:
return source.replace("@HELPERS@", base.HELPERS).replace("@MIC@", MIC).replace("@SCREENS@", json.dumps(base.SCREENS)).replace("@HEARD@", HEARD)
def fake_mic() -> str:
return FAKE_MIC.replace("@WAV@", base64.b64encode(FIXTURE.read_bytes()).decode())
@unittest.skipUnless(shutil.which(os.environ.get("CHROMIUM", "chromium")), "no Chromium")
class TestVoiceInputE2E(HubCase):
drive = base.TestOrchestratorE2E.drive
def setUp(self):
super().setUp()
OUT.mkdir(parents=True, exist_ok=True)
def steps(self, browser: str) -> list[dict]:
steps = []
for width, height in WIDTHS:
url = f"http://localhost:{self.port}/?{width}#stream"
steps += [
{"width": width, "height": height, "url": url, "wait": 1500, "probe": probe(SCAN), "shot": str(OUT / f"{browser}-{width}-idle.png"), "gesture": True},
{"width": width, "height": height, "wait": 200, "probe": probe(RECORD), "shot": str(OUT / f"{browser}-{width}-recording.png"), "gesture": True},
{"width": width, "height": height, "wait": 1200, "probe": probe(STOP), "gesture": True},
{"width": width, "height": height, "wait": 200, "probe": probe(CANCEL), "gesture": True},
]
return steps
def deaf_steps(self) -> list[dict]:
return [{"width": width, "height": height, "url": f"http://localhost:{self.port}/?{width}#stream", "wait": 1500, "probe": probe(DEAF), "gesture": True} for width, height in WIDTHS]
def deafen(self) -> None:
speech = self.httpd.hub.speech
patch = mock.patch.object(speech, "engines", dataclasses.replace(speech.engines, whisper_model=self.outside / "ggml-missing.bin"))
patch.start()
self.addCleanup(patch.stop)
def errors(self, data: dict) -> list[str]:
return [error for error in data["errors"] if "/favicon.ico" not in error]
def check(self, data: dict, browser: str) -> None:
self.assertEqual(self.errors(data), [])
for index, (width, _) in enumerate(WIDTHS):
scan, recording, stopped, cancelled = data["results"][index * 4 : index * 4 + 4]
with self.subTest(browser=browser, width=width):
for found in (scan, recording, stopped, cancelled):
self.assertNotIn("error", found, found)
self.assertEqual(scan["missing"], {}, scan)
self.assertGreater(scan["scanned"], 0, scan)
idle = scan["idle"]
self.assertEqual((idle["pressed"], idle["disabled"], idle["live"]), ("false", False, None), idle)
self.assertEqual((recording["pressed"], recording["opened"]), ("true", 1), recording)
self.assertIn("Recording", recording["live"] or "", recording)
self.assertEqual(stopped["value"], f"start {HEARD} end", stopped)
self.assertEqual((stopped["caret"], stopped["focused"], stopped["posts"], stopped["pressed"], stopped["live"]), (len(f"start {HEARD}"), True, 1, "false", None), stopped)
self.assertEqual((cancelled["recording"], cancelled["same"], cancelled["posts"], cancelled["pressed"], cancelled["live"]), (True, True, 0, "false", None), cancelled)
if width < 640:
for shown in (idle, recording):
self.assertGreaterEqual(min(shown["width"], shown["height"]), 44, shown)
def check_deaf(self, data: dict, browser: str) -> None:
self.assertEqual(self.errors(data), [])
for (width, _), found in zip(WIDTHS, data["results"], strict=True):
with self.subTest(browser=browser, width=width):
self.assertNotIn("error", found, found)
self.assertEqual((found["disabled"], found["pressed"], found["opened"], found["posts"], found["live"]), (True, "false", 0, 0, None), found)
self.assertIn(f"the operator row {engines.MODEL_ROW} installs it", found["title"], found)
def test_chromium_records_from_a_fake_microphone_and_inserts_the_transcript_at_the_cursor_at_1920_and_390(self):
data = self.drive(self.steps("chromium"), None, init=fake_mic())
(OUT / "chromium-voice-input.json").write_text(json.dumps(data, indent=1), encoding="utf-8")
self.check(data, "chromium")
@unittest.skipUnless(shutil.which(os.environ.get("FIREFOX", "firefox")), "no Firefox")
def test_firefox_records_from_a_fake_microphone_and_inserts_the_transcript_at_the_cursor_at_1920_and_390(self):
data = self.drive(self.steps("firefox"), None, "bidi.mjs", os.environ.get("FIREFOX", "firefox"), init=fake_mic())
(OUT / "firefox-voice-input.json").write_text(json.dumps(data, indent=1), encoding="utf-8")
self.check(data, "firefox")
def test_chromium_disables_the_microphone_and_names_the_operator_row_without_a_model(self):
self.deafen()
data = self.drive(self.deaf_steps(), None, init=fake_mic())
(OUT / "chromium-voice-input-deaf.json").write_text(json.dumps(data, indent=1), encoding="utf-8")
self.check_deaf(data, "chromium")
@unittest.skipUnless(shutil.which(os.environ.get("FIREFOX", "firefox")), "no Firefox")
def test_firefox_disables_the_microphone_and_names_the_operator_row_without_a_model(self):
self.deafen()
data = self.drive(self.deaf_steps(), None, "bidi.mjs", os.environ.get("FIREFOX", "firefox"), init=fake_mic())
(OUT / "firefox-voice-input-deaf.json").write_text(json.dumps(data, indent=1), encoding="utf-8")
self.check_deaf(data, "firefox")
if __name__ == "__main__":
unittest.main()
Binary file not shown.
Binary file not shown.
+7 -4
View File
@@ -31,13 +31,16 @@ with wave.open(args.output_file, "wb") as out:
""" """
FAKE_WHISPER = """ FAKE_WHISPER = """
import json, os, sys import json, os, sys
argv = sys.argv[1:]
data = sys.stdin.buffer.read() data = sys.stdin.buffer.read()
with open(os.path.join(os.environ["HOME"], "whisper.jsonl"), "a", encoding="utf-8") as log: with open(os.path.join(os.environ["HOME"], "whisper.jsonl"), "a", encoding="utf-8") as log:
log.write(json.dumps({"argv": sys.argv[1:], "sha256": __import__("hashlib").sha256(data).hexdigest()}) + "\\n") log.write(json.dumps({"argv": argv, "sha256": __import__("hashlib").sha256(data).hexdigest()}) + "\\n")
if b"FAIL" in data: if b"FAIL" in data:
sys.exit(4) sys.exit(4)
print(" hello from the [BLANK_AUDIO] mic") language = argv[argv.index("--language") + 1]
print("second line") segments = [{"text": " hello from the [BLANK_AUDIO] mic", "tokens": [{"text": "[_BEG_]", "p": 0.1}, {"text": " hello", "p": 0.9}]}, {"text": "second line", "tokens": [{"text": " second", "p": 0.8}]}]
with open(argv[argv.index("--output-file") + 1] + ".json", "w", encoding="utf-8") as out:
json.dump({"result": {"language": "en" if language == "auto" else language}, "transcription": segments}, out)
""" """
@@ -53,7 +56,7 @@ def write_engines(base: Path) -> server.Engines:
(voices / f"{model}.onnx").write_bytes(b"onnx") (voices / f"{model}.onnx").write_bytes(b"onnx")
(voices / f"{model}.onnx.json").write_text(json.dumps(config), encoding="utf-8") (voices / f"{model}.onnx.json").write_text(json.dumps(config), encoding="utf-8")
(voices / "broken.onnx").write_bytes(b"onnx") (voices / "broken.onnx").write_bytes(b"onnx")
model = base / "ggml-base.bin" model = base / "ggml-small-q8_0.bin"
model.write_bytes(b"ggml") model.write_bytes(b"ggml")
return server.Engines(piper=bin_dir / "piper", voices=voices, operator_voice=OPERATOR_VOICE, whisper=bin_dir / "whisper-cli", whisper_model=model, scratch=scratch, service=server.Service(url="", key="")) return server.Engines(piper=bin_dir / "piper", voices=voices, operator_voice=OPERATOR_VOICE, whisper=bin_dir / "whisper-cli", whisper_model=model, scratch=scratch, service=server.Service(url="", key=""))
+14 -9
View File
@@ -18,6 +18,8 @@ from hub_engines import OPERATOR_VOICE, VOICES, log_rows, wav_bytes
from hub_support import HOST_STATE, HUB, server from hub_support import HOST_STATE, HUB, server
from kvb_orchestrator import council, engines, exchange, handler, phrasing from kvb_orchestrator import council, engines, exchange, handler, phrasing
ROW = f"; the operator row {engines.MODEL_ROW} installs it"
class TestHubSpeech(HubCase): class TestHubSpeech(HubCase):
def test_voices_come_from_the_models_and_stick_to_their_session(self): def test_voices_come_from_the_models_and_stick_to_their_session(self):
@@ -52,9 +54,9 @@ class TestHubSpeech(HubCase):
({"piper": self.outside / "piper"}, "tts_off", f"the Piper executable {self.outside / 'piper'} is missing or not executable"), ({"piper": self.outside / "piper"}, "tts_off", f"the Piper executable {self.outside / 'piper'} is missing or not executable"),
({"voices": self.outside}, "tts_off", f"the voice directory {self.outside} holds no <model>.onnx voice"), ({"voices": self.outside}, "tts_off", f"the voice directory {self.outside} holds no <model>.onnx voice"),
({"voices": lone}, "tts_off", f"the voice config {lone / 'solo.onnx'}.json is missing, unreadable or names no 1 to {engines.SPEAKER_LIMIT} speakers"), ({"voices": lone}, "tts_off", f"the voice config {lone / 'solo.onnx'}.json is missing, unreadable or names no 1 to {engines.SPEAKER_LIMIT} speakers"),
({"whisper": self.outside / "whisper-cli"}, "stt_off", f"the whisper.cpp executable {self.outside / 'whisper-cli'} is missing or not executable"), ({"whisper": self.outside / "whisper-cli"}, "stt_off", f"the whisper.cpp executable {self.outside / 'whisper-cli'} is missing or not executable{ROW}"),
({"whisper_model": self.outside / "model.bin"}, "stt_off", f"the whisper.cpp model {self.outside / 'model.bin'} is missing or not readable"), ({"whisper_model": self.outside / "model.bin"}, "stt_off", f"the whisper.cpp model {self.outside / 'model.bin'} is missing or not readable{ROW}"),
({"whisper_model": self.outside}, "stt_off", f"the whisper.cpp model {self.outside} is not a file"), ({"whisper_model": self.outside}, "stt_off", f"the whisper.cpp model {self.outside} is not a file{ROW}"),
) )
for number, (change, field, message) in enumerate(cases): for number, (change, field, message) in enumerate(cases):
with self.subTest(change=change): with self.subTest(change=change):
@@ -70,7 +72,7 @@ class TestHubSpeech(HubCase):
self.addCleanup(folder.chmod, 0o755) self.addCleanup(folder.chmod, 0o755)
image = engines.Speech(dataclasses.replace(self.engines, whisper=whisper), self.state / "image").describe() image = engines.Speech(dataclasses.replace(self.engines, whisper=whisper), self.state / "image").describe()
self.assertEqual({key: image[key] for key in ("tts", "voices", "operator", "tts_off")}, {"tts": False, "voices": 0, "operator": "", "tts_off": f"the voice directory {voices} is missing or not searchable"}) self.assertEqual({key: image[key] for key in ("tts", "voices", "operator", "tts_off")}, {"tts": False, "voices": 0, "operator": "", "tts_off": f"the voice directory {voices} is missing or not searchable"})
self.assertEqual((image["stt"], image["stt_off"]), (False, f"the whisper.cpp executable {whisper} lies under {whisper.parent}, which the hub may not search")) self.assertEqual((image["stt"], image["stt_off"]), (False, f"the whisper.cpp executable {whisper} lies under {whisper.parent}, which the hub may not search{ROW}"))
status, _, body = self.post_json("/api/tts", {"text": "radio check", "voice": OPERATOR_VOICE}, self.guard(self.host)) status, _, body = self.post_json("/api/tts", {"text": "radio check", "voice": OPERATOR_VOICE}, self.guard(self.host))
self.assertEqual((status, body.decode()), (503, f"speech output is off: the voice directory {voices} is missing or not searchable\n")) self.assertEqual((status, body.decode()), (503, f"speech output is off: the voice directory {voices} is missing or not searchable\n"))
@@ -147,7 +149,7 @@ class TestHubSpeech(HubCase):
with self.subTest(content_type=content_type): with self.subTest(content_type=content_type):
self.assertEqual(self.call("POST", "/api/stt", clip, {**good, "Content-Type": content_type})[0], 415) self.assertEqual(self.call("POST", "/api/stt", clip, {**good, "Content-Type": content_type})[0], 415)
self.assertEqual(self.call("POST", "/api/stt", clip, good)[0], 415) self.assertEqual(self.call("POST", "/api/stt", clip, good)[0], 415)
self.assertEqual(self.call("POST", "/api/stt", b"RIFF", {**good, "Content-Type": "audio/wav", "Content-Length": str(engines.STT_LIMIT + 1)})[0], 413) self.assertEqual(self.call("POST", "/api/stt", b"RIFF", {**good, "Content-Type": "audio/wav", "Content-Length": str(engines.INPUT_LIMIT + 1)})[0], 413)
for broken in ( for broken in (
b"not a wav", b"not a wav",
wav_bytes(1, 2, 44100, b"\x00\x00" * 100), wav_bytes(1, 2, 44100, b"\x00\x00" * 100),
@@ -157,14 +159,17 @@ class TestHubSpeech(HubCase):
): ):
with self.subTest(broken=broken[:40]): with self.subTest(broken=broken[:40]):
self.assertEqual(post(broken, good)[0], 400) self.assertEqual(post(broken, good)[0], 400)
with mock.patch.object(engines, "STT_SECONDS", 1): with mock.patch.object(engines, "INPUT_SECONDS", 1):
self.assertEqual(post(wav_bytes(1, 2, engines.STT_RATE, b"\x00\x00" * (engines.STT_RATE + 1)), good)[0], 400) self.assertEqual(post(wav_bytes(1, 2, engines.STT_RATE, b"\x00\x00" * (engines.STT_RATE + 1)), good)[0], 400)
self.assertEqual(log_rows(calls), []) self.assertEqual(log_rows(calls), [])
status, headers, body = post(clip, good) status, headers, body = post(clip, good)
self.assertEqual((status, headers["content-type"]), (200, "application/json"), body) self.assertEqual((status, headers["content-type"]), (200, "application/json"), body)
self.assertEqual(json.loads(body), {"text": "hello from the mic second line"}) self.assertEqual(json.loads(body), {"text": "hello from the mic second line", "language": "en"})
argv = ["--model", str(self.engines.whisper_model), "--file", "-", "--output-txt", "--language", "auto", "--no-prints", "--suppress-nst"] [row] = log_rows(calls)
self.assertEqual(log_rows(calls), [{"argv": argv, "sha256": hashlib.sha256(clip).hexdigest()}]) target = PurePosixPath(row["argv"][row["argv"].index("--output-file") + 1])
argv = ["--model", str(self.engines.whisper_model), "--file", "-", "--output-json", "--output-json-full", "--output-file", str(target), "--language", "auto", "--no-prints", "--suppress-nst"]
self.assertEqual(row, {"argv": argv, "sha256": hashlib.sha256(clip).hexdigest()})
self.assertEqual((target.name, str(target.parent.parent)), ("auto", str(self.engines.scratch)))
long = wav_bytes(1, 2, engines.STT_RATE, b"\x00\x00" * engines.STT_RATE * 40) long = wav_bytes(1, 2, engines.STT_RATE, b"\x00\x00" * engines.STT_RATE * 40)
self.assertGreater(len(long), handler.BODY_LIMIT) self.assertGreater(len(long), handler.BODY_LIMIT)
self.assertEqual(post(long, good)[0], 200) self.assertEqual(post(long, good)[0], 200)
+1 -1
View File
@@ -169,7 +169,7 @@ class TestHubState(HubCase):
self.assertTrue(all(row["voice"] in VOICES and row["voice"] != OPERATOR_VOICE for row in state["radio"])) self.assertTrue(all(row["voice"] in VOICES and row["voice"] != OPERATOR_VOICE for row in state["radio"]))
self.assertEqual(state["speech"], { self.assertEqual(state["speech"], {
"tts": True, "stt": True, "tts_off": "", "stt_off": "", "voices": len(VOICES), "operator": OPERATOR_VOICE, "sample": "all stations, this is Operator, radio check, out", "tts": True, "stt": True, "tts_off": "", "stt_off": "", "voices": len(VOICES), "operator": OPERATOR_VOICE, "sample": "all stations, this is Operator, radio check, out",
"limit": phrasing.SPEECH_LIMIT, "rate": engines.STT_RATE, "seconds": engines.STT_SECONDS, "gate": {"threshold": 50, "cpu": None, "open": False}, "limit": phrasing.SPEECH_LIMIT, "rate": engines.STT_RATE, "seconds": engines.STT_SECONDS, "input": engines.INPUT_SECONDS, "gate": {"threshold": 50, "cpu": None, "open": False},
}) })
self.assertAlmostEqual(state["now"] - state["since"], view.RADIO_WINDOW) self.assertAlmostEqual(state["now"] - state["since"], view.RADIO_WINDOW)
self.assertEqual((state["night"], state["silence"]), ("day", None)) self.assertEqual((state["night"], state["silence"]), ("day", None))
+14 -1
View File
@@ -74,12 +74,25 @@ class TestVoiceModule(unittest.TestCase):
def test_the_recording_is_encoded_as_16_khz_mono_pcm(self): def test_the_recording_is_encoded_as_16_khz_mono_pcm(self):
body = base64.b64decode(self.data["wav"]) body = base64.b64decode(self.data["wav"])
self.assertEqual(engines.wav_seconds(body), 7 / engines.STT_RATE) self.assertEqual(engines.wav_seconds(body, engines.STT_SECONDS), 7 / engines.STT_RATE)
with wave.open(io.BytesIO(body)) as reader: with wave.open(io.BytesIO(body)) as reader:
self.assertEqual((reader.getnchannels(), reader.getsampwidth(), reader.getframerate(), reader.getnframes()), (1, 2, 16000, 7)) self.assertEqual((reader.getnchannels(), reader.getsampwidth(), reader.getframerate(), reader.getnframes()), (1, 2, 16000, 7))
self.assertEqual(struct.unpack("<7h", reader.readframes(7)), (0, 16383, -16384, 32767, -32768, 32767, -32768)) self.assertEqual(struct.unpack("<7h", reader.readframes(7)), (0, 16383, -16384, 32767, -32768, 32767, -32768))
self.assertEqual(self.data["frames"], [24000, 16000 * 120, 0]) self.assertEqual(self.data["frames"], [24000, 16000 * 120, 0])
def test_the_transcript_lands_at_the_cursor_with_one_space_on_each_side(self):
cases = [["start end", 6, 6, "hello"], ["", 0, 0, " hello "], ["abc", 3, 3, "x"], ["ab cd", 1, 4, "X"], ["keep", 2, 2, " "], ["a\nb", 2, 2, "line"]]
probe = "const V = require(process.argv[1]); console.log(JSON.stringify(JSON.parse(process.argv[2]).map((one) => V.spliced(...one))));"
done = subprocess.run(["node", "-e", probe, str(VOICE), json.dumps(cases)], capture_output=True, text=True, timeout=60, check=True)
self.assertEqual(json.loads(done.stdout), [
{"value": "start hello end", "caret": 11},
{"value": "hello", "caret": 5},
{"value": "abc x", "caret": 5},
{"value": "a X d", "caret": 3},
{"value": "keep", "caret": 2},
{"value": "a\nline b", "caret": 6},
])
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+150
View File
@@ -0,0 +1,150 @@
from __future__ import annotations
import dataclasses
import hashlib
import json
import re
import sys
import unittest
from pathlib import Path
from unittest import mock
from hub_case import HubCase
from hub_engines import log_rows, wav_bytes
from kvb_orchestrator import engines
FIXTURES = Path(__file__).resolve().parent / "fixtures" / "voice"
DUTCH = wav_bytes(1, 2, engines.STT_RATE, b"\x02\x00" * engines.STT_RATE)
FRENCH = wav_bytes(1, 2, engines.STT_RATE, b"\x03\x00" * engines.STT_RATE)
HEARD = {
"hallo-welt.wav": {"auto": ["de", "Hallo Welt."], "de": [0.9, "Hallo Welt."], "en": [0.3, "Hollow felt."]},
"hello-world.wav": {"auto": ["en", "Hello world."], "de": [0.2, "Hello Wörld."], "en": [0.9, "Hello world."]},
"dutch": {"auto": ["nl", "Hallo wereld."], "de": [0.7, "Hallo Welt."], "en": [0.4, "Hello world."]},
"french": {"auto": ["fr", "Bonjour le monde."], "de": [0.2, "Bonschur."], "en": [0.6, "Bonjour world."]},
}
FAKE_WHISPER = """
import hashlib, json, os, sys
argv = sys.argv[1:]
data = sys.stdin.buffer.read()
home = os.environ["HOME"]
with open(os.path.join(home, "heard.json"), encoding="utf-8") as table:
clip = json.load(table).get(hashlib.sha256(data).hexdigest(), {"auto": ["en", ""], "de": [0.0, ""], "en": [0.0, ""]})
with open(os.path.join(home, "whisper.jsonl"), "a", encoding="utf-8") as log:
log.write(json.dumps({"argv": argv, "cwd": os.getcwd(), "tmp": os.environ["TMPDIR"]}) + "\\n")
with open(os.path.join(os.environ["TMPDIR"], "engine-spill.wav"), "wb") as spill:
spill.write(data)
if b"FAIL" in data:
sys.exit(4)
language = argv[argv.index("--language") + 1]
detected, text, sure = (*clip["auto"], 0.5) if language == "auto" else (language, clip[language][1], clip[language][0])
tokens = [{"text": "[_BEG_]", "p": 0.01}, *({"text": f" {word}", "p": sure} for word in text.split())]
with open(argv[argv.index("--output-file") + 1] + ".json", "w", encoding="utf-8") as out:
json.dump({"result": {"language": detected}, "transcription": [{"text": text, "tokens": tokens}]}, out)
"""
KEPT = ("heard.json", "whisper.jsonl")
def spoken(text: str) -> str:
return " ".join(re.sub(r"[^\w\s]", " ", text).lower().split())
class TestVoiceInput(HubCase):
def setUp(self):
super().setUp()
self.engines.whisper.write_text(f"#!{sys.executable}\n{FAKE_WHISPER}", encoding="utf-8")
clips = {"dutch": DUTCH, "french": FRENCH}
heard = {hashlib.sha256(clips[name] if name in clips else (FIXTURES / name).read_bytes()).hexdigest(): table for name, table in HEARD.items()}
(self.engines.scratch / "heard.json").write_text(json.dumps(heard), encoding="utf-8")
self.calls = self.engines.scratch / "whisper.jsonl"
def post(self, body: bytes, headers: dict[str, str] | None = None, length: int | None = None) -> tuple[int, dict, bytes]:
merged = {**(self.guard(self.host) if headers is None else headers), "Content-Type": "audio/wav"}
if length is not None:
merged["Content-Length"] = str(length)
return self.call("POST", "/api/stt", body, merged)
def leftovers(self) -> list[str]:
return sorted(path.name for path in self.engines.scratch.iterdir() if path.name not in KEPT)
def languages(self) -> list[str]:
return [row["argv"][row["argv"].index("--language") + 1] for row in log_rows(self.calls)]
def test_the_german_and_the_english_fixture_come_back_as_their_words_with_the_language_detected(self):
for name, language, words in (("hallo-welt.wav", "de", "hallo welt"), ("hello-world.wav", "en", "hello world")):
with self.subTest(name=name):
status, _, body = self.post((FIXTURES / name).read_bytes())
self.assertEqual(status, 200, body)
answer = json.loads(body)
self.assertEqual((answer["language"], spoken(answer["text"])), (language, words))
self.assertEqual(self.languages(), ["auto", "auto"])
for row in log_rows(self.calls):
self.assertEqual(row["argv"].count("--language"), 1, row)
def test_a_third_language_is_heard_again_in_the_surer_of_german_and_english(self):
for clip, language, words in ((DUTCH, "de", "hallo welt"), (FRENCH, "en", "bonjour world")):
with self.subTest(language=language):
status, _, body = self.post(clip)
self.assertEqual(status, 200, body)
answer = json.loads(body)
self.assertEqual((answer["language"], spoken(answer["text"])), (language, words))
self.assertEqual(self.languages(), ["auto", "de", "en", "auto", "de", "en"])
self.assertEqual(self.leftovers(), [])
def test_the_temp_directory_is_empty_after_every_request_including_a_failed_one(self):
self.assertEqual(self.post((FIXTURES / "hello-world.wav").read_bytes())[0], 200)
self.assertEqual(self.leftovers(), [])
status, _, body = self.post(wav_bytes(1, 2, engines.STT_RATE, b"FAIL" * 400))
self.assertEqual(status, 502, body)
self.assertEqual(self.leftovers(), [])
rows = log_rows(self.calls)
self.assertEqual(len(rows), 2)
for row in rows:
used = Path(row["tmp"])
self.assertEqual((Path(row["cwd"]), used.parent), (used, self.engines.scratch), row)
self.assertFalse(used.exists(), row)
def test_a_recording_over_60_seconds_or_5_mb_is_refused_and_not_transcribed(self):
rate = engines.STT_RATE
self.assertEqual((engines.INPUT_SECONDS, engines.INPUT_LIMIT), (60, 5 << 20))
self.assertEqual(self.httpd.hub.speech.describe()["input"], engines.INPUT_SECONDS)
self.assertEqual(self.post(wav_bytes(1, 2, rate, b"\x01\x00" * rate * 60))[0], 200)
self.assertEqual(len(log_rows(self.calls)), 1)
status, _, body = self.post(wav_bytes(1, 2, rate, b"\x01\x00" * (rate * 60 + 1)))
self.assertEqual((status, body), (400, b"the recording must last more than 0 and at most 60 seconds\n"))
self.assertEqual(self.post(b"\x00" * engines.INPUT_LIMIT)[0], 400)
self.assertEqual(self.post(b"RIFF", length=engines.INPUT_LIMIT + 1)[0], 413)
self.assertEqual(len(log_rows(self.calls)), 1)
self.assertEqual(self.leftovers(), [])
def test_another_origin_host_or_a_missing_csrf_header_is_refused_and_not_transcribed(self):
clip = (FIXTURES / "hallo-welt.wav").read_bytes()
good = self.guard(self.host)
refused = (
{key: value for key, value in good.items() if key != "X-Kvb-Orchestrator"},
{key: value for key, value in good.items() if key != "Origin"},
{**good, "Origin": "http://evil.example"},
{**good, "Origin": f"http://localhost:{self.port + 1}"},
{**good, "Host": "evil.example"},
)
for headers in refused:
with self.subTest(headers=headers):
self.assertEqual(self.post(clip, headers)[0], 403)
self.assertEqual(log_rows(self.calls), [])
self.assertEqual(self.post(clip, good)[0], 200)
def test_without_a_model_the_endpoint_and_the_button_state_name_the_operator_row(self):
missing = self.outside / "ggml-missing.bin"
speech = self.httpd.hub.speech
with mock.patch.object(speech, "engines", dataclasses.replace(self.engines, whisper_model=missing)):
state = speech.describe()
status, _, body = self.post((FIXTURES / "hallo-welt.wav").read_bytes())
self.assertFalse(state["stt"])
self.assertEqual(state["stt_off"], f"the whisper.cpp model {missing} is missing or not readable; the operator row {engines.MODEL_ROW} installs it")
self.assertEqual(status, 503, body)
self.assertIn(f"the operator row {engines.MODEL_ROW} installs it".encode(), body)
self.assertEqual(log_rows(self.calls), [])
self.assertEqual(engines.MODEL_ROW, "make orchestrator-install")
if __name__ == "__main__":
unittest.main()
+36
View File
@@ -0,0 +1,36 @@
from __future__ import annotations
import dataclasses
import json
import os
import unittest
from pathlib import Path
from unittest import mock
from hub_case import HubCase
from test_hub_voice_input import FIXTURES, spoken
WHISPER = Path(os.environ.get("KVB_WHISPER_CLI", "/nonexistent/whisper-cli"))
MODEL = Path(os.environ.get("KVB_WHISPER_MODEL", "/nonexistent/ggml-small-q8_0.bin"))
QUEUED = os.environ.get("KVB_VOICE_INTEGRATION") == "1"
READY = os.access(WHISPER, os.X_OK) and MODEL.is_file()
REASON = (
"real Whisper run: needs KVB_WHISPER_CLI and KVB_WHISPER_MODEL from the operator rows of requirement 013 "
"and KVB_VOICE_INTEGRATION=1, which only the queue.py hold command line sets"
)
@unittest.skipUnless(QUEUED and READY, REASON)
class TestVoiceModel(HubCase):
def test_the_real_model_transcribes_the_german_and_the_english_fixture(self):
speech = self.httpd.hub.speech
with mock.patch.object(speech, "engines", dataclasses.replace(speech.engines, whisper=WHISPER, whisper_model=MODEL)):
for name, words in (("hallo-welt.wav", "hallo welt"), ("hello-world.wav", "hello world")):
with self.subTest(name=name):
status, _, body = self.call("POST", "/api/stt", (FIXTURES / name).read_bytes(), {**self.guard(self.host), "Content-Type": "audio/wav"})
self.assertEqual(status, 200, body)
self.assertEqual(spoken(json.loads(body)["text"]), words)
if __name__ == "__main__":
unittest.main()
+1 -1
View File
@@ -12,7 +12,7 @@ VOICE = "huggingface.co/rhasspy/piper-voices/resolve/e7229f7d33a10e145e109b886d6
PINNED = { PINNED = {
VOICE: "9127a559e11603f10b366d1a20ac7426826081dbc521de4c2420c57728d73f0f", VOICE: "9127a559e11603f10b366d1a20ac7426826081dbc521de4c2420c57728d73f0f",
f"{VOICE}.json": "2efdc6d7f954588b8180132cbd9b8001933fdd00932c92bc92fd0d2028a9eb3d", f"{VOICE}.json": "2efdc6d7f954588b8180132cbd9b8001933fdd00932c92bc92fd0d2028a9eb3d",
"huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-base-q8_0.bin": "c577b9a86e7e048a0b7eada054f4dd79a56bbfa911fbdacf900ac5b567cbb7d9", "huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-small-q8_0.bin": "49c8fb02b65e6049d5fa6c04f81f53b867b5ec9540406812c643f177317f779f",
} }
ARCHIVE = (f"codeload.github.com/ggml-org/whisper.cpp/tar.gz/{COMMIT}", "645445da884df6dbd52b30f82d7be7c01d1767c938714e477e786b88d529e9f4") ARCHIVE = (f"codeload.github.com/ggml-org/whisper.cpp/tar.gz/{COMMIT}", "645445da884df6dbd52b30f82d7be7c01d1767c938714e477e786b88d529e9f4")
CLONE = ("clone", "https://github.com/ggml-org/whisper.cpp", f"v1.9.5@{COMMIT}") CLONE = ("clone", "https://github.com/ggml-org/whisper.cpp", f"v1.9.5@{COMMIT}")