Merge branch 'feature/hub-voice-input' into unify/skills-20261114
# Conflicts: # README.md # docs/requirements/README.md
This commit is contained in:
commit
b95e305db0
27 files changed
+861
-149
No files matched your search
@@ -468,7 +468,7 @@ Both commands also patch the target's `.claude/settings.local.json` — the mach
|
||||
|
||||
The council root is the one directory every `kvb-org` script, hook and the hub share: the register, radio, queue, claims, operator rows, score journal, gate receipts, security cache and night state, plus `pages/` (the `kvb-git-diff` review pages the hub reads), `approvals/` (the replay stores of `approval.py verify`) and `gates/` (gate logs). `skills/kvb-org-organize/scripts/councilroot.py` alone resolves it: `$AGENTS_COUNCIL_ROOT`, else `$XDG_STATE_HOME/agents-shared`, else `~/.local/state/agents-shared`, so it survives a reboot; `councilroot.py root` prints it and `councilroot.py dir <part>` creates and prints a part. It and `agents-release` below are the only state paths a sandbox's `allowWrite` may name. Its sibling `~/.local/state/agents-council` (`councilroot.py durable`) is host-only and never belongs in `allowWrite`: only hooks, the runner and the systemd units write it (`gates.db`, the watch hook's copies of the receipt, parent, acceptance and queue-history journals, `queue-cleanup.json`, `resources/`, `subagent-model.jsonl`), and the hub mounts it read-only; no directory is both agent-writable and host-written except the council root, where every host writer opens files through `nofollow.py`. The `kvb-release` drafts lie in `agents-release/<version>` beside both (`councilroot.py release <version>`). The global install first runs `councilroot.py preflight` and stops before it copies anything unless `~/.claude/settings.json` lists both `agents-shared` and `agents-release` in `sandbox.filesystem.allowWrite`, no `allowWrite` entry reaches `agents-council` in the managed file, its `managed-settings.d` drop-ins, the user or user-local file (the list `councilroot.py` shares with the queue's permission check), in a project `.claude/settings.json` or `.claude/settings.local.json` below the pkgmgr repositories and worktrees directories (directory and file links followed), or in the start directory of any session listed in `~/.claude/projects`, and the move below would succeed; it warns while the agents are not suspended. Fixing those entries is the operator's step. It then builds the skills, stops the hub, the runner, the offices, resources and queue-cleanup units, runs `councilroot.py migrate`, which moves an earlier council root in `/tmp/agents-council` into the council root, copies the skills, runs `migrate --catch-up` for what the old skills wrote meanwhile, and starts the units again; the hub image is built before the container is replaced, and a failed build leaves the hub down and fails the install until `bash scripts/orchestrator.sh install` succeeds. Migrate refuses while a lock there is held or a queue job waits, and, checked under those locks, moves nothing (exit 4) while a file cannot be moved or a whole file both roots hold differs; the install then stops with the earlier root intact. Each file is renamed to a `.migrating` name before it is copied, so a writer that opens the old name meanwhile makes a new file the catch-up moves. An identical file leaves one copy, a `.jsonl` journal gets the rows it lacks (by id, else each exact line as often as the earlier root holds it more often, in `ts` order), links are skipped, and a second run moves nothing and duplicates no row. The catch-up keeps a differing whole file beside its namesake as `<name>.legacy-<hash>` (exit 3); a file that fails after the check stays behind (exit 5). After exit 3 or 5 the install goes on so the new skills read the new root, names every such file and exits 1 at the end. `~/.local/state/agents-council/migrate.started` exists from the first move to the end of a run; when a crash leaves it, the install copies the new skills anyway, names it and exits 1, and the next completed migrate removes it. Because the council root survives reboots, each `kvb-org-queue-cleanup` round first runs `councilroot.py retire`, which moves gate logs and review pages older than 30 days into `archive/<date>/`, and the radio appends the rows it drops past its last 500 to `archive/news-<date>.jsonl`; no reader opens `archive/`, which `kvb-cleanup` offers for deletion. Run the install with every agent suspended (`kvb-org-suspend`) and the queue empty, so nothing writes either root during the move.
|
||||
|
||||
The global install also runs the Agent Orchestrator, a local review page in which the operator approves or rejects the agents' staged commits (the `kvb-git-diff` skill): it copies `orchestrator/` to `~/.local/share/kvb-orchestrator`, starts it there with `docker compose up -d --build` (restart `unless-stopped`, bound to `127.0.0.1` only, state in `~/.local/state/kvb-orchestrator`) and registers its MCP server for Claude Code at user scope. Its port is `KVB_ORCHESTRATOR_PORT` in `orchestrator/.env`; the hub page is `http://localhost:<port>/`, the MCP endpoint `/mcp`. Without docker the hub is skipped and the commit skills ask in chat. The hub reads new radio traffic aloud in maritime phraseology and takes push-to-talk input; its image builds both engines from pinned, sha256-verified sources and runs each as a separate process: Piper (`piper-tts`, GPL-3.0-or-later) with the `en_US-libritts-high` voice (904 speakers, trained on LibriTTS, CC BY 4.0, from `rhasspy/piper-voices`) and whisper.cpp (MIT) with the multilingual `ggml-base-q8_0` model (OpenAI Whisper weights, MIT, 82 MB). The operator's own voice is `KVB_ORCHESTRATOR_OPERATOR_VOICE` in `orchestrator/.env`. Its Tree reads only the file `pkgmgr branch tree` writes: the install clones `PKGMGR_REPO` (default the GitHub repository, never the sandbox-writable local checkout), builds the newest `v*` tag a fully trusted GPG key signed into `~/.local/share/kvb-orchestrator/pkgmgr`, and enables the `kvb-orchestrator-tree.timer` user timer, which writes `~/.local/state/pkgmgr/tree/branch-tree.json` a minute after its previous run while the hub page polls, on at most half a core; the hub mounts that directory read-only and starts no git. A signed tag without `branch tree` leaves the timer off.
|
||||
The global install also runs the Agent Orchestrator, a local review page in which the operator approves or rejects the agents' staged commits (the `kvb-git-diff` skill): it copies `orchestrator/` to `~/.local/share/kvb-orchestrator`, starts it there with `docker compose up -d --build` (restart `unless-stopped`, bound to `127.0.0.1` only, state in `~/.local/state/kvb-orchestrator`) and registers its MCP server for Claude Code at user scope. Its port is `KVB_ORCHESTRATOR_PORT` in `orchestrator/.env`; the hub page is `http://localhost:<port>/`, the MCP endpoint `/mcp`. Without docker the hub is skipped and the commit skills ask in chat. The hub reads new radio traffic aloud in maritime phraseology and takes push-to-talk input; its image builds both engines from pinned, sha256-verified sources and runs each as a separate process: Piper (`piper-tts`, GPL-3.0-or-later) with the `en_US-libritts-high` voice (904 speakers, trained on LibriTTS, CC BY 4.0, from `rhasspy/piper-voices`) and whisper.cpp (MIT) with the multilingual `ggml-small-q8_0` model (OpenAI Whisper weights, MIT, 264 MB). The operator's own voice is `KVB_ORCHESTRATOR_OPERATOR_VOICE` in `orchestrator/.env`. Its Tree reads only the file `pkgmgr branch tree` writes: the install clones `PKGMGR_REPO` (default the GitHub repository, never the sandbox-writable local checkout), builds the newest `v*` tag a fully trusted GPG key signed into `~/.local/share/kvb-orchestrator/pkgmgr`, and enables the `kvb-orchestrator-tree.timer` user timer, which writes `~/.local/state/pkgmgr/tree/branch-tree.json` a minute after its previous run while the hub page polls, on at most half a core; the hub mounts that directory read-only and starts no git. A signed tag without `branch tree` leaves the timer off.
|
||||
|
||||
Stream → Radio (`#stream/radio`) is the hub's radio interface. Its station list shows one line per station, the full name in the tooltip, with its state and a play or stop button; when the browser blocks autoplay, the stream fails or the CSP refuses its origin, the row and a toast say so, and one click starts the station. New stations play daily around the clock. While the hub reads a message aloud, the speaking agent pulses there and in the Orbit and Chord views, under the caption "<Agent> speaking" with the message's first line; under `prefers-reduced-motion` the ring stays still. The speech bubble icon in the header (a hub setting, kept across reloads) puts a bubble at each speaking agent in the Agents graph, 3D, Hierarchy, Radio and Stream views, or in a caption bar at the bottom when the view does not show that agent: the agent being read gets the largest bubble and its words appear in time with the audio, waiting messages show a smaller `…` bubble per agent, and without speech output the same rows run at 180 words a minute, at least 1.5 s each. The composer sends on all seven channels: post and call to one agent, newsletter, broadcast, pan-pan, security and mayday to all. "Record" records with MediaRecorder; the hub transcribes the recording with whisper.cpp off the request thread (at most 120 s of audio, a 120 s timeout, four at a time), keeps it for replay in `~/.local/state/kvb-orchestrator/voice` (the newest 50) and puts the transcript into the field to edit; the message goes out as text, with the recording's host path and sha256 only when "Attach recording" is ticked. The hub transcribes at most six recordings a minute and 600 s of audio an hour. Every operator message is passkey-signed, and the hub verifies the signature against the pinned keys, each once, before it relays; `operator_message(id, text, to)` answers `match` only for the signed text and recipient; for a denial the signature covers the recipient the inbox resolved at signing, and the answer's text is only the signed reason, never the agent's own words the radio row quotes. "Cancel mayday", Evening, Sunset, Sunrise and a settings save that adds a stream URL take the passkey the same way.
|
||||
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
# 013 - Voice Input
|
||||
|
||||
## User Story
|
||||
|
||||
As the operator, I want a microphone on every text field of the hub that
|
||||
turns my speech into text on this machine, so that I can work without a
|
||||
keyboard on a desktop and on a small screen.
|
||||
|
||||
## Decisions (operator-confirmed 2026-11-14; do not re-litigate)
|
||||
|
||||
1. **Everywhere**: one shared microphone component sits on every `textarea`
|
||||
and text input of the hub (composer, radio, feedback popup, marks,
|
||||
filters, settings), present by default. Rejected: an opt-in per field.
|
||||
2. **Local Whisper**: the browser records with MediaRecorder and sends the
|
||||
audio to the hub on this machine, which transcribes it with a local
|
||||
Whisper model; the audio never leaves the machine and is deleted right
|
||||
after the transcription. Rejected: the browser's speech recognition
|
||||
(Chrome sends audio to Google, Firefox has none); a browser fallback.
|
||||
3. **Insert**: the text lands at the cursor; German and English are detected
|
||||
automatically; a second tap stops the recording, Escape cancels it, and a
|
||||
visible indicator shows while it records.
|
||||
4. **Model**: the one-time model download is an operator row; without a
|
||||
model the button is disabled and its tooltip names that row.
|
||||
5. **Limits**: a recording is capped at 60 s and 5 MB; the transcription
|
||||
endpoint accepts requests only from the hub page, with the same origin
|
||||
and CSRF checks as every other POST of the hub.
|
||||
6. **Responsive**: touch targets of at least 44 px at 390 px; the component
|
||||
works with mouse, touch and keyboard.
|
||||
7. **Order**: built in parallel with 012 in its own sidetree; 012's
|
||||
feedback popup and marks get the microphone through the shared component.
|
||||
|
||||
## Approach
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph P["🧭 0 · Preparation"]
|
||||
P1["📋 Write requirement"] --> P2{"✅ Operator approves?"}
|
||||
P2 -- "✏️ corrections" --> P1
|
||||
end
|
||||
P2 -- "🤖 yes" --> S1
|
||||
subgraph S["🗣️ A · Local transcription"]
|
||||
S1["📥 Operator row: model download"] --> S2["🧩 /api/stt: caps, origin, CSRF, temp cleanup"]
|
||||
S2 --> S3["🧠 Local Whisper, German and English"]
|
||||
S3 --> ST{"🧪 Unit and fixture audio green?"}
|
||||
ST -- "🔴 no: fix" --> S2
|
||||
end
|
||||
ST -- "🟢 yes" --> M1
|
||||
subgraph M["🎙️ B · Microphone everywhere"]
|
||||
M1["🧩 Shared component on every text field"] --> M2["⏺️ Record, stop, cancel, indicator"]
|
||||
M2 --> M3["✍️ Insert at the cursor"]
|
||||
M3 --> MT{"🧪 E2E with a fake microphone green?"}
|
||||
MT -- "🔴 no: fix" --> M1
|
||||
MT -- "🟢 yes" --> MF{"💬 Operator feedback on screenshots?"}
|
||||
MF -- "✏️ changes" --> M1
|
||||
end
|
||||
MF -- "👍 fits" --> E1["🏁 Review, commit with operator OK, hand-over"]
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
### Transcription
|
||||
|
||||
- [ ] A POST of the German fixture WAV to `/api/stt` returns "hallo welt",
|
||||
the English fixture returns "hello world".
|
||||
- [ ] The transcription's temp directory is empty after each request,
|
||||
including a failed one.
|
||||
- [ ] A request from another origin, without the CSRF token, over 60 s or
|
||||
over 5 MB is refused with 4xx and is not transcribed.
|
||||
- [ ] Without an installed model, `/api/stt` answers with the operator row
|
||||
that installs it, and the button is disabled with that row in its
|
||||
tooltip.
|
||||
|
||||
### Microphone
|
||||
|
||||
- [ ] A DOM scan of every hub view finds no `textarea` or text input without
|
||||
the microphone button.
|
||||
- [ ] Pressing the button records, a second press stops and the transcript
|
||||
appears at the cursor; Escape cancels without inserting text; an
|
||||
indicator shows while it records.
|
||||
- [ ] At 390 px the button's touch target is at least 44 by 44 px.
|
||||
|
||||
### Tests
|
||||
|
||||
- [ ] Unit: the endpoint caps, the origin and CSRF checks, the temp cleanup
|
||||
and the language selection.
|
||||
- [ ] Integration: the real model transcribes both fixture files, run
|
||||
through `queue.py hold`.
|
||||
- [ ] E2E: Chromium and Firefox with a fake microphone stream fed from a
|
||||
WAV file, at 1920 and 390 px; the transcript appears in the composer;
|
||||
negative control: with the model missing the button is disabled.
|
||||
@@ -16,4 +16,5 @@ skill defines the format.
|
||||
- [010 - Security Scan Cache per File](010-security-scan-cache-per-file.md)
|
||||
- [011 - Test Result Cache](011-test-result-cache.md)
|
||||
- [012 - Match Review](012-match-review.md)
|
||||
- [013 - Voice Input](013-voice-input.md)
|
||||
- [014 - Header Rates](014-header-rates.md)
|
||||
@@ -50,9 +50,9 @@ ADD --checksum=sha256:2efdc6d7f954588b8180132cbd9b8001933fdd00932c92bc92fd0d2028
|
||||
${INFINITO_ARTEFACT_BASE:-https:/}/huggingface.co/rhasspy/piper-voices/resolve/e7229f7d33a10e145e109b886d661984201ac3c4/en/en_US/libritts/high/en_US-libritts-high.onnx.json \
|
||||
/opt/voices/en_US-libritts-high.onnx.json
|
||||
# hadolint ignore=DL3020
|
||||
ADD --checksum=sha256:c577b9a86e7e048a0b7eada054f4dd79a56bbfa911fbdacf900ac5b567cbb7d9 --chmod=0644 \
|
||||
${INFINITO_ARTEFACT_BASE:-https:/}/huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-base-q8_0.bin \
|
||||
/opt/whisper/ggml-base-q8_0.bin
|
||||
ADD --checksum=sha256:49c8fb02b65e6049d5fa6c04f81f53b867b5ec9540406812c643f177317f779f --chmod=0644 \
|
||||
${INFINITO_ARTEFACT_BASE:-https:/}/huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-small-q8_0.bin \
|
||||
/opt/whisper/ggml-small-q8_0.bin
|
||||
COPY --from=whisper-cli /out/whisper-cli /opt/whisper/whisper-cli
|
||||
# ADD --chmod=0644 also gives the directories it creates mode 0644; without the search bit the non-root hub finds no voice and no whisper.cpp
|
||||
RUN chmod 0755 /opt/voices /opt/whisper
|
||||
|
||||
@@ -77,7 +77,7 @@ services:
|
||||
- --whisper
|
||||
- /opt/whisper/whisper-cli
|
||||
- --whisper-model
|
||||
- /opt/whisper/ggml-base-q8_0.bin
|
||||
- /opt/whisper/ggml-small-q8_0.bin
|
||||
- --scratch
|
||||
- /scratch
|
||||
- --speech-url
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
import struct
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import wave
|
||||
@@ -30,6 +32,12 @@ STT_RATE = 16000
|
||||
STT_SECONDS = 120
|
||||
STT_LIMIT = 44 + STT_RATE * 2 * STT_SECONDS
|
||||
STT_TIMEOUT = 120
|
||||
INPUT_SECONDS = 60
|
||||
INPUT_LIMIT = 5 << 20
|
||||
MODEL_ROW = "make orchestrator-install"
|
||||
HEARING_DIR = "stt-"
|
||||
LANGUAGES = ("de", "en")
|
||||
SPECIAL = "[_"
|
||||
HEARING_WINDOW = 60
|
||||
HEARING_RATE = 6
|
||||
HEARING_HOUR = 3600
|
||||
@@ -100,7 +108,15 @@ def voice_of(key: str, pool: list[str], operator: str) -> str:
|
||||
return pool[int(digest(key), 16) % len(pool)]
|
||||
|
||||
|
||||
def wav_seconds(body: bytes) -> float:
|
||||
def wav_seconds(body: bytes, seconds: int) -> float:
|
||||
"""Args:
|
||||
body: the recording.
|
||||
seconds: the longest recording accepted.
|
||||
|
||||
Returns:
|
||||
Its length in seconds once it is 16 kHz mono 16-bit PCM WAV lasting
|
||||
more than 0 and at most seconds; otherwise it raises ToolError.
|
||||
"""
|
||||
try:
|
||||
with wave.open(io.BytesIO(body), "rb") as reader:
|
||||
shape = (reader.getnchannels(), reader.getsampwidth(), reader.getframerate(), reader.getcomptype())
|
||||
@@ -109,8 +125,8 @@ def wav_seconds(body: bytes) -> float:
|
||||
raise ToolError("the recording must be a WAV file") from None
|
||||
if shape != (1, 2, STT_RATE, "NONE"):
|
||||
raise ToolError(f"the recording must be {STT_RATE} Hz mono 16-bit PCM")
|
||||
if not 0 < frames <= STT_RATE * STT_SECONDS:
|
||||
raise ToolError(f"the recording must last more than 0 and at most {STT_SECONDS} seconds")
|
||||
if not 0 < frames <= STT_RATE * seconds:
|
||||
raise ToolError(f"the recording must last more than 0 and at most {seconds} seconds")
|
||||
return frames / STT_RATE
|
||||
|
||||
|
||||
@@ -156,6 +172,27 @@ def heard(output: str) -> str:
|
||||
return " ".join(NOISE.sub(" ", output).split())[:TEXT_LIMIT]
|
||||
|
||||
|
||||
def transcript(path: Path) -> tuple[str, str, float]:
|
||||
"""Args:
|
||||
path: the JSON whisper.cpp writes with --output-json-full.
|
||||
|
||||
Returns:
|
||||
(the language it names, its segment texts joined, the mean
|
||||
probability of its text tokens, 0.0 without any); a file that is
|
||||
missing or not of that shape raises EngineError.
|
||||
"""
|
||||
try:
|
||||
data = json.loads(path.read_bytes().decode("utf-8", errors="replace"))
|
||||
language, segments = data["result"]["language"], data["transcription"]
|
||||
texts = [segment["text"] for segment in segments]
|
||||
sure = [token["p"] for segment in segments for token in segment.get("tokens", []) if not str(token["text"]).startswith(SPECIAL)]
|
||||
except (OSError, ValueError, RecursionError, KeyError, IndexError, TypeError, AttributeError):
|
||||
raise EngineError("whisper.cpp wrote no readable JSON transcript") from None
|
||||
if not isinstance(language, str) or not all(isinstance(text, str) for text in texts) or not all(isinstance(p, int | float) and not isinstance(p, bool) for p in sure):
|
||||
raise EngineError("whisper.cpp wrote no readable JSON transcript")
|
||||
return language, " ".join(texts), sum(sure) / len(sure) if sure else 0.0
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Engines:
|
||||
piper: Path
|
||||
@@ -207,14 +244,15 @@ class Speech:
|
||||
def input_gap(self) -> str:
|
||||
"""Returns:
|
||||
"" when speech input works, otherwise why not, naming the
|
||||
whisper.cpp executable or model with its path; always "" with the
|
||||
speech service.
|
||||
whisper.cpp executable or model with its path and the operator row
|
||||
MODEL_ROW that installs both; always "" with the speech service.
|
||||
"""
|
||||
if self.engines.service.enabled():
|
||||
return ""
|
||||
model = self.engines.whisper_model
|
||||
gap = unreachable(self.engines.whisper, "the whisper.cpp executable", os.X_OK, "executable") or unreachable(model, "the whisper.cpp model", os.R_OK, "readable")
|
||||
return gap or ("" if os.path.isfile(model) else f"the whisper.cpp model {model} is not a file")
|
||||
gap = gap or ("" if os.path.isfile(model) else f"the whisper.cpp model {model} is not a file")
|
||||
return f"{gap}; the operator row {MODEL_ROW} installs it" if gap else ""
|
||||
|
||||
def voice_of(self, key: str) -> str:
|
||||
"""Args:
|
||||
@@ -233,7 +271,7 @@ class Speech:
|
||||
silent, deaf = self.output_gap(), self.input_gap()
|
||||
return {
|
||||
"tts": not silent, "stt": not deaf, "tts_off": silent, "stt_off": deaf, "voices": len(self.voices), "operator": operator, "sample": sample,
|
||||
"limit": SPEECH_LIMIT, "rate": STT_RATE, "seconds": STT_SECONDS,
|
||||
"limit": SPEECH_LIMIT, "rate": STT_RATE, "seconds": STT_SECONDS, "input": INPUT_SECONDS,
|
||||
}
|
||||
|
||||
def synthesize(self, body: bytes, admit: Callable[[str], None]) -> bytes:
|
||||
@@ -333,21 +371,40 @@ class Speech:
|
||||
gap = self.input_gap()
|
||||
if gap:
|
||||
raise Unavailable(f"speech input is off: {gap}")
|
||||
self.admit(wav_seconds(body))
|
||||
self.admit(wav_seconds(body, INPUT_SECONDS))
|
||||
return self.whisper(body)
|
||||
|
||||
def whisper(self, body: bytes) -> dict:
|
||||
"""Returns:
|
||||
{"text"} from the speech service, otherwise {"text", "language"}
|
||||
in one of LANGUAGES.
|
||||
"""
|
||||
if self.engines.service.enabled():
|
||||
with self.hearing:
|
||||
return {"text": heard(self.engines.service.hear(body))}
|
||||
command = [str(self.engines.whisper), "--model", str(self.engines.whisper_model), "--file", "-", "--output-txt", "--language", "auto", "--no-prints", "--suppress-nst"]
|
||||
with self.hearing:
|
||||
try:
|
||||
done = subprocess.run(command, input=body, capture_output=True, timeout=STT_TIMEOUT, env=self.env(), cwd=self.engines.scratch, check=False)
|
||||
except subprocess.TimeoutExpired:
|
||||
raise EngineError(f"whisper.cpp took longer than {STT_TIMEOUT} seconds") from None
|
||||
except OSError as error:
|
||||
raise EngineError(f"whisper.cpp did not start: {error.strerror}") from None
|
||||
with self.hearing, tempfile.TemporaryDirectory(prefix=HEARING_DIR, dir=self.engines.scratch) as folder:
|
||||
language, text, _ = self.listen(body, Path(folder), "auto")
|
||||
if language not in LANGUAGES:
|
||||
language, text, _ = max((self.listen(body, Path(folder), choice) for choice in LANGUAGES), key=lambda found: found[2])
|
||||
return {"text": heard(text), "language": language}
|
||||
|
||||
def listen(self, body: bytes, folder: Path, language: str) -> tuple[str, str, float]:
|
||||
"""Args:
|
||||
body: the recording.
|
||||
folder: the request's own directory, cwd and TMPDIR of the run.
|
||||
language: "auto" to detect it, otherwise the language to hear.
|
||||
|
||||
Returns:
|
||||
transcript() of the JSON whisper.cpp writes into folder.
|
||||
"""
|
||||
target = folder / language
|
||||
command = [str(self.engines.whisper), "--model", str(self.engines.whisper_model), "--file", "-", "--output-json", "--output-json-full", "--output-file", str(target), "--language", language, "--no-prints", "--suppress-nst"]
|
||||
try:
|
||||
done = subprocess.run(command, input=body, capture_output=True, timeout=STT_TIMEOUT, env={**self.env(), "TMPDIR": str(folder)}, cwd=folder, check=False)
|
||||
except subprocess.TimeoutExpired:
|
||||
raise EngineError(f"whisper.cpp took longer than {STT_TIMEOUT} seconds") from None
|
||||
except OSError as error:
|
||||
raise EngineError(f"whisper.cpp did not start: {error.strerror}") from None
|
||||
if done.returncode != 0:
|
||||
raise EngineError(f"whisper.cpp failed with exit code {done.returncode}: {done.stderr.decode('utf-8', errors='replace').strip()[-300:]}")
|
||||
return {"text": heard(done.stdout.decode("utf-8", errors="replace"))}
|
||||
return transcript(target.with_name(f"{language}.json"))
|
||||
@@ -8,7 +8,7 @@ from http import HTTPStatus
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from .diffs import DIFF_BODY_LIMIT
|
||||
from .engines import STT_LIMIT
|
||||
from .engines import INPUT_LIMIT, STT_LIMIT
|
||||
from .exchange import PLAIN_CSP, Exchange
|
||||
from .rates import rated
|
||||
from .resources import RESOURCES, sampled
|
||||
@@ -294,7 +294,7 @@ class Handler(Exchange):
|
||||
|
||||
def do_POST(self) -> None:
|
||||
path = urlsplit(self.path).path
|
||||
length = self.body_length({UPLOADS: UPLOAD_LIMIT, STT_ROUTE: STT_LIMIT, VOICE_ROUTE: STT_LIMIT, "/mcp": DIFF_BODY_LIMIT, "/api/asks": DIFF_BODY_LIMIT}.get(path, BODY_LIMIT))
|
||||
length = self.body_length({UPLOADS: UPLOAD_LIMIT, STT_ROUTE: INPUT_LIMIT, VOICE_ROUTE: STT_LIMIT, "/mcp": DIFF_BODY_LIMIT, "/api/asks": DIFF_BODY_LIMIT}.get(path, BODY_LIMIT))
|
||||
if length is None:
|
||||
return
|
||||
host = self.host()
|
||||
|
||||
@@ -10,7 +10,8 @@ from collections import Counter
|
||||
from collections.abc import Callable, Iterator
|
||||
from pathlib import Path
|
||||
|
||||
from .engines import Speech, wav_seconds
|
||||
from . import engines
|
||||
from .engines import Speech
|
||||
from .errors import EngineError, Missing, ToolError, Unavailable
|
||||
from .records import write_atomic
|
||||
|
||||
@@ -44,7 +45,7 @@ class VoiceNotes:
|
||||
gap = self.speech.input_gap()
|
||||
if gap:
|
||||
raise Unavailable(f"speech input is off: {gap}")
|
||||
seconds = wav_seconds(body)
|
||||
seconds = engines.wav_seconds(body, engines.STT_SECONDS)
|
||||
with self.lock:
|
||||
if sum(1 for note in self.notes.values() if note["status"] == "pending") >= HEARING_LIMIT:
|
||||
raise Unavailable(f"{HEARING_LIMIT} recordings are already waiting for their transcript")
|
||||
|
||||
@@ -320,7 +320,16 @@ textarea { width: 100%; min-height: 84px; padding: 8px 10px; border: 1px solid v
|
||||
.palette-box input { width: 100%; padding: 14px 16px; border: 0; background: transparent; font-size: var(--fs-lg); outline: none; }
|
||||
.palette-row { display: flex; align-items: center; gap: 6px; padding-right: 10px; border-bottom: 1px solid var(--line); }
|
||||
.palette-row input { flex: 1; min-width: 0; }
|
||||
.mic[aria-pressed="true"] { background: var(--bad-soft); color: var(--bad); }
|
||||
.mic { flex: none; }
|
||||
.mic[aria-pressed="true"] { background: var(--bad-soft); color: var(--bad); animation: mic-pulse 1.2s ease-in-out infinite; }
|
||||
.mic-auto { margin-left: 4px; vertical-align: middle; }
|
||||
textarea + .mic-auto { display: flex; margin: 4px 0 0 auto; }
|
||||
.mic-live { position: fixed; left: 50%; bottom: calc(18px + env(safe-area-inset-bottom, 0px)); z-index: 60; display: flex; align-items: center; gap: 8px; max-width: calc(100vw - 32px); padding: 8px 14px; transform: translateX(-50%); border-radius: 999px; background: var(--bad); color: #fff; font-weight: 650; box-shadow: var(--shadow-lift); }
|
||||
.mic-live[hidden] { display: none; }
|
||||
.mic-live::before { content: ""; flex: none; width: 10px; height: 10px; border-radius: 50%; background: #fff; animation: mic-pulse 1s ease-in-out infinite; }
|
||||
@keyframes mic-pulse { 50% { opacity: .45; } }
|
||||
@media (max-width: 640px) { .mic { min-width: 44px; min-height: 44px; } }
|
||||
@media (prefers-reduced-motion: reduce) { .mic[aria-pressed="true"], .mic-live::before { animation: none; } }
|
||||
.palette-list { max-height: 55vh; margin: 0; padding: 6px; overflow: auto; list-style: none; }
|
||||
.palette-list li { display: flex; align-items: center; gap: 10px; padding: 8px 10px; border-radius: 8px; cursor: pointer; }
|
||||
.palette-list li[aria-selected="true"] { background: var(--accent-soft); color: var(--accent); }
|
||||
@@ -617,6 +626,8 @@ pre.notes { margin: 0; padding: 8px 10px; background: var(--sunken); border-radi
|
||||
|
||||
{% include "page-speech.js" %}
|
||||
|
||||
{% include "page-mic.js" %}
|
||||
|
||||
{% include "page-bubbles.js" %}
|
||||
|
||||
{% include "page-alerts.js" %}
|
||||
|
||||
@@ -94,7 +94,7 @@
|
||||
const classes = choice("to-one", TO_ONE, "post");
|
||||
const form = h("form", {class: "popup-form"}, text, file, classes,
|
||||
h("p", {class: "muted"}, "Post arrives at the end of the turn, Call at once with a read receipt; your passkey signs, verifiable with operator_message."),
|
||||
h("div", {class: "actions"}, h("span", {class: "spacer"}), micButton(text), button));
|
||||
h("div", {class: "actions"}, h("span", {class: "spacer"}), micButton(text, false), button));
|
||||
form.addEventListener("submit", async (event) => {
|
||||
event.preventDefault();
|
||||
if (!text.value.trim()) {
|
||||
|
||||
@@ -39,6 +39,7 @@
|
||||
window.addEventListener("resize", placePopup);
|
||||
window.addEventListener("resize", placeStations);
|
||||
skyPanel();
|
||||
micWatch();
|
||||
radioSpot();
|
||||
radio.element = radioElement();
|
||||
drawSky();
|
||||
|
||||
@@ -78,7 +78,7 @@
|
||||
throw new Error("no recording");
|
||||
}
|
||||
parts.status.textContent = "Converting …";
|
||||
const stored = await send("/api/voice", await wavOf(blob, sound), "audio/wav");
|
||||
const stored = await send("/api/voice", await wavOf(blob, sound, state.speech.seconds), "audio/wav");
|
||||
if (!stored.ok || !stored.data) {
|
||||
throw new Error(`${stored.status} ${stored.text}`.trim());
|
||||
}
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
const MIC_HINT = "Voice input: tap to record, tap again to insert the text at the cursor; Escape cancels";
|
||||
const MIC_NODES = `${Voice.FIELDS}, button.mic`;
|
||||
const micButtons = new WeakMap();
|
||||
const micFields = new WeakMap();
|
||||
let micCount = 0;
|
||||
|
||||
function micLive() {
|
||||
let live = document.getElementById("mic-live");
|
||||
if (!live) {
|
||||
live = h("div", {id: "mic-live", class: "mic-live", role: "status", "aria-live": "polite", hidden: true});
|
||||
document.body.append(live);
|
||||
}
|
||||
return live;
|
||||
}
|
||||
|
||||
function micState(button) {
|
||||
const reason = micGap();
|
||||
button.disabled = Boolean(reason);
|
||||
button.title = reason || MIC_HINT;
|
||||
}
|
||||
|
||||
function micLook(button, look) {
|
||||
const recording = look === "recording";
|
||||
button.setAttribute("aria-pressed", String(recording));
|
||||
button.setAttribute("aria-label", recording ? "Stop voice input" : "Voice input");
|
||||
button.replaceChildren(icon(recording ? "bi-stop-fill" : look === "busy" ? "bi-hourglass-split" : "bi-mic"));
|
||||
micLive().hidden = !recording;
|
||||
if (look === "busy") {
|
||||
button.disabled = true;
|
||||
} else {
|
||||
micState(button);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {HTMLInputElement|HTMLTextAreaElement} field the text field the transcript lands in.
|
||||
* @param {boolean} auto whether the shared component placed the button right after the field and keeps it there.
|
||||
* @returns {HTMLButtonElement} the microphone button of the field.
|
||||
*/
|
||||
function micButton(field, auto) {
|
||||
if (!field.id) {
|
||||
micCount += 1;
|
||||
field.id = `mic-field-${micCount}`;
|
||||
}
|
||||
const button = h("button", {type: "button", class: auto ? "btn small ghost icon-only mic mic-auto" : "btn small ghost icon-only mic", "aria-controls": field.id}, icon("bi-mic"));
|
||||
micButtons.set(field, button);
|
||||
micFields.set(button, field);
|
||||
field.dataset.mic = "on";
|
||||
micLook(button, "idle");
|
||||
button.addEventListener("click", () => toggleMic(field, button));
|
||||
return button;
|
||||
}
|
||||
|
||||
function micSweep() {
|
||||
for (const field of document.querySelectorAll(Voice.FIELDS)) {
|
||||
const button = micButtons.get(field);
|
||||
if (field.readOnly || (button && !button.classList.contains("mic-auto"))) {
|
||||
continue;
|
||||
}
|
||||
if (!button) {
|
||||
field.after(micButton(field, true));
|
||||
} else if (button.previousElementSibling !== field) {
|
||||
field.after(button);
|
||||
}
|
||||
}
|
||||
for (const button of document.querySelectorAll("button.mic-auto")) {
|
||||
if (!micFields.get(button).isConnected) {
|
||||
button.remove();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function micTouched(records) {
|
||||
return records.some((record) => [...record.addedNodes, ...record.removedNodes].some((node) => node instanceof Element && (node.matches(MIC_NODES) || Boolean(node.querySelector(MIC_NODES)))));
|
||||
}
|
||||
|
||||
function micWatch() {
|
||||
micSweep();
|
||||
new MutationObserver((records) => micTouched(records) && micSweep()).observe(document.body, {childList: true, subtree: true});
|
||||
window.addEventListener("keydown", (event) => {
|
||||
if (event.key === "Escape" && mic.button && !mic.busy) {
|
||||
event.preventDefault();
|
||||
event.stopImmediatePropagation();
|
||||
stopMic(true);
|
||||
}
|
||||
}, true);
|
||||
}
|
||||
|
||||
function stopMic(cancel) {
|
||||
mic.cancel = cancel;
|
||||
window.clearInterval(mic.timer);
|
||||
if (mic.recorder && mic.recorder.state === "recording") {
|
||||
mic.recorder.stop();
|
||||
}
|
||||
}
|
||||
|
||||
function micTick() {
|
||||
const seconds = Math.floor((Date.now() - mic.started) / 1000);
|
||||
micLive().textContent = `Recording ${seconds} s of at most ${state.speech.input} s: tap the microphone again to insert, Escape cancels`;
|
||||
if (seconds >= state.speech.input) {
|
||||
stopMic(false);
|
||||
}
|
||||
}
|
||||
|
||||
function micDone() {
|
||||
Object.assign(mic, {recorder: null, button: null, field: null, busy: false, cancel: false});
|
||||
}
|
||||
|
||||
async function toggleMic(field, button) {
|
||||
if (mic.button) {
|
||||
if (mic.button === button && mic.recorder) {
|
||||
stopMic(false);
|
||||
}
|
||||
return;
|
||||
}
|
||||
Object.assign(mic, {button, field, cancel: false});
|
||||
let media = null;
|
||||
try {
|
||||
media = await navigator.mediaDevices.getUserMedia({audio: {channelCount: 1, echoCancellation: true, noiseSuppression: true}});
|
||||
} catch (error) {
|
||||
micDone();
|
||||
toast("Microphone not available", [String(error.message || error)], false);
|
||||
return;
|
||||
}
|
||||
if (mic.cancel) {
|
||||
media.getTracks().forEach((track) => track.stop());
|
||||
micDone();
|
||||
return;
|
||||
}
|
||||
const recorder = new MediaRecorder(media);
|
||||
const chunks = [];
|
||||
recorder.addEventListener("dataavailable", (event) => {
|
||||
if (event.data.size) {
|
||||
chunks.push(event.data);
|
||||
}
|
||||
});
|
||||
recorder.addEventListener("stop", () => {
|
||||
media.getTracks().forEach((track) => track.stop());
|
||||
window.clearInterval(mic.timer);
|
||||
const cancelled = mic.cancel;
|
||||
micLook(button, "idle");
|
||||
if (cancelled) {
|
||||
micDone();
|
||||
return;
|
||||
}
|
||||
mic.recorder = null;
|
||||
micTranscribe(new Blob(chunks, {type: recorder.mimeType}), field, button);
|
||||
}, {once: true});
|
||||
Object.assign(mic, {recorder, started: Date.now()});
|
||||
recorder.start();
|
||||
micLook(button, "recording");
|
||||
micTick();
|
||||
mic.timer = window.setInterval(micTick, 250);
|
||||
}
|
||||
|
||||
function micInsert(field, text) {
|
||||
const start = field.selectionStart ?? field.value.length;
|
||||
const spliced = Voice.spliced(field.value, start, field.selectionEnd ?? start, text);
|
||||
field.value = spliced.value;
|
||||
field.focus();
|
||||
field.setSelectionRange(spliced.caret, spliced.caret);
|
||||
field.dispatchEvent(new Event("input", {bubbles: true}));
|
||||
}
|
||||
|
||||
async function micTranscribe(blob, field, button) {
|
||||
const sound = context();
|
||||
if (!sound || !blob.size) {
|
||||
micDone();
|
||||
return;
|
||||
}
|
||||
mic.busy = true;
|
||||
micLook(button, "busy");
|
||||
try {
|
||||
const result = await send("/api/stt", await wavOf(blob, sound, state.speech.input), "audio/wav");
|
||||
if (!result.ok || !result.data) {
|
||||
throw new Error(`${result.status} ${result.text}`.trim());
|
||||
}
|
||||
micInsert(field, result.data.text || "");
|
||||
} catch (error) {
|
||||
toast("Voice input failed", [String(error.message || error)], false);
|
||||
} finally {
|
||||
micDone();
|
||||
micLook(button, "idle");
|
||||
}
|
||||
}
|
||||
@@ -142,7 +142,7 @@
|
||||
runPalette(palette.index);
|
||||
}
|
||||
});
|
||||
box.replaceChildren(h("div", {class: "palette-box", role: "dialog", "aria-label": "Commands"}, h("div", {class: "palette-row"}, palette.input, micButton(palette.input)), palette.list));
|
||||
box.replaceChildren(h("div", {class: "palette-box", role: "dialog", "aria-label": "Commands"}, h("div", {class: "palette-row"}, palette.input, micButton(palette.input, false)), palette.list));
|
||||
box.hidden = false;
|
||||
drawPalette();
|
||||
palette.input.focus();
|
||||
|
||||
@@ -309,84 +309,10 @@
|
||||
: !window.MediaRecorder ? "This browser cannot record" : !state.speech.stt ? `Voice input is off: ${state.speech.stt_off}` : "";
|
||||
}
|
||||
|
||||
function micButton(field) {
|
||||
const button = h("button", {type: "button", class: "btn small ghost icon-only mic", "aria-pressed": "false", ...named("Voice input", "Voice input: hold and speak; the text lands in the field and is not sent")}, icon("bi-mic"));
|
||||
const reason = micGap();
|
||||
if (reason) {
|
||||
button.disabled = true;
|
||||
button.title = `Voice input: ${reason}`;
|
||||
return button;
|
||||
}
|
||||
const press = (event) => {
|
||||
event.preventDefault();
|
||||
record(field, button);
|
||||
};
|
||||
button.addEventListener("pointerdown", press);
|
||||
for (const name of ["pointerup", "pointerleave", "pointercancel"]) {
|
||||
button.addEventListener(name, release);
|
||||
}
|
||||
button.addEventListener("keydown", (event) => {
|
||||
if ((event.key === " " || event.key === "Enter") && !event.repeat) {
|
||||
press(event);
|
||||
}
|
||||
});
|
||||
button.addEventListener("keyup", (event) => {
|
||||
if (event.key === " " || event.key === "Enter") {
|
||||
event.preventDefault();
|
||||
release();
|
||||
}
|
||||
});
|
||||
return button;
|
||||
}
|
||||
|
||||
async function record(field, button) {
|
||||
if (mic.held || mic.recorder || mic.busy) {
|
||||
return;
|
||||
}
|
||||
mic.held = true;
|
||||
let media = null;
|
||||
try {
|
||||
media = await navigator.mediaDevices.getUserMedia({audio: {channelCount: 1, echoCancellation: true, noiseSuppression: true}});
|
||||
} catch (error) {
|
||||
mic.held = false;
|
||||
toast("Microphone not available", [String(error.message || error)], false);
|
||||
return;
|
||||
}
|
||||
if (!mic.held) {
|
||||
media.getTracks().forEach((track) => track.stop());
|
||||
return;
|
||||
}
|
||||
const recorder = new MediaRecorder(media);
|
||||
const chunks = [];
|
||||
recorder.addEventListener("dataavailable", (event) => {
|
||||
if (event.data.size) {
|
||||
chunks.push(event.data);
|
||||
}
|
||||
});
|
||||
recorder.addEventListener("stop", () => {
|
||||
media.getTracks().forEach((track) => track.stop());
|
||||
mic.recorder = null;
|
||||
button.setAttribute("aria-pressed", "false");
|
||||
transcribe(new Blob(chunks, {type: recorder.mimeType}), field, button);
|
||||
}, {once: true});
|
||||
mic.recorder = recorder;
|
||||
recorder.start();
|
||||
button.setAttribute("aria-pressed", "true");
|
||||
mic.timer = window.setTimeout(release, state.speech.seconds * 1000);
|
||||
}
|
||||
|
||||
function release() {
|
||||
mic.held = false;
|
||||
window.clearTimeout(mic.timer);
|
||||
if (mic.recorder && mic.recorder.state === "recording") {
|
||||
mic.recorder.stop();
|
||||
}
|
||||
}
|
||||
|
||||
async function wavOf(blob, sound) {
|
||||
async function wavOf(blob, sound, seconds) {
|
||||
const decoded = await sound.decodeAudioData(await blob.arrayBuffer());
|
||||
const rate = state.speech.rate;
|
||||
const length = Voice.frames(decoded.duration, rate, state.speech.seconds);
|
||||
const length = Voice.frames(decoded.duration, rate, seconds);
|
||||
if (length < rate / 4) {
|
||||
throw new Error("too short: speak for at least a quarter second");
|
||||
}
|
||||
@@ -398,28 +324,3 @@
|
||||
const rendered = await offline.startRendering();
|
||||
return Voice.wav(rendered.getChannelData(0), rate);
|
||||
}
|
||||
|
||||
async function transcribe(blob, field, button) {
|
||||
const sound = context();
|
||||
if (!sound || !blob.size) {
|
||||
return;
|
||||
}
|
||||
mic.busy = true;
|
||||
button.disabled = true;
|
||||
try {
|
||||
const result = await send("/api/stt", await wavOf(blob, sound), "audio/wav");
|
||||
if (!result.ok || !result.data) {
|
||||
throw new Error(`${result.status} ${result.text}`.trim());
|
||||
}
|
||||
if (result.data.text) {
|
||||
field.value = [field.value.trim(), result.data.text].filter(Boolean).join(" ");
|
||||
field.dispatchEvent(new Event("input", {bubbles: true}));
|
||||
}
|
||||
field.focus();
|
||||
} catch (error) {
|
||||
toast("Voice input failed", [String(error.message || error)], false);
|
||||
} finally {
|
||||
mic.busy = false;
|
||||
button.disabled = false;
|
||||
}
|
||||
}
|
||||
@@ -47,7 +47,7 @@
|
||||
const radio = {element: null, url: "", level: 0, timer: 0, playing: false, waiting: false, index: -1, pick: null, sig: "", state: "", short: "", note: "", told: null};
|
||||
const speech = {pending: [], busy: false, warned: 0, current: null, source: null, loading: null, timer: 0, item: null};
|
||||
let radioSet = RadioSet.valid(stored(RadioSet.KEY, null));
|
||||
const mic = {recorder: null, held: false, busy: false, timer: 0};
|
||||
const mic = {recorder: null, button: null, field: null, busy: false, cancel: false, timer: 0, started: 0};
|
||||
const space = {
|
||||
views: {graph: {...state.settings.space.graph}, sky: {...state.settings.space.sky}}, glides: {graph: null, sky: null}, edits: 0, saved: 0, timer: 0, look: false, compass: null,
|
||||
};
|
||||
|
||||
@@ -3,6 +3,7 @@ const Voice = (() => {
|
||||
const RANKS = {mayday: 0, "pan-pan": 1, security: 2};
|
||||
const ALARMS = new Set(["mayday", "pan-pan"]);
|
||||
const ROUTINE = 3;
|
||||
const FIELDS = 'textarea, input:not([type]), input[type="text"], input[type="search"]';
|
||||
|
||||
function toneOf(row, human) {
|
||||
if (ALARMS.has(row.class)) {
|
||||
@@ -59,7 +60,26 @@ const Voice = (() => {
|
||||
return buffer;
|
||||
}
|
||||
|
||||
return {toneOf, loudness, beep, enqueue, frames, wav};
|
||||
/**
|
||||
* @param {string} value the text of the field.
|
||||
* @param {number} start the start of its selection.
|
||||
* @param {number} end the end of its selection.
|
||||
* @param {string} text the transcript.
|
||||
* @returns {{value: string, caret: number}} value with the trimmed transcript in place of the selection, one space on each side where a neighbour is no whitespace, and the caret right after the transcript.
|
||||
*/
|
||||
function spliced(value, start, end, text) {
|
||||
const words = String(text).trim();
|
||||
if (!words) {
|
||||
return {value, caret: end};
|
||||
}
|
||||
const before = value.slice(0, start);
|
||||
const after = value.slice(end);
|
||||
const lead = before && !/\s$/.test(before) ? " " : "";
|
||||
const tail = after && !/^\s/.test(after) ? " " : "";
|
||||
return {value: `${before}${lead}${words}${tail}${after}`, caret: before.length + lead.length + words.length};
|
||||
}
|
||||
|
||||
return {FIELDS, toneOf, loudness, beep, enqueue, frames, wav, spliced};
|
||||
})();
|
||||
if (typeof module === "object" && module.exports) {
|
||||
module.exports = Voice;
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import dataclasses
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
sys.path[:0] = [str(HERE.parent), str(HERE)]
|
||||
|
||||
import test_orchestrator_e2e as base
|
||||
from hub_case import HubCase
|
||||
from kvb_orchestrator import engines
|
||||
|
||||
OUT = Path(os.environ.get("E2E_OUT", "/tmp/voice-input-shots"))
|
||||
WIDTHS = ((1920, 1080), (390, 844))
|
||||
FIXTURE = HERE.parent / "fixtures" / "voice" / "hello-world.wav"
|
||||
HEARD = "hello from the mic second line"
|
||||
FAKE_MIC = """() => {
|
||||
const wav = Uint8Array.from(atob("@WAV@"), (char) => char.charCodeAt(0));
|
||||
const posted = [];
|
||||
window.__voice = {posted, opened: 0};
|
||||
const fetched = window.fetch.bind(window);
|
||||
window.fetch = (path, init) => {
|
||||
if (String(path) === "/api/stt") {
|
||||
posted.push(Date.now());
|
||||
}
|
||||
return fetched(path, init);
|
||||
};
|
||||
if (navigator.mediaDevices) {
|
||||
navigator.mediaDevices.getUserMedia = async () => {
|
||||
window.__voice.opened += 1;
|
||||
const sound = new AudioContext();
|
||||
sound.resume();
|
||||
const source = sound.createBufferSource();
|
||||
source.buffer = await sound.decodeAudioData(wav.slice().buffer);
|
||||
source.loop = true;
|
||||
const sink = sound.createMediaStreamDestination();
|
||||
source.connect(sink);
|
||||
source.start();
|
||||
return sink.stream;
|
||||
};
|
||||
}
|
||||
}"""
|
||||
MIC = """
|
||||
const FIELDS = 'textarea, input:not([type]), input[type="text"], input[type="search"]';
|
||||
const micOf = (field) => (field && field.id ? document.querySelector(`button.mic[aria-controls="${CSS.escape(field.id)}"]`) : null);
|
||||
const bare = () => all(FIELDS).filter((field) => !field.readOnly && !micOf(field)).map((field) => field.getAttribute("aria-label") || field.placeholder || field.outerHTML.slice(0, 80));
|
||||
const composer = () => document.querySelector('#radio-spot textarea[aria-label="Message"]');
|
||||
const live = () => {
|
||||
const found = document.getElementById("mic-live");
|
||||
return found && !found.hidden ? text(found) : null;
|
||||
};
|
||||
const look = (button) => {
|
||||
const box = button.getBoundingClientRect();
|
||||
return {pressed: button.getAttribute("aria-pressed"), disabled: button.disabled, title: button.title, width: box.width, height: box.height, live: live()};
|
||||
};
|
||||
const escape = () => document.dispatchEvent(new KeyboardEvent("keydown", {key: "Escape", bubbles: true, cancelable: true}));
|
||||
const openComposer = async () => {
|
||||
const spot = document.getElementById("radio-spot");
|
||||
if (spot.hidden) {
|
||||
document.getElementById("radio-open").click();
|
||||
}
|
||||
const field = await until(() => composer(), 4000);
|
||||
field.scrollIntoView({block: "center"});
|
||||
return field;
|
||||
};
|
||||
"""
|
||||
SCAN = """(async () => {@HELPERS@@MIC@
|
||||
const missing = {};
|
||||
let scanned = 0;
|
||||
for (const screen of @SCREENS@) {
|
||||
location.hash = `#${screen}`;
|
||||
await screenShown(8000);
|
||||
await sleep(250);
|
||||
scanned += all(FIELDS).length;
|
||||
const found = bare();
|
||||
if (found.length) {
|
||||
missing[screen] = found;
|
||||
}
|
||||
}
|
||||
for (const [name, opener] of [["settings", "settings-open"], ["palette", "palette-open"]]) {
|
||||
document.getElementById(opener).click();
|
||||
await sleep(500);
|
||||
scanned += all(FIELDS).length;
|
||||
const found = bare();
|
||||
if (found.length) {
|
||||
missing[name] = found;
|
||||
}
|
||||
escape();
|
||||
await sleep(300);
|
||||
}
|
||||
location.hash = "#stream";
|
||||
await screenShown(8000);
|
||||
const field = await openComposer();
|
||||
field.value = "start end";
|
||||
field.focus();
|
||||
field.setSelectionRange(6, 6);
|
||||
await sleep(400);
|
||||
return {missing, scanned, idle: look(micOf(field))};
|
||||
})()"""
|
||||
RECORD = """(async () => {@HELPERS@@MIC@
|
||||
const button = micOf(composer());
|
||||
button.click();
|
||||
await until(() => button.getAttribute("aria-pressed") === "true" && live(), 8000);
|
||||
await sleep(300);
|
||||
return {...look(button), opened: window.__voice.opened};
|
||||
})()"""
|
||||
STOP = """(async () => {@HELPERS@@MIC@
|
||||
const field = composer();
|
||||
const button = micOf(field);
|
||||
button.click();
|
||||
await until(() => field.value.includes("@HEARD@"), 20000);
|
||||
await sleep(200);
|
||||
return {value: field.value, caret: field.selectionStart, focused: document.activeElement === field, posts: window.__voice.posted.length, ...look(button)};
|
||||
})()"""
|
||||
CANCEL = """(async () => {@HELPERS@@MIC@
|
||||
const field = composer();
|
||||
const button = micOf(field);
|
||||
const before = [field.value, window.__voice.posted.length];
|
||||
button.click();
|
||||
const recording = Boolean(await until(() => button.getAttribute("aria-pressed") === "true" && live(), 8000));
|
||||
await sleep(800);
|
||||
escape();
|
||||
await until(() => button.getAttribute("aria-pressed") === "false" && !live(), 8000);
|
||||
await sleep(1500);
|
||||
return {recording, same: field.value === before[0], posts: window.__voice.posted.length - before[1], ...look(button)};
|
||||
})()"""
|
||||
DEAF = """(async () => {@HELPERS@@MIC@
|
||||
await screenShown(8000);
|
||||
const field = await openComposer();
|
||||
const button = micOf(field);
|
||||
button.click();
|
||||
await sleep(800);
|
||||
return {...look(button), opened: window.__voice.opened, posts: window.__voice.posted.length};
|
||||
})()"""
|
||||
|
||||
|
||||
def probe(source: str) -> str:
|
||||
return source.replace("@HELPERS@", base.HELPERS).replace("@MIC@", MIC).replace("@SCREENS@", json.dumps(base.SCREENS)).replace("@HEARD@", HEARD)
|
||||
|
||||
|
||||
def fake_mic() -> str:
|
||||
return FAKE_MIC.replace("@WAV@", base64.b64encode(FIXTURE.read_bytes()).decode())
|
||||
|
||||
|
||||
@unittest.skipUnless(shutil.which(os.environ.get("CHROMIUM", "chromium")), "no Chromium")
|
||||
class TestVoiceInputE2E(HubCase):
|
||||
drive = base.TestOrchestratorE2E.drive
|
||||
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def steps(self, browser: str) -> list[dict]:
|
||||
steps = []
|
||||
for width, height in WIDTHS:
|
||||
url = f"http://localhost:{self.port}/?{width}#stream"
|
||||
steps += [
|
||||
{"width": width, "height": height, "url": url, "wait": 1500, "probe": probe(SCAN), "shot": str(OUT / f"{browser}-{width}-idle.png"), "gesture": True},
|
||||
{"width": width, "height": height, "wait": 200, "probe": probe(RECORD), "shot": str(OUT / f"{browser}-{width}-recording.png"), "gesture": True},
|
||||
{"width": width, "height": height, "wait": 1200, "probe": probe(STOP), "gesture": True},
|
||||
{"width": width, "height": height, "wait": 200, "probe": probe(CANCEL), "gesture": True},
|
||||
]
|
||||
return steps
|
||||
|
||||
def deaf_steps(self) -> list[dict]:
|
||||
return [{"width": width, "height": height, "url": f"http://localhost:{self.port}/?{width}#stream", "wait": 1500, "probe": probe(DEAF), "gesture": True} for width, height in WIDTHS]
|
||||
|
||||
def deafen(self) -> None:
|
||||
speech = self.httpd.hub.speech
|
||||
patch = mock.patch.object(speech, "engines", dataclasses.replace(speech.engines, whisper_model=self.outside / "ggml-missing.bin"))
|
||||
patch.start()
|
||||
self.addCleanup(patch.stop)
|
||||
|
||||
def errors(self, data: dict) -> list[str]:
|
||||
return [error for error in data["errors"] if "/favicon.ico" not in error]
|
||||
|
||||
def check(self, data: dict, browser: str) -> None:
|
||||
self.assertEqual(self.errors(data), [])
|
||||
for index, (width, _) in enumerate(WIDTHS):
|
||||
scan, recording, stopped, cancelled = data["results"][index * 4 : index * 4 + 4]
|
||||
with self.subTest(browser=browser, width=width):
|
||||
for found in (scan, recording, stopped, cancelled):
|
||||
self.assertNotIn("error", found, found)
|
||||
self.assertEqual(scan["missing"], {}, scan)
|
||||
self.assertGreater(scan["scanned"], 0, scan)
|
||||
idle = scan["idle"]
|
||||
self.assertEqual((idle["pressed"], idle["disabled"], idle["live"]), ("false", False, None), idle)
|
||||
self.assertEqual((recording["pressed"], recording["opened"]), ("true", 1), recording)
|
||||
self.assertIn("Recording", recording["live"] or "", recording)
|
||||
self.assertEqual(stopped["value"], f"start {HEARD} end", stopped)
|
||||
self.assertEqual((stopped["caret"], stopped["focused"], stopped["posts"], stopped["pressed"], stopped["live"]), (len(f"start {HEARD}"), True, 1, "false", None), stopped)
|
||||
self.assertEqual((cancelled["recording"], cancelled["same"], cancelled["posts"], cancelled["pressed"], cancelled["live"]), (True, True, 0, "false", None), cancelled)
|
||||
if width < 640:
|
||||
for shown in (idle, recording):
|
||||
self.assertGreaterEqual(min(shown["width"], shown["height"]), 44, shown)
|
||||
|
||||
def check_deaf(self, data: dict, browser: str) -> None:
|
||||
self.assertEqual(self.errors(data), [])
|
||||
for (width, _), found in zip(WIDTHS, data["results"], strict=True):
|
||||
with self.subTest(browser=browser, width=width):
|
||||
self.assertNotIn("error", found, found)
|
||||
self.assertEqual((found["disabled"], found["pressed"], found["opened"], found["posts"], found["live"]), (True, "false", 0, 0, None), found)
|
||||
self.assertIn(f"the operator row {engines.MODEL_ROW} installs it", found["title"], found)
|
||||
|
||||
def test_chromium_records_from_a_fake_microphone_and_inserts_the_transcript_at_the_cursor_at_1920_and_390(self):
|
||||
data = self.drive(self.steps("chromium"), None, init=fake_mic())
|
||||
(OUT / "chromium-voice-input.json").write_text(json.dumps(data, indent=1), encoding="utf-8")
|
||||
self.check(data, "chromium")
|
||||
|
||||
@unittest.skipUnless(shutil.which(os.environ.get("FIREFOX", "firefox")), "no Firefox")
|
||||
def test_firefox_records_from_a_fake_microphone_and_inserts_the_transcript_at_the_cursor_at_1920_and_390(self):
|
||||
data = self.drive(self.steps("firefox"), None, "bidi.mjs", os.environ.get("FIREFOX", "firefox"), init=fake_mic())
|
||||
(OUT / "firefox-voice-input.json").write_text(json.dumps(data, indent=1), encoding="utf-8")
|
||||
self.check(data, "firefox")
|
||||
|
||||
def test_chromium_disables_the_microphone_and_names_the_operator_row_without_a_model(self):
|
||||
self.deafen()
|
||||
data = self.drive(self.deaf_steps(), None, init=fake_mic())
|
||||
(OUT / "chromium-voice-input-deaf.json").write_text(json.dumps(data, indent=1), encoding="utf-8")
|
||||
self.check_deaf(data, "chromium")
|
||||
|
||||
@unittest.skipUnless(shutil.which(os.environ.get("FIREFOX", "firefox")), "no Firefox")
|
||||
def test_firefox_disables_the_microphone_and_names_the_operator_row_without_a_model(self):
|
||||
self.deafen()
|
||||
data = self.drive(self.deaf_steps(), None, "bidi.mjs", os.environ.get("FIREFOX", "firefox"), init=fake_mic())
|
||||
(OUT / "firefox-voice-input-deaf.json").write_text(json.dumps(data, indent=1), encoding="utf-8")
|
||||
self.check_deaf(data, "firefox")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Vendored
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
@@ -31,13 +31,16 @@ with wave.open(args.output_file, "wb") as out:
|
||||
"""
|
||||
FAKE_WHISPER = """
|
||||
import json, os, sys
|
||||
argv = sys.argv[1:]
|
||||
data = sys.stdin.buffer.read()
|
||||
with open(os.path.join(os.environ["HOME"], "whisper.jsonl"), "a", encoding="utf-8") as log:
|
||||
log.write(json.dumps({"argv": sys.argv[1:], "sha256": __import__("hashlib").sha256(data).hexdigest()}) + "\\n")
|
||||
log.write(json.dumps({"argv": argv, "sha256": __import__("hashlib").sha256(data).hexdigest()}) + "\\n")
|
||||
if b"FAIL" in data:
|
||||
sys.exit(4)
|
||||
print(" hello from the [BLANK_AUDIO] mic")
|
||||
print("second line")
|
||||
language = argv[argv.index("--language") + 1]
|
||||
segments = [{"text": " hello from the [BLANK_AUDIO] mic", "tokens": [{"text": "[_BEG_]", "p": 0.1}, {"text": " hello", "p": 0.9}]}, {"text": "second line", "tokens": [{"text": " second", "p": 0.8}]}]
|
||||
with open(argv[argv.index("--output-file") + 1] + ".json", "w", encoding="utf-8") as out:
|
||||
json.dump({"result": {"language": "en" if language == "auto" else language}, "transcription": segments}, out)
|
||||
"""
|
||||
|
||||
|
||||
@@ -53,7 +56,7 @@ def write_engines(base: Path) -> server.Engines:
|
||||
(voices / f"{model}.onnx").write_bytes(b"onnx")
|
||||
(voices / f"{model}.onnx.json").write_text(json.dumps(config), encoding="utf-8")
|
||||
(voices / "broken.onnx").write_bytes(b"onnx")
|
||||
model = base / "ggml-base.bin"
|
||||
model = base / "ggml-small-q8_0.bin"
|
||||
model.write_bytes(b"ggml")
|
||||
return server.Engines(piper=bin_dir / "piper", voices=voices, operator_voice=OPERATOR_VOICE, whisper=bin_dir / "whisper-cli", whisper_model=model, scratch=scratch, service=server.Service(url="", key=""))
|
||||
|
||||
|
||||
@@ -18,6 +18,8 @@ from hub_engines import OPERATOR_VOICE, VOICES, log_rows, wav_bytes
|
||||
from hub_support import HOST_STATE, HUB, server
|
||||
from kvb_orchestrator import council, engines, exchange, handler, phrasing
|
||||
|
||||
ROW = f"; the operator row {engines.MODEL_ROW} installs it"
|
||||
|
||||
|
||||
class TestHubSpeech(HubCase):
|
||||
def test_voices_come_from_the_models_and_stick_to_their_session(self):
|
||||
@@ -52,9 +54,9 @@ class TestHubSpeech(HubCase):
|
||||
({"piper": self.outside / "piper"}, "tts_off", f"the Piper executable {self.outside / 'piper'} is missing or not executable"),
|
||||
({"voices": self.outside}, "tts_off", f"the voice directory {self.outside} holds no <model>.onnx voice"),
|
||||
({"voices": lone}, "tts_off", f"the voice config {lone / 'solo.onnx'}.json is missing, unreadable or names no 1 to {engines.SPEAKER_LIMIT} speakers"),
|
||||
({"whisper": self.outside / "whisper-cli"}, "stt_off", f"the whisper.cpp executable {self.outside / 'whisper-cli'} is missing or not executable"),
|
||||
({"whisper_model": self.outside / "model.bin"}, "stt_off", f"the whisper.cpp model {self.outside / 'model.bin'} is missing or not readable"),
|
||||
({"whisper_model": self.outside}, "stt_off", f"the whisper.cpp model {self.outside} is not a file"),
|
||||
({"whisper": self.outside / "whisper-cli"}, "stt_off", f"the whisper.cpp executable {self.outside / 'whisper-cli'} is missing or not executable{ROW}"),
|
||||
({"whisper_model": self.outside / "model.bin"}, "stt_off", f"the whisper.cpp model {self.outside / 'model.bin'} is missing or not readable{ROW}"),
|
||||
({"whisper_model": self.outside}, "stt_off", f"the whisper.cpp model {self.outside} is not a file{ROW}"),
|
||||
)
|
||||
for number, (change, field, message) in enumerate(cases):
|
||||
with self.subTest(change=change):
|
||||
@@ -70,7 +72,7 @@ class TestHubSpeech(HubCase):
|
||||
self.addCleanup(folder.chmod, 0o755)
|
||||
image = engines.Speech(dataclasses.replace(self.engines, whisper=whisper), self.state / "image").describe()
|
||||
self.assertEqual({key: image[key] for key in ("tts", "voices", "operator", "tts_off")}, {"tts": False, "voices": 0, "operator": "", "tts_off": f"the voice directory {voices} is missing or not searchable"})
|
||||
self.assertEqual((image["stt"], image["stt_off"]), (False, f"the whisper.cpp executable {whisper} lies under {whisper.parent}, which the hub may not search"))
|
||||
self.assertEqual((image["stt"], image["stt_off"]), (False, f"the whisper.cpp executable {whisper} lies under {whisper.parent}, which the hub may not search{ROW}"))
|
||||
status, _, body = self.post_json("/api/tts", {"text": "radio check", "voice": OPERATOR_VOICE}, self.guard(self.host))
|
||||
self.assertEqual((status, body.decode()), (503, f"speech output is off: the voice directory {voices} is missing or not searchable\n"))
|
||||
|
||||
@@ -147,7 +149,7 @@ class TestHubSpeech(HubCase):
|
||||
with self.subTest(content_type=content_type):
|
||||
self.assertEqual(self.call("POST", "/api/stt", clip, {**good, "Content-Type": content_type})[0], 415)
|
||||
self.assertEqual(self.call("POST", "/api/stt", clip, good)[0], 415)
|
||||
self.assertEqual(self.call("POST", "/api/stt", b"RIFF", {**good, "Content-Type": "audio/wav", "Content-Length": str(engines.STT_LIMIT + 1)})[0], 413)
|
||||
self.assertEqual(self.call("POST", "/api/stt", b"RIFF", {**good, "Content-Type": "audio/wav", "Content-Length": str(engines.INPUT_LIMIT + 1)})[0], 413)
|
||||
for broken in (
|
||||
b"not a wav",
|
||||
wav_bytes(1, 2, 44100, b"\x00\x00" * 100),
|
||||
@@ -157,14 +159,17 @@ class TestHubSpeech(HubCase):
|
||||
):
|
||||
with self.subTest(broken=broken[:40]):
|
||||
self.assertEqual(post(broken, good)[0], 400)
|
||||
with mock.patch.object(engines, "STT_SECONDS", 1):
|
||||
with mock.patch.object(engines, "INPUT_SECONDS", 1):
|
||||
self.assertEqual(post(wav_bytes(1, 2, engines.STT_RATE, b"\x00\x00" * (engines.STT_RATE + 1)), good)[0], 400)
|
||||
self.assertEqual(log_rows(calls), [])
|
||||
status, headers, body = post(clip, good)
|
||||
self.assertEqual((status, headers["content-type"]), (200, "application/json"), body)
|
||||
self.assertEqual(json.loads(body), {"text": "hello from the mic second line"})
|
||||
argv = ["--model", str(self.engines.whisper_model), "--file", "-", "--output-txt", "--language", "auto", "--no-prints", "--suppress-nst"]
|
||||
self.assertEqual(log_rows(calls), [{"argv": argv, "sha256": hashlib.sha256(clip).hexdigest()}])
|
||||
self.assertEqual(json.loads(body), {"text": "hello from the mic second line", "language": "en"})
|
||||
[row] = log_rows(calls)
|
||||
target = PurePosixPath(row["argv"][row["argv"].index("--output-file") + 1])
|
||||
argv = ["--model", str(self.engines.whisper_model), "--file", "-", "--output-json", "--output-json-full", "--output-file", str(target), "--language", "auto", "--no-prints", "--suppress-nst"]
|
||||
self.assertEqual(row, {"argv": argv, "sha256": hashlib.sha256(clip).hexdigest()})
|
||||
self.assertEqual((target.name, str(target.parent.parent)), ("auto", str(self.engines.scratch)))
|
||||
long = wav_bytes(1, 2, engines.STT_RATE, b"\x00\x00" * engines.STT_RATE * 40)
|
||||
self.assertGreater(len(long), handler.BODY_LIMIT)
|
||||
self.assertEqual(post(long, good)[0], 200)
|
||||
|
||||
@@ -169,7 +169,7 @@ class TestHubState(HubCase):
|
||||
self.assertTrue(all(row["voice"] in VOICES and row["voice"] != OPERATOR_VOICE for row in state["radio"]))
|
||||
self.assertEqual(state["speech"], {
|
||||
"tts": True, "stt": True, "tts_off": "", "stt_off": "", "voices": len(VOICES), "operator": OPERATOR_VOICE, "sample": "all stations, this is Operator, radio check, out",
|
||||
"limit": phrasing.SPEECH_LIMIT, "rate": engines.STT_RATE, "seconds": engines.STT_SECONDS, "gate": {"threshold": 50, "cpu": None, "open": False},
|
||||
"limit": phrasing.SPEECH_LIMIT, "rate": engines.STT_RATE, "seconds": engines.STT_SECONDS, "input": engines.INPUT_SECONDS, "gate": {"threshold": 50, "cpu": None, "open": False},
|
||||
})
|
||||
self.assertAlmostEqual(state["now"] - state["since"], view.RADIO_WINDOW)
|
||||
self.assertEqual((state["night"], state["silence"]), ("day", None))
|
||||
|
||||
+14
-1
@@ -74,12 +74,25 @@ class TestVoiceModule(unittest.TestCase):
|
||||
|
||||
def test_the_recording_is_encoded_as_16_khz_mono_pcm(self):
|
||||
body = base64.b64decode(self.data["wav"])
|
||||
self.assertEqual(engines.wav_seconds(body), 7 / engines.STT_RATE)
|
||||
self.assertEqual(engines.wav_seconds(body, engines.STT_SECONDS), 7 / engines.STT_RATE)
|
||||
with wave.open(io.BytesIO(body)) as reader:
|
||||
self.assertEqual((reader.getnchannels(), reader.getsampwidth(), reader.getframerate(), reader.getnframes()), (1, 2, 16000, 7))
|
||||
self.assertEqual(struct.unpack("<7h", reader.readframes(7)), (0, 16383, -16384, 32767, -32768, 32767, -32768))
|
||||
self.assertEqual(self.data["frames"], [24000, 16000 * 120, 0])
|
||||
|
||||
def test_the_transcript_lands_at_the_cursor_with_one_space_on_each_side(self):
|
||||
cases = [["start end", 6, 6, "hello"], ["", 0, 0, " hello "], ["abc", 3, 3, "x"], ["ab cd", 1, 4, "X"], ["keep", 2, 2, " "], ["a\nb", 2, 2, "line"]]
|
||||
probe = "const V = require(process.argv[1]); console.log(JSON.stringify(JSON.parse(process.argv[2]).map((one) => V.spliced(...one))));"
|
||||
done = subprocess.run(["node", "-e", probe, str(VOICE), json.dumps(cases)], capture_output=True, text=True, timeout=60, check=True)
|
||||
self.assertEqual(json.loads(done.stdout), [
|
||||
{"value": "start hello end", "caret": 11},
|
||||
{"value": "hello", "caret": 5},
|
||||
{"value": "abc x", "caret": 5},
|
||||
{"value": "a X d", "caret": 3},
|
||||
{"value": "keep", "caret": 2},
|
||||
{"value": "a\nline b", "caret": 6},
|
||||
])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,150 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
from hub_case import HubCase
|
||||
from hub_engines import log_rows, wav_bytes
|
||||
from kvb_orchestrator import engines
|
||||
|
||||
FIXTURES = Path(__file__).resolve().parent / "fixtures" / "voice"
|
||||
DUTCH = wav_bytes(1, 2, engines.STT_RATE, b"\x02\x00" * engines.STT_RATE)
|
||||
FRENCH = wav_bytes(1, 2, engines.STT_RATE, b"\x03\x00" * engines.STT_RATE)
|
||||
HEARD = {
|
||||
"hallo-welt.wav": {"auto": ["de", "Hallo Welt."], "de": [0.9, "Hallo Welt."], "en": [0.3, "Hollow felt."]},
|
||||
"hello-world.wav": {"auto": ["en", "Hello world."], "de": [0.2, "Hello Wörld."], "en": [0.9, "Hello world."]},
|
||||
"dutch": {"auto": ["nl", "Hallo wereld."], "de": [0.7, "Hallo Welt."], "en": [0.4, "Hello world."]},
|
||||
"french": {"auto": ["fr", "Bonjour le monde."], "de": [0.2, "Bonschur."], "en": [0.6, "Bonjour world."]},
|
||||
}
|
||||
FAKE_WHISPER = """
|
||||
import hashlib, json, os, sys
|
||||
argv = sys.argv[1:]
|
||||
data = sys.stdin.buffer.read()
|
||||
home = os.environ["HOME"]
|
||||
with open(os.path.join(home, "heard.json"), encoding="utf-8") as table:
|
||||
clip = json.load(table).get(hashlib.sha256(data).hexdigest(), {"auto": ["en", ""], "de": [0.0, ""], "en": [0.0, ""]})
|
||||
with open(os.path.join(home, "whisper.jsonl"), "a", encoding="utf-8") as log:
|
||||
log.write(json.dumps({"argv": argv, "cwd": os.getcwd(), "tmp": os.environ["TMPDIR"]}) + "\\n")
|
||||
with open(os.path.join(os.environ["TMPDIR"], "engine-spill.wav"), "wb") as spill:
|
||||
spill.write(data)
|
||||
if b"FAIL" in data:
|
||||
sys.exit(4)
|
||||
language = argv[argv.index("--language") + 1]
|
||||
detected, text, sure = (*clip["auto"], 0.5) if language == "auto" else (language, clip[language][1], clip[language][0])
|
||||
tokens = [{"text": "[_BEG_]", "p": 0.01}, *({"text": f" {word}", "p": sure} for word in text.split())]
|
||||
with open(argv[argv.index("--output-file") + 1] + ".json", "w", encoding="utf-8") as out:
|
||||
json.dump({"result": {"language": detected}, "transcription": [{"text": text, "tokens": tokens}]}, out)
|
||||
"""
|
||||
KEPT = ("heard.json", "whisper.jsonl")
|
||||
|
||||
|
||||
def spoken(text: str) -> str:
|
||||
return " ".join(re.sub(r"[^\w\s]", " ", text).lower().split())
|
||||
|
||||
|
||||
class TestVoiceInput(HubCase):
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
self.engines.whisper.write_text(f"#!{sys.executable}\n{FAKE_WHISPER}", encoding="utf-8")
|
||||
clips = {"dutch": DUTCH, "french": FRENCH}
|
||||
heard = {hashlib.sha256(clips[name] if name in clips else (FIXTURES / name).read_bytes()).hexdigest(): table for name, table in HEARD.items()}
|
||||
(self.engines.scratch / "heard.json").write_text(json.dumps(heard), encoding="utf-8")
|
||||
self.calls = self.engines.scratch / "whisper.jsonl"
|
||||
|
||||
def post(self, body: bytes, headers: dict[str, str] | None = None, length: int | None = None) -> tuple[int, dict, bytes]:
|
||||
merged = {**(self.guard(self.host) if headers is None else headers), "Content-Type": "audio/wav"}
|
||||
if length is not None:
|
||||
merged["Content-Length"] = str(length)
|
||||
return self.call("POST", "/api/stt", body, merged)
|
||||
|
||||
def leftovers(self) -> list[str]:
|
||||
return sorted(path.name for path in self.engines.scratch.iterdir() if path.name not in KEPT)
|
||||
|
||||
def languages(self) -> list[str]:
|
||||
return [row["argv"][row["argv"].index("--language") + 1] for row in log_rows(self.calls)]
|
||||
|
||||
def test_the_german_and_the_english_fixture_come_back_as_their_words_with_the_language_detected(self):
|
||||
for name, language, words in (("hallo-welt.wav", "de", "hallo welt"), ("hello-world.wav", "en", "hello world")):
|
||||
with self.subTest(name=name):
|
||||
status, _, body = self.post((FIXTURES / name).read_bytes())
|
||||
self.assertEqual(status, 200, body)
|
||||
answer = json.loads(body)
|
||||
self.assertEqual((answer["language"], spoken(answer["text"])), (language, words))
|
||||
self.assertEqual(self.languages(), ["auto", "auto"])
|
||||
for row in log_rows(self.calls):
|
||||
self.assertEqual(row["argv"].count("--language"), 1, row)
|
||||
|
||||
def test_a_third_language_is_heard_again_in_the_surer_of_german_and_english(self):
|
||||
for clip, language, words in ((DUTCH, "de", "hallo welt"), (FRENCH, "en", "bonjour world")):
|
||||
with self.subTest(language=language):
|
||||
status, _, body = self.post(clip)
|
||||
self.assertEqual(status, 200, body)
|
||||
answer = json.loads(body)
|
||||
self.assertEqual((answer["language"], spoken(answer["text"])), (language, words))
|
||||
self.assertEqual(self.languages(), ["auto", "de", "en", "auto", "de", "en"])
|
||||
self.assertEqual(self.leftovers(), [])
|
||||
|
||||
def test_the_temp_directory_is_empty_after_every_request_including_a_failed_one(self):
|
||||
self.assertEqual(self.post((FIXTURES / "hello-world.wav").read_bytes())[0], 200)
|
||||
self.assertEqual(self.leftovers(), [])
|
||||
status, _, body = self.post(wav_bytes(1, 2, engines.STT_RATE, b"FAIL" * 400))
|
||||
self.assertEqual(status, 502, body)
|
||||
self.assertEqual(self.leftovers(), [])
|
||||
rows = log_rows(self.calls)
|
||||
self.assertEqual(len(rows), 2)
|
||||
for row in rows:
|
||||
used = Path(row["tmp"])
|
||||
self.assertEqual((Path(row["cwd"]), used.parent), (used, self.engines.scratch), row)
|
||||
self.assertFalse(used.exists(), row)
|
||||
|
||||
def test_a_recording_over_60_seconds_or_5_mb_is_refused_and_not_transcribed(self):
|
||||
rate = engines.STT_RATE
|
||||
self.assertEqual((engines.INPUT_SECONDS, engines.INPUT_LIMIT), (60, 5 << 20))
|
||||
self.assertEqual(self.httpd.hub.speech.describe()["input"], engines.INPUT_SECONDS)
|
||||
self.assertEqual(self.post(wav_bytes(1, 2, rate, b"\x01\x00" * rate * 60))[0], 200)
|
||||
self.assertEqual(len(log_rows(self.calls)), 1)
|
||||
status, _, body = self.post(wav_bytes(1, 2, rate, b"\x01\x00" * (rate * 60 + 1)))
|
||||
self.assertEqual((status, body), (400, b"the recording must last more than 0 and at most 60 seconds\n"))
|
||||
self.assertEqual(self.post(b"\x00" * engines.INPUT_LIMIT)[0], 400)
|
||||
self.assertEqual(self.post(b"RIFF", length=engines.INPUT_LIMIT + 1)[0], 413)
|
||||
self.assertEqual(len(log_rows(self.calls)), 1)
|
||||
self.assertEqual(self.leftovers(), [])
|
||||
|
||||
def test_another_origin_host_or_a_missing_csrf_header_is_refused_and_not_transcribed(self):
|
||||
clip = (FIXTURES / "hallo-welt.wav").read_bytes()
|
||||
good = self.guard(self.host)
|
||||
refused = (
|
||||
{key: value for key, value in good.items() if key != "X-Kvb-Orchestrator"},
|
||||
{key: value for key, value in good.items() if key != "Origin"},
|
||||
{**good, "Origin": "http://evil.example"},
|
||||
{**good, "Origin": f"http://localhost:{self.port + 1}"},
|
||||
{**good, "Host": "evil.example"},
|
||||
)
|
||||
for headers in refused:
|
||||
with self.subTest(headers=headers):
|
||||
self.assertEqual(self.post(clip, headers)[0], 403)
|
||||
self.assertEqual(log_rows(self.calls), [])
|
||||
self.assertEqual(self.post(clip, good)[0], 200)
|
||||
|
||||
def test_without_a_model_the_endpoint_and_the_button_state_name_the_operator_row(self):
|
||||
missing = self.outside / "ggml-missing.bin"
|
||||
speech = self.httpd.hub.speech
|
||||
with mock.patch.object(speech, "engines", dataclasses.replace(self.engines, whisper_model=missing)):
|
||||
state = speech.describe()
|
||||
status, _, body = self.post((FIXTURES / "hallo-welt.wav").read_bytes())
|
||||
self.assertFalse(state["stt"])
|
||||
self.assertEqual(state["stt_off"], f"the whisper.cpp model {missing} is missing or not readable; the operator row {engines.MODEL_ROW} installs it")
|
||||
self.assertEqual(status, 503, body)
|
||||
self.assertIn(f"the operator row {engines.MODEL_ROW} installs it".encode(), body)
|
||||
self.assertEqual(log_rows(self.calls), [])
|
||||
self.assertEqual(engines.MODEL_ROW, "make orchestrator-install")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,36 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import json
|
||||
import os
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
from hub_case import HubCase
|
||||
from test_hub_voice_input import FIXTURES, spoken
|
||||
|
||||
WHISPER = Path(os.environ.get("KVB_WHISPER_CLI", "/nonexistent/whisper-cli"))
|
||||
MODEL = Path(os.environ.get("KVB_WHISPER_MODEL", "/nonexistent/ggml-small-q8_0.bin"))
|
||||
QUEUED = os.environ.get("KVB_VOICE_INTEGRATION") == "1"
|
||||
READY = os.access(WHISPER, os.X_OK) and MODEL.is_file()
|
||||
REASON = (
|
||||
"real Whisper run: needs KVB_WHISPER_CLI and KVB_WHISPER_MODEL from the operator rows of requirement 013 "
|
||||
"and KVB_VOICE_INTEGRATION=1, which only the queue.py hold command line sets"
|
||||
)
|
||||
|
||||
|
||||
@unittest.skipUnless(QUEUED and READY, REASON)
|
||||
class TestVoiceModel(HubCase):
|
||||
def test_the_real_model_transcribes_the_german_and_the_english_fixture(self):
|
||||
speech = self.httpd.hub.speech
|
||||
with mock.patch.object(speech, "engines", dataclasses.replace(speech.engines, whisper=WHISPER, whisper_model=MODEL)):
|
||||
for name, words in (("hallo-welt.wav", "hallo welt"), ("hello-world.wav", "hello world")):
|
||||
with self.subTest(name=name):
|
||||
status, _, body = self.call("POST", "/api/stt", (FIXTURES / name).read_bytes(), {**self.guard(self.host), "Content-Type": "audio/wav"})
|
||||
self.assertEqual(status, 200, body)
|
||||
self.assertEqual(spoken(json.loads(body)["text"]), words)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -12,7 +12,7 @@ VOICE = "huggingface.co/rhasspy/piper-voices/resolve/e7229f7d33a10e145e109b886d6
|
||||
PINNED = {
|
||||
VOICE: "9127a559e11603f10b366d1a20ac7426826081dbc521de4c2420c57728d73f0f",
|
||||
f"{VOICE}.json": "2efdc6d7f954588b8180132cbd9b8001933fdd00932c92bc92fd0d2028a9eb3d",
|
||||
"huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-base-q8_0.bin": "c577b9a86e7e048a0b7eada054f4dd79a56bbfa911fbdacf900ac5b567cbb7d9",
|
||||
"huggingface.co/ggerganov/whisper.cpp/resolve/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-small-q8_0.bin": "49c8fb02b65e6049d5fa6c04f81f53b867b5ec9540406812c643f177317f779f",
|
||||
}
|
||||
ARCHIVE = (f"codeload.github.com/ggml-org/whisper.cpp/tar.gz/{COMMIT}", "645445da884df6dbd52b30f82d7be7c01d1767c938714e477e786b88d529e9f4")
|
||||
CLONE = ("clone", "https://github.com/ggml-org/whisper.cpp", f"v1.9.5@{COMMIT}")
|
||||
|
||||
Reference in new issue
Block a user