dsh-audio-input

Readme

# dsh-audio-input

**Speak into the DeepSeek Harness Web GUI.** A DSH web plugin that adds a
microphone button to the message composer. Dictation uses the **native Apple
Speech framework** (SFSpeechRecognizer) running as a small helper process on
the Mac whose microphone you use — no cloud API key, no audio leaves the
machine.

## How it works

- **Browser half** — the mic button and the live-transcript strip
  (conversation.input.right / conversation.input.dock). Click to start,
  click again to stop; the final transcript is inserted into the composer
  draft, ready for you to press send.
- **Server half** — a cordis plugin that exposes small JSON routes on the
  web server and relays dictation control and transcripts between the
  browser and the helper process.
- **Helper process** — dsh-audio-speech, a ~130 KB native binary built from
  native/SpeechCapture.swift. It captures the microphone with AVAudioEngine
  and recognizes speech with SFSpeechRecognizer on the machine it runs on.

Two modes:

| Mode | Helper runs on | Use when |
|---|---|---|
| **local** | the dsh server Mac | you are sitting at the server Mac |
| **remote** | another Mac on the network | you access the GUI from another Mac — mic + recognition happen there, only text travels |

The browser automatically uses **remote mode when a helper is connected**,
otherwise local mode.

## Requirements

- dsh server: macOS, the Web profile (dsh web)
- Remote client machine: any OS with a browser — the mic button works from
  anywhere. **Remote dictation additionally needs macOS on the client** (the
  helper is a Mac binary); the one-time helper setup below is done on that
  Mac.
- Xcode Command Line Tools are needed **only** to rebuild the helper from
  source — the repo ships the compiled binary.

---

## Setup — Server side

Run these on the **server Mac**, once.

### 1. Install the plugin

From the plugin checkout (after cloning):

```bash
dsh plugin --profile web add link:/path/to/dsh-audio-input
```

This installs the package into ~/.dsh/profiles/web, appends it to the
profile's bundle layers, and its cordis.patch.yml inserts the audio-input
host row. (A GitHub install without a local checkout also works:
`dsh plugin --profile web add git+ssh://git@github.com/<you>/dsh-audio-input.git`
— the built lib/ and native/bin/ artifacts are committed, so no build step
runs on install.)

### 2. Configure the remote token

Edit ~/.dsh/profiles/web/cordis.patch.yml and add an **id-targeted config
patch** (the audio-input row already exists — do **not** use `- insert:`,
which would create a broken duplicate):

```yaml
# dsh-audio-input: sets the shared secret for the remote helper.
- id: audio-input
  config:
    remoteToken: 'choose-any-secret-string'
```

The token is a password you invent — it must match exactly on the remote
helper (see below). Leaving it unset disables remote dictation entirely.

### 3. Restart

Stop dsh web (Ctrl-C in its terminal), then start it again the same way.

Restarting is required after installing the plugin, after config changes,
and after updating the plugin.

### 4. Verify (optional but recommended)

```bash
# token-gated route — the bearer value must match your config
curl -H 'Authorization: Bearer choose-any-secret-string' http://127.0.0.1:3080/audio-input/remote/commands
# expected: {"ok":true,"command":null}
```

---

## Setup — Remote side

Run these on the **remote Mac** (the one you dictate at), once. The only
thing you need is the server's address — e.g. 192.168.0.101 (find it on the
server with `ipconfig getifaddr en0`, or in the URL line printed when
`dsh web` starts).

### 1. Download the helper straight from the server

The server serves the compiled binary — no file-transfer tooling needed:

```bash
curl -o ~/dsh-audio-speech http://<SERVER-IP>:3080/audio-input/helper
chmod +x ~/dsh-audio-speech
```

**Verify it is the binary, not a web page** (the GUI answers unknown paths
with HTML, which produces a confusing "syntax error" when run):

```bash
file ~/dsh-audio-speech
# must print: Mach-O 64-bit executable arm64
# if it prints something about HTML/text, the download was wrong — retry
```

> Alternative without the server: copy native/bin/dsh-audio-speech from the
> plugin repo, or build it there with `pnpm build:native` (requires Xcode
> Command Line Tools). Apple-silicon (arm64) only.

### 2. Run the helper

```bash
~/dsh-audio-speech --remote http://<SERVER-IP>:3080 --token 'choose-any-secret-string'
```

- Replace <SERVER-IP> with the server's real address and the token with the
  exact value from step 2 on the server. **Do not keep the angle brackets** —
  `<` and `>` are shell redirection operators.
- The helper polls the server for start/stop commands every 0.5 s and stays
  alive across dictation sessions. Keep its terminal window open (or use
  `nohup ~/dsh-audio-speech ... &`); closing it disables remote dictation.

### 3. First-use permissions (on the remote Mac)

macOS asks once for **Speech Recognition** and **Microphone** access,
attributed to the app that launched the helper (Terminal, iTerm, SSH, ...):

- System Settings → Privacy & Security → **Speech Recognition** → allow
- System Settings → Privacy & Security → **Microphone** → allow

If the helper was launched over SSH (no GUI prompt), it reports
`speech-permission-pending` — grant both permissions in System Settings on
the remote Mac, then simply click the mic button again (the helper keeps
running and retries).

---

## Daily use

1. Click the **mic button** at the right end of the composer tool row.
2. Speak — partials appear live in the strip above the composer.
3. Click the mic button **again** to stop; the final transcript is inserted
   into the composer draft.
4. Review, press **send**.

Tips:

- One dictation at a time; switching sessions while recording keeps the
  button in other sessions disabled until it finishes.
- The strip above the composer shows errors (permissions, helper offline,
  ...) instead of failing silently.

## Configuration reference

Set these under `- id: audio-input` + `config:` in the profile's
cordis.patch.yml (all optional):

| Key | Default | Meaning |
|---|---|---|
| locale | 'en-US' | BCP 47 recognizer locale, e.g. 'zh-CN', 'de-DE' |
| onDevice | false | true forces on-device recognition (fully offline; narrower language support) |
| remoteToken | '' | Shared secret for the remote helper. Empty = remote dictation disabled |

## Troubleshooting

| Symptom | Cause / fix |
|---|---|
| `line 1: syntax error ... '<!doctype html>'` when running the helper | The downloaded file is the web GUI's HTML. Re-fetch with the exact /audio-input/helper URL and check with `file` (see Remote side, step 1). |
| Mic click shows `remote-offline` | No helper is connected. Start the helper on the remote Mac (--remote + matching --token), or check it did not exit. |
| `unauthorized` / dictation never starts | Token mismatch. Compare the helper's --token with remoteToken in the profile patch — they must be identical; restart `dsh web` after changing the patch. |
| `speech-permission-pending` | Permission prompt never appeared (SSH/agent launch). Grant Speech Recognition + Microphone in System Settings on that Mac, then click the mic again. |
| `speech-permission-denied` / `mic-permission-denied` | Permissions were denied. Allow them in System Settings → Privacy & Security, then retry. |
| Second mic click reports `dictation already in phase recording` | Outdated client bundle — update the plugin and **restart `dsh web`** (the restart loads the new browser bundle). |
| Stop click seems to hang for ~8 s | The remote helper did not answer the stop command (it may have died). Restart the helper; check its terminal for errors. |
| Local mode: mic opens on the wrong machine | Local mode uses the server Mac's mic. Prefer remote mode (start the helper on your Mac) whenever you are not at the server. |

## Security notes

- `remoteToken` is the only fence between the two machines and the rest of
  your LAN. Anyone with the token can start dictation and inject
  transcripts; anyone *without* it gets `401`. Treat it like a password (a
  random string is best: `openssl rand -hex 16`).
- The token sits in the helper's command line and in the profile patch — do
  not reuse a valuable password for it.
- The remote endpoints only exist when `remoteToken` is non-empty; the
  default (empty) rejects every remote request.

## Build from source

```bash
pnpm build          # tsdown: lib/index.js (server half) + lib/client.js (browser half)
pnpm build:native   # swiftc: native/bin/dsh-audio-speech (macOS SDK required)
pnpm typecheck      # tsc --noEmit
node scripts/smoke-host.mjs   # end-to-end smoke test (routes + helper)
```

lib/ and native/bin/ are committed so installs from GitHub work without a
build step; rebuild after source changes.

## Debugging

```bash
# every route answers directly — no browser needed:
curl -s -X POST http://127.0.0.1:3080/audio-input/status
curl -s -X POST http://127.0.0.1:3080/audio-input/start -H 'content-type: application/json' -d '{}'
curl -s -X POST http://127.0.0.1:3080/audio-input/stop
curl -s -o ~/x http://127.0.0.1:3080/audio-input/helper   # the binary
# remote helper traffic (token required):
curl -s -H 'Authorization: Bearer <token>' http://127.0.0.1:3080/audio-input/remote/commands
curl -s -X POST -H 'Authorization: Bearer <token>' -H 'content-type: application/json' \
  -d '{"type":"error","code":"probe","message":"hi"}' http://127.0.0.1:3080/audio-input/remote/events
```

Override the helper binary the server spawns (local mode) with
`DSH_AUDIO_SPEECH_BIN=/path/to/binary` on the `dsh web` process.

## License

MIT