dsh-plugin-elevenlabs-callback

Readme

# dsh-plugin-elevenlabs-callback

[![dsh-plugin](https://img.shields.io/badge/dsh-plugin-blue)](https://github.com/deepseek-ai/deepseek-harness)
[![dsh 0.1.1-rc.2](https://img.shields.io/badge/dsh-0.1.1--rc.2-4c8dff)](https://www.npmjs.com/package/@deepseek-ai/dsh)
[![node ≥22](https://img.shields.io/badge/node-%E2%89%A522-339933)](package.json)
[![license](https://img.shields.io/badge/license-MIT-green)](LICENSE)

A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) plugin that calls you back by voice.

You give the harness a long task and walk away. When the turn finishes, fails, or stops to ask for approval, the plugin sends a link to your phone. Opening it starts a conversation with an [ElevenLabs](https://elevenlabs.io/agents) voice agent that already knows what happened. It reads you the short version, answers questions about the run, and — once you confirm the wording — sends your next instruction (or your approve/deny) straight back into the session through webhook tools served by the plugin.

Built in a weekend on top of the ElevenLabs Agents API and the dsh plugin API. dsh is in developer preview; this plugin was written and tested against `@deepseek-ai/dsh` **0.1.1-rc.2** and will need updating when the plugin API changes.

[Prerequisites](#prerequisites) · [Install](#install) · [First callback](#first-callback) · [Configuration](#configuration) · [Try it without a harness](#try-it-without-a-harness) · [Real phone calls](#real-phone-calls-optional) · [Troubleshooting](#troubleshooting)

## What a callback sounds like

> **Agent:** Hi, this is your DeepSeek Harness calling about retry-refactor. It has finished. Done. I moved the retry logic into http/retry.ts, added exponential backoff with jitter, and wrote six tests for it. Want more detail, or should I tell it what to do next?
> **You:** Tell it to run the tests again and commit if they pass.
> **Agent:** So, you want me to tell it to run the tests again and commit if they pass?
> **You:** Yes.
> *→ `send_instruction` → the message lands in the session as a follow-up turn*
> **Agent:** I've sent the instruction. It will call back when it's done.

(Transcript from `pnpm live-check`, text mode, against the real agent and real webhook tools.)

The same loop was then run against a real dsh session (DeepSeek-V4-Flash, a 40-second turn with 14 tool calls that fixed a failing test and committed): the plugin sent the link, the voice agent summarised the run from the session log, and the spoken instruction arrived in the session as a follow-up turn — visible in the dsh UI as `Context injection · elevenlabs-callback · instruction given by voice callback`. Replies came 0.9–1.9 s after each message (average 1.3 s); the webhook was answered in 2.6 ms. A later run from a real handset (“improve the function”, spoken) produced a 29-second follow-up turn that refactored the function, passed the tests and committed, after which the plugin called back again.

## How it works

```mermaid
sequenceDiagram
    participant H as dsh session
    participant P as plugin (in the dsh process)
    participant N as your phone
    participant E as ElevenLabs Agents

    H->>P: agent/status idle (turn/end in the session log)
    P->>P: gather last message, tool calls, duration → debrief
    P->>N: link (terminal + QR, ntfy push, desktop, or any command)
    N->>P: open link, tap Answer → conversation token
    N->>E: WebRTC call with dynamic variables (summary, status, debrief id)
    E->>N: spoken summary
    N->>E: "tell it to rerun the tests and commit"
    E->>P: POST /tools/send_instruction (X-Tool-Token, debrief id filled by the platform)
    P->>H: agent.followup(message) — or steer() if it is still running
    E->>N: "Sent. It will call back when done." → end_call
```

Three webhook tools are wired to the agent: `get_run_details` (the full last message and every tool call of the turn), `send_instruction`, and `resolve_approval`. The `debrief_id` argument is marked as a dynamic variable in the tool schema, so the platform fills it from the conversation itself and the language model never has to repeat an identifier.

Approvals are a race: when a tool is waiting for permission, the plugin sends a link **and** lets the normal keyboard prompt run. Whichever answers first wins; the other side is told it was already handled.

## Prerequisites

| you need | why |
|---|---|
| **Node.js ≥ 22** and **pnpm** on your PATH | `dsh plugin` drives pnpm to install plugins into a profile |
| **DeepSeek Harness** `@deepseek-ai/dsh` 0.1.1-rc.2, already working | `npx @deepseek-ai/dsh web` must start and run a turn on its own (model provider and API key configured) before you add this plugin — it only listens to sessions, it does not create them |
| An **ElevenLabs account** and **API key** | the plugin creates one agent, three webhook tools and one workspace secret in your workspace on first start. Create the key under *Settings → API Keys*; the free tier is enough for a few callbacks a day (about 15 agent-minutes a month) |
| A **public HTTPS URL** for the plugin's small server: either [`cloudflared`](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/installation/) on your PATH (no account needed; the plugin opens a quick tunnel by itself) or your own tunnel / reverse proxy set as `publicUrl` | ElevenLabs must be able to reach the webhook tools, and your phone must be able to load the page. Browsers only allow the microphone on HTTPS |
| A **phone** (or any device) with a modern browser and a microphone | the page runs the call over WebRTC using `@elevenlabs/client` |
| Optional: the [ntfy](https://ntfy.sh) app | to have links pushed to your phone instead of reading a QR code off the terminal |
| Optional: a Twilio number | for a real ringing phone instead of a link — see [Real phone calls](#real-phone-calls-optional) |

Tested on macOS with dsh 0.1.1-rc.2 (the npm `latest` at the time of writing). Desktop notifications use `osascript` on macOS and `notify-send` on Linux.

## Install

Not on npm yet — install from GitHub or from a local checkout.

**From GitHub** (pnpm fetches the sources and builds them with the package's `prepare` script):

```sh
dsh plugin --profile web add github:StephenEvenson/dsh-plugin-elevenlabs-callback
```

pnpm refuses to run build scripts of git dependencies until you allow them. If the first `add` prints a warning about ignored build scripts, add this to `~/.dsh/profiles/web/pnpm-workspace.yaml` and run the same `add` again:

```yaml
allowBuilds:
  dsh-plugin-elevenlabs-callback: true
```

Only grant this to packages you trust — it runs the package's build at install time.

**From a local checkout** (no build permission needed; the checkout is linked into the profile, so `git pull && pnpm build` updates it):

```sh
git clone https://github.com/StephenEvenson/dsh-plugin-elevenlabs-callback.git
cd dsh-plugin-elevenlabs-callback && pnpm install && pnpm build
dsh plugin --profile web add "$PWD"
```

**Check the layer** without booting:

```sh
dsh --profile web --dump-config | grep -A2 "id: elevenlabs-callback"
```

**Give it the API key and start the profile.** The plugin reads `ELEVENLABS_API_KEY` from the process environment (dsh's own `~/.dsh/.env` is not exported to plugins), or `apiKey` from the profile patch — see [Configuration](#configuration):

```sh
export ELEVENLABS_API_KEY=sk_...
dsh web
```

On start-up the plugin binds `127.0.0.1:7331`, opens the tunnel, then creates (or updates) the agent and its three tools on ElevenLabs. You will see:

```
[elevenlabs-callback] ready — agent agent_…, links will point at https://….trycloudflare.com
```

Every later start reuses the same agent and tools (their ids are kept in `~/.dsh/elevenlabs-callback/agent.json`) and only pushes the current settings and public URL.

**Update:** in a linked checkout, `git pull && pnpm build`; for a GitHub install, re-run the `add github:…` command. Restart `dsh web` afterwards — bundle membership only changes on restart.

**Uninstall:**

```sh
dsh plugin --profile web remove dsh-plugin-elevenlabs-callback
```

then restart `dsh web`. The plugin's state stays in `~/.dsh/elevenlabs-callback/` and the agent stays in your ElevenLabs workspace (delete it under *Agents*); remove both if you want a clean slate.

## First callback

1. In the dsh web UI, give the harness something that takes longer than 20 seconds (a refactor with tests, a scripted change across files) and step away.
2. When the turn ends, the terminal running `dsh web` prints the link with a QR code:

   ```
   [elevenlabs-callback] Harness: my-project — Finished, waiting for you
   [elevenlabs-callback] open on your phone: https://….trycloudflare.com/d/2p4yc6?k=…
   ```

   With `notify.ntfyTopic` set, the same link is pushed to the ntfy app; with `notify.desktop`, it also appears as a desktop notification.
3. Open the link on your phone, tap **Answer**, and allow the microphone. The agent speaks first: which project, what happened, and the first sentence of the harness's last message.
4. Ask for detail (“what did it change?”), or tell it what to do next. The agent reads your instruction back and only sends it after you say yes.
5. The instruction arrives in the dsh session as a follow-up turn (as a steer message if the harness is still running). When that turn ends, you get the next link.

For approvals the flow is the same, except that the agent asks the question (“It wants to run `rm -rf build`. Allow it?”) and calls `resolve_approval`; the keyboard prompt in dsh keeps working in parallel.

## Configuration

Settings live in your profile's `cordis.patch.yml` (`~/.dsh/profiles/web/cordis.patch.yml`). Target the row by its id, `elevenlabs-callback`. A patch replaces the whole `config` object, so repeat every key you want to keep:

```yaml
- id: elevenlabs-callback
  config:
    apiKey: sk_...                                   # or leave out and export ELEVENLABS_API_KEY
    triggers: { finished: true, error: true, approval: true, minTurnSeconds: 20 }
    notify: { terminal: true, qr: false, desktop: true, ntfyTopic: my-private-topic, ntfyServer: https://ntfy.sh, command: "" }
```

| key | default | meaning |
|---|---|---|
| `apiKey` | `""` | ElevenLabs API key; empty = read `ELEVENLABS_API_KEY` from the environment |
| `publicUrl` | `""` | https URL where the plugin server is reachable; empty = start a cloudflared quick tunnel |
| `host`, `port` | `127.0.0.1`, `7331` | where the plugin server listens (`0` = free port) |
| `stateDir` | `$DSH_HOME/elevenlabs-callback` | agent/tool ids and the generated tool token |
| `agent.llm`, `agent.voiceId`, `agent.ttsModel` | `gemini-2.5-flash`, Charlie, `eleven_flash_v2` | agent settings pushed at start-up (English agents need `eleven_flash_v2` or `eleven_turbo_v2`) |
| `triggers.finished` / `.error` / `.approval` | `true` | which events send a link |
| `triggers.minTurnSeconds` | `20` | completed turns shorter than this are ignored — you were probably watching |
| `linkTtlMinutes` | `120` | how long a link can start a call |
| `notify.terminal`, `notify.qr` | `true` | print the link (and a QR code) to the terminal |
| `notify.desktop` | `false` | macOS / Linux desktop notification |
| `notify.ntfyTopic`, `notify.ntfyServer` | `""`, `https://ntfy.sh` | push the link to the ntfy app |
| `notify.command` | `""` | any shell command; gets `DSH_CALLBACK_URL`, `_TITLE`, `_BODY` in its environment |
| `phone.toNumber`, `phone.fromNumber` | `""` | ring a real phone instead of only sending a link — see below |

Patch edits to `config` hot-reload; the plugin restarts its server and re-provisions the agent with the new settings.

## Try it without a harness

```sh
cp .env.example .env            # ELEVENLABS_API_KEY, TOOL_SHARED_SECRET
pnpm install
pnpm demo                       # --status finished | needs_approval | error
```

`pnpm demo` runs the same server, tunnel and provisioning with one made-up debrief and a fake harness that prints whatever the voice agent sends. It uses its own agent (`… (standalone demo)`) so it never rewires the plugin's. Open the printed link on your phone and talk to it; or drive it from the terminal in text mode, which spends no audio minutes:

```sh
pnpm live-check --link "https://…/d/abc123?k=…"
```

This opens a real conversation over WebSocket, sends a four-line script, and checks that `send_instruction` was called through the public URL. Observed on the run recorded above: agent replies 1.0–2.1 s after each message (average 1.4 s); the plugin answered the webhook in 0.2 ms.

## Real phone calls (optional)

Links work everywhere; a ringing phone is nicer. ElevenLabs does not sell numbers itself, so you bring one from Twilio (or any SIP trunk):

1. In the [Twilio console](https://console.twilio.com): buy a number with voice capability (a US number is instant; other countries need address/identity documents). ElevenLabs cannot import numbers from a Twilio **trial** account (Twilio error 20003), so the account must be upgraded to pay-as-you-go first.
2. In the ElevenLabs dashboard: Agents → Phone numbers → *Import from Twilio*, using the **Account SID** (`AC…`) and **Auth Token** from the Twilio console (not an API key).
3. In the profile patch:

   ```yaml
   - id: elevenlabs-callback
     config:
       phone: { toNumber: "+61400000000", fromNumber: "+15550001111" }
   ```

From then on every callback rings your phone through `POST /v1/convai/twilio/outbound-call` with the same dynamic variables the web page would use — the agent speaks first, the tools work identically — and the link is still sent as a fallback. `pnpm demo --call +61…` (with `PHONE_FROM` in `.env`) tries it without a harness. Twilio charges per minute on top of ElevenLabs' agent minutes.

## Troubleshooting

- **`cloudflared: command not found` / “tunnel did not report a URL”** — install cloudflared, or run your own tunnel and set `publicUrl`. The plugin refuses non-https URLs because ElevenLabs will not call them.
- **No link after a turn** — the turn was shorter than `triggers.minTurnSeconds`, was interrupted/aborted, or ended before the plugin started (only turns that end after start-up count). Check the `[elevenlabs-callback]` lines in the terminal; `ctx.logger` output is not shown by `dsh web`, so everything user-facing is printed with plain `console.log`.
- **The page loads but Answer fails** — the microphone needs HTTPS and a permission grant; on iOS Safari the first tap must be a user gesture. A link older than `linkTtlMinutes` answers 404.
- **Tools answer 401** — the shared token in `stateDir/tool-secret` no longer matches the workspace secret on ElevenLabs. Restart the profile; the plugin re-pushes the secret value on every start.
- **The agent talks over you or in the wrong voice/model** — change `agent.*` in the patch; the plugin PATCHes the live agent on the next start. English agents must use `eleven_flash_v2` or `eleven_turbo_v2`.
- **“Unusual activity” / quota errors from ElevenLabs** — the free tier shares its monthly agent minutes across every agent in the workspace; `pnpm live-check` uses text mode and spends none.

## Development

```sh
pnpm install
pnpm test        # 40 tests: debrief model, HTTP surface, session-log parsing, config schema,
                 # and the whole loop inside a real Cordis context with a fake agent registry
pnpm typecheck   # against the published @deepseek-ai/dsh-* type definitions
pnpm build       # tsc → lib/
```

Every webhook call is logged with its latency (`tool=send_instruction status=ok latency_ms=0.2 …`) and visible at `/api/tool-log`. Tool endpoints answer `200` with a structured `ok`/`found` field even when something is wrong, so the voice agent can say what happened instead of stalling on an HTTP error.

To iterate without reinstalling: link the checkout as above; after `pnpm build`, restart `dsh web`.

## Security notes

- The tunnel exposes only this plugin's server, never the dsh web UI.
- Each link carries its own random key; without it the page and the token endpoint answer 404. Links expire.
- ElevenLabs authenticates to the tools with a token generated on first run, stored as a workspace secret on their side and in `stateDir` on yours; anything else gets 401.
- The agent is private (`enable_auth`), capped at one concurrent call, 50 calls a day and 5 minutes per call.
- An instruction is only delivered after the agent has read it back and heard a yes; a turn accepts one instruction per callback.
- Nothing is sent anywhere but ElevenLabs (and ntfy / Twilio if you enable them); the debrief contains the harness's last message and tool-call summaries, so treat the link like the session itself.

## Compatibility

| dsh | status |
|---|---|
| `0.1.1-rc.2` (npm `latest`) | tested end to end, including a real handset call |
| `0.1.2-alpha.x` (GitHub main) | not tested; the plugin API is a developer preview and is expected to break |

Peer ranges in `package.json` are open-ended (`>=0.1.1-rc.2`); `pnpm typecheck` runs against the pinned rc.2 type definitions.

The plugin does not resume a session that has been disposed; the link still works for reading, but `send_instruction` will say the session is gone.

## Layout

```
src/index.ts              the Cordis plugin: config schema, triggers, approval race, delivery
src/harness.ts            turns a dsh session log into a debrief (pure, unit-tested)
src/debrief.ts            debrief store, per-link keys, spoken opening, dynamic variables
src/server.ts             node:http server: phone page, token endpoint, three webhook tools
src/notify.ts             terminal + QR, desktop, ntfy, shell command
src/tunnel.ts             cloudflared quick tunnel
src/elevenlabs/           agent definition as code, minimal API client, idempotent provisioning
web/                      the phone page (plain HTML/JS on the @elevenlabs/client IIFE bundle)
scripts/                  demo (no harness needed), live-check (text mode, real tools), provision
cordis.patch.yml          the layer `dsh plugin add` merges into a profile
```

## License

MIT