dsh-vision-patch

Readme

# dsh-vision-patch

A plugin with two halves on the `llm-pi-ai` settings namespace. The client
card on the Models page declares which models accept image input, per model
and per route, and picks a per-route **vision fallback model**. The host half
uses that fallback to describe images so a turn never dies on an image the
main model cannot read.

The card renders inside every provider card the Models page draws for a
`llm-pi-ai` route (saved rows, first-run setup rows, add-provider drafts) and
writes through the settings Remote. The host half adds no HTTP route of its
own; it registers a single `agent/pre-step` interceptor.

## What it does

- **Per-model checkbox** — `input: ['text', 'image']` on the model's stored
  entry. Unchecked *removes* the `input` key entirely, so the model inherits
  the installed catalog's capability (it never writes `['text']`, which would
  forbid images outright).
- **Route-wide checkbox** — `defaultInput: ['text', 'image']` (set/unset via
  path ops) for every model on the route that declares no list of its own.
- **Vision fallback model** — a per-route dropdown of the route's
  vision-capable models (those that declare images themselves or inherit them
  from the route). The picked model is stored as `visionFallbackModel`.
- **Model filter** — a search box appears once a route lists more than 8
  models; the count line shows how many models currently declare images.
- **Live refresh** — the card re-reads after each write and follows
  `settings/document-updated`, so edits made elsewhere on the page converge.

A provider with no stored profile (never saved from the Models page) shows
guidance text instead of checkboxes.

## Vision fallback (host)

A provider route in the `llm-pi-ai` settings namespace may carry a
`visionFallbackModel` field: `{ provider?: string; model: string }`. The
`provider` defaults to the route itself; `model` names a vision-capable model
on it. An absent field turns the feature off for that route. The Models card's
dropdown writes this field; the host reads the same stored profile, so the two
sides always agree.

When the host's `agent/pre-step` interceptor sees a step whose user message
contains an `image` content block AND the step's route (from
`agent.options.provider`) has a configured `visionFallbackModel`, it:

1. streams the turn to the fallback vision model via
   `ctx.llm.stream({ provider, model, messages, signal })` to describe the
   image(s);
2. replaces each `image` block in the user message with a `text` block holding
   that description; and
3. returns the rewritten messages, so the main model continues on the
   description.

It does not inspect the active main model's modality: it describes whenever a
fallback is configured and an image is present. If the describe fails, times
out, no fallback is configured, or `ctx.llm` is unavailable, the interceptor
delegates unchanged (`next()`), so the image falls through to the harness's
default handling and the turn is never blocked.

### Durability & logging

When the interceptor describes an image it rewrites the user's own message,
replacing each `image` content block with a `text` block holding the
description. That rewritten text is **not stamped with a plugin `source`
kind**. The whole host seam is validated only against mocks — there is no live
host to confirm a new, non-catalog `source` kind is accepted — and rewriting an
already-present user message (rather than appending a fresh one) risks
clobbering the harness's own `source` marker on that message. Durability of the
injected wording is therefore the harness's call; this plugin makes the
injection **observable instead**, via the host logger:

- `info` when a describe is attempted (an image is present, a fallback model is
  configured, and `ctx.llm.stream` is available);
- `debug` when the describe yields no usable text and the interceptor delegates
  unchanged (`next()`);
- `debug` when a description is actually injected and the turn enters rewritten;
- `debug` when a describe is skipped because one is already in flight for the
  same agent (the per-agent concurrency guard).

All logging is guarded (`ctx?.logger?.info?.(...)` / `debug` / `warn?.(...)`), so
an absent logger is harmless. A silent describe is never silent in the logs: if
the user believes an image was described but no text assembled (for example the
real `ctx.llm.stream` chunk shape differs from the mock), the debug line names
the agent that fell back to `next()` unchanged.

## Install

```bash
dsh plugin --profile web add github:ryasrk/dsh-vision-patch
```

The profile's `dsh.profile.bundles` list picks the package up automatically
(its `package.json` declares `dsh.bundle.patch`). Restart the web host so the
new bundle layer composes, then reload the Models page.

## Layout

| Path | Role |
| --- | --- |
| `cordis.patch.yml` | inserts the `dsh-vision-patch` loader row |
| `lib/index.js` | host entry — `apply(ctx)` registers the `agent/pre-step` interceptor via `lib/host.js` |
| `lib/host.js` | host helpers: the `agent/pre-step` handler that describes images and rewrites the turn, plus the settings fallback reader |
| `client/client.js` | browser bundle: the card (image-input checkboxes + vision fallback dropdown), registered on the `settings.models.provider-card` slot keyed `llm-pi-ai` |
| `package.json` | `dsh.bundle.patch` + `dsh.client` (web platform, inject edges) |

The client bundle requires only `react` (a platform seed word); it needs no
build step and has no runtime dependencies.

## How it writes

All writes go through `ctx.remote.settings` against the `llm-pi-ai` namespace,
carrying the revision the last read observed (`settings-conflict` on a lost
race):

- route-wide toggle → `mutate` path ops on `['providers', <route>, 'defaultInput']`
- vision fallback → `mutate` path ops on `['providers', <route>, 'visionFallbackModel']` (`set` the `{ model }` choice, `unset` to clear)
- per-model → `update` merge restating the whole stored `models` list (path
  ops cannot address array elements), every untouched field merge-preserved.

Writes that land return the namespace's post-write view, which folds straight
back into the card.

## Uninstall

```bash
dsh plugin --profile web remove dsh-vision-patch
```

(Then restart the web host so the bundle layer leaves the composition.)