dsh-image-vision

Readme

# dsh-image-vision

Seamless image understanding for **DeepSeek Harness** — lets pure-text main
models read pasted/dropped images and Feishu/Lark images via a configurable
vision model, **without modifying any host code**.

A pure "everything is a plugin" implementation: it wraps a documented `llm`
service method and listens to official agent/tool waterfalls. No `packages/`
files are touched, so it portably runs wherever DeepSeek Harness runs.

> 简体中文说明见 [README.zh.md](./README.zh.md)

---

## What it does

- **Paste / drop an image in chat** — even when the main model is text-only
  (e.g. DeepSeek V4 Flash routed through pi-ai), the image is accepted and
  described to the model as text.
- **Feishu / Lark document images** — `<image token="..."/>` produced by
  `lark_read_doc` and friends is downloaded through `lark-cli` and described;
  the result is injected back as tool output.
- **`describe_image` model tool** — describes a local image file path.
- **Full description, not just OCR** — people, scenes, tables, charts and
  verbatim text are all described, driven by a prompt you can override.

Pick a vision model from **ModelScope free community API** (e.g.
`Qwen/Qwen3-VL-8B-Instruct`) or **SiliconFlow** (e.g.
`Qwen/Qwen3-VL-32B-Instruct`). Routes are auto-discovered from your `settings`.

---

## Why no host code changes

The web API proxy gates image upload/model-switch by calling
`llm.resolveModelInfo(...).inputModalities` and rejecting when the model does
not declare `image`. Since DSH is "everything is a plugin", this plugin:

1. **Wraps `ctx.llm.resolveModelInfo`** — when the target model lacks `image`
   input and a vision route is configured, the returned capabilities gain
   `image`, so the admission gate lets the image through.
2. **Listens on `agent/pre-step`** — before the real model request, each image
   block is transcribed by the vision model and replaced with its text, so the
   text-only main model only ever receives plain text.

Both steps use documented DSH extension points. No core files are changed.

---

## Requirements

- DeepSeek Harness `0.1.0-rc.6` (**source checkout or installed release** —
  both are supported, see *Installation*).
- Node.js >= 22.19
- A configured **vision model provider** (see *Configuration*).

---

## Installation

Two deployment styles are supported: a **source checkout** (clone the
deepseek-harness repo and run `pnpm dsh web`) and the **installed release**
(`npm i -g @deepseek-ai/dsh`). Both go through the same `dsh plugin` manager;
`@deepseek-ai/*` dependencies are provided by the host and resolved from the
DSH installation anchor.

### From GitHub (recommended)

```bash
npx @deepseek-ai/dsh plugin --profile web add github:VeryInt/dsh-image-vision
```

### From a local path (development / validation)

```bash
git clone https://github.com/VeryInt/dsh-image-vision.git /path/to/dsh-image-vision
npx @deepseek-ai/dsh plugin --profile web add /path/to/dsh-image-vision
```

> Replace `<profile>` with whichever profile you use (`web`, `headless`, ...).
> If pnpm blocks a git dependency's build/prepare script, allowlist the exact
> key it prints under `allowBuilds` in `<profile>/pnpm-workspace.yaml`, then
> re-run.

Restart Harness after installing. The plugin then appears under
**Settings → Plugins**.

---

## Configuration

Image recognition needs a **vision model provider**. Add an OpenAI-compatible
vision endpoint in the DSH Models page (or `~/.dsh/settings.yaml`), and — this
is the part the Models page does not show — declare `input: [text, image]` on
the model entry.

Example `settings.yaml`:

```yaml
llm-pi-ai:
  providers:
    modelscope:
      displayName: ModelScope
      apiKeyEnv: MODELSCOPE_API_KEY
      api: openai-completions
      baseURL: https://api-inference.modelscope.cn/v1
      models:
        - id: Qwen/Qwen3-VL-8B-Instruct
          name: Qwen3-VL-8B
          input: [ text, image ]
    siliconflow:
      displayName: 硅基流动
      apiKeyEnv: SILICONFLOW_API_KEY
      api: openai-completions
      baseURL: https://api.siliconflow.cn/v1
      models:
        - id: Qwen/Qwen3-VL-32B-Instruct
          name: Qwen3-VL-32B-Instruct
          input: [ text, image ]
```

Then provide the API key in `~/.credentials.yaml` or as an environment
variable:

```bash
# ModelScope community access: generate at https://modelscope.cn
MODELSCOPE_API_KEY=ms-xxxxxx
# SiliconFlow: generate at https://cloud.siliconflow.cn
SILICONFLOW_API_KEY=sk-xxxxxx
```

> **Use a vision model (VL suffix).** Text-only models such as
> `Qwen/Qwen3.5-27B` return empty results even when declared image-capable.

### Plugin vision route

The plugin's `cordis.patch.yml` carries two relevant fields:

```yaml
- insert:
    - id: dsh-image-vision
      name: dsh-image-vision
      config:
        provider: siliconflow      # vision provider route (empty = auto-discover)
        model: Qwen/Qwen3-VL-32B-Instruct   # vision model id (empty = auto-discover)
        # prompt: ...
        # maxImagesPerMessage: 4
        # feishuImages: true
        # maxFeishuImages: 5
```

Rules:

- **Both `provider` and `model` set**: used as-is (highest priority).
- **Both empty**: scan every configured provider and pick the first model
  declaring `input: [text, image]` (this is the common case).
- Changing the model = update these two fields **and** make sure the model
  declares `input: [text, image]` in `settings.yaml`.

---

## Verification

After a restart:

1. **Paste an image in chat** — the main model should describe its content
   (rendered as `[图片内容] …`).
2. **Read a Feishu/Lark doc with images** — tool output gains
   `[飞书图片 <token>] …` descriptions.
3. **`describe_image` tool** — ask the model to describe a local image path.

On a transient vision failure the reply still comes back with a
`[图片内容识别失败：…]` / `[飞书图片 … 识别失败：…]` placeholder rather than
failing the whole turn.

---

## Options

| Field | Default | Description |
| --- | --- | --- |
| `provider` | empty | Vision provider route; empty = auto-discover |
| `model` | empty | Vision model id; empty = auto-discover |
| `prompt` | (full description) | Instruction sent to the vision model |
| `maxImagesPerMessage` | 4 | Max images bridged per message; beyond throws |
| `feishuImages` | true | Auto-describe Feishu `<image token>` |
| `maxFeishuImages` | 5 | Max Feishu images per tool result |

---

## How this differs from oil-oil/dsh-vision

We looked at the community plugin [oil-oil/dsh-vision](https://github.com/oil-oil/dsh-vision)
before writing this one. Key differences:

| | **oil-oil/dsh-vision** | **dsh-image-vision** |
|---|---|---|
| Mechanism | Replaces the official `deepseek-official` **adapter** with a `VisionBridgeAdapter`; installer disables the built-in `llm-deepseek` | Wraps only `llm.resolveModelInfo` and uses the official `agent/pre-step` waterfall — no adapter replacement, no built-in plugin disabled |
| Works with | Models routed through `deepseek-official` | Any pure-text route (pi-ai `deepseek`, etc.) because it hooks the `llm` service, not a specific route |
| Vision backends | Bundles several (zenmux / Bailian / see config / local OCR) | Reuses your `settings.yaml` / Models vision providers (input: [text, image]) |
| Settings UI | Ships a "Vision Recognition" card | None — config lives in the plugin `cordis.patch.yml` |
| Packaging | TypeScript + tsdown build, many peer deps | Plain JS ESM, minimal peers, no build step |

In short: oil-oil *replaces the DeepSeek adapter* (so it only affects
`deepseek-official` and requires disabling the built-in plugin); this plugin
*only wraps a llm capability lookup and uses the official pre-step* — it works
for any text-only main model route and makes no destructive changes, which fits
"everything is a plugin" more literally.

---

## License

[MIT](./LICENSE)