dsh-vision-pro-bridge

Readme

# dsh-vision-pro-bridge

Give a text-only **DeepSeek-V4-Pro** brain real image understanding — **without switching models, without Ollama, and without a second API key**.

[![npm](https://img.shields.io/npm/v/dsh-vision-pro-bridge)](https://www.npmjs.com/package/dsh-vision-pro-bridge)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Node](https://img.shields.io/badge/node-%3E%3D22-43853d)](#)

> DeepSeek Harness (dsh) plugin. English · [简体中文](README.zh-CN.md)

## The problem

`deepseek-v4-pro` (and `deepseek-v4-flash`) are **text-only models**: their API rejects image bytes. In DeepSeek Harness, pasting a screenshot into a Pro session fails with *"does not accept image input"*. Your only official option is to switch the whole conversation to the weaker, Flash-tier `deepseek-v4-flash-vision-exp`.

## What this plugin does

It registers a **twin provider route** — `deepseek-vision-pro` (model `deepseek-v4-pro-vision`) — that advertises image support. When you attach an image, the plugin first asks **`deepseek-v4-flash-vision-exp`** to transcribe it into text (using your existing `DEEPSEEK_API_KEY`), then hands that text — plus a stable local file path — to **`deepseek-v4-pro`**. The DeepSeek wire route never sees an image; V4-Pro keeps its coding brain and still "sees" the picture.

```
paste screenshot  →  [image block admitted]
   → transcribe via deepseek-v4-flash-vision-exp (text)
   → text + local path  →  deepseek-v4-pro answers
```

## Why this one

- **Zero third-party dependencies.** It reuses only the harness's own `@deepseek-ai/dsh-llm` and `@deepseek-ai/dsh-llm-deepseek`. No `openai`, no `sharp`, no `schemastery`, no Ollama.
- **DeepSeek-only, single provider.** Images go only to DeepSeek (`deepseek-v4-flash-vision-exp`), through the same `DEEPSEEK_API_KEY`. No second VLM vendor, no local model, no anonymous endpoints.
- **Official image pipeline, byte-for-byte.** It subclasses the official `DeepSeekAdapter`, so image normalization, Files-API upload, and inline-base64 fallback are identical to official behavior — no hand-rolled HTTP.
- **Minimal & auditable.** One file, ~250 lines. Easy to read, verify, and fork.
- **Content-addressed persistence.** Images are saved under `~/.dsh/vision-pro-bridge/images/` and the stable local path is passed to the model, so V4-Pro can reference it.
- **Pro-focused.** One model, `deepseek-v4-pro-vision`: keep V4-Pro's coding ability, add vision.

## Install

```bash
dsh plugin --profile web add dsh-vision-pro-bridge
# restart dsh web
```

## Usage

1. In the model picker, choose **DeepSeek-V4-Pro (视觉桥)**.
2. Paste or drag a screenshot and ask normally.

Make it the default for new sessions (`~/.dsh/settings.yaml`):

```yaml
agent-default-model:
  provider: deepseek-vision-pro
  model: deepseek-v4-pro-vision
```

## Configuration

Environment variables (all optional):

| Variable | Default | Purpose |
| --- | --- | --- |
| `DEEPSEEK_API_KEY` | credentials service | DeepSeek key (same as the official route) |
| `DEEPSEEK_BASE_URL` | `https://api.deepseek.com` | Endpoint (same as the official route) |
| `DSH_VISION_PRO_BRIDGE_PROMPT` | built-in English transcription prompt | Override the transcription prompt (e.g. Chinese UI analysis) |

Images are cached content-addressed under `~/.dsh/vision-pro-bridge/images/` (re-attaching the same image reuses the transcription); an activity log is written to `~/.dsh/vision-pro-bridge-activity.jsonl`.

## How it works

1. `apply(ctx)` registers a `DeepSeekAdapter` subclass for the provider route `deepseek-vision-pro`.
2. `listModels` / `resolveModel` / **`prepareCall`** all declare `inputModalities: ['text', 'image']`, so the harness's image gates (paste admission, thumbnails, `read_image`) open. Overriding `prepareCall` is required — otherwise the LLM runtime projects images to placeholders before they ever reach the adapter.
3. `stream()` walks every message (including nested `tool-result` content), and for each image block reads its bytes, saves them content-addressed, and calls `super.stream()` against `deepseek-v4-flash-vision-exp` to produce a text transcription.
4. The image block is replaced with `[图片（视觉桥分析 by …）]\n<transcription>\n\n本地路径：<path>` and the now text-only conversation is streamed to `deepseek-v4-pro`.

With no image, `stream()` forwards directly to Pro — zero overhead.

## Limitations

- Transcription is **lossy**: great for layout / OCR / UI review, not for exact pixel coordinates.
- The transcription model is fixed to `deepseek-v4-flash-vision-exp`.
- The endpoint is resolved from `DEEPSEEK_BASE_URL` (not the settings-page `llm-deepseek` overrides).
- No fallback chain: if the transcription call fails, the image becomes a placeholder and the turn continues.

## Alternatives

For a feature-rich take (fallback VLM chain, local Ollama, settings-page UI, image downscaling, classified errors), see [`dsh-vision-proxy`](https://www.npmjs.com/package/dsh-vision-proxy). This plugin is the minimal, DeepSeek-only, zero-dependency version of the same idea.

## License

[MIT](LICENSE)