dsh-llm-auto-route

Readme

# dsh-llm-auto-route

`dsh-llm-auto-route` is a community Cordis plugin for DeepSeek Harness. It discovers which already-configured `dsh-llm-pi-ai` route should handle a request, explains the decision, and performs failover only before the first visible output.

It is deliberately a routing policy layer. It does not implement HTTP protocols, ship provider SDKs, or register `openai`, `anthropic`, `deepseek`, or other adapter routes. The official `@deepseek-ai/dsh-llm-pi-ai` plugin remains the owner of those routes.

> Community project: this package is not an official DeepSeek Harness package and does not imply DeepSeek endorsement.

## Requirements

- Node.js `>=22.19.0`
- DeepSeek Harness `0.1.0-rc.5` or a compatible `0.1.x` release
- `@deepseek-ai/dsh-llm-pi-ai` configured with the route names you want to select

The package is tested against the npm `0.1.0-rc.6` companion packages while keeping the peer range compatible with `rc.5`.

## Install

```bash
pnpm add dsh-llm-auto-route
```

The official base bundle already contains `@deepseek-ai/dsh-llm-pi-ai`. If you compose plugins manually, install and load that official adapter before this package. Load the shipped `cordis.patch.yml` with the normal DeepSeek Harness/Cordis composition command used by your deployment.

The patch adds one plugin named `llm-auto-route`; it does not add or replace any official adapter route.

## Configure the official adapter first

The route names in this plugin must already be registered by `dsh-llm-pi-ai`. The following is an abbreviated official-adapter configuration:

```yaml
- id: llm
  name: '@deepseek-ai/dsh-llm-pi-ai'
  config:
    providers:
      deepseek:
        apiKeyEnv: DEEPSEEK_API_KEY
      openai:
        apiKeyEnv: OPENAI_API_KEY
      anthropic:
        apiKeyEnv: ANTHROPIC_API_KEY
      ollama:
        baseURL: http://127.0.0.1:11434/v1
        api: openai-completions
        models:
          - id: llama3.1
            contextWindow: 131072
            maxTokens: 8192
      vllm:
        baseURL: http://127.0.0.1:8000/v1
        api: openai-completions
        models:
          - id: local-model
            contextWindow: 32768
            maxTokens: 4096
      openai-compatible:
        apiKeyEnv: GATEWAY_API_KEY
        baseURL: https://gateway.example.test/v1
        api: openai-completions
        models:
          - id: gateway-model
            contextWindow: 65536
            maxTokens: 8192
```

`dsh-llm-pi-ai` owns credentials, model metadata, transport, and stream conversion. The auto-route plugin only sees the route directory and selects one of those route keys.

## Automatic selection

An automatic request uses `provider: auto`, or omits the provider when the plugin configuration's provider token is `auto`:

```ts
const options = {
  provider: 'auto',
  model: 'deepseek-chat',
  messages,
}
```

The fixed default precedence is:

```text
explicit → provider_env → base_url → model_prefix
```

- A non-`auto` provider is always preserved. Explicit requests are never silently rerouted.
- `DEEPSEEK_API_KEY`, `OPENAI_API_KEY`, and `ANTHROPIC_API_KEY` are detection signals only; the key value is never written to a log.
- `LLM_BASE_URL` selects the generic `openai-compatible` rule. `11434` identifies Ollama and `8000` identifies a vLLM/OpenAI-compatible local endpoint.
- `deepseek-*`, `gpt-*`, `o1-*`, `o3-*`, and `claude-*` provide model-prefix hints.
- A candidate is usable only when the official adapter has registered the same route.
- Equal-priority candidates at the same stage return `AMBIGUOUS_ROUTE`; the plugin does not guess.
- A route's `defaultModel` is used only when a matching rule supplies one and the request omits its model.

Every selection can be explained without exposing credentials:

```text
已选择 deepseek/deepseek-chat；原因：发现 DEEPSEEK_API_KEY
```

Use an explicit provider or a route `priority` when an environment intentionally contains multiple credentials.

## Configuration

The shipped patch contains the default rules. An application can override them through its Cordis configuration:

```yaml
provider: auto
precedence:
  - explicit
  - provider_env
  - base_url
  - model_prefix
healthCheck:
  mode: adaptive       # off | adaptive | probe
  timeoutMs: 3000
  cacheTtlMs: 30000
failover:
  enabled: true
  maxAttempts: 3
diagnostics: info      # silent | error | info
routes:
  deepseek:
    apiKeyEnv: DEEPSEEK_API_KEY
    defaultModel: deepseek-chat
    priority: 10
  openai-compatible:
    apiKeyEnv: GATEWAY_API_KEY
    baseURLEnv: LLM_BASE_URL
    modelPrefixes: [gateway-]
```

Route fields are hints, not adapter configuration:

| Field | Meaning |
| --- | --- |
| `provider` | Registered route id to return; defaults to the `routes` key. |
| `apiKeyEnv` | Non-empty environment variable used as a provider-env signal and for discovery. |
| `baseURL` / `baseURLEnv` | Exact or user-supplied endpoint hint. `baseURLEnv` also supports unknown OpenAI-compatible hosts. |
| `baseURLPatterns` | Additional normalized URL prefixes. |
| `ports` | Local ports that identify this route when a base URL is supplied. |
| `modelPrefixes` | Model id prefixes for automatic selection. |
| `defaultModel` | Model to use when the request does not name one. |
| `priority` | Tie breaker within one matching stage; higher wins. |

Do not put an API key directly in route configuration. Use the official adapter's credential reference, normally `apiKeyEnv`, and keep this plugin's `apiKeyEnv` aligned with it.

## Health checks and failover

`adaptive` health checks reuse the official `dsh-llm` model-discovery seam when it is available, with a bounded timeout and an in-memory cache. If a deployment cannot expose discovery, route/model resolution is used as the local fallback and the real stream remains the final availability check. `off` skips preflight checks. `probe` asks the discovery seam whenever the adapter exposes it.

Failover is intentionally conservative:

- it is allowed only before text, reasoning, tool-call, or block output has been emitted;
- buffered protocol metadata from a failed attempt is discarded before trying the next route;
- aborts, explicit providers, configuration errors, and requests that already emitted output are not retried;
- a later provider never receives a partial assistant response from an earlier provider.

The plugin observes `agent/request`, `agent/request-error`, and `llm/stream`. It returns a new immutable request configuration and never mutates a frozen request object.

## Public API

The package exports `AutoRouteConfig`, `RouteRule`, `RouteDecision`, `MatchStage`, and the pure helpers `normalizeConfig`, `normalizeBaseURL`, and `resolveRoute`:

```ts
import { normalizeConfig, resolveRoute } from 'dsh-llm-auto-route'

const decision = resolveRoute(normalizeConfig(), {
  model: 'deepseek-chat',
  env: { DEEPSEEK_API_KEY: 'present' },
  registeredProviders: new Set(['deepseek']),
})

if (decision.kind === 'matched') {
  console.log(decision.candidate.provider, decision.candidate.model, decision.stage)
}
```

`AutoRouteError.code` is stable for `AMBIGUOUS_ROUTE`, `MISSING_MODEL`, `NO_CANDIDATE`, and `NO_REGISTERED_ROUTE`.

## Troubleshooting

**`NO_REGISTERED_ROUTE`** — the route exists in this plugin's default hints but not in `ctx.llm.listProviders()`. Add the same key under the official adapter's `providers` configuration.

**`AMBIGUOUS_ROUTE`** — more than one candidate matched at the same stage and priority. Set `provider` explicitly, remove an unused environment variable, or give one route a higher `priority`.

**`MISSING_CREDENTIAL` / `INVALID_CREDENTIAL`** — detection and the official adapter must agree on the environment variable. Check the variable name, not its value in logs or issue reports.

**Local gateway is not selected** — set `LLM_BASE_URL`, or provide a URL containing port `11434` (Ollama) or `8000` (vLLM). The official adapter still needs a route with that key and a model catalog.

**A response was not retried** — this is expected after the first text, reasoning, tool-call, or block output, after cancellation, for an explicit provider, or for a configuration failure. Retrying then could duplicate or splice an assistant response.

## Compatibility and project status

This is an independent ecosystem plugin for the DeepSeek Harness developer preview. It follows the current upstream guidance for community plugins, `dsh-plugin` topics, Discussions announcements, and standalone repositories. It does not open a pull request against the official Harness repository.

Upstream references:

- [`dsh-llm-pi-ai` README](https://github.com/deepseek-ai/deepseek-harness/tree/master/packages/llm/llm-pi-ai)
- [LLM interface](https://github.com/deepseek-ai/deepseek-harness/blob/master/packages/llm/llm/README.md)
- [Streaming and routing extension points](https://github.com/deepseek-ai/deepseek-harness/blob/master/docs/subsystems/llm-streaming.md)
- [Contribution guide](https://github.com/deepseek-ai/deepseek-harness/blob/master/CONTRIBUTING.zh.md)

## License

MIT. See [LICENSE](LICENSE).