dsh-compat-probe

Readme

# dsh-compat-probe

A DeepSeek Harness plugin with one tool. `compat_probe` sends five tiny fixture
requests to one OpenAI-compatible server and returns a protocol card: a fixed
set of booleans saying what that server actually implements, rather than what
its docs claim.

The card answers the questions that break a harness at runtime. Is the server
up? Does `GET /models` answer? Does `stream: true` come back as Server-Sent
Events, or as one non-streaming JSON completion because the build ignored the
flag? Does it accept `max_tokens`, `max_completion_tokens`, both, or neither?

It pairs with [dsh-llamacpp](https://github.com/jwilson411/dsh-llamacpp), which
does the actual serving. This package only asks questions.

## What this is not

- It does **not** ship GGUF weights, and it does not download any.
- It does **not** start `llama-server` or any other process. Point it at a
  server you already run.
- It is **not** Ollama, and it does not speak the Ollama API.
- It is **not** a load tester. Five requests, once, when the model calls the
  tool.
- It is **not** a GPU benchmark. It measures nothing about throughput,
  latency, or hardware.
- It is **not** a client for hosted OpenAI or hosted Anthropic. The default
  target is loopback, and a non-loopback target is refused unless the
  deployment opts in.

## Install

```sh
dsh plugin --profile web add github:jwilson411/dsh-compat-probe
```

The installer reads `dsh.bundle.patch` from `package.json` and appends
`cordis.patch.yml` to the profile's ordered bundle list. The bundle inserts one
row, `compat-probe`, pointing at loopback.

### Pin the harness packages

Install `@deepseek-ai/dsh-*` at `0.1.1-rc.2` explicitly. The `latest` dist-tag
on those packages is older than the release this plugin builds against, so an
unpinned install resolves backwards and `defineTool` will not have the output
schema shape used here.

```sh
npm install @deepseek-ai/dsh-tools@0.1.1-rc.2
```

## Configuration

Set these in the profile's `cordis.patch.yml`, or in a `--patch` overlay, on
the row with `id: compat-probe`. An id-targeted patch replaces the row's whole
`config`, so restate every field you mean to keep.

| Key | Default | Env fallback | Notes |
| --- | --- | --- | --- |
| `baseURL` | `http://127.0.0.1:8080/v1` | `DSH_COMPAT_PROBE_BASE_URL` | Includes the API prefix, so `/v1` is never appended twice. This is where `llama-server --port 8080` listens. |
| `apiKey` | unset | `DSH_COMPAT_PROBE_API_KEY` | Sent as `Authorization: Bearer` when set, and never logged. Prefer the environment variable over writing a credential into the profile. Never a tool argument. |
| `allowRemote` | `false` | none, on purpose | Permit a non-loopback `baseURL`. No env fallback: widening where the harness sends requests should take an edit to the profile, not an exported shell variable. |
| `model` | `probe` | `DSH_COMPAT_PROBE_MODEL` | The model id named on the fixture requests. A single-model server ignores it. |
| `timeoutMs` | `3000` | `DSH_COMPAT_PROBE_TIMEOUT_MS` | Per-request budget in milliseconds. |

The tool itself declares no parameters. Where to probe and which credential to
send are deployment decisions, so a model cannot aim the harness at a host the
operator did not name, and has no argument slot to smuggle a key into.

## The card

| Field | Type | Meaning |
| --- | --- | --- |
| `reachable` | boolean | Any fixture request received an HTTP response at all. |
| `models_ok` | boolean | `GET {baseURL}/models` answered 2xx. |
| `stream_sse` | boolean | A `stream: true` chat completion came back as Server-Sent Events (`text/event-stream`, or `data:` frames). |
| `stream_ignored_json` | boolean | A `stream: true` chat completion came back as one non-streaming JSON `chat.completion`, meaning the server ignored `stream`. |
| `honors_max_tokens` | boolean | A POST carrying `max_tokens: 1` avoided a 4xx or 5xx. |
| `honors_max_completion_tokens` | boolean | A POST carrying `max_completion_tokens: 1` avoided a 4xx or 5xx. |
| `http_status` | integer or null | The first non-2xx status seen in probe order, otherwise the last successful status, otherwise null when nothing connected. |
| `error_code` | string or null | Null when every fixture answered 2xx. `UNREACHABLE` when a request got no answer, `HTTP_ERROR` when one came back non-2xx. |

Failures are fields, not exceptions. A refused connection, a DNS failure, a
timeout, and a 500 each land in the card, so the model reads one shape whether
the server is healthy or dead.

## Default-deny for non-loopback targets

Only `http:` and `https:` on a loopback host (`localhost`, `::1`, anything in
`127.0.0.0/8`) are probed. Anything else is refused before a socket is opened,
and the refusal is thrown rather than returned so it cannot be ignored by
forgetting to read a boolean. To probe a server on another host, set
`allowRemote: true` on the plugin row.

Loopback is judged as written, not as resolved. A hostname that happens to
resolve to `127.0.0.1` is still remote to this check.

Denials carry a stable `reason`: `INVALID_URL`, `SCHEME_DENIED`, or
`NOT_LOOPBACK`.

## Tests

```sh
npm test
```

The suite is offline. Every server it probes is a `node:http` mock bound to
`127.0.0.1:0`, so the kernel picks a free port and nothing outside the machine
is contacted. No GGUF weights, no llama.cpp process, no GPU, no credential.
Unreachability is produced by binding a port and releasing it, not by waiting
on a timeout.

## Topics

`dsh-plugin`, `deepseek-harness`, `llama-cpp`

## License

MIT. See [LICENSE](LICENSE).