dsh-qwen38-ninfer-compaction-fix

Readme

# Qwen3.8 (NInfer) Compaction Fix

Fix for a local **qwen3.8-27b** gateway running on the **NInfer** engine that fails to compact. The model thinks at `xhigh`, spends the entire output token budget on reasoning before reaching a conclusion, and the compaction checkpoint comes back truncated. This [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (`dsh`) plugin disables thinking **for compaction calls only** and applies, to those calls, the sampling parameters the model recommends when thinking is off.

> **Scope: NInfer only.** This package is written for gateways served by the
> [NInfer](https://github.com/Neroued/ninfer) engine (`ninfer-serve`, OpenAI-compatible API): the wire
> fields it rewrites (`reasoning_effort`, `presence_penalty`, `repetition_penalty`, `min_p`) are the
> ones the NInfer gateway interprets. If your qwen3.8 is launched by something else
> (llama.cpp, vLLM, FastMTP, ...), the same idea — turn thinking off for dsh's compaction and
> session-title calls so `xhigh` reasoning cannot burn the entire output budget — still applies,
> but each engine speaks different wire parameters, so this package does not claim to fix those
> launch methods.

## Install

```sh
dsh plugin --profile web add dsh-qwen38-ninfer-compaction-fix
```

Then restart `dsh web` (or refresh the GUI page). The plugin is also listed in the DSH
[plugin market](https://github.com/dsh-market/dsh-market) (Settings → Plugin Market → search "qwen38").

## The symptom

When context pressure builds, dsh compacts the conversation into a checkpoint. With a local **qwen3.8-27b** running at its default reasoning level, compaction sometimes fails, and the session is left with exactly this line:

> summarization truncated at the token cap (incomplete checkpoint)

That is dsh-compaction-basic's verdict that the summarizer hit its output token cap without finishing. The conversation is condensed only into the truncated text: everything the summary never reached is effectively lost from context.

## Why it happens

Local **qwen3.8-27b** deployments think at their default effort: `xhigh` in a typical route config, on every call that does not explicitly set a reasoning effort. Compaction is one such call. The model is asked to write a long summary of the whole conversation. At `xhigh`, it spends the **entire `max_tokens` output budget on reasoning tokens before ever writing a word of the summary**. The response hits the cap with no conclusion, and you get the truncated checkpoint above.

## The fix

1. **Disable thinking for compaction only.** Stamps `reasoning_effort: "off"` onto the compaction call (waterfall layer, resolved through the model's `reasoningEfforts` declaration to the wire value `reasoning_effort: "none"`): zero reasoning tokens, and the whole output budget is available for the summary. Every other call, whether a normal turn, a subagent, or a request for any other model, keeps the route's `xhigh`. Thinking is never disabled globally.
2. **Apply the sampling parameters recommended when thinking is off.** **qwen3.8-27b** publishes different recommended sampling parameters for thinking mode and for the mode with thinking disabled. The plugin writes the second set (`temperature: 0.7, top_p: 0.8, top_k: 20, min_p: 0.0, presence_penalty: 1.5, repetition_penalty: 1.0`) into the compaction request body, so the request actually matches the mode it is in.
3. **Restore the output budget.** Raises the wire `max_tokens` of compaction bodies up to a configurable floor (default `16384`; the value is never lowered), undoing the context clamp in pi-ai that can collapse `max_tokens` on large conversations.
4. **Fix session titles (secondary).** Session title generation runs on the same route with a tiny output budget (`maxTokens: 64`), so `xhigh` thinking truncates titles too. The plugin writes the configured `reasoning_effort` wire value into title request bodies. The title provider locks its LLM options in place before the waterfall, so only the wire body is reachable.

Everything else is untouched. Every gate checks the purpose and the model (waterfall), or the signature and the model (HTTP), so conversation turns, subagents, and any other model pass through unchanged, byte for byte, with the route defaults. Every guard fails open.

## The model name must match: read this first

The plugin only acts when the `model` field of the outgoing request matches an id in the `models` allowlist: default exactly `qwen3.8-27b`, case sensitive. It is an **exact id match, not a family or substring match**, because the sampling parameters written into the body are specific to this model and would be wrong for any other model.

**What id is being compared:** the model `id` your route serves, the one declared under `llm-pi-ai.providers.<provider>.models[].id` in `$DSH_HOME/settings.yaml`. That is the value dsh puts in the `model` field of every outgoing request to your gateway. It is not whatever name your gateway internally calls the model.

**Make it match: either of two places works:**

1. The `qwen38-compaction-fix:` section of `$DSH_HOME/settings.yaml` (applies live, no restart):

   ```yaml
   qwen38-compaction-fix:
     models: [your-model-id]
   ```

2. The `config:` block of the plugin row in your profile's `cordis.patch.yml` (applies on the next GUI load):

   ```yaml
   - id: qwen38-compaction-fix
     config:
       models: [your-model-id]
   ```

If your gateway serves the model under a different id (a custom model name, a suffixed variant, different casing), add *that* id to `models`. The sampling values stay the same; only the id has to match.

**If nothing matches, the plugin silently does nothing:** every request passes through unchanged, byte for byte, and there is no warning on a miss. If you installed this plugin and you still see the truncated checkpoint, check the model id first.

## Configuration

All keys are optional; defaults are applied by the schema. Precedence (highest first):

1. `qwen38-compaction-fix:` section of `$DSH_HOME/settings.yaml` (applies live, no restart)
2. the `config:` block of the plugin row in your profile's `cordis.patch.yml`
3. the defaults shipped with the plugin

Example `settings.yaml`:

```yaml
qwen38-compaction-fix:
  effort: off            # "" disables the effort policy
  models: [qwen3.8-27b]  # exact ids; [] disables the whole policy
  sampling:              # wire field names, written verbatim into the body
    temperature: 0.7
    top_p: 0.8
    top_k: 20
    min_p: 0.0
    presence_penalty: 1.5
    repetition_penalty: 1.0
  maxTokensFloor: 16384  # 0 disables the floor
  titleReasoning: none   # "" disables the title gate
```

| Key | Default | Meaning |
|---|---|---|
| `effort` | `"off"` | Reasoning effort stamped onto matched calls. Preference order: configured, then `off`, then `low`; a model offering none of them keeps its own default (a single warning). `""` disables the effort policy. |
| `purposes` | `["compaction"]` | `purpose` tags of LLM calls the waterfall layer applies to. |
| `models` | `["qwen3.8-27b"]` | Exact model ids (case sensitive) the policy applies to, checked at every layer. Empty list disables the whole policy. See the model name section above. |
| `sampling.*` | `{}` | Sampling settings written verbatim into compaction request bodies. Absent keys stay absent. |
| `maxTokensFloor` | `16384` | The wire `max_tokens`/`max_completion_tokens` of compaction bodies is raised to at least this value; never lowered. `0` disables. |
| `titleReasoning` | `"none"` | Wire `reasoning_effort` value written into session title request bodies. `""` disables the gate. |

An explicit `reasoningEffort` on the call itself always wins over the plugin default.

## How it works

Four cooperating layers:

1. **`llm/stream` waterfall:** for calls whose `purpose` is in `purposes` and whose `options.model` is in `models`, resolves the model's offered reasoning efforts and stamps the chosen effort in place before dispatch.
2. **HTTP sampling:** wraps the process `fetch`; when a chat completion body carries the compaction engine's final instruction (a stable signature) and its `model` field is allowed, applies the configured sampling entries — wire fields as spelled for the NInfer gateway.
3. **HTTP max_tokens floor:** same gate; raises the output cap to the floor; never lowers it.
4. **HTTP session title reasoning:** when a body carries the title provider's system prompt (stable signature) and its `model` is allowed, writes the configured `reasoning_effort` wire value.

Identity at the HTTP layer relies on the instruction text shipped by dsh-compaction-basic and dsh-session-title-llm. If a future dsh release changes those instructions, the matching HTTP gate silently stops matching and the body keeps its wire defaults. The other gates are unaffected, and nothing ever breaks LLM traffic, because every guard fails open.

## Limitations

- **Engine scope: NInfer.** The wire fields this plugin rewrites are the ones the NInfer gateway interprets. Gateways served by other engines (llama.cpp, vLLM, FastMTP, ...) may ignore or spell these fields differently; the same thinking-off idea can be ported to them, but that port is a different package.
- Model matching is an **exact id match** against the id declared under `llm-pi-ai.providers.<provider>.models[].id` in `settings.yaml`. See the model name section above.
- The HTTP layer signatures track specific dsh releases; see "How it works" for what happens when a signature stops matching.
- This plugin shapes requests for the *local gateway* you run. It does not change the harness's own routing or the server's real capacity limits (the server still enforces them).

## Testing

The gating logic is covered by a smoke test (see [CHANGELOG.md](./CHANGELOG.md#verification)): allowed model bodies are rewritten with the expected sampling, floor, and reasoning values; disallowed model, missing model, and empty allowlist bodies pass through unchanged, byte for byte.

## License

MIT. See [LICENSE](./LICENSE).