dsh-fetch-data

Readme

# dsh-fetch-data

🌐 **English** | [中文](README.zh.md)

![dsh-fetch-data](docs/banner.svg)

**Structured data extractor for DeepSeek Harness** — intercepts a page's real data APIs (XHR/fetch JSON) and returns **precise fields** (comment like-counts, rankings, prices, tables) that text extractors lose when flattening HTML.

**read_url answers "what does this page say?" — fetch_data answers "what are the exact numbers/data behind it?"**

## Why it exists

| | read_url (text extractor) | **fetch_data** (this plugin) |
|---|---|---|
| Reads | cleaned page text / Markdown | the page's **underlying JSON APIs** |
| Output | prose the model reads | **structured fields** (`{title, view}` pairs) |
| Strength | reading articles/docs | **exact field attribution** — which number belongs to which row |
| Weakness | flattened text loses number↔item association | requires playwright (browser engine) |

Real example: on a Xiaoheihe post, `read_url` returned `"60125"` with no way to know if it meant 60 likes + 125 favorites or anything else. `fetch_data` intercepts `/bbs/app/link/tree` and returns `{user, up, content}` — unambiguous.

## Tool

**`fetch_data(url, api?, fields?, maxItems?)`** — capture the page's data APIs and extract fields

| Param | Type | Default | Description |
|---|---|---|---|
| `url` | string | required | http(s) page whose data APIs to capture |
| `api` | string | auto | Pin a specific endpoint from the structure-mode menu (e.g. `"/x/web-interface/ranking/v2"`); auto-selected when omitted |
| `fields` | string | structure mode | Comma-separated field paths to extract; arrays via `[]`: `"data.list[].title,data.list[].view"` |
| `maxItems` | number | 20 | Max array items per extracted field (1–100) |

**Two modes:**

1. **Structure mode** (no `fields`) — returns the auto-picked endpoint's 2-level structure plus a **menu of all JSON APIs** (path · size · array?), so the model can re-call with `api=` to pin one:
```
页面 25 个 JSON 接口
选中: /x/web-interface/ranking/v2 (137763B)
结构: { code: number, message: string, ttl: number, data: { note: string, list: [100] { aid: number, ... } } }

接口清单（可传 api=<路径> 指定其中一个再提取字段）:
  /x/web-interface/nav · 249B
  /x/vip/ads/materials · 1140B · 含数组
  ...
```

2. **Field mode** (with `fields`) — extracts exact values, arrays truncated at `maxItems`:
```
data.list[].title (前5条):
  用MC还原《神的随波逐流》 【B萌应援】
  WasteTheFallen丨首曝PV＆实机演示：凝视深渊，人性渐泯
  ...
data.list[].stat.view (前5条):
  2428730
  10326863
  ...
```

## Auto-pick logic

Ranks captured JSON responses: **largest one containing an array wins** (data endpoints are usually big and array-bearing; tracking/config endpoints are small). Falls back to the largest JSON. Manual `api=` overrides.

## Real-world verification (2026-08-16, v0.1.3)

14-site sweep: **12 OK / 2 static-site expected errors / 0 crashes** — driven by `multi-site.mjs` (committed). Auto-scroll (lazy-load capture) + JSONP parsing verified live.

| Site | Result |
|---|---|
| Bilibili popular ranking | ✅ auto-picked `/x/web-interface/ranking/v2` (139KB, `list[100]`), extracted titles + play counts, 1:1 attributed |
| Juejin feed | ✅ auto-picked `/recommend_api/v1/article/recommend_all_feed`, extracted article titles |
| Weibo | ✅ captured `/ajax/feed/hottimeline` (240KB) + `/ajax/statuses/config` (745KB, oversized config edge case) |
| QQ news | ✅ captured `/getQNChannels` (304KB, 36 APIs) |
| Douban | ✅ captured `/rexxar/api/v2/search/hots` |
| Taobao | ✅ captured 20 JSON APIs incl. mtop config (824KB, oversized-config edge case) |
| JD | ✅ captured `/wp-json/news/list` + `/category/get` |
| Zhihu / Baidu / CSDN / Netease / Xiaoheihe | ✅ captured their JSON endpoints (some are config/menu APIs — use structure mode to pick the data one) |
| example.com / ruanyifeng.com (static) | ✅ clear "no JSON API captured" error, no crash |

- **18 zero-dep assertions** (field-path extractor, pick logic, structure summary, static-asset filter, truncation precision, description-length guard, JSONP parsing) + **22 live-interception assertions** (incl. scroll-triggered lazy capture + JSONP extraction) all green.

## Why it saves tokens

- **Two-phase**: structure mode returns a small "menu" (not the 137KB payload); field mode returns only the requested columns — never dumps the full JSON;
- **Compact fixed cost**: tool description trimmed to ~300 chars (the one thing sent on every call); static schema (KV-cache friendly);
- **Sorted menu**: array-bearing + biggest endpoints listed first, so the model finds the data API immediately; static-asset JSON (Bilibili `/bfs/svg-next/...`) filtered out and the count shown inline;
- Sizes rendered as `136KB` not `139070B`; arrays truncated at `maxItems`; values capped at 200 chars;
- Compact text render; clear one-line errors.

## Architecture (DSH-aligned)

- Browser singleton launched once, **closed via `ctx.effect`** on unload (temporal composability);
- Fresh browser context per call — no cookie/state leakage across calls (avoids anti-bot flakiness);
- `domcontentloaded` + settle wait instead of `networkidle` (heartbeat-polling sites never idle);
- **Auto-scroll** triggers lazy-loaded data APIs (feeds / infinite lists); **JSONP** responses auto-unwrapped;
- Cooperative timeout: `timeoutMs` + `exec.signal`;
- Zero runtime deps beyond Node built-ins; **playwright is the required engine** (it's the interception layer, not an optional enhancement).

## Install

```bash
# playwright is REQUIRED for this plugin (network interception core)
cd <DSH profile dir>
npm i playwright && npx playwright install chromium

# add the plugin
dsh plugin --profile web add github:2672243194/dsh-fetch-data
```

## Boundaries

- **Login-walled APIs are not accessible** (same as read_url);
- Each site has its own endpoint structure — use the structure-mode menu to discover it;
- Auto-scroll + JSONP cover most lazy-loading/JS-delivered data; remaining gaps (login walls, SSR-only pages) return a clear error, never a guess.

## Support

If dsh-fetch-data helps you, give it a ⭐ Star on GitHub. Free and open source (MIT); star count is how I decide where to keep investing.

## License

MIT