Skip to content

Commit 44a96d3

Browse files
committed
feat: AI News 周报润色 skill + weekly-review v2.3 + 多项修复
Merge PR #61: adds skills/ai-news-weekly-polish for iterative weekly report polishing with 11-dimension audit rubric, v2 weekly_summary.py with event clustering, weekly-review skill v2.3, and various fixes including FreshRSS cache recovery, HuggingFace retry, bilibili upload, and Zotero linked_file sync. Closes #4
2 parents 29127e7 + fdb5cb4 commit 44a96d3

45 files changed

Lines changed: 11902 additions & 643 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.claude/commands/zotero-notebooklm.md

Lines changed: 0 additions & 81 deletions
This file was deleted.

.env.example

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,3 +99,18 @@ DAILYINFO_DATA_ROOT=
9999
# Used when DeepSeek V4 Pro returns empty responses after retries.
100100
# The fallback routes through OpenRouter. Default: moonshotai/kimi-k2.5
101101
# DAILYINFO_FALLBACK_MODEL=
102+
103+
# -----------------------------------------------------------------------
104+
# Zotero Web API (for zotero_sync.py linked_file import)
105+
# -----------------------------------------------------------------------
106+
# Create an API key at https://www.zotero.org/settings/keys
107+
# with "Allow library access" and "Allow notes access" enabled.
108+
ZOTERO_API_KEY=
109+
# Your numeric Zotero user ID (visible at https://www.zotero.org/settings/keys)
110+
ZOTERO_LIBRARY_ID=
111+
# Local path to Google Drive papers folder where ZotMoov stores linked PDFs.
112+
# This is the same folder set as "Linked Attachment Base Directory" in
113+
# Zotero Preferences > Advanced > Files and Folders.
114+
GDRIVE_PAPERS_PATH=
115+
# Windows example: G:\我的云端硬盘\Documents\Papers\Zotero_Papers
116+
# or: C:\Users\owen\Google Drive\我的云端硬盘\Documents\Papers\Zotero_Papers

.gitignore

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,15 @@ docs/docs/
4141
# 本地数据卷
4242
workspace/
4343
data/
44+
output/
4445

4546
# 本地工具运行时(CodeGraph 数据库 / oh-my-codex 状态)
4647
.codegraph/
4748
.omx/
4849
.claude/
50+
51+
# Playwright MCP 浏览器工作目录
52+
.playwright-mcp/
53+
54+
# Firecrawl 输出
55+
.firecrawl/

CLAUDE.md

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,23 @@ dailyinfo push -d 2026-04-22 # Specific date
4545
dailyinfo status # Show today's briefing/pushed counts
4646
dailyinfo start/stop/restart # FreshRSS Docker container
4747
dailyinfo logs # Tail execution log
48+
dailyinfo clean-cache # Delete FreshRSS cache files older than 24h
49+
dailyinfo clean-cache --dry-run # Preview what would be deleted
50+
51+
# Download PDFs (agent-operated, requires Playwright MCP `mcp__plugin_playwright_playwright__*`)
52+
# Deterministic patterns per publisher — see skills/download-pdf/SKILL.md for full flow:
53+
# Nature (OA): navigate → click "Download PDF" → Chrome native download
54+
# Nature (inst): navigate → WAYF login (user does SSO) → click "Download PDF"
55+
# Wiley/AGU (all): navigate to pdfdirect?download=true → Chrome native download
56+
# Cloudflare blocks: pause, tell user to pass challenge, wait for "done"
57+
dailyinfo download-pdf 10.1016/j.jhydrol.2024.132471 # Print download instructions for the skill
58+
python scripts/download_pdf.py verify <pdf> # Verify PDF and extract metadata
59+
python scripts/download_pdf.py detect <url> # Detect publisher from URL
60+
61+
# Sync downloaded PDF to Zotero (linked_file, zero cloud quota)
62+
# ⚠️ MUST use `uv run python` — conda Python lacks pyzotero
63+
uv run python scripts/zotero_sync.py <pdf> <doi> --json # Copy to GDrive + create Zotero item
64+
uv run python scripts/zotero_sync.py <pdf> <doi> --dry-run # Preview without creating
4865

4966
# Zotero -> NotebookLM (agent-operated)
5067
# Prefer the Claude Code slash command:
@@ -122,6 +139,8 @@ Scrape sources with custom parsing need matching `if self.name == "..."` dispatc
122139
Required: `DEEPSEEK_API_KEY`, `DISCORD_BOT_TOKEN`
123140
Optional: `OPENROUTER_API_KEY` (fallback model), `DISCORD_CHANNEL_PAPERS/AI_NEWS/CODE/RESOURCE`, `FRESHRSS_USER/PASSWORD`, `DAILYINFO_DATA_ROOT` (default: `~/.myagentdata/dailyinfo`), `DAILYINFO_FALLBACK_MODEL`
124141

142+
Zotero sync optional: `ZOTERO_API_KEY`, `ZOTERO_LIBRARY_ID`, `GDRIVE_PAPERS_PATH` (for `zotero_sync.py` linked_file attachment)
143+
125144
## Testing Conventions
126145

127146
- **Autouse `tmp_data_root`** in `conftest.py` redirects all filesystem writes to `tmp_path` and sets `DISCORD_BOT_TOKEN`
@@ -141,6 +160,70 @@ Issues are tracked as GitHub issues on `iHeadWater/dailyinfo`. See `docs/agents/
141160

142161
Default label vocabulary (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
143162

163+
### Download PDF (download-pdf)
164+
165+
Download academic PDFs through institutional access (DUT SSO) using Playwright browser automation. Zotero sync via linked_file attachment. See `skills/download-pdf/SKILL.md`.
166+
167+
**New-machine Playwright setup** (one-time, ~5 min):
168+
169+
```bash
170+
# 1. Enable the official Playwright plugin (provides mcp__plugin_playwright_playwright__* tools)
171+
# In ~/.claude/settings.json → enabledPlugins → "playwright@claude-plugins-official": true
172+
173+
# 2. Install Chromium (if plugin auto-download fails)
174+
npx playwright install chromium
175+
176+
# 3. Install @playwright/mcp globally (provides the CLI that the plugin wraps)
177+
npm install -g @playwright/mcp@latest
178+
```
179+
180+
The plugin provides `mcp__plugin_playwright_playwright__*` tools (standalone Chromium, NOT the user's personal Chrome).
181+
Browser profile (cookies, WAYF/SSO sessions) persists in `.playwright-mcp/` under the project directory.
182+
Nature login survives across Claude Code restarts; Wiley/AGU Cloudflare challenge must be passed once per session.
183+
184+
**MCP tools to use:**
185+
- `mcp__plugin_playwright_playwright__browser_navigate` — navigate to URL
186+
- `mcp__plugin_playwright_playwright__browser_click` — click element (use ref from snapshot)
187+
- `mcp__plugin_playwright_playwright__browser_snapshot` — get page accessibility tree
188+
- `mcp__plugin_playwright_playwright__browser_type` — type text into field
189+
- `mcp__plugin_playwright_playwright__browser_press_key` — press keyboard key
190+
- `mcp__plugin_playwright_playwright__browser_wait_for` — wait for text or time
191+
- `mcp__plugin_playwright_playwright__browser_run_code_unsafe` — run arbitrary Playwright code
192+
- `mcp__plugin_playwright_playwright__browser_tabs` — manage browser tabs
193+
194+
**DO NOT use:**
195+
- `mcp__plugin_ecc_playwright__*` — requires Chrome extension bridge, needs separate setup
196+
- `browser_evaluate` + `readAsDataURL()` — crashes MCP on PDFs >1MB
197+
- `browser_run_code` + `require('fs')``require` is not defined in the MCP runtime
198+
199+
### Bilibili Upload (bilibili-upload)
200+
201+
Upload podcast audio to Bilibili as video (audio + auto-generated cover → MP4 via ffmpeg → biliup upload). See `skills/bilibili-upload/SKILL.md`.
202+
203+
**One-time setup:**
204+
205+
```bash
206+
winget install --id=ForgQi.biliup-rs -e
207+
biliup -u ~/.bilibili/cookies.json login # scan QR code, valid ~2 years
208+
```
209+
210+
**Usage:**
211+
212+
```bash
213+
# Upload audio (cover auto-generated)
214+
dailyinfo bilibili-upload "output/weekly-review/2026-06-28/podcast/audio_hydrology.mp3" \
215+
--title "水文AI周报 2026-W26" \
216+
--tags "AI,水文,科研"
217+
218+
# Preview only (no upload)
219+
dailyinfo bilibili-upload audio.mp3 --title "Test" --dry-run
220+
221+
# In Claude Code, just say: "上传这周的水文周报音频到B站"
222+
```
223+
224+
biliup cookie at `~/.bilibili/cookies.json` persists for ~2 years.
225+
If upload fails with code 601, wait a few minutes and retry (rate limit).
226+
144227
### Domain docs
145228

146229
Single-context — one `CONTEXT.md` + `docs/adr/` at the repo root. See `docs/agents/domain.md`.

README.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,22 @@ dailyinfo push
133133
| `dailyinfo zotero-brief --collection water --artifact audio` | Process the `water` collection and request Audio Overview |
134134
| `dailyinfo zotero-brief --artifact video` | Request NotebookLM Video Overview |
135135
| `dailyinfo zotero-brief --manual-only` | Prepare local materials without calling NotebookLM |
136+
| `dailyinfo download-pdf <doi>` | PDF download via Claude Code (see [docs](docs/download-pdf.md)) |
137+
| `uv run python scripts/zotero_sync.py <pdf> <doi>` | Zotero linked_file sync (see [docs](docs/zotero-sync.md)) |
138+
139+
## Download PDF → Zotero Sync
140+
141+
> **Requires Claude Code + Playwright MCP plugin.** For setup, supported publishers, and failure handling, see [PDF Download](docs/download-pdf.md) and [Zotero Sync](docs/zotero-sync.md).
142+
143+
Quick reference:
144+
```bash
145+
# One-time setup
146+
npx playwright install chromium
147+
uv pip install pyzotero
148+
149+
# Then in Claude Code:
150+
/download-pdf 10.1038/s41586-026-10704-3
151+
```
136152

137153
## Zotero -> NotebookLM Agent Workflow
138154

@@ -175,6 +191,9 @@ See:
175191
| `DAILYINFO_ENV` | Environment: `prod` / `dev` / `staging` (default `prod`) |
176192
| `DAILYINFO_FALLBACK_MODEL` | Fallback model when DeepSeek returns empty (default `moonshotai/kimi-k2.5`) |
177193
| `ZOTERO_LOCAL_BASE_URL` | Zotero local API base URL, default `http://127.0.0.1:23119` |
194+
| `ZOTERO_API_KEY` | Zotero Web API key (for `zotero_sync.py` linked_file sync) |
195+
| `ZOTERO_LIBRARY_ID` | Numeric Zotero user library ID (for `zotero_sync.py` linked_file sync) |
196+
| `GDRIVE_PAPERS_PATH` | Local path to Google Drive papers folder (ZotMoov / Linked Attachment Base Directory) |
178197
| `NOTEBOOKLM_HOME` | NotebookLM profile/auth directory used by `notebooklm-py` |
179198

180199
## Scheduling and Agents
@@ -195,6 +214,8 @@ DailyInfo intentionally avoids owning the scheduler. Recommended ownership:
195214
- [Architecture](docs/architecture.md)
196215
- [CLI Reference](docs/cli.md)
197216
- [Agent Config](docs/agent-config.md)
217+
- [PDF Download (Institutional Access)](docs/download-pdf.md)
218+
- [Zotero Sync (linked_file)](docs/zotero-sync.md)
198219
- [Zotero NotebookLM Workflow](docs/zotero-notebooklm.md)
199220
- [Zotero NotebookLM 工作流](docs/zotero-notebooklm.zh.md)
200221
- [Information Sources](docs/sources.md)

README.zh-CN.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,24 @@ dailyinfo push
131131
| `dailyinfo zotero-brief --collection water --artifact audio` | 处理 `water` collection 并请求音频概览 |
132132
| `dailyinfo zotero-brief --artifact video` | 请求 NotebookLM 视频概览 |
133133
| `dailyinfo zotero-brief --manual-only` | 只生成本地素材,不调用 NotebookLM |
134+
| `dailyinfo download-pdf <doi>` | PDF 下载(依赖 Claude Code,见[文档](docs/download-pdf.md)|
135+
| `uv run python scripts/zotero_sync.py <pdf> <doi>` | Zotero linked_file 同步(见[文档](docs/zotero-sync.md)|
136+
137+
## 下载 PDF → Zotero 同步
138+
139+
> **依赖 Claude Code + Playwright MCP 插件。** 完整配置、支持的出版商和故障处理见 [PDF 下载](docs/download-pdf.md)[Zotero 同步](docs/zotero-sync.md)
140+
141+
快速参考:
142+
```bash
143+
# 一次性配置
144+
npx playwright install chromium
145+
uv pip install pyzotero
146+
147+
# 在 Claude Code 中输入:
148+
/download-pdf 10.1038/s41586-026-10704-3
149+
```
150+
151+
## Zotero -> NotebookLM Agent 工作流
134152

135153
## Zotero -> NotebookLM Agent 工作流
136154

@@ -171,6 +189,9 @@ NotebookLM 登录有意保留人工参与。Agent 可以打开浏览器或提示
171189
| `DAILYINFO_DATA_ROOT` | 覆盖默认数据根目录 |
172190
| `DAILYINFO_FALLBACK_MODEL` | 主模型空响应时的备用模型 |
173191
| `ZOTERO_LOCAL_BASE_URL` | Zotero 本地 API 地址,默认 `http://127.0.0.1:23119` |
192+
| `ZOTERO_API_KEY` | Zotero Web API 密钥(`zotero_sync.py` linked_file 同步用) |
193+
| `ZOTERO_LIBRARY_ID` | 数字型 Zotero 用户库 ID(`zotero_sync.py` linked_file 同步用) |
194+
| `GDRIVE_PAPERS_PATH` | Google Drive 论文文件夹本地路径(ZotMoov / 链接附件根目录) |
174195
| `NOTEBOOKLM_HOME` | `notebooklm-py` 使用的 NotebookLM profile/auth 目录 |
175196

176197
## 调度和 Agent 分工
@@ -191,6 +212,8 @@ DailyInfo 不负责调度。推荐分工如下:
191212
- [系统架构](docs/architecture.md)
192213
- [CLI 参考](docs/cli.md)
193214
- [Agent 配置](docs/agent-config.md)
215+
- [PDF 下载(机构访问)](docs/download-pdf.md)
216+
- [Zotero 同步(linked_file)](docs/zotero-sync.md)
194217
- [Zotero NotebookLM Workflow](docs/zotero-notebooklm.md)
195218
- [Zotero NotebookLM 工作流](docs/zotero-notebooklm.zh.md)
196219
- [数据源说明](docs/sources.md)

docs/cli.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,6 +158,34 @@ automation. If Google changes the NotebookLM UI/API or auth is not ready, the
158158
command degrades to the local material package so the run can be completed
159159
manually in the NotebookLM web UI.
160160

161+
### Download PDF (institutional access)
162+
163+
Requires Claude Code + Playwright MCP plugin. The CLI is a thin helper — real browser
164+
orchestration happens through the agent skill. See [PDF Download](download-pdf.md) for
165+
full setup, supported publishers, and failure handling.
166+
167+
```bash
168+
dailyinfo download-pdf <doi|pii|url> # Print download instructions for the skill
169+
python scripts/download_pdf.py detect <url> # Detect publisher from URL
170+
python scripts/download_pdf.py verify <pdf> # Verify PDF and extract metadata
171+
python scripts/download_pdf.py doi <doi> # Resolve DOI to article URL + publisher
172+
```
173+
174+
### Sync PDF to Zotero (linked_file)
175+
176+
Creates a Zotero item with a linked_file attachment. See [Zotero Sync](zotero-sync.md)
177+
for prerequisites, setup, and design rationale.
178+
179+
```bash
180+
uv run python scripts/zotero_sync.py <pdf> <doi> --json # Copy to GDrive + create Zotero item
181+
uv run python scripts/zotero_sync.py <pdf> <doi> --dry-run # Preview only
182+
```
183+
184+
Design rationale: uses pyzotero Web API with `linked_file` attachment
185+
(`attachments:<filename>` scheme) rather than `imported_file`, because Zotero's
186+
300 MB cloud quota is too small for institutional PDFs (10-90 MB each).
187+
See [ADR-001](decisions/adr-001-zotero-linked-file-sync.md).
188+
161189
### Weekly Recap
162190

163191
```bash
@@ -198,6 +226,9 @@ FRESHRSS_PASSWORD=freshrss123
198226
| `FRESHRSS_PASSWORD` | FreshRSS password |
199227
| `DAILYINFO_FALLBACK_MODEL` | Fallback LLM when the primary model returns empty (default `moonshotai/kimi-k2.5`) |
200228
| `ZOTERO_LOCAL_BASE_URL` | Zotero local API base URL for `zotero-brief` (default `http://127.0.0.1:23119`) |
229+
| `ZOTERO_API_KEY` | Zotero Web API key for `zotero_sync.py` linked_file import (create at https://www.zotero.org/settings/keys) |
230+
| `ZOTERO_LIBRARY_ID` | Numeric Zotero user library ID for `zotero_sync.py` |
231+
| `GDRIVE_PAPERS_PATH` | Local path to Google Drive papers folder (ZotMoov target, same as Zotero's Linked Attachment Base Directory) |
201232
| `NOTEBOOKLM_HOME` | NotebookLM profile/auth directory used by `notebooklm-py`; must match the login run |
202233

203234
## Scheduling

0 commit comments

Comments
 (0)