feat: HF Daily Papers + arXiv 关键词/语义双通道论文信源 - #80
Open
CylenLC wants to merge 1 commit into
Open
Conversation
…r sources Implements the two-channel paper discovery described in iHeadWater#12. Channel A -- Hugging Face Daily Papers (community heat): * New api source hf_daily_papers, fetched per publication day via /api/daily_papers?date=YYYY-MM-DD so a run reports one specific day rather than the rolling 50-entry board the bare endpoint returns. * weekdays_only: HF publishes Mon-Fri; Sat/Sun return HTTP 200 with an empty list. Guarded upstream of fetch so no briefing file is written at all -- an empty fetch would otherwise reach the placeholder branch and push a "no updates" notice to Discord on weekends. * No count cap: top_percent 100 keeps the full day (13-26 papers observed) instead of the top N by upvotes. Channel B -- arXiv keyword + embedding retrieval: * scripts/paper_retrieval.py: keyword/phrase matching with word-boundary semantics, plus lexical_embedding_union strategy. * scripts/embedding_retrieval.py + qwen_embedding_service.py: optional local llama.cpp embedding backend for semantic recall. * scripts/text_match.py: one phrase matcher shared by both pipelines. * exclude_phrases acts as a cross-channel hard veto, so a vetoed paper cannot re-enter through the embedding channel. Both channels write to the shared arxiv category and dedupe against per-source seen state, so the same paper is not pushed twice. Verified: pytest 251 passed / 1 skipped (skip = fastapi absent), ruff clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #12
概述
按 #12 的双通道设计实现论文信源：通道 A 社区热度（HF Daily Papers） + 通道 B 关键词/语义筛选（arXiv），两通道并列生成、各自去重，同属
arxiv分类推送。通道 A：HuggingFace Daily Papers
新增
hf_daily_papers（type: api），三个要点：按发布日抓取。 走
/api/daily_papers?date=YYYY-MM-DD取某一天的页面，而不是无参端点返回的滚动 50 条热榜。实测差异：无参端点的publishedAt跨 5 天，会把几天前的论文混进当天简报。周末不产出（
weekdays_only）。 HF 只在周一至周五发布，周六日返回HTTP 200+ 空列表（实测：周六/周日 0 篇，周一 13 / 周五 26）。这一层拦截必须放在fetch()之前——如果只让 fetch 返回空列表，会落进_process_regular_source的 placeholder 分支，push_to_discord再把占位文件当"📭 今日无文章更新"发到频道，等于周末照样推送。所以用DataSource.skips_today()在两条调用路径（通用路径 +run_pipeline_arxiv，后者先 fetch）前短路，周末一个文件都不写。不限数量。
top_percent: 100，取当天全量而非投票前 N 篇。注意解析处仅在max_items键缺失时才不设上限（兄弟解析器默认 25/20），所以是删键生效。max_articles_per_batch: 10保留，它限制单次 AI 调用的 prompt 规模，不限覆盖范围。通道 B：arXiv 关键词 + 向量召回
scripts/paper_retrieval.py— 关键词/短语匹配，词边界语义（watermark不误命中watermarking），含lexical_embedding_union策略scripts/embedding_retrieval.py+qwen_embedding_service.py— 可选的本地 llama.cpp 向量后端scripts/text_match.py— 两条 pipeline 共用一个短语匹配器exclude_phrases是跨通道硬否决，被否决的论文不会从向量通道绕回来两通道各自维护 seen 状态，同一篇论文不会重复推送。
验证
pytest— 251 passed, 1 skipped（跳过项为fastapi未安装，与本 PR 无关）ruff check scripts/ tests/— All checks passedassert 1 == 0），因此它钉住的是"不写文件、不推送"这个行为，而非仅返回值这个分支的 diff 包含了 OpenReview 顶会 pipeline（
scripts/conference.py+1840、scripts/openreview_provider.py+452 及其测试），也就是 #79 的范围。原因是两条工作线共享祖先1da1d0c，而run_pipelines.py:27无条件from conference import ...，upstream/main又没有conference.py，剔除会直接 import 失败——因此无法在本分支内干净剥离。而且本分支的 conference 代码是较旧的一支：#79 的
conference.py有 2686 行，本分支只有 1840 行。若先合本 PR，会覆盖 #79 上更新的实现。建议先合 #79，我再把本 PR rebase 到其之上，只保留 #12 的双通道改动（
paper_retrieval.py/embedding_retrieval.py/text_match.py/datasource.py/config/sources.json等，这些文件均不依赖 conference 模块，已验证）。听 maintainer 安排。