BGGG Reddit Data
Use Reddit's public JSON listings through a logged-in or public Chrome tab when direct shell access is blocked. Save every response before transforming it, then normalize locally with the bundled script.
VOC Project Layout（bggg 系列共用）
bggg VOC 系列 skill（bggg-data-amazon / bggg-data-reddit / bggg-data-x / bggg-voc-report / industry-orchestrator）共用一个项目文件夹，让多平台数据规整到同一处、下游分析零改路径。开工先确定项目根目录 <project>（用户指定，或新建 voc-<产品或主题slug>/），并从 <project> 根目录执行本 skill 的全部命令（下文相对路径都基于它）：
text<project>/ PROJECT.md # 研究简报 + 决策日志（编排 skill 维护；单独使用可省） config/ # 采集目标：amazon_targets.tsv / reddit_queries.tsv / x_queries.tsv / keywords.txt work/<platform>/… # 各平台原始证据、attempt 日志、request plan、manifest data/raw/ # 各平台规范化 JSONL（统一行契约，分析共用层） data/clean|coded/ # 下游清洗与编码（industry-orchestrator 维护） output/ # 报告与交付物（bggg-voc-report 写 output/report/）
本 skill 的落点：config/reddit_queries.tsv → work/reddit/（request plan 与 manifest）+ data/raw/source_json/（未修改源响应）→ data/raw/reddit_<lang>_<date>.jsonl。
Workflow
- Define a small query file and build a request plan:
bashpython3 scripts/build_request_plan.py \ --queries config/reddit_queries.tsv \ --output work/reddit/request_plan.json
Use tab-separated columns query, lang, and round. lang and round are optional.
-
Open
https://www.reddit.com/in Chrome. Fetch each planned URL from that Reddit page with same-originfetch, including pagination throughdata.after. Do not read or export browser cookies, local storage, profile data, or credentials. -
Save each unmodified response under
data/raw/source_json/and append one manifest row per saved file. Follow references/chrome_same_origin.md for the browser loop and manifest format. -
Rank directly relevant parent posts by engagement and topical fit. Download complete comment trees for the highest-value parents with
/comments/{post_id}.json?limit=500&depth=10&sort=top&raw_json=1. Treat replies as clustered under their parent thread; do not count thousands of replies from one viral thread as independent market prevalence. -
Normalize and validate:
bashpython3 scripts/normalize_reddit.py \ --manifest work/reddit/source_manifest.jsonl \ --output data/raw/reddit_EN_2026-07-25.jsonl \ --summary work/reddit/normalize_summary.json
- Report query count, parent-post count, comment count, unique rows, deleted/removed rows skipped, unresolved
morenodes, HTTP failures, and parent-thread concentration.
Quality Rules
- Keep raw source JSON immutable. Write retries to new files rather than overwriting evidence.
- Use the Reddit fullname (
t3_…ort1_…) asnative_id; derivesource_idwith SHA-256. - Preserve exact
text_raw. Put translations or labels in downstream clean/coded files. - Skip
[deleted],[removed], empty bodies, ads, and promoted posts; count every exclusion. - Search listings are discovery samples, not complete census data. Record sort, time window, query, page cursor, and collection time.
- Stop pagination only when
afteris null, the requested cap is reached, or repeated cursors/no-new-ID protection fires. - Respect login walls, rate limits, and challenge pages. Back off and record the failure; never bypass authentication or change the user's account.
- Deduplicate exact native IDs first. Use similarity dedupe only in a later cleaning stage so raw evidence remains auditable.
- For prevalence statistics, cap or weight rows per parent thread and disclose the rule.
Degradation
If Chrome same-origin fetching is unavailable, try a normal public Reddit JSON request with a conservative user agent. If it is blocked, deliver the request plan and collection manifest with the exact failure; do not substitute search-engine snippets for full Reddit comments.
Resources
scripts/build_request_plan.py: create deterministic search request and manifest templates.scripts/normalize_reddit.py: parse saved search listings and nested comment trees into JSONL.references/chrome_same_origin.md: Chrome collection loop, checkpointing, and manifest contract.references/schema.md: normalized row and audit requirements.
