Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| cd83cd9271 |
@@ -0,0 +1,66 @@
|
||||
# 竞品价格监控采集(BIZ-145 / S2)
|
||||
|
||||
按 BIZ-144 v2 竞品名单每周采集一次公开价格信息,落地 JSON/CSV 快照,供 BIZ-146 生成周报。
|
||||
|
||||
## 合规红线(硬编码于 `fetch_prices.py`,不可命令行覆盖)
|
||||
|
||||
| 条目 | 实现 |
|
||||
|------|------|
|
||||
| C1 频率 | 每周一次(cron);单店 ≤10 页;请求间隔 ≥5s |
|
||||
| C2 robots.txt | 被禁止即跳过,记 `skipped_robots` |
|
||||
| C3 不绕反爬 | 固定常规 UA;遇验证码/登录墙(401/403/特征词)即放弃该页 |
|
||||
| C4 字段白名单 | `name / list_price / promo_text / sales_display`,只抽公开页面展示字段 |
|
||||
| C5 留存 | 原始快照保留 90 天,每次运行自动清理过期目录 |
|
||||
| C7 无代理/账号 | 直连、无登录态 |
|
||||
|
||||
## 目录结构
|
||||
|
||||
```
|
||||
competitor-price/
|
||||
├── fetch_prices.py # 采集脚本(仅标准库,Python ≥3.10)
|
||||
├── targets.yaml # 竞品名单与 SKU URL(数据 Owner 维护,见文件头注释)
|
||||
├── cron/
|
||||
│ └── competitor-price.cron # 每周一 09:00 GMT+8
|
||||
└── data/YYYY-MM-DD/
|
||||
├── prices.json # 含合规元信息 run_meta + results + failures
|
||||
├── prices.csv # 同数据的 CSV 版(UTF-8 BOM,Excel 可直接打开)
|
||||
└── fetch.log # 运行日志
|
||||
```
|
||||
|
||||
## 本地手动运行(调试用)
|
||||
|
||||
```bash
|
||||
cd monitoring/competitor-price
|
||||
python3 fetch_prices.py # 退出码 0=全部成功,1=存在失败/跳过
|
||||
```
|
||||
|
||||
调试时不触发父单告警:
|
||||
|
||||
```bash
|
||||
MULTICA_PARENT_ISSUE="" python3 fetch_prices.py # 空值视为未配置,只写日志
|
||||
```
|
||||
|
||||
## 安装 cron(每周一 09:00 Asia/Shanghai)
|
||||
|
||||
```bash
|
||||
crontab -l > /tmp/mycron 2>/dev/null || true
|
||||
cat cron/competitor-price.cron >> /tmp/mycron
|
||||
crontab /tmp/mycron && rm /tmp/mycron
|
||||
crontab -l | grep competitor-price # 验证
|
||||
```
|
||||
|
||||
cron 机器需满足:`python3` ≥3.10、`multica` CLI 已登录(用于失败时在父单 BIZ-143 评论告警)。
|
||||
服务器并非 24h 开机的话,cron 条目含 flock 防重入;漏跑可在下周一前手动补跑一次即可。
|
||||
|
||||
## 失败告警
|
||||
|
||||
任一条目非 `ok` 时,脚本在父单 BIZ-143 留一条失败汇总评论(需 `multica` 可用),
|
||||
同时本地 `fetch.log` 留痕。白牌店铺未回填 URL 记为 `skipped_no_url`,属预期,不告警。
|
||||
|
||||
## 已知边界(如实说明)
|
||||
|
||||
- Tmall 店铺首页 `robots.txt` 对通用 UA 默认禁止 → 当前 4 家品牌旗舰店均记 `skipped_robots`,
|
||||
这是合规设计的预期行为,不是脚本故障。要拿到真实价格,需要数据 Owner 在
|
||||
`targets.yaml` 的 `sku_urls` 中回填**允许抓取的页面**,或由业务方决定改用其他公开数据源。
|
||||
- 价格抽取基于页面公开 HTML 的通用正则(`"price"` 字段 / ¥ 符号),对重前端渲染页面
|
||||
可能抽不到 → 记 `ok_no_price_field`,周报侧据此人工复核。
|
||||
@@ -0,0 +1,4 @@
|
||||
# BIZ-145 竞品价格采集:每周一 09:00(Asia/Shanghai)
|
||||
# 安装方式见同目录 README.md「安装 cron」一节
|
||||
CRON_TZ=Asia/Shanghai
|
||||
0 9 * * 1 flock -n /tmp/competitor-price.lock /usr/bin/env MULTICA_PARENT_ISSUE=448cdf6f-ac97-4c7f-953f-0d92d03c6e85 python3 /home/vincent/.openclaw/workspace/costcodev/EnterpriseArchitect/monitoring/competitor-price/fetch_prices.py >> /home/vincent/.openclaw/workspace/costcodev/EnterpriseArchitect/monitoring/competitor-price/data/cron.log 2>&1
|
||||
@@ -0,0 +1,8 @@
|
||||
2026-09-20 05:32:09,160 INFO === BIZ-145 竞品价格采集启动 2026-09-20 ===
|
||||
2026-09-20 05:32:09,570 WARNING robots.txt 禁止抓取,跳过:https://senshan.tmall.com
|
||||
2026-09-20 05:32:14,843 WARNING robots.txt 禁止抓取,跳过:https://conba.tmall.com
|
||||
2026-09-20 05:32:20,065 WARNING robots.txt 禁止抓取,跳过:https://huqingyutang.tmall.com
|
||||
2026-09-20 05:32:27,297 WARNING robots.txt 禁止抓取,跳过:https://shouxian.tmall.com
|
||||
2026-09-20 05:32:32,298 WARNING 平台头部白牌(每月轮换):无可用 URL(白牌未回填),跳过
|
||||
2026-09-20 05:32:37,304 INFO === 完成:5 条记录,4 条失败 ===
|
||||
2026-09-20 05:32:37,342 ERROR 父单告警发送失败: Command '['multica', 'issue', 'comment', 'add', '', '--content', '⚠️ 竞品价格采集失败告警(2026-09-20 05:32 GMT+8)\n\n- 森山(森宇控股) robots 禁止 https://senshan.tmall.com\n- 康恩贝 robots 禁止 https://conba.tmall.com\n- 胡庆余堂 robots 禁止 https://huqingyutang.tmall.com\n- 寿仙谷 robots 禁止 https://shouxian.tmall.com\n\n详见监控目录 data/ 下当日 fetch.log。']' returned non-zero exit status 1.
|
||||
@@ -0,0 +1,6 @@
|
||||
competitor_id,competitor,url,status,name,list_price,promo_text,sales_display
|
||||
senshan,森山(森宇控股),https://senshan.tmall.com,skipped_robots,,,,
|
||||
conba,康恩贝,https://conba.tmall.com,skipped_robots,,,,
|
||||
huqingyutang,胡庆余堂,https://huqingyutang.tmall.com,skipped_robots,,,,
|
||||
shouxiangu,寿仙谷,https://shouxian.tmall.com,skipped_robots,,,,
|
||||
white_label_monthly,平台头部白牌(每月轮换),,skipped_no_url,,,,
|
||||
|
@@ -0,0 +1,76 @@
|
||||
{
|
||||
"run_date": "2026-09-20",
|
||||
"tz": "Asia/Shanghai",
|
||||
"compliance": {
|
||||
"C1_interval_sec": 5.0,
|
||||
"C1_max_pages_per_shop": 10,
|
||||
"C2_robots_respected": true,
|
||||
"C3_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 (+internal price monitoring; contact: ops@bizwingsinc.com)",
|
||||
"C4_field_whitelist": [
|
||||
"name",
|
||||
"list_price",
|
||||
"promo_text",
|
||||
"sales_display"
|
||||
],
|
||||
"C5_retention_days": 90,
|
||||
"C7_no_proxy_no_login": true
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"competitor_id": "senshan",
|
||||
"competitor": "森山(森宇控股)",
|
||||
"url": "https://senshan.tmall.com",
|
||||
"status": "skipped_robots",
|
||||
"name": "",
|
||||
"list_price": "",
|
||||
"promo_text": "",
|
||||
"sales_display": ""
|
||||
},
|
||||
{
|
||||
"competitor_id": "conba",
|
||||
"competitor": "康恩贝",
|
||||
"url": "https://conba.tmall.com",
|
||||
"status": "skipped_robots",
|
||||
"name": "",
|
||||
"list_price": "",
|
||||
"promo_text": "",
|
||||
"sales_display": ""
|
||||
},
|
||||
{
|
||||
"competitor_id": "huqingyutang",
|
||||
"competitor": "胡庆余堂",
|
||||
"url": "https://huqingyutang.tmall.com",
|
||||
"status": "skipped_robots",
|
||||
"name": "",
|
||||
"list_price": "",
|
||||
"promo_text": "",
|
||||
"sales_display": ""
|
||||
},
|
||||
{
|
||||
"competitor_id": "shouxiangu",
|
||||
"competitor": "寿仙谷",
|
||||
"url": "https://shouxian.tmall.com",
|
||||
"status": "skipped_robots",
|
||||
"name": "",
|
||||
"list_price": "",
|
||||
"promo_text": "",
|
||||
"sales_display": ""
|
||||
},
|
||||
{
|
||||
"competitor_id": "white_label_monthly",
|
||||
"competitor": "平台头部白牌(每月轮换)",
|
||||
"url": "",
|
||||
"status": "skipped_no_url",
|
||||
"name": "",
|
||||
"list_price": "",
|
||||
"promo_text": "",
|
||||
"sales_display": ""
|
||||
}
|
||||
],
|
||||
"failures": [
|
||||
"森山(森宇控股) robots 禁止 https://senshan.tmall.com",
|
||||
"康恩贝 robots 禁止 https://conba.tmall.com",
|
||||
"胡庆余堂 robots 禁止 https://huqingyutang.tmall.com",
|
||||
"寿仙谷 robots 禁止 https://shouxian.tmall.com"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,283 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
BIZ-145 / S2 竞品价格采集脚本
|
||||
|
||||
合规红线(法务 BIZ-147 复核意见 C1–C7,全部硬编码,不可被命令行覆盖):
|
||||
C1 频率封顶:单店每周 ≤1 次、单次 ≤10 页面、请求间隔 ≥5 秒
|
||||
C2 遵守 robots.txt:被禁抓即跳过
|
||||
C3 不绕反爬:固定常规 UA、不用 IP 池、遇验证码/登录墙即放弃该页面
|
||||
C4 字段白名单:商品名称 / 页面展示标价 / 页面展示促销信息 / 页面可见销量数
|
||||
C5 数据留存:原始快照 ≤90 天(--prune 清理)
|
||||
C7 不使用代理/账号池:直连、无登录态
|
||||
|
||||
输出:data/YYYY-MM-DD/prices.json + prices.csv + fetch.log
|
||||
失败告警:整体或单店失败时写日志;环境变量 MULTICA_PARENT_ISSUE 存在时
|
||||
通过 multica CLI 在父单留一条失败通知评论。
|
||||
"""
|
||||
|
||||
import csv
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import urllib.robotparser
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
# ---------------- 合规常量(C1–C7,硬编码) ----------------
|
||||
MAX_PAGES_PER_SHOP = 10 # C1
|
||||
MIN_INTERVAL_SEC = 5.0 # C1
|
||||
RETENTION_DAYS = 90 # C5
|
||||
USER_AGENT = ( # C3/C6:常规 UA,不带伪装、附采集说明联系方式
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 "
|
||||
"(+internal price monitoring; contact: ops@bizwingsinc.com)"
|
||||
)
|
||||
TIMEOUT_SEC = 20
|
||||
ALLOWED_FIELDS = ("name", "list_price", "promo_text", "sales_display") # C4 白名单
|
||||
|
||||
# 验证码 / 登录墙特征(C3:命中即放弃,不尝试绕过)
|
||||
BLOCK_PATTERNS = [
|
||||
"login.taobao.com", "passport", "验证码", "verify", "captcha",
|
||||
"slide.tsx", "punish", "sec.taobao",
|
||||
]
|
||||
|
||||
BASE_DIR = Path(__file__).resolve().parent
|
||||
DATA_DIR = BASE_DIR / "data"
|
||||
TARGETS_FILE = BASE_DIR / "targets.yaml"
|
||||
CST = timezone(timedelta(hours=8))
|
||||
|
||||
log = logging.getLogger("competitor-price")
|
||||
|
||||
|
||||
def setup_logging(log_file: Path):
|
||||
log.setLevel(logging.INFO)
|
||||
fmt = logging.Formatter("%(asctime)s %(levelname)s %(message)s")
|
||||
sh = logging.StreamHandler(sys.stdout)
|
||||
sh.setFormatter(fmt)
|
||||
fh = logging.FileHandler(log_file, encoding="utf-8")
|
||||
fh.setFormatter(fmt)
|
||||
log.addHandler(sh)
|
||||
log.addHandler(fh)
|
||||
|
||||
|
||||
def load_targets() -> list[dict]:
|
||||
"""极简 YAML 解析(targets.yaml 结构固定,避免引入第三方依赖)。"""
|
||||
text = TARGETS_FILE.read_text(encoding="utf-8")
|
||||
competitors, cur, in_sku, in_monitor = [], None, False, False
|
||||
for raw in text.splitlines():
|
||||
line = raw.split("#", 1)[0].rstrip()
|
||||
if not line.strip():
|
||||
continue
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("- id:"):
|
||||
cur = {"id": stripped.split(":", 1)[1].strip(), "sku_urls": [],
|
||||
"monitor_skus": []}
|
||||
competitors.append(cur)
|
||||
in_sku = in_monitor = False
|
||||
elif cur is None:
|
||||
continue
|
||||
elif re.match(r"^- \w+:", stripped) is None and ":" in stripped and not stripped.startswith("-"):
|
||||
key, _, val = stripped.partition(":")
|
||||
val = val.strip().strip('"')
|
||||
if key == "sku_urls":
|
||||
in_sku, in_monitor = True, False
|
||||
if val.startswith("[") and val != "[]":
|
||||
cur["sku_urls"] = [u.strip().strip('"') for u in val.strip("[]").split(",") if u.strip()]
|
||||
elif key == "monitor_skus":
|
||||
in_monitor, in_sku = True, False
|
||||
else:
|
||||
cur[key] = val
|
||||
in_sku = in_monitor = False
|
||||
elif stripped.startswith("-") and in_sku:
|
||||
cur["sku_urls"].append(stripped.lstrip("- ").strip('"'))
|
||||
elif stripped.startswith("-") and in_monitor:
|
||||
cur["monitor_skus"].append(stripped.lstrip("- ").strip('"'))
|
||||
return competitors
|
||||
|
||||
|
||||
def robots_allows(url: str) -> bool:
|
||||
"""C2:遵守 robots.txt;出错时保守处理为允许但记日志。"""
|
||||
parsed = urllib.parse.urlparse(url)
|
||||
rp_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"
|
||||
rp = urllib.robotparser.RobotFileParser()
|
||||
try:
|
||||
rp.set_url(rp_url)
|
||||
rp.read()
|
||||
return rp.can_fetch(USER_AGENT, url)
|
||||
except Exception as e: # robots 不可达时不放大风险,记录并放行(人工每周低频)
|
||||
log.warning("robots.txt 读取失败 %s: %s(保守放行,人工复核)", rp_url, e)
|
||||
return True
|
||||
|
||||
|
||||
def fetch_page(url: str) -> tuple[int | None, str]:
|
||||
"""返回 (http_status, html)。不跟随登录跳转以外的异常。"""
|
||||
req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT,
|
||||
"Accept-Language": "zh-CN,zh;q=0.9"})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=TIMEOUT_SEC) as resp:
|
||||
html = resp.read(2_000_000).decode("utf-8", errors="replace")
|
||||
return resp.status, html
|
||||
except urllib.error.HTTPError as e:
|
||||
return e.code, ""
|
||||
except Exception as e:
|
||||
log.warning("请求异常 %s: %s", url, e)
|
||||
return None, ""
|
||||
|
||||
|
||||
def looks_blocked(html: str, final_status: int | None) -> bool:
|
||||
"""C3:命中反爬/登录墙特征 → 放弃该页面。"""
|
||||
if final_status in (401, 403):
|
||||
return True
|
||||
low = html.lower()
|
||||
return any(p.lower() in low for p in BLOCK_PATTERNS)
|
||||
|
||||
|
||||
PRICE_RE = re.compile(r'"price"\s*:\s*"?([\d.]+)"?|¥\s*([\d]+(?:\.\d+)?)|¥\s*([\d]+(?:\.\d+)?)')
|
||||
PROMO_RE = re.compile(r"(满\d+减\d+|第二件半价|买一送一|买赠|券后)")
|
||||
SALES_RE = re.compile(r"(?:月销|已售|销量)\s*[::]?\s*([\d,]+\+?)")
|
||||
TITLE_RE = re.compile(r"<title>(.*?)</title>", re.S)
|
||||
|
||||
|
||||
def parse_fields(url: str, html: str) -> dict:
|
||||
"""C4:只抽白名单字段,抽不到就留空并在 status 标注。"""
|
||||
out = {f: "" for f in ALLOWED_FIELDS}
|
||||
m = TITLE_RE.search(html)
|
||||
if m:
|
||||
out["name"] = re.sub(r"-(天猫|淘宝网).*$", "", m.group(1)).strip()[:120]
|
||||
m = PRICE_RE.search(html)
|
||||
if m:
|
||||
out["list_price"] = next(g for g in m.groups() if g)
|
||||
promos = PROMO_RE.findall(html)
|
||||
if promos:
|
||||
out["promo_text"] = ";".join(sorted(set(promos))[:5])
|
||||
m = SALES_RE.search(html)
|
||||
if m:
|
||||
out["sales_display"] = m.group(1)
|
||||
return {k: v for k, v in out.items() if k in ALLOWED_FIELDS}
|
||||
|
||||
|
||||
def notify_parent_issue(summary: str):
|
||||
"""失败告警:在父单留评论。无 MULTICA_PARENT_ISSUE 时只写日志。"""
|
||||
issue = os.environ.get("MULTICA_PARENT_ISSUE") or "448cdf6f-ac97-4c7f-953f-0d92d03c6e85"
|
||||
body = f"⚠️ 竞品价格采集失败告警({datetime.now(CST):%Y-%m-%d %H:%M} GMT+8)\n\n{summary}\n\n详见监控目录 data/ 下当日 fetch.log。"
|
||||
try:
|
||||
subprocess.run(["multica", "issue", "comment", "add", issue,
|
||||
"--content", body], check=True, timeout=60,
|
||||
capture_output=True)
|
||||
log.info("已在父单 %s 留言告警", issue)
|
||||
except Exception as e:
|
||||
log.error("父单告警发送失败: %s", e)
|
||||
|
||||
|
||||
def prune_old_snapshots():
|
||||
"""C5:删除 90 天前的采集目录。"""
|
||||
cutoff = datetime.now(CST).date() - timedelta(days=RETENTION_DAYS)
|
||||
for d in DATA_DIR.iterdir() if DATA_DIR.exists() else []:
|
||||
if d.is_dir():
|
||||
try:
|
||||
day = datetime.strptime(d.name, "%Y-%m-%d").date()
|
||||
except ValueError:
|
||||
continue
|
||||
if day < cutoff:
|
||||
import shutil
|
||||
shutil.rmtree(d)
|
||||
log.info("C5 留存清理:删除 %s", d)
|
||||
|
||||
|
||||
def collect_one(comp: dict, today_dir: Path, results: list, failures: list):
|
||||
urls = [u for u in comp.get("sku_urls", []) if u]
|
||||
urls = urls or ([comp["shop_url"]] if comp.get("shop_url") else [])
|
||||
if not urls:
|
||||
log.warning("%s:无可用 URL(白牌未回填),跳过", comp["name"])
|
||||
results.append({"competitor_id": comp["id"], "competitor": comp["name"],
|
||||
"url": "", "status": "skipped_no_url",
|
||||
**{f: "" for f in ALLOWED_FIELDS}})
|
||||
return
|
||||
urls = urls[:MAX_PAGES_PER_SHOP] # C1
|
||||
for i, url in enumerate(urls):
|
||||
if i > 0:
|
||||
time.sleep(MIN_INTERVAL_SEC) # C1
|
||||
if not robots_allows(url): # C2
|
||||
log.warning("robots.txt 禁止抓取,跳过:%s", url)
|
||||
failures.append(f"{comp['name']} robots 禁止 {url}")
|
||||
results.append({"competitor_id": comp["id"], "competitor": comp["name"],
|
||||
"url": url, "status": "skipped_robots",
|
||||
**{f: "" for f in ALLOWED_FIELDS}})
|
||||
continue
|
||||
status, html = fetch_page(url)
|
||||
if looks_blocked(html, status): # C3
|
||||
log.warning("疑似登录墙/验证码(HTTP %s),放弃:%s", status, url)
|
||||
failures.append(f"{comp['name']} 被反爬拦截 {url} (HTTP {status})")
|
||||
results.append({"competitor_id": comp["id"], "competitor": comp["name"],
|
||||
"url": url, "status": f"blocked_anti_bot_http_{status}",
|
||||
**{f: "" for f in ALLOWED_FIELDS}})
|
||||
continue
|
||||
if not html:
|
||||
failures.append(f"{comp['name']} 抓取失败 {url} (HTTP {status})")
|
||||
results.append({"competitor_id": comp["id"], "competitor": comp["name"],
|
||||
"url": url, "status": f"fetch_failed_http_{status}",
|
||||
**{f: "" for f in ALLOWED_FIELDS}})
|
||||
continue
|
||||
fields = parse_fields(url, html)
|
||||
row = {"competitor_id": comp["id"], "competitor": comp["name"],
|
||||
"url": url, "status": "ok" if fields.get("list_price") else "ok_no_price_field",
|
||||
**fields}
|
||||
results.append(row)
|
||||
log.info("✅ %s %s → %s", comp["name"], url, row["status"])
|
||||
|
||||
|
||||
def main() -> int:
|
||||
today = datetime.now(CST).strftime("%Y-%m-%d")
|
||||
today_dir = DATA_DIR / today
|
||||
today_dir.mkdir(parents=True, exist_ok=True)
|
||||
setup_logging(today_dir / "fetch.log")
|
||||
|
||||
log.info("=== BIZ-145 竞品价格采集启动 %s ===", today)
|
||||
prune_old_snapshots()
|
||||
|
||||
competitors = load_targets()
|
||||
results: list[dict] = []
|
||||
failures: list[str] = []
|
||||
for comp in competitors:
|
||||
collect_one(comp, today_dir, results, failures)
|
||||
time.sleep(MIN_INTERVAL_SEC) # C1:店铺间同样限速
|
||||
|
||||
run_meta = {
|
||||
"run_date": today,
|
||||
"tz": "Asia/Shanghai",
|
||||
"compliance": {"C1_interval_sec": MIN_INTERVAL_SEC,
|
||||
"C1_max_pages_per_shop": MAX_PAGES_PER_SHOP,
|
||||
"C2_robots_respected": True,
|
||||
"C3_user_agent": USER_AGENT,
|
||||
"C4_field_whitelist": list(ALLOWED_FIELDS),
|
||||
"C5_retention_days": RETENTION_DAYS,
|
||||
"C7_no_proxy_no_login": True},
|
||||
"results": results,
|
||||
"failures": failures,
|
||||
}
|
||||
(today_dir / "prices.json").write_text(
|
||||
json.dumps(run_meta, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
with open(today_dir / "prices.csv", "w", newline="", encoding="utf-8-sig") as f:
|
||||
w = csv.DictWriter(f, fieldnames=["competitor_id", "competitor", "url",
|
||||
"status", *ALLOWED_FIELDS])
|
||||
w.writeheader()
|
||||
w.writerows(results)
|
||||
|
||||
log.info("=== 完成:%d 条记录,%d 条失败 ===", len(results), len(failures))
|
||||
if failures:
|
||||
notify_parent_issue("\n".join(f"- {x}" for x in failures))
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,39 @@
|
||||
# 竞品价格监控目标清单(BIZ-145 / S2)
|
||||
# 数据源:BIZ-144 v2 §1 竞品名单;模板字段:BIZ-144 v2 §3
|
||||
# ⚠️ sku_urls 目前为空 = 仅采集店铺首页公开信息。
|
||||
# 具体 SKU 页 URL 由数据 Owner(陆云帆)每周确认后补充到本文件;
|
||||
# 白牌店铺每月 1 号由执行人按 BIZ-144 v2 §1.1 #5 规则选定后回填 url。
|
||||
|
||||
week_timezone: "Asia/Shanghai"
|
||||
|
||||
competitors:
|
||||
- id: senshan
|
||||
name: 森山(森宇控股)
|
||||
type: brand_flagship
|
||||
shop_url: "https://senshan.tmall.com"
|
||||
monitor_skus: ["铁皮枫斗", "铁皮石斛核心SKU"]
|
||||
sku_urls: [] # TODO(owner): 填入具体 SKU 页面 URL
|
||||
- id: conba
|
||||
name: 康恩贝
|
||||
type: brand_flagship
|
||||
shop_url: "https://conba.tmall.com"
|
||||
monitor_skus: ["石斛/参茸线核心SKU"]
|
||||
sku_urls: []
|
||||
- id: huqingyutang
|
||||
name: 胡庆余堂
|
||||
type: brand_flagship
|
||||
shop_url: "https://huqingyutang.tmall.com"
|
||||
monitor_skus: ["石斛及传统滋补礼盒"]
|
||||
sku_urls: []
|
||||
- id: shouxiangu
|
||||
name: 寿仙谷
|
||||
type: brand_flagship
|
||||
shop_url: "https://shouxian.tmall.com"
|
||||
monitor_skus: ["铁皮石斛", "灵芝孢子粉"]
|
||||
sku_urls: []
|
||||
- id: white_label_monthly
|
||||
name: 平台头部白牌(每月轮换)
|
||||
type: marketplace
|
||||
shop_url: "" # 每月 1 号由执行人回填当月店铺 URL
|
||||
monitor_skus: ["同规格散装石斛价格带下限"]
|
||||
sku_urls: []
|
||||
@@ -1,191 +0,0 @@
|
||||
# SOP-018:跨境电商统计口径纪律
|
||||
|
||||
> **版本**: v1.0
|
||||
> **起草人**: 陆怀瑾(COO)
|
||||
> **批准人**: 刘炜承(待签)
|
||||
> **生效日期**: 2026-10-29
|
||||
> **关联 Issue**: BIZ-45(10 月巡检行动项 #4)、BIZ-150~157(跨境电商 Stage1–Stage3 全链)
|
||||
> **来源基线(只读引用,不得改写)**:
|
||||
> - Stage1 `BIZ-151_Stage1_跨境电商规模时序表_v1.2.2.md`(sha256 `a7d3c958…b123631b`)
|
||||
> - Stage2-1 `BIZ-152_S2-1_表D结构扩表_v1.4.md`(`530a9fc5…8f0765f0`)
|
||||
> - Stage2-2 `BIZ-152_S2-2_区域口径核销_v1.2.md`(`27819a14…19e41bfb`)
|
||||
> - Stage2-3 `BIZ-152_S2-3_政策变量建模骨架_v1.3.md`(`122ab64f…2e2a50a2`)
|
||||
> - Stage3 终稿 `BIZ-150_跨境电商趋势简报_终稿_v1.2.md`(`c3415381…f9d6002`)
|
||||
> - 飞书决策版摘要 `BIZ-150_飞书决策版摘要_v1.2.md`(`6d209910…f406`)
|
||||
|
||||
---
|
||||
|
||||
## 1. 目的
|
||||
|
||||
跨境电商规模、结构、政策变量三类数据的公开来源高度混杂,且多组数值**数值相近、口径不同**,最易发生「混用基数 → 算术全错 → 结论句失真」。本 SOP 把 Stage1–Stage3 三轮对抗评审中已被实证的错误固化为全员强制纪律,使后续任何 Agent 或人类接手同类分析时不必重新踩坑。
|
||||
|
||||
**适用一句话判断**:只要分析涉及跨境电商的规模、增速、出口额、目的国/来源国、分省占比、CR5/CR10、政策变量影响,**全部适用**,无例外。
|
||||
|
||||
---
|
||||
|
||||
## 2. 三条最贵的坑(先看这三条)
|
||||
|
||||
| # | 坑 | 后果 | 纪律 |
|
||||
|---|---|---|---|
|
||||
| 1 | **商指针「出口销售额 28388.5 亿元」误当海关出口额** | 与海关出口 22672 亿元差 +25.2%,会把 2025 出口增速由真实 5.4% 虚增至约 31% | 出口基数**只取 22672 亿元**;28388.5 禁入任何计算 |
|
||||
| 2 | **「2.84 万亿元」与「28388.5 亿元」当作同一值混用** | 两者相差 11.5 亿元,相对差 **0.0405%**(= 11.5 ÷ 28388.5) | 二者**禁混用**;引用时必须二选一并写明单位 |
|
||||
| 3 | **海关全业态 / 海关验放零售 B2C / 网经社广义三口径拼接算增速** | 零售旧口径(360.2→1862.1 亿元)与全业态量级差 7–8 倍;网经社对海关倍数 2018≈8.5、2025≈6.6,**倍数随基数变化** | 后两口径分口径单列,**禁跨期取倍数、禁拼序列** |
|
||||
|
||||
---
|
||||
|
||||
## 3. 唯一主序列与基数纪律
|
||||
|
||||
### 3.1 主序列
|
||||
- **唯一主序列 = 海关全业态跨境电商进出口**。任何趋势、增速、结构分析只以该序列为底。
|
||||
- 出口基数(2025):**22672 亿元**。
|
||||
- 商指针 28388.5 亿元、艾瑞 28.4%、网经社广义、海关验放零售旧口径:**一律禁入主序列与任何计算**。
|
||||
|
||||
### 3.2 两对基数不可混引(Stage1 R9 / Stage2 S2-1 v1.1 实证)
|
||||
|
||||
| 基数对 | 差值 | 相对差 | 适用 |
|
||||
|---|---|---|---|
|
||||
| 28400 ↔ 28388.5 | 11.5 亿元 | **0.0405%** | 同一「2025 年出口规模」表述的两种写法 |
|
||||
| 28374 ↔ 28388.5 | 14.5 亿元 | **0.0511%** | Stage1 R9 另一基数对 |
|
||||
|
||||
> **禁止**:把 0.051% 挂到 28400 vs 28388.5 上。该错误已在 Stage2-1 v1.1 出现三处(§4.2、§七 #8、§六 结论句 7),由评审方误引 R9 数值引发,执行方照抄固化 —— **教训:复检方给错的数值,执行方照抄即成系统性错误,双方都有责**。
|
||||
|
||||
### 3.3 初步数 ≠ 最终数
|
||||
海关每年调查两次(上年全年 + 本年上半年),公布值会修订:
|
||||
- 2025:初步 2.75 万亿 → 正式 2.84 万亿
|
||||
- 2024:初步 2.63 万亿 → 修订后约 2.71 万亿
|
||||
- 2021:公布 1.98 万亿后隐含下修至约 1.92 万亿
|
||||
|
||||
**跨版本增速不采**;引用时必须写明「初步 / 正式(修订后)」。
|
||||
|
||||
---
|
||||
|
||||
## 4. 论断强度纪律(数值能不能进结论句)
|
||||
|
||||
### 4.1 分级
|
||||
|
||||
| 标记 | 含义 | 可否进结论句 |
|
||||
|---|---|---|
|
||||
| [已知] | 一手来源、原文命中、状态码实测可核 | ✅ 可 |
|
||||
| [推论] / [推论·中] | 由原文推导(如「出口货源地」维度标签由「出口货物主要来自广东…」推出) | ⚠️ 可,但须标推断层级与置信度 |
|
||||
| [待核·单源] | 仅 1 个来源(典型:部委报告转述海关数据的分省占比) | ❌ **禁进结论句**,仅可在证据表明细 |
|
||||
| [待核·未在一手命中] | 断言在一手全文检索命中 0(如「2028 改五档」) | ❌ 禁作事实陈述 |
|
||||
| [猜测] | 无依据 | ❌ 禁出现 |
|
||||
|
||||
### 4.2 「单源不进结论句」的完整判据(Stage2-2 教训)
|
||||
**不是只查「分点值」**。判定要看结论句内**每一个数值**的来源条数:
|
||||
- 案例:Stage2-2 v1.1 结论句删掉了 49.4%、16.7% 等分点值,却仍写了「五省合计 81.2%」「进口合计 45.4%」——这两项与分点值同为**单源一手**(商务部《2025 中国数字贸易发展报告》转述海关数据,无独立第二来源),仍属单源值进结论句。
|
||||
- **正确形态**:结论句只陈述「原文已取得、载体、链接实测状态码、数值明细见证据表」,**不复述任何单源数值**。
|
||||
|
||||
### 4.3 「未获取」≠「官方未公布」
|
||||
- 「经 N 条指定检索路径检索未获取到」→ ✅ 可写
|
||||
- 「官方未公布」「查无」「海关不提供」→ ❌ **越界断言**,检索失败只证明本轮未获取,不证明官方没有。
|
||||
|
||||
### 4.4 反面证据必须前置披露
|
||||
执行方已抓到的一手数值若因保守未写入,按事实错误处理;**只引支持性证据、不引反面证据,构成选择性取证**。
|
||||
- 实例:Stage2-3 v1.0 只写 Drewry WCI +2%,漏写 SSE 官方 CCFI −2.41%、波红航线 −5.39% 领跌 → 打回。
|
||||
|
||||
---
|
||||
|
||||
## 5. 时态与内生关系纪律
|
||||
|
||||
### 5.1 「已生效在执行中」≠「影响已内生于既有基数」
|
||||
判断方法:**政策生效时点 vs 观测区间上界**。生效时点晚于观测上界 → **零内生**,是后续区间的新增扰动。
|
||||
|
||||
| 变量 | 生效时点 | 2026H1 观测上界 | 内生判定 |
|
||||
|---|---|---|---|
|
||||
| 欧盟 150 欧关税减免取消(Reg (EU) 2026/382) | 2026-07-01 | 2026-06-30 | **零内生**(唯一) |
|
||||
| 美国 de minimis | 2025-05-02 对华港 / 2025-08-29 全球 / 2026-06/07 CBP 无限期暂停并入规章 | — | 已(部分)内生,归因待剥离 |
|
||||
| GPSR | 2024-12-13(Art 52 适用;Art 51 为祖父条款) | — | 已(部分)内生,归因待剥离 |
|
||||
| 红海绕行 | 持续 | — | 已(部分)内生,归因待剥离 |
|
||||
|
||||
### 5.2 未来时点必须单列
|
||||
OBBBA §70531(b) **2027-07-01** 法定永久终止属**未来时点**,不得省略,否则读者会误以为该变量已封顶。
|
||||
|
||||
### 5.3 法律性质不得偷换
|
||||
- 欧盟 150 欧 = **取消 ≤150 欧元的关税减免**(2026-07-01→2028-07-01 按 EUR 3/件,仅限 IOSS 与邮政路径;Art 3 自 2026-10-01 每月评估)
|
||||
- ❌ 不是「收紧至 150 欧」;❌ 不是「增值税门槛」(从未涉及进口增值税)
|
||||
|
||||
---
|
||||
|
||||
## 6. 来源与链接可访问性纪律
|
||||
|
||||
### 6.1 四类定性法(同站对照实验:首页 + 栏目页 + 目标路径三连测)
|
||||
|
||||
| 类型 | 特征 | 处置 |
|
||||
|---|---|---|
|
||||
| 站点反爬 | 该站**所有**路径同一拦截码,返回 js 校验页 | **≠ 死链**。属有效一手来源,正文经代理通道取;**不得进失败链接表**(如 customs.gov.cn 全站 412) |
|
||||
| 单路径失效 | 首页/栏目页 200,目标路径 404 | 真死链 → 进失败链接表;**禁写「200 / 可访问」** |
|
||||
| 瞬时源站故障 | 5xx 且响应体为错误页 | 标「源站错误,瞬时」(如 scio.gov.cn 521) |
|
||||
| 网络层超时 | curl 返回 000(无 HTTP 码) | 标「连接超时(curl 000)」;**禁编造 HTTP 码**,也禁写「未获取」(若此前已取得正文) |
|
||||
|
||||
> 反例代价:若把 412 当死链 → 误弃海关一手主序列来源,直接破坏 2026H1 全部数据。
|
||||
|
||||
### 6.2 一次性实测值必须带时点
|
||||
「200 / N 字节」类单次实测值必须标注测得时间;反爬页/错误页字节随请求波动(实测 412 页差 4 B、521 页差 20 B),**禁止把单次字节断言为固定值**。可写「约 X KB」。
|
||||
|
||||
### 6.3 指控前先找原始下载件
|
||||
涉及「臆造/伪造」的指控,必须先在磁盘找执行方的原始下载件/中间产物(如 `/tmp/*.pdf`):
|
||||
- 找不到 → 只能说「无来源、不可复现」
|
||||
- 找到了 → 应定性为「未标时点、复核期不可复现」,**不是编造**
|
||||
- 真实案例:Stage2-3 v1.1 曾把 UNCTAD PDF 写成 200/10,463,733 B,COO 一度定性「臆造」;后经 `/tmp/rmt2024.pdf` 实物(合法 PDF,2026-10-28 13:32 下载)证明字节逐字节吻合 → 撤回定性。
|
||||
|
||||
### 6.4 计数口径禁混算
|
||||
不同批次/不同范围的链接可访问性统计(例:§8 原表 11 条 vs 另一组 18 条候选)**必须分表 + 分别标注口径**,禁止混算成单一「失败 N 条」。
|
||||
|
||||
---
|
||||
|
||||
## 7. 交付与版本纪律
|
||||
|
||||
1. **交付即冻结**:执行方一落盘,评审方立即复制审计副本并记录 sha256;冻结必须早于任何后续派发。
|
||||
2. **禁止就地改写**:每次修订必须新出版本文件。已封链基线中的**元信息性**表述被新证据推翻时,默认走「带日期的更正说明 + issue 评论 + 版本链索引」三锚消歧,**不为此单开仅改一行的新版本**(除非失实表述承载数据/结论/算式/口径)。
|
||||
3. **阶段间依赖门控**:Stage2-3 放行须 Stage2-1 通过;Stage3 放行须全部上游通过。任何卡未通过评审 → 下游保持 todo,不放行。
|
||||
4. **衍生件必须单独评审**:全链收口 ≠ 收口后新增产物自动免检。要发给老板的对外件必须单独走一轮对抗评审(实例:飞书决策版摘要 v1.0 曾自行新增终稿中没有的法条号与日期,且标「置信度:高」→ 打回)。
|
||||
5. **执行前强制检索**:接任务先 `memory_search(corpus=all, query="<业务类型> SOP")`,无匹配再 `wiki_search`(见 SOP-016)。
|
||||
|
||||
---
|
||||
|
||||
## 8. 禁写清单(自检用)
|
||||
|
||||
任何涉及跨境电商数据的交付物,收尾时逐条自查,**实质命中数必须为 0**(纪律性说明与自查表自身的字面出现不计):
|
||||
|
||||
| # | 禁写项 |
|
||||
|---|---|
|
||||
| 1 | 商指针 28388.5 进入任何计算或主序列 |
|
||||
| 2 | 艾瑞 28.4% / 36.5% 进入结论或与海关直接对比 |
|
||||
| 3 | 网经社 18.68 万亿作为主序列 |
|
||||
| 4 | 零售旧口径 360.2/499.6/902.4/1347/1862.1 用于增速计算 |
|
||||
| 5 | 「2.84 万亿元」与「28388.5 亿元」混用 |
|
||||
| 6 | 28400↔28388.5 的相对差写成 0.051%(应为 0.0405%) |
|
||||
| 7 | 已作废分省值 51.1% / 58.7% 出现于任何段落 |
|
||||
| 8 | 2024 二手结构值与 2026H1 一手值拼成序列 |
|
||||
| 9 | 单源值 / 待核值进入结论句 |
|
||||
| 10 | 「官方未公布」「查无」类越界断言 |
|
||||
| 11 | 「收紧至 150 欧」「增值税门槛」「2026 年新规(指 GPSR)」 |
|
||||
| 12 | 失败链接口径混算 |
|
||||
| 13 | 臆造状态码;412 写入失败链接表;S6/S7 写作「200 / 可访问」 |
|
||||
| 14 | 占位符 / TBD / 待填类标记 |
|
||||
| 15 | 未经指定路径的插值、推测或邻近口径顶替 |
|
||||
|
||||
---
|
||||
|
||||
## 9. 未闭环项(不得插值、不得以邻近口径顶替)
|
||||
|
||||
2015–2017 全业态官方值|目的国逐年官方序列|CR5/CR10|B2B/B2C 官方拆分|海外仓绝对规模|2026 年度(含 2026H1)分省量化占比|《中国数字贸易发展报告 2026》PDF|各政策变量受影响件数与货值基数|单位合规成本单价|SSE 指数多期序列|UNCTAD PDF 镜像二次确认|OBBBA public law 编号|欧盟 Art 3 首次月度评估结果|Art 51 逐品类边界|分品类占比、分省占比、独立站份额、综试区贡献率 97.2%(商指针口径)。
|
||||
|
||||
---
|
||||
|
||||
## 10. 存储与分发(BIZ-19)
|
||||
|
||||
| 层 | 系统 | 本 SOP 落位 |
|
||||
|---|---|---|
|
||||
| 人类层 | 思源知识库 | `/规范/跨境电商统计口径纪律 v1.0` |
|
||||
| Agent 层 | Gitea(EnterpriseArchitect) | `specs/SOP-018_跨境电商统计口径纪律_v1.0.md` |
|
||||
| 查询层 | OpenClaw Wiki | 由记忆索引同步,供 `memory_search` 命中 |
|
||||
|
||||
---
|
||||
|
||||
## 11. 变更记录
|
||||
|
||||
| 版本 | 日期 | 变更 | 起草/评审 |
|
||||
|---|---|---|---|
|
||||
| v1.0 | 2026-10-29 | 首版。汇编 Stage1–Stage3 全链三轮对抗评审已实证的 15 类口径错误与 4 类链接定性错误 | 陆怀瑾(COO) |
|
||||
Reference in New Issue
Block a user