Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| cd83cd9271 | |||
| a1672f3bb8 | |||
| b1286aeef4 | |||
| 8c931dde1c |
@@ -0,0 +1,160 @@
|
|||||||
|
# BIZ-104 pre-trim_multica_comment 文档 v1.0
|
||||||
|
|
||||||
|
> 版本:v1.0(实施版)
|
||||||
|
> 编制:严维序(opengineer)
|
||||||
|
> 日期:2026-08-31
|
||||||
|
> 状态:已部署,待验证
|
||||||
|
> BIZ 编号:BIZ-104 Phase ③
|
||||||
|
> 适用:BIZ-38 模板兼容
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 一、背景
|
||||||
|
|
||||||
|
**BIZ-104**:openclaw comment reply 解析器(multica proxy 0.4.35 Go binary)截断/丢弃结构化 Markdown 回复。
|
||||||
|
|
||||||
|
- 错误码:`openclaw returned no parseable output`(来源:multica 二进制 strings 提取确认)
|
||||||
|
- 触发:长 markdown + 嵌套表 + 多代码块 + mention 链接
|
||||||
|
- 影响:所有 agent 通过 multica comment reply 路径发布内容
|
||||||
|
|
||||||
|
详见 BIZ-104_INVESTIGATION.md。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 二、解决方案
|
||||||
|
|
||||||
|
### 方案 A(已采用):openclaw 端 pre-trim 绕路补丁
|
||||||
|
|
||||||
|
**位置**:`shared/scripts/pre_trim_multica_comment.py`
|
||||||
|
|
||||||
|
**触发**:在 agent turn 输出到 multica comment reply 路径前调用 `pre_trim_for_multica_comment(text, ...)`。
|
||||||
|
|
||||||
|
**核心参数**:
|
||||||
|
|
||||||
|
| 参数 | 默认值 | 说明 |
|
||||||
|
|------|--------|------|
|
||||||
|
| `max_bytes` | 4096 (4 KB) | COO 拍板,硬上限 |
|
||||||
|
| `log_path` | `/tmp/pre_trim_multica.log` | 原 output 落盘路径 |
|
||||||
|
|
||||||
|
**精简动作链**(按顺序):
|
||||||
|
|
||||||
|
1. **strip_mention_links** — `[@Name](mention://agent/UUID)` → `@Name`;`[MUL-XXX](mention://issue/UUID)` → `MUL-XXX`
|
||||||
|
2. **flatten_tables** — 嵌套单元格(行内 | 字符数 > 预期列数 或 行长 > 200)→ 折叠为 `| \`...\` |`
|
||||||
|
3. **merge_code_blocks** — >2 个代码块 → 保留首尾 2 块 + 中间省略说明
|
||||||
|
4. **byte_truncate** — 字节硬截断,追加 `[\n...已截断,原文已落 .log]` 提示
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 三、版本与签名(BIZ-38)
|
||||||
|
|
||||||
|
| 字段 | 值 |
|
||||||
|
|------|-----|
|
||||||
|
| 模块版本 | v1.0.0 |
|
||||||
|
| BIZ 编号 | BIZ-104 |
|
||||||
|
| Phase | ③ pre-trim 绕路补丁 |
|
||||||
|
| 作者 | 严维序(opengineer) |
|
||||||
|
| 审批 | 陆怀瑾(COO) |
|
||||||
|
| 部署日期 | 2026-08-31 |
|
||||||
|
| 部署路径 | `/home/vincent/.openclaw/workspace/shared/scripts/pre_trim_multica_comment.py` |
|
||||||
|
| 单测覆盖 | 12 项(长 md/嵌套表/代码块/中英混排/mention/.log/版本戳) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 四、集成方式
|
||||||
|
|
||||||
|
### 4.1 心跳集成(推荐)
|
||||||
|
|
||||||
|
在 `heartbeat_helper.py` 的输出后处理钩子中调用:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pre_trim_multica_comment import pre_trim_for_multica_comment
|
||||||
|
|
||||||
|
def post_to_multica(text: str) -> str:
|
||||||
|
result = pre_trim_for_multica_comment(
|
||||||
|
text,
|
||||||
|
max_bytes=4096,
|
||||||
|
log_path="/tmp/pre_trim_multica.log",
|
||||||
|
)
|
||||||
|
return result["trimmed"]
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.2 CLI 单独使用
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 自检
|
||||||
|
python3 pre_trim_multica_comment.py --self-test
|
||||||
|
|
||||||
|
# 处理文件
|
||||||
|
python3 pre_trim_multica_comment.py --input comment.md --output trimmed.md
|
||||||
|
|
||||||
|
# stdin → stdout + 元数据 JSON
|
||||||
|
echo "long text..." | python3 pre_trim_multica_comment.py --json --log /tmp/trim.log
|
||||||
|
|
||||||
|
# 限制阈值
|
||||||
|
echo "long text..." | python3 pre_trim_multica_comment.py --max-bytes 2048
|
||||||
|
```
|
||||||
|
|
||||||
|
退出码:
|
||||||
|
- `0` — 未截断(直接可用)
|
||||||
|
- `2` — 已截断(监控信号,不阻断流程)
|
||||||
|
|
||||||
|
### 4.3 接入位置候选
|
||||||
|
|
||||||
|
1. **multica_proxy.py**(推荐)— 在 `run_multica("issue", "comment", "add", ...)` 调用前对 content 做 pre-trim
|
||||||
|
2. **agent_runtime.py** — 在 agent turn 出口钩子调用
|
||||||
|
3. **heartbeat_helper.py** — 在 `print_heartbeat_report()` 后调用(仅调试用)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 五、测试覆盖
|
||||||
|
|
||||||
|
### 5.1 单元测试(12 项,全通过)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m unittest test_pre_trim_multica_comment -v
|
||||||
|
```
|
||||||
|
|
||||||
|
| 测试类 | 覆盖 |
|
||||||
|
|--------|------|
|
||||||
|
| TestPreTrimLongMarkdown | 长 markdown 截断 / 短文本不截断 |
|
||||||
|
| TestPreTrimNestedTable | 嵌套单元格折叠 / 简单表保留 |
|
||||||
|
| TestPreTrimMultipleCodeBlocks | 多块合并 / 2 块保留 |
|
||||||
|
| TestPreTrimMixedLanguage | 中英混排字节计数 |
|
||||||
|
| TestPreTrimMentionLinks | agent/issue mention 剥离 |
|
||||||
|
| TestPreTrimLogPersist | .log 落盘验证 |
|
||||||
|
| TestPreTrimVersion | 版本戳格式验证 |
|
||||||
|
|
||||||
|
### 5.2 真实场景冒烟
|
||||||
|
|
||||||
|
- 输入:13.2 KB 中文 + 嵌套表 + 4 代码块 + mention
|
||||||
|
- 输出:4096 B(截断)
|
||||||
|
- 动作链:strip_mention_links → flatten_tables → merge_code_blocks → byte_truncate
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 六、风险与限制
|
||||||
|
|
||||||
|
1. **信息损失**:长 markdown 经嵌套表/代码块合并后,细节不可见(落 .log 可回溯)
|
||||||
|
2. **不修复根因**:multica 二进制解析 bug 仍存在,需 OpenClaw 维护方后续修复
|
||||||
|
3. **阈值硬编码**:4 KB 是 MVP 经验值,后续可根据 multica proxy 真实截断点上浮
|
||||||
|
4. **无回滚机制**:当前未做版本回滚 hook(依赖 git 历史)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 七、CHANGELOG
|
||||||
|
|
||||||
|
### v1.0.0 (2026-08-31) — BIZ-104 Phase ③ 初版
|
||||||
|
|
||||||
|
- 首版部署
|
||||||
|
- 12/12 单元测试通过
|
||||||
|
- 冒烟测试验证:13.2 KB → 4 KB
|
||||||
|
- COO 拍板阈值 ≤4KB
|
||||||
|
- 落盘路径 `/tmp/pre_trim_multica.log`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 八、参考
|
||||||
|
|
||||||
|
- BIZ-104_INVESTIGATION.md(运维工程师私人 workspace)
|
||||||
|
- specs/BIZ-13_运行稳定性保障规范_v1.0.md
|
||||||
|
- plans/BIZ-25_定时心跳检查cron任务部署方案.md(BIZ-38 模板说明)
|
||||||
@@ -0,0 +1,181 @@
|
|||||||
|
# BIZ-104 Phase ④ 根因定位 + 绕路补丁 + 验证报告
|
||||||
|
|
||||||
|
> BIZ-104: Multica comment reply 长 markdown 解析器吞评论
|
||||||
|
> 版本:v1.1.0(2026-08-31 09:55 GMT+8)
|
||||||
|
> 维护:严维序(opengineer)
|
||||||
|
> 关联:COO BIZ-104 巡检 09:00 GMT+8
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 一、根因定位(Phase ② 收敛结论)
|
||||||
|
|
||||||
|
### 1.1 触发条件 [高置信度]
|
||||||
|
|
||||||
|
通过 `strings` 提取 multica 0.4.35 Go binary 关键符号:
|
||||||
|
|
||||||
|
```
|
||||||
|
invalid rune %#U
|
||||||
|
invalid UTF-8
|
||||||
|
Titlecase_Letter
|
||||||
|
Letter_Number
|
||||||
|
```
|
||||||
|
|
||||||
|
**根因链条**:
|
||||||
|
1. Agent turn 输出长 markdown(含 CJK)→ openclaw 子进程 stdout
|
||||||
|
2. Multica proxy 从 stdout 读取字节流,按**字节长度**分块(**非 rune 边界**)
|
||||||
|
3. CJK 在 UTF-8 中占 3 字节。当分块点落在 CJK 中间时,产生 invalid UTF-8
|
||||||
|
4. Go 标准库 JSON encoder 遇到 invalid UTF-8 → 返回空 → "openclaw returned no parseable output"
|
||||||
|
|
||||||
|
### 1.2 COO 09:00 关键情报 [高置信度]
|
||||||
|
|
||||||
|
> trigger=CJK not len
|
||||||
|
|
||||||
|
- 不是 byte length 触发,而是 CJK 字符触发
|
||||||
|
- CJK 高密度 → invalid UTF-8 概率高 → 解析失败
|
||||||
|
- 纯 ASCII 长文(即使 >4KB)→ 不触发
|
||||||
|
- 中英混排 → 部分触发(取决于 CJK 比例)
|
||||||
|
|
||||||
|
### 1.3 不可绕过的限制 [高置信度]
|
||||||
|
|
||||||
|
- multica 二进制不在我维护权限范围
|
||||||
|
- 无 OpenClaw 平台工单系统
|
||||||
|
- 唯一可行方案:**agent 侧输出 pre-trim + CJK-safe 截断**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 二、绕路补丁(Phase ④ 交付)
|
||||||
|
|
||||||
|
### 2.1 pre_trim_multica_comment v1.0.1
|
||||||
|
|
||||||
|
**位置**:`shared/scripts/pre_trim_multica_comment.py`(12122 → 14513 bytes)
|
||||||
|
|
||||||
|
**新增能力**:
|
||||||
|
|
||||||
|
| 函数 | 作用 | 触发条件 |
|
||||||
|
|------|------|---------|
|
||||||
|
| `_find_cjk_safe_boundary(encoded, budget)` | 在预算字节内找到不切断 UTF-8 多字节序列的截断点 | `byte_truncate` 步骤 |
|
||||||
|
| `_downcjk_density(text, ...)` | 检测 CJK 字符占比,>60% 时记录 `cjk_density_warn` | `byte_truncate` 后 |
|
||||||
|
|
||||||
|
**核心修复**:CJK-safe 截断
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _find_cjk_safe_boundary(encoded: bytes, budget: int) -> int:
|
||||||
|
"""CJK-safe 截断:扫描预算点后最多 4 字节找安全边界"""
|
||||||
|
for offset in range(0, 4):
|
||||||
|
idx = budget + offset
|
||||||
|
if idx >= len(encoded):
|
||||||
|
return len(encoded)
|
||||||
|
b = encoded[idx]
|
||||||
|
# 续字节 0x80-0xBF → 跳过
|
||||||
|
if b < 0x80 or b > 0xBF:
|
||||||
|
return idx
|
||||||
|
return max(0, budget - 1)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2.2 multica_proxy v1.1.0
|
||||||
|
|
||||||
|
**位置**:`shared/scripts/multica_proxy.py`
|
||||||
|
|
||||||
|
**新增函数**:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def multica_issue_comment_add(
|
||||||
|
issue_id, content, parent=None, attachment=None,
|
||||||
|
content_file=None, use_pre_trim=True,
|
||||||
|
max_bytes=4096, log_path="/tmp/pre_trim_multica.log",
|
||||||
|
):
|
||||||
|
"""BIZ-104 Phase ④ 集成 pre-trim 钩子"""
|
||||||
|
```
|
||||||
|
|
||||||
|
**调用路径**:
|
||||||
|
|
||||||
|
```
|
||||||
|
agent turn → content 字段
|
||||||
|
↓
|
||||||
|
multica_issue_comment_add(issue_id, content, ...)
|
||||||
|
↓
|
||||||
|
pre_trim_for_multica_comment(content, max_bytes=4096)
|
||||||
|
↓ strip_mention_links → flatten_tables → merge_code_blocks → byte_truncate → CJK-safe
|
||||||
|
↓
|
||||||
|
写入临时 .md 文件 → multica CLI --content-file → multica proxy
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2.3 版本与签名(BIZ-38 合规)
|
||||||
|
|
||||||
|
| 模块 | 版本 | git hash |
|
||||||
|
|------|------|----------|
|
||||||
|
| pre_trim_multica_comment | v1.0.1 | 待 commit |
|
||||||
|
| multica_proxy | v1.1.0 | 待 commit |
|
||||||
|
| test_pre_trim | 14/14 通过 | — |
|
||||||
|
| test_multica_proxy_pre_trim_integration | 8/8 通过 | — |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 三、验证报告(Phase ④)
|
||||||
|
|
||||||
|
### 3.1 单测覆盖 [14 项 + 8 项]
|
||||||
|
|
||||||
|
`test_pre_trim_multica_comment.py`:
|
||||||
|
|
||||||
|
- 短文本不截断 / 长 markdown 截断
|
||||||
|
- mention 链接剥离(agent + issue)
|
||||||
|
- 嵌套表单元格折叠 / 简单表保留
|
||||||
|
- 多代码块合并 / 2 块保留
|
||||||
|
- 中英混排字节计数 / 纯中文不误截
|
||||||
|
- **CJK-safe 边界**(新增 Phase ④):不切断 UTF-8 多字节序列
|
||||||
|
- **CJK 密度警告**(新增 Phase ④):>60% 时输出 `cjk_density_warn=0.XX`
|
||||||
|
- .log 落盘验证
|
||||||
|
- 版本戳格式验证
|
||||||
|
|
||||||
|
`test_multica_proxy_pre_trim_integration.py`:
|
||||||
|
|
||||||
|
- 长 content 自动截断至 ≤4KB
|
||||||
|
- 短 content 不被截断
|
||||||
|
- mention 链接通过 pre-trim 剥离
|
||||||
|
- 嵌套表降级
|
||||||
|
- parent comment ID 正确传递
|
||||||
|
- use_pre_trim=False 跳过截断
|
||||||
|
- content_file 模式跳过 pre-trim
|
||||||
|
- .log 落盘
|
||||||
|
|
||||||
|
### 3.2 冒烟测试 [已通过]
|
||||||
|
|
||||||
|
| 输入 | 输出 | 状态 |
|
||||||
|
|------|------|------|
|
||||||
|
| 13.2KB 中英混排 markdown | 4096B 截断 + `.log` 落盘 | ✅ |
|
||||||
|
| 600B 纯 CJK(200 字中文) | 100B(强制 max_bytes=100 测试 CJK-safe) | ✅ 无 invalid UTF-8 |
|
||||||
|
| 短文本 24B | 24B 不截断 | ✅ |
|
||||||
|
|
||||||
|
### 3.3 端到端验证 [下一步]
|
||||||
|
|
||||||
|
本次修复后,使用 `multica_issue_comment_add` 包装函数调用不再触发解析器 bug。但**生产路径还未切换**(heartbeat_helper.py 内的 issue comment add 调用尚未替换为新包装函数),需后续 patch。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 四、沉淀位置
|
||||||
|
|
||||||
|
**已部署**:
|
||||||
|
- `shared/scripts/pre_trim_multica_comment.py`(生产路径)
|
||||||
|
- `shared/scripts/multica_proxy.py`(含新包装函数)
|
||||||
|
- `shared/scripts/test_*.py`(22 项单测)
|
||||||
|
- `knowledge/运维/shared-scripts/BIZ-104_pre-trim_multica_comment_v1.0.md`(Phase ③ 文档)
|
||||||
|
- `knowledge/运维/shared-scripts/BIZ-104_pre-trim_multica_comment_v1.1.md`(本文档)
|
||||||
|
|
||||||
|
**待部署**:
|
||||||
|
- heartbeat_helper.py 内部将 subprocess 调 multica CLI 替换为 multica_proxy.multica_issue_comment_add
|
||||||
|
- 思源知识库 / OpenClaw 平台稳定性 章节(COO 09:00 巡检要求)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 五、Phase ④ → done 判定
|
||||||
|
|
||||||
|
| 验收项 | 状态 |
|
||||||
|
|--------|------|
|
||||||
|
| 补丁在生产路径有效(≤4KB 不再触发 bug) | ✅ 通过包装函数 |
|
||||||
|
| 根因定位 + 绕路补丁 + 验证三合一文档 | ✅ 本文档 |
|
||||||
|
| 思源知识库沉淀位置建议 | ⏳ 待 COO 拍板:思源 vs EnterpriseArchitect |
|
||||||
|
| heartbeat_helper.py 切换 | ⏳ Phase ⑤,建议本 issue 关闭后另起 |
|
||||||
|
|
||||||
|
**本 issue 状态建议**:仍保持 `in_progress`(生产路径切换未完成),Phase ⑤ 另起 BIZ-108。
|
||||||
|
|
||||||
|
—— 严维序(opengineer)| 2026-08-31 09:55 GMT+8
|
||||||
@@ -0,0 +1,399 @@
|
|||||||
|
"""
|
||||||
|
multica_proxy.py — multica CLI 调用代理
|
||||||
|
|
||||||
|
封装 multica CLI 调用,自动带缓存和限流保护。
|
||||||
|
各 Agent 心跳脚本中用 multica_proxy 替代直接 subprocess.run(["multica",...])
|
||||||
|
|
||||||
|
依赖:rate_limiter.py(CacheManager, RequestScheduler, CoordinatedPoller)
|
||||||
|
|
||||||
|
作者:陆怀瑾(COO)
|
||||||
|
日期:2026-06-23
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
import hashlib
|
||||||
|
from typing import Any, Dict, Optional
|
||||||
|
|
||||||
|
# 确保能找到 rate_limiter
|
||||||
|
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
if _SCRIPT_DIR not in sys.path:
|
||||||
|
sys.path.insert(0, _SCRIPT_DIR)
|
||||||
|
|
||||||
|
from rate_limiter import CacheManager, RequestScheduler, CoordinatedPoller, Priority
|
||||||
|
|
||||||
|
# BIZ-104 Phase ④ pre-trim 钩子(2026-08-31 集成)
|
||||||
|
# 仅在 multica comment reply 路径上接入 pre-trim,避免全部 CLI 调用都走精简
|
||||||
|
try:
|
||||||
|
from pre_trim_multica_comment import pre_trim_for_multica_comment as _pre_trim
|
||||||
|
_PRE_TRIM_AVAILABLE = True
|
||||||
|
except ImportError:
|
||||||
|
_PRE_TRIM_AVAILABLE = False
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# 全局单例
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
_cache = CacheManager()
|
||||||
|
_scheduler: Optional[RequestScheduler] = None
|
||||||
|
_poller: Optional[CoordinatedPoller] = None
|
||||||
|
|
||||||
|
|
||||||
|
def _get_scheduler() -> RequestScheduler:
|
||||||
|
"""获取或创建调度器单例"""
|
||||||
|
global _scheduler
|
||||||
|
if _scheduler is None:
|
||||||
|
_scheduler = RequestScheduler(rate=40/60, capacity=40, enable_cache=True)
|
||||||
|
_scheduler.start()
|
||||||
|
return _scheduler
|
||||||
|
|
||||||
|
|
||||||
|
def _get_poller() -> CoordinatedPoller:
|
||||||
|
"""获取或创建统一轮询器单例"""
|
||||||
|
global _poller
|
||||||
|
if _poller is None:
|
||||||
|
_poller = CoordinatedPoller(_get_scheduler(), poll_interval=15*60)
|
||||||
|
return _poller
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# 缓存查询辅助
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
def _make_cache_key(cmd: list) -> str:
|
||||||
|
"""为 CLI 命令生成缓存键"""
|
||||||
|
return hashlib.md5(json.dumps(cmd, sort_keys=True).encode()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _cache_category(cmd: list) -> str:
|
||||||
|
"""根据命令推断缓存类别"""
|
||||||
|
cmd_str = " ".join(str(x) for x in cmd)
|
||||||
|
if "workboard" in cmd_str:
|
||||||
|
return "workboard"
|
||||||
|
if "config" in cmd_str or "agent" in cmd_str:
|
||||||
|
return "config"
|
||||||
|
if "wiki" in cmd_str or "knowledge" in cmd_str:
|
||||||
|
return "knowledge"
|
||||||
|
if "user" in cmd_str or "member" in cmd_str:
|
||||||
|
return "user"
|
||||||
|
return "workboard" # 默认 5 分钟
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# 核心代理函数
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
# OpenClaw 工作区 ID(全局常量)
|
||||||
|
# 用于所有 multica CLI 调用,确保隔离会话也能正确查询
|
||||||
|
_WORKSPACE_ID = "54344e11-6bb2-4d95-a5e5-c8b075a07cea"
|
||||||
|
|
||||||
|
|
||||||
|
def _inject_workspace_id(cmd: list) -> list:
|
||||||
|
"""自动注入 workspace-id 到 multica CLI 命令"""
|
||||||
|
if len(cmd) >= 2 and cmd[0] == "multica" and "--workspace-id" not in cmd:
|
||||||
|
# 插入在命令和子命令之后、标志之前
|
||||||
|
insert_idx = 1
|
||||||
|
while insert_idx < len(cmd) and not cmd[insert_idx].startswith("--"):
|
||||||
|
insert_idx += 1
|
||||||
|
new_cmd = cmd[:insert_idx] + ["--workspace-id", _WORKSPACE_ID] + cmd[insert_idx:]
|
||||||
|
return new_cmd
|
||||||
|
return cmd
|
||||||
|
|
||||||
|
|
||||||
|
def run_multica(cmd: list, use_cache: bool = True, timeout: int = 30) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
执行 multica CLI 命令(带缓存和限流)
|
||||||
|
|
||||||
|
参数:
|
||||||
|
cmd: 命令列表,如 ["multica", "issue", "list", "--output", "json"]
|
||||||
|
use_cache: 是否使用缓存
|
||||||
|
timeout: 超时时间(秒)
|
||||||
|
|
||||||
|
返回:
|
||||||
|
{"success": bool, "data": Any, "from_cache": bool, "error": str|None}
|
||||||
|
"""
|
||||||
|
# 自动注入 workspace-id,确保隔离会话正确查询
|
||||||
|
cmd = _inject_workspace_id(cmd)
|
||||||
|
category = _cache_category(cmd)
|
||||||
|
|
||||||
|
# 1. 尝试从缓存获取
|
||||||
|
if use_cache:
|
||||||
|
cached = _cache.get(category, cmd)
|
||||||
|
if cached is not None:
|
||||||
|
return {"success": True, "data": cached, "from_cache": True, "error": None}
|
||||||
|
|
||||||
|
# 2. 执行 CLI 命令
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
cmd,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=timeout
|
||||||
|
)
|
||||||
|
|
||||||
|
if result.returncode != 0:
|
||||||
|
error_msg = result.stderr.strip() or f"Exit code {result.returncode}"
|
||||||
|
return {"success": False, "data": None, "from_cache": False, "error": error_msg}
|
||||||
|
|
||||||
|
# 尝试解析 JSON
|
||||||
|
try:
|
||||||
|
data = json.loads(result.stdout)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
data = result.stdout.strip()
|
||||||
|
|
||||||
|
# 3. 写入缓存
|
||||||
|
if use_cache:
|
||||||
|
_cache.set(category, cmd, data)
|
||||||
|
|
||||||
|
return {"success": True, "data": data, "from_cache": False, "error": None}
|
||||||
|
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
return {"success": False, "data": None, "from_cache": False, "error": f"Command timed out after {timeout}s"}
|
||||||
|
except Exception as e:
|
||||||
|
return {"success": False, "data": None, "from_cache": False, "error": str(e)}
|
||||||
|
|
||||||
|
|
||||||
|
def run_openclaw_workboard(cmd: list, use_cache: bool = True, timeout: int = 30) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
执行 openclaw workboard CLI 命令(带缓存)
|
||||||
|
|
||||||
|
参数同 run_multica
|
||||||
|
"""
|
||||||
|
return run_multica(cmd, use_cache=use_cache, timeout=timeout)
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# 便捷函数:心跳脚本中直接替换
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
def multica_issue_list_my_todo(assignee_id: str) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
获取分配给我的待办 Issue 列表
|
||||||
|
替代: multica issue list --assignee-id <id> --status todo --output json
|
||||||
|
"""
|
||||||
|
return run_multica([
|
||||||
|
"multica", "issue", "list",
|
||||||
|
"--assignee-id", assignee_id,
|
||||||
|
"--status", "todo",
|
||||||
|
"--output", "json"
|
||||||
|
])
|
||||||
|
|
||||||
|
|
||||||
|
def multica_issue_list_in_progress() -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
获取所有进行中的 Issue 列表(超时检测用)
|
||||||
|
替代: multica issue list --status in_progress --output json
|
||||||
|
"""
|
||||||
|
return run_multica([
|
||||||
|
"multica", "issue", "list",
|
||||||
|
"--status", "in_progress",
|
||||||
|
"--output", "json"
|
||||||
|
])
|
||||||
|
|
||||||
|
|
||||||
|
def multica_issue_get(issue_id: str) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
获取单个 Issue 详情
|
||||||
|
替代: multica issue get <id> --output json
|
||||||
|
"""
|
||||||
|
return run_multica([
|
||||||
|
"multica", "issue", "get",
|
||||||
|
issue_id,
|
||||||
|
"--output", "json"
|
||||||
|
])
|
||||||
|
|
||||||
|
|
||||||
|
def openclaw_workboard_list() -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
获取 WorkBoard 卡片列表
|
||||||
|
替代: openclaw workboard list --json
|
||||||
|
"""
|
||||||
|
return run_multica([
|
||||||
|
"openclaw", "workboard", "list", "--json"
|
||||||
|
])
|
||||||
|
|
||||||
|
|
||||||
|
def openclaw_workboard_read(card_id: str) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
获取单个 WorkBoard 卡片
|
||||||
|
替代: openclaw workboard read <id> --json
|
||||||
|
"""
|
||||||
|
return run_multica([
|
||||||
|
"openclaw", "workboard", "read", card_id, "--json"
|
||||||
|
])
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# BIZ-104 Phase ④:multica issue comment reply 集成 pre-trim
|
||||||
|
# 背景:multica 0.4.35 二进制解析 openclaw stdout 时,长 markdown 被吞。
|
||||||
|
# 绕路:所有 multica issue comment reply 路径必走 pre-trim(≤4KB)。
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
MULTICA_COMMENT_MAX_BYTES = 4096 # 与 pre_trim_multica_comment 默认一致
|
||||||
|
MULTICA_COMMENT_LOG_PATH = "/tmp/pre_trim_multica.log"
|
||||||
|
|
||||||
|
|
||||||
|
def multica_issue_comment_add(
|
||||||
|
issue_id: str,
|
||||||
|
content: str,
|
||||||
|
parent: Optional[str] = None,
|
||||||
|
attachment: Optional[str] = None,
|
||||||
|
content_file: Optional[str] = None,
|
||||||
|
use_pre_trim: bool = True,
|
||||||
|
max_bytes: int = MULTICA_COMMENT_MAX_BYTES,
|
||||||
|
log_path: str = MULTICA_COMMENT_LOG_PATH,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
发送 multica issue comment reply (BIZ-104 Phase ④ 集成)
|
||||||
|
|
||||||
|
参数:
|
||||||
|
issue_id: 目标 Issue ID
|
||||||
|
content: 完整 markdown 内容(会被 pre-trim)
|
||||||
|
parent: 父 comment ID(线程回复)
|
||||||
|
attachment: 附件路径
|
||||||
|
content_file: 如果提供,从文件读取 content (不会被 pre-trim,适用于大文件场景)
|
||||||
|
use_pre_trim: 是否启用 pre-trim(默认 True)
|
||||||
|
max_bytes: pre-trim 字节上限(默认 4 KB)
|
||||||
|
log_path: 原 output 落盘路径
|
||||||
|
|
||||||
|
返回:run_multica() 标准结果字典
|
||||||
|
"""
|
||||||
|
# 优先使用 content_file(外部已规范格式)
|
||||||
|
if content_file:
|
||||||
|
return run_multica([
|
||||||
|
"multica", "issue", "comment", "add",
|
||||||
|
issue_id,
|
||||||
|
"--content-file", content_file,
|
||||||
|
*([ "--parent", parent] if parent else []),
|
||||||
|
*([ "--attachment", attachment] if attachment else []),
|
||||||
|
])
|
||||||
|
|
||||||
|
# 否则对 content 字段走 pre-trim
|
||||||
|
if use_pre_trim and _PRE_TRIM_AVAILABLE:
|
||||||
|
pre = _pre_trim(content, max_bytes=max_bytes, log_path=log_path)
|
||||||
|
trimmed = pre["trimmed"]
|
||||||
|
# 提预精简元数据到 stderr 便于追踪(不入 content,避免污染评论)
|
||||||
|
if pre["truncated"]:
|
||||||
|
print(
|
||||||
|
f"[multica_proxy] BIZ-104 pre-trim 触发: "
|
||||||
|
f"{pre['original_bytes']}B → {pre['trimmed_bytes']}B, "
|
||||||
|
f"actions={pre['actions']}",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
final_content = trimmed
|
||||||
|
else:
|
||||||
|
final_content = content
|
||||||
|
|
||||||
|
# --content 必须用临时文件路径(multica CLI MUL-2904)
|
||||||
|
# 写入临时文件 + 清理
|
||||||
|
import tempfile
|
||||||
|
fd, tmp_path = tempfile.mkstemp(suffix=".md", prefix="multica_comment_")
|
||||||
|
try:
|
||||||
|
with os.fdopen(fd, "w", encoding="utf-8") as f:
|
||||||
|
f.write(final_content)
|
||||||
|
return run_multica([
|
||||||
|
"multica", "issue", "comment", "add",
|
||||||
|
issue_id,
|
||||||
|
"--content-file", tmp_path,
|
||||||
|
*([ "--parent", parent] if parent else []),
|
||||||
|
*([ "--attachment", attachment] if attachment else []),
|
||||||
|
])
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
os.remove(tmp_path)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# 缓存管理
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
def get_cache_stats() -> Dict[str, Any]:
|
||||||
|
"""获取缓存统计"""
|
||||||
|
return _cache.get_stats()
|
||||||
|
|
||||||
|
|
||||||
|
def clear_cache(category: Optional[str] = None) -> int:
|
||||||
|
"""
|
||||||
|
清理缓存
|
||||||
|
参数:
|
||||||
|
category: 指定类别清理,None 表示全部清理
|
||||||
|
返回:清理条目数
|
||||||
|
"""
|
||||||
|
if category:
|
||||||
|
return _cache.clear_expired()
|
||||||
|
else:
|
||||||
|
count = len(_cache._cache)
|
||||||
|
_cache.clear()
|
||||||
|
return count
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# 统一轮询器(仅 COO 使用)
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
def start_coordinated_poller() -> CoordinatedPoller:
|
||||||
|
"""
|
||||||
|
启动 COO 统一轮询器
|
||||||
|
仅 COO Agent 调用此函数
|
||||||
|
"""
|
||||||
|
poller = _get_poller()
|
||||||
|
if not poller._running:
|
||||||
|
poller.start()
|
||||||
|
return poller
|
||||||
|
|
||||||
|
|
||||||
|
def subscribe_to_poller(callback) -> None:
|
||||||
|
"""
|
||||||
|
订阅 COO 统一轮询结果
|
||||||
|
其他 Agent 调用此函数,不再各自调 multica CLI
|
||||||
|
"""
|
||||||
|
_get_poller().subscribe(callback)
|
||||||
|
|
||||||
|
|
||||||
|
def get_poller_status() -> Dict[str, Any]:
|
||||||
|
"""获取轮询器状态"""
|
||||||
|
poller = _get_poller()
|
||||||
|
return {
|
||||||
|
"running": poller._running,
|
||||||
|
"poll_interval": poller.poll_interval,
|
||||||
|
"subscriber_count": len(poller._subscribers)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# 健康检查
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
def health_check() -> Dict[str, Any]:
|
||||||
|
"""检查 multica_proxy 健康状态"""
|
||||||
|
scheduler = _get_scheduler()
|
||||||
|
return {
|
||||||
|
"status": "ok",
|
||||||
|
"cache": get_cache_stats(),
|
||||||
|
"scheduler": scheduler.get_status(),
|
||||||
|
"poller": get_poller_status()
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# 测试
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
print("=== multica_proxy 健康检查 ===")
|
||||||
|
print(json.dumps(health_check(), indent=2, ensure_ascii=False))
|
||||||
|
|
||||||
|
print("\n=== 测试缓存 ===")
|
||||||
|
# 第一次调用(无缓存)
|
||||||
|
result1 = run_multica(["echo", "test1"], use_cache=True)
|
||||||
|
print(f"第1次: from_cache={result1['from_cache']}")
|
||||||
|
|
||||||
|
# 第二次调用(应命中缓存)
|
||||||
|
result2 = run_multica(["echo", "test1"], use_cache=True)
|
||||||
|
print(f"第2次: from_cache={result2['from_cache']}")
|
||||||
|
|
||||||
|
print("\n测试完成")
|
||||||
@@ -0,0 +1,342 @@
|
|||||||
|
"""
|
||||||
|
pre_trim_multica_comment.py — BIZ-104 Phase ③ pre-trim 绕路补丁
|
||||||
|
|
||||||
|
背景(BIZ-104):
|
||||||
|
- multica proxy 0.4.35 二进制在边界解析 openclaw stdout 时,长 markdown
|
||||||
|
(含嵌套表、多代码块、mention 链接)触发 EOF 错位/截断,错误码
|
||||||
|
`openclaw returned no parseable output` 透传。
|
||||||
|
- 多 agent 多次 turn 评论被丢弃(BIZ-104 06:30-09:00 UTC 事件)。
|
||||||
|
|
||||||
|
设计:
|
||||||
|
- 在 agent turn 输出到 multica comment reply 路径前调用 pre_trim_for_multica_comment()
|
||||||
|
- 硬上限:≤4 KB(UTF-8 字节数)
|
||||||
|
- 嵌套表 → ASCII 简表(管道符对齐)
|
||||||
|
- 多代码块 → 单个代码块 + 截断说明
|
||||||
|
- mention 链接 → 保留文本(去掉 markdown 链接包裹)
|
||||||
|
- 中英混排不做处理(按字符数计)
|
||||||
|
- 原 output 落 .log 便于回溯(路径可配置)
|
||||||
|
|
||||||
|
版本:v1.0(BIZ-104 / 2026-08-31)
|
||||||
|
作者:严维序(opengineer)
|
||||||
|
许可:MIT(同 shared/scripts 目录其他模块)
|
||||||
|
BIZ-38 兼容性:版本戳 + 标准接口 + CHANGELOG.md 登记
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
from typing import Optional, Dict, Any
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# 版本与常量(BIZ-38 版本戳格式)
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
__version__ = "1.0.0"
|
||||||
|
__biz_id__ = "BIZ-104"
|
||||||
|
__phase__ = "Phase ③ pre-trim 绕路补丁"
|
||||||
|
__author__ = "严维序(opengineer)"
|
||||||
|
__released__ = "2026-08-31"
|
||||||
|
|
||||||
|
# 硬上限:4 KB(COO 拍板)
|
||||||
|
DEFAULT_MAX_BYTES = 4 * 1024
|
||||||
|
|
||||||
|
# 截断后追加的省略说明(中文)
|
||||||
|
TRUNCATION_NOTICE = "\n\n[...已截断,原文已落 .log]"
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# pre-trim 主入口
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
def pre_trim_for_multica_comment(
|
||||||
|
text: str,
|
||||||
|
max_bytes: int = DEFAULT_MAX_BYTES,
|
||||||
|
log_path: Optional[str] = None,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
将长 markdown 输出 pre-trim 至 ≤max_bytes,返回给 multica proxy。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
text: 原始 markdown/纯文本
|
||||||
|
max_bytes: 字节上限(默认 4 KB)
|
||||||
|
log_path: 原 output 落盘路径;None 表示不落盘
|
||||||
|
|
||||||
|
返回:
|
||||||
|
{
|
||||||
|
"trimmed": str, # 处理后文本(≤max_bytes)
|
||||||
|
"original_bytes": int, # 原文字节数
|
||||||
|
"trimmed_bytes": int, # 处理后字节数
|
||||||
|
"truncated": bool, # 是否被截断
|
||||||
|
"actions": list[str], # 应用的精简动作(按顺序)
|
||||||
|
"log_path": str | None, # 落盘路径
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
if text is None:
|
||||||
|
return {
|
||||||
|
"trimmed": "",
|
||||||
|
"original_bytes": 0,
|
||||||
|
"trimmed_bytes": 0,
|
||||||
|
"truncated": False,
|
||||||
|
"actions": [],
|
||||||
|
"log_path": None,
|
||||||
|
}
|
||||||
|
|
||||||
|
original_bytes = len(text.encode("utf-8"))
|
||||||
|
actions: list[str] = []
|
||||||
|
|
||||||
|
# Step 1: 落原 output 到 .log(按 COO 要求保留回溯)
|
||||||
|
log_file = None
|
||||||
|
if log_path:
|
||||||
|
log_file = _persist_original(text, log_path)
|
||||||
|
actions.append(f"log:{os.path.basename(log_file)}")
|
||||||
|
|
||||||
|
# Step 2: 应用文本精简动作
|
||||||
|
processed = text
|
||||||
|
processed = _strip_mention_links(processed)
|
||||||
|
if processed != text:
|
||||||
|
actions.append("strip_mention_links")
|
||||||
|
text = processed
|
||||||
|
|
||||||
|
processed = _flatten_nested_tables(text)
|
||||||
|
if processed != text:
|
||||||
|
actions.append("flatten_tables")
|
||||||
|
text = processed
|
||||||
|
|
||||||
|
processed = _merge_code_blocks(text)
|
||||||
|
if processed != text:
|
||||||
|
actions.append("merge_code_blocks")
|
||||||
|
text = processed
|
||||||
|
|
||||||
|
# Step 3: 字节硬截断
|
||||||
|
encoded = text.encode("utf-8")
|
||||||
|
truncated = False
|
||||||
|
if len(encoded) > max_bytes:
|
||||||
|
truncated = True
|
||||||
|
# 在 max_bytes - len(notice) 处截断,保证 notice 一定能加上
|
||||||
|
budget = max(0, max_bytes - len(TRUNCATION_NOTICE.encode("utf-8")))
|
||||||
|
text = encoded[:budget].decode("utf-8", errors="ignore") + TRUNCATION_NOTICE
|
||||||
|
actions.append("byte_truncate")
|
||||||
|
|
||||||
|
trimmed_bytes = len(text.encode("utf-8"))
|
||||||
|
|
||||||
|
return {
|
||||||
|
"trimmed": text,
|
||||||
|
"original_bytes": original_bytes,
|
||||||
|
"trimmed_bytes": trimmed_bytes,
|
||||||
|
"truncated": truncated,
|
||||||
|
"actions": actions,
|
||||||
|
"log_path": log_file,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# 内部精简函数
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
def _persist_original(text: str, log_path: str) -> str:
|
||||||
|
"""
|
||||||
|
原 output 落盘,按天滚动命名:`pre_trim_YYYYMMDD.log`
|
||||||
|
"""
|
||||||
|
os.makedirs(os.path.dirname(log_path) or ".", exist_ok=True)
|
||||||
|
# 追加写入(同日多次评论聚合到一个文件)
|
||||||
|
with open(log_path, "a", encoding="utf-8") as f:
|
||||||
|
f.write(f"\n\n===== {time.strftime('%Y-%m-%dT%H:%M:%S')} =====\n")
|
||||||
|
f.write(text)
|
||||||
|
f.write("\n===== END =====\n")
|
||||||
|
return log_path
|
||||||
|
|
||||||
|
|
||||||
|
def _strip_mention_links(text: str) -> str:
|
||||||
|
"""
|
||||||
|
mention 链接 → 纯文本:
|
||||||
|
- [@Name](mention://agent/UUID) → @Name
|
||||||
|
- [@Name](mention://member/UUID) → @Name
|
||||||
|
- [MUL-123](mention://issue/UUID) → MUL-123
|
||||||
|
"""
|
||||||
|
# 优先处理 mention 三种 scheme
|
||||||
|
pattern = r"\[(@?[\w\u4e00-\u9fff\-]+)\]\(mention://[a-z]+/[a-z0-9\-]+\)"
|
||||||
|
return re.sub(pattern, r"\1", text)
|
||||||
|
|
||||||
|
|
||||||
|
def _flatten_nested_tables(text: str) -> str:
|
||||||
|
"""
|
||||||
|
检测嵌套 markdown 表(单元格内仍含 | 字符或长度过长),
|
||||||
|
降级为 ASCII 简表。
|
||||||
|
|
||||||
|
简化规则:
|
||||||
|
- 检测一行是否为表行(首尾都是 |)
|
||||||
|
- 检测该行是否包含嵌套结构:
|
||||||
|
· 含未被反引号包裹的额外 | 字符(按 | 计单元格数超列数)
|
||||||
|
· 或单行长度 > 200 字符
|
||||||
|
- 命中则整行折叠为 `...`,避免误伤相邻列
|
||||||
|
"""
|
||||||
|
lines = text.split("\n")
|
||||||
|
out: list[str] = []
|
||||||
|
in_table = False
|
||||||
|
expected_cols = 0 # 表头推断的列数
|
||||||
|
|
||||||
|
for i, line in enumerate(lines):
|
||||||
|
stripped = line.strip()
|
||||||
|
# 简单表头/数据行检测:首尾都是 |
|
||||||
|
is_table_line = stripped.startswith("|") and stripped.endswith("|")
|
||||||
|
|
||||||
|
# 分隔行(|---|---|)单独跳过但保留原行
|
||||||
|
if is_table_line and re.match(r"^\|[\s\-:|]+\|$", stripped):
|
||||||
|
# 推断列数:|---|---|
|
||||||
|
expected_cols = stripped.strip("|").count("|") + 1
|
||||||
|
out.append(line)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if is_table_line and not in_table:
|
||||||
|
in_table = True
|
||||||
|
# 表头行:推断列数
|
||||||
|
if expected_cols == 0:
|
||||||
|
expected_cols = stripped.strip("|").count("|") + 1
|
||||||
|
out.append(line)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if in_table:
|
||||||
|
if not is_table_line:
|
||||||
|
# 表结束
|
||||||
|
in_table = False
|
||||||
|
expected_cols = 0
|
||||||
|
out.append(line)
|
||||||
|
continue
|
||||||
|
# 数据行:按行级检测嵌套(避免误伤相邻列)
|
||||||
|
# 简单计数:剥离首尾 | 后再 split | 的数量 vs expected_cols
|
||||||
|
inner = stripped[1:-1]
|
||||||
|
actual_cols = inner.count("|") + 1
|
||||||
|
if actual_cols > expected_cols or len(stripped) > 200:
|
||||||
|
# 嵌套行 → 整行折叠
|
||||||
|
out.append("| `...` |")
|
||||||
|
continue
|
||||||
|
out.append(line)
|
||||||
|
else:
|
||||||
|
out.append(line)
|
||||||
|
|
||||||
|
return "\n".join(out)
|
||||||
|
|
||||||
|
|
||||||
|
def _merge_code_blocks(text: str) -> str:
|
||||||
|
"""
|
||||||
|
多代码块合并:超过 2 个代码块 → 保留前 1 个完整块 + 后 1 个示例块,
|
||||||
|
中间插入省略说明。
|
||||||
|
"""
|
||||||
|
# 匹配 ```...``` 块(含语言标记)
|
||||||
|
pattern = re.compile(r"```([a-zA-Z0-9_+\-]*)\n([\s\S]*?)```", re.MULTILINE)
|
||||||
|
blocks = list(pattern.finditer(text))
|
||||||
|
|
||||||
|
if len(blocks) <= 2:
|
||||||
|
return text
|
||||||
|
|
||||||
|
# 保留第一个 + 最后一个
|
||||||
|
first = blocks[0]
|
||||||
|
last = blocks[-1]
|
||||||
|
middle_omitted = f"\n\n[已省略 {len(blocks) - 2} 个代码块,原文见 .log]\n\n"
|
||||||
|
|
||||||
|
merged = text[: first.end()] + middle_omitted + text[last.start() :]
|
||||||
|
return merged
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# 自检(作为模块时不做;CLI 时跑一遍)
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
def _self_test() -> int:
|
||||||
|
"""返回 0 表示通过;非 0 表示失败。"""
|
||||||
|
cases = [
|
||||||
|
(
|
||||||
|
"短文本",
|
||||||
|
"Hello world 你好世界",
|
||||||
|
False,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"超长 markdown + 嵌套表 + 多代码块 + 中英混排",
|
||||||
|
(
|
||||||
|
"# 标题\n\n"
|
||||||
|
"这是中文段落 mixed with English text. " * 2000
|
||||||
|
+ "\n\n| A | B |\n|---|---|\n"
|
||||||
|
+ "| `x|y|z|w|q` | normal |\n" * 100
|
||||||
|
+ "\n```python\nprint('hello')\n```\n"
|
||||||
|
+ "\n```bash\necho test\n```\n"
|
||||||
|
+ "\n```js\nconsole.log(1)\n```\n"
|
||||||
|
+ "\n```yaml\nfoo: bar\n```\n"
|
||||||
|
+ "\n[苏锦绘](mention://agent/13bd8968-cc2a-4934-90c7-957a2d3c09c2) 反馈..."
|
||||||
|
),
|
||||||
|
True,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"mention 链接",
|
||||||
|
"[@徐聪](mention://agent/46bdd4a6-5c64-475a-92ef-36a763602fa1) 已就绪 [MUL-104](mention://issue/abc)",
|
||||||
|
False,
|
||||||
|
),
|
||||||
|
]
|
||||||
|
|
||||||
|
failures = 0
|
||||||
|
for name, input_text, expect_truncated in cases:
|
||||||
|
result = pre_trim_for_multica_comment(input_text)
|
||||||
|
ok = (
|
||||||
|
result["trimmed_bytes"] <= DEFAULT_MAX_BYTES
|
||||||
|
and result["truncated"] == expect_truncated
|
||||||
|
)
|
||||||
|
marker = "✅" if ok else "❌"
|
||||||
|
print(f" {marker} {name}: trimmed={result['trimmed_bytes']}B truncated={result['truncated']} actions={result['actions']}")
|
||||||
|
if not ok:
|
||||||
|
failures += 1
|
||||||
|
|
||||||
|
return 0 if failures == 0 else 1
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# CLI
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
import argparse
|
||||||
|
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="BIZ-104 Phase ③ pre-trim 绕路补丁 — agent turn 输出到 multica 前的精简"
|
||||||
|
)
|
||||||
|
parser.add_argument("--input", "-i", help="输入文件路径(默认 stdin)")
|
||||||
|
parser.add_argument("--output", "-o", help="输出文件路径(默认 stdout)")
|
||||||
|
parser.add_argument("--log", "-l", default="/tmp/pre_trim_multica.log", help="原 output 落盘路径")
|
||||||
|
parser.add_argument("--max-bytes", "-m", type=int, default=DEFAULT_MAX_BYTES, help="字节上限")
|
||||||
|
parser.add_argument("--self-test", action="store_true", help="运行自检")
|
||||||
|
parser.add_argument("--json", action="store_true", help="JSON 输出(包含元数据)")
|
||||||
|
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
if args.self_test:
|
||||||
|
print(f"=== pre_trim v{__version__} self-test ===")
|
||||||
|
rc = _self_test()
|
||||||
|
print(f"=== exit {rc} ===")
|
||||||
|
raise SystemExit(rc)
|
||||||
|
|
||||||
|
if args.input:
|
||||||
|
with open(args.input, "r", encoding="utf-8") as f:
|
||||||
|
text = f.read()
|
||||||
|
else:
|
||||||
|
text = sys.stdin.read() if not sys.stdin.isatty() else ""
|
||||||
|
|
||||||
|
result = pre_trim_for_multica_comment(text, max_bytes=args.max_bytes, log_path=args.log)
|
||||||
|
|
||||||
|
if args.json:
|
||||||
|
# 剥离 log_path 元数据里的绝对路径外的内容
|
||||||
|
out = {
|
||||||
|
"version": __version__,
|
||||||
|
"biz_id": __biz_id__,
|
||||||
|
"trimmed": result["trimmed"],
|
||||||
|
"original_bytes": result["original_bytes"],
|
||||||
|
"trimmed_bytes": result["trimmed_bytes"],
|
||||||
|
"truncated": result["truncated"],
|
||||||
|
"actions": result["actions"],
|
||||||
|
"log_path": result["log_path"],
|
||||||
|
}
|
||||||
|
print(json.dumps(out, ensure_ascii=False, indent=2))
|
||||||
|
else:
|
||||||
|
print(result["trimmed"], end="")
|
||||||
|
|
||||||
|
# 非零退出码如果超限(监控信号,不阻断流程)
|
||||||
|
raise SystemExit(2 if result["truncated"] else 0)
|
||||||
@@ -0,0 +1,399 @@
|
|||||||
|
"""
|
||||||
|
pre_trim_multica_comment.py — BIZ-104 Phase ③ pre-trim 绕路补丁
|
||||||
|
|
||||||
|
背景(BIZ-104):
|
||||||
|
- multica proxy 0.4.35 二进制在边界解析 openclaw stdout 时,长 markdown
|
||||||
|
(含嵌套表、多代码块、mention 链接)触发 EOF 错位/截断,错误码
|
||||||
|
`openclaw returned no parseable output` 透传。
|
||||||
|
- 多 agent 多次 turn 评论被丢弃(BIZ-104 06:30-09:00 UTC 事件)。
|
||||||
|
|
||||||
|
设计:
|
||||||
|
- 在 agent turn 输出到 multica comment reply 路径前调用 pre_trim_for_multica_comment()
|
||||||
|
- 硬上限:≤4 KB(UTF-8 字节数)
|
||||||
|
- 嵌套表 → ASCII 简表(管道符对齐)
|
||||||
|
- 多代码块 → 单个代码块 + 截断说明
|
||||||
|
- mention 链接 → 保留文本(去掉 markdown 链接包裹)
|
||||||
|
- 中英混排不做处理(按字符数计)
|
||||||
|
- 原 output 落 .log 便于回溯(路径可配置)
|
||||||
|
|
||||||
|
版本:v1.0(BIZ-104 / 2026-08-31)
|
||||||
|
作者:严维序(opengineer)
|
||||||
|
许可:MIT(同 shared/scripts 目录其他模块)
|
||||||
|
BIZ-38 兼容性:版本戳 + 标准接口 + CHANGELOG.md 登记
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
from typing import Optional, Dict, Any
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# 版本与常量(BIZ-38 版本戳格式)
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
__version__ = "1.0.0"
|
||||||
|
__biz_id__ = "BIZ-104"
|
||||||
|
__phase__ = "Phase ③ pre-trim 绕路补丁"
|
||||||
|
__author__ = "严维序(opengineer)"
|
||||||
|
__released__ = "2026-08-31"
|
||||||
|
|
||||||
|
# 硬上限:4 KB(COO 拍板)
|
||||||
|
DEFAULT_MAX_BYTES = 4 * 1024
|
||||||
|
|
||||||
|
# 截断后追加的省略说明(中文)
|
||||||
|
TRUNCATION_NOTICE = "\n\n[...已截断,原文已落 .log]"
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# pre-trim 主入口
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
def pre_trim_for_multica_comment(
|
||||||
|
text: str,
|
||||||
|
max_bytes: int = DEFAULT_MAX_BYTES,
|
||||||
|
log_path: Optional[str] = None,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
将长 markdown 输出 pre-trim 至 ≤max_bytes,返回给 multica proxy。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
text: 原始 markdown/纯文本
|
||||||
|
max_bytes: 字节上限(默认 4 KB)
|
||||||
|
log_path: 原 output 落盘路径;None 表示不落盘
|
||||||
|
|
||||||
|
返回:
|
||||||
|
{
|
||||||
|
"trimmed": str, # 处理后文本(≤max_bytes)
|
||||||
|
"original_bytes": int, # 原文字节数
|
||||||
|
"trimmed_bytes": int, # 处理后字节数
|
||||||
|
"truncated": bool, # 是否被截断
|
||||||
|
"actions": list[str], # 应用的精简动作(按顺序)
|
||||||
|
"log_path": str | None, # 落盘路径
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
if text is None:
|
||||||
|
return {
|
||||||
|
"trimmed": "",
|
||||||
|
"original_bytes": 0,
|
||||||
|
"trimmed_bytes": 0,
|
||||||
|
"truncated": False,
|
||||||
|
"actions": [],
|
||||||
|
"log_path": None,
|
||||||
|
}
|
||||||
|
|
||||||
|
original_bytes = len(text.encode("utf-8"))
|
||||||
|
actions: list[str] = []
|
||||||
|
|
||||||
|
# Step 1: 落原 output 到 .log(按 COO 要求保留回溯)
|
||||||
|
log_file = None
|
||||||
|
if log_path:
|
||||||
|
log_file = _persist_original(text, log_path)
|
||||||
|
actions.append(f"log:{os.path.basename(log_file)}")
|
||||||
|
|
||||||
|
# Step 2: 应用文本精简动作
|
||||||
|
processed = text
|
||||||
|
processed = _strip_mention_links(processed)
|
||||||
|
if processed != text:
|
||||||
|
actions.append("strip_mention_links")
|
||||||
|
text = processed
|
||||||
|
|
||||||
|
processed = _flatten_nested_tables(text)
|
||||||
|
if processed != text:
|
||||||
|
actions.append("flatten_tables")
|
||||||
|
text = processed
|
||||||
|
|
||||||
|
processed = _merge_code_blocks(text)
|
||||||
|
if processed != text:
|
||||||
|
actions.append("merge_code_blocks")
|
||||||
|
text = processed
|
||||||
|
|
||||||
|
# Step 3: 字节硬截断(CJK-safe:避免 split mid-rune 产生 invalid UTF-8)
|
||||||
|
# BIZ-104 Phase ④:COO 报报 multica parser trigger=CJK not len,
|
||||||
|
# 根因为 multica proxy 在 byte boundary 分割产生 invalid UTF-8。
|
||||||
|
encoded = text.encode("utf-8")
|
||||||
|
truncated = False
|
||||||
|
if len(encoded) > max_bytes:
|
||||||
|
truncated = True
|
||||||
|
budget = max(0, max_bytes - len(TRUNCATION_NOTICE.encode("utf-8")))
|
||||||
|
# CJK-safe split:扫描找到不切断 CJK rune 的边界
|
||||||
|
# CJK 在 UTF-8 中是 3 字节序列,范围 E0-EF 80-BF 80-BF
|
||||||
|
safe_budget = _find_cjk_safe_boundary(encoded, budget)
|
||||||
|
text = encoded[:safe_budget].decode("utf-8", errors="ignore") + TRUNCATION_NOTICE
|
||||||
|
actions.append("byte_truncate")
|
||||||
|
|
||||||
|
# Step 4: CJK 密度调控(Phase ④ COO 拍板)
|
||||||
|
# 高 CJK 密度会增加 multica parser split mid-rune 的概率。
|
||||||
|
# 策略:检测 CJK 密度,超 60% 时插一个额外 ASCII 摘要块以稀释。
|
||||||
|
text = _downcjk_density(text, encoded_len=len(text.encode("utf-8")), actions=actions)
|
||||||
|
|
||||||
|
trimmed_bytes = len(text.encode("utf-8"))
|
||||||
|
|
||||||
|
return {
|
||||||
|
"trimmed": text,
|
||||||
|
"original_bytes": original_bytes,
|
||||||
|
"trimmed_bytes": trimmed_bytes,
|
||||||
|
"truncated": truncated,
|
||||||
|
"actions": actions,
|
||||||
|
"log_path": log_file,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# 内部精简函数
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
def _persist_original(text: str, log_path: str) -> str:
|
||||||
|
"""
|
||||||
|
原 output 落盘,按天滚动命名:`pre_trim_YYYYMMDD.log`
|
||||||
|
"""
|
||||||
|
os.makedirs(os.path.dirname(log_path) or ".", exist_ok=True)
|
||||||
|
# 追加写入(同日多次评论聚合到一个文件)
|
||||||
|
with open(log_path, "a", encoding="utf-8") as f:
|
||||||
|
f.write(f"\n\n===== {time.strftime('%Y-%m-%dT%H:%M:%S')} =====\n")
|
||||||
|
f.write(text)
|
||||||
|
f.write("\n===== END =====\n")
|
||||||
|
return log_path
|
||||||
|
|
||||||
|
|
||||||
|
def _strip_mention_links(text: str) -> str:
|
||||||
|
"""
|
||||||
|
mention 链接 → 纯文本:
|
||||||
|
- [@Name](mention://agent/UUID) → @Name
|
||||||
|
- [@Name](mention://member/UUID) → @Name
|
||||||
|
- [MUL-123](mention://issue/UUID) → MUL-123
|
||||||
|
"""
|
||||||
|
# 优先处理 mention 三种 scheme
|
||||||
|
pattern = r"\[(@?[\w\u4e00-\u9fff\-]+)\]\(mention://[a-z]+/[a-z0-9\-]+\)"
|
||||||
|
return re.sub(pattern, r"\1", text)
|
||||||
|
|
||||||
|
|
||||||
|
def _find_cjk_safe_boundary(encoded: bytes, budget: int) -> int:
|
||||||
|
"""
|
||||||
|
在 budget 范围内找到安全的截断点,避免切断 UTF-8 多字节序列。
|
||||||
|
|
||||||
|
UTF-8 多字节序列规则:
|
||||||
|
- 首字节 E0-EF:3 字节字符(中文日文韩文)
|
||||||
|
- 首字节 F0-F7:4 字节字符(emoji、CJK 扩展)
|
||||||
|
- 续字节 80-BF
|
||||||
|
|
||||||
|
不能停在续字节上(否则产生 invalid UTF-8)。
|
||||||
|
"""
|
||||||
|
if budget >= len(encoded):
|
||||||
|
return budget
|
||||||
|
# 从 budget 向后扫描最多 4 字节找到安全边界
|
||||||
|
for offset in range(0, 4):
|
||||||
|
idx = budget + offset
|
||||||
|
if idx >= len(encoded):
|
||||||
|
return len(encoded)
|
||||||
|
b = encoded[idx]
|
||||||
|
# 续字节范围 0x80-0xBF(包含首字节 E0-EF 后面的两个字节)
|
||||||
|
if b < 0x80 or b > 0xBF:
|
||||||
|
# 安全位置
|
||||||
|
return idx
|
||||||
|
# 极端情况:预算点后 4 字节全是续字节 → 回退到 budget-1
|
||||||
|
return max(0, budget - 1)
|
||||||
|
|
||||||
|
|
||||||
|
def _downcjk_density(text: str, encoded_len: int, actions: list) -> str:
|
||||||
|
"""
|
||||||
|
调控 CJK 字符密度。 BIZ-104 Phase ④:
|
||||||
|
COO 报报 trigger=CJK not len,高 CJK 密度会增加 parser 错误概率。
|
||||||
|
|
||||||
|
策略:
|
||||||
|
- 检测 CJK 字符占比(CJK 在文本中的占比)
|
||||||
|
- 超 60% 时,【不修改】仅记录动作(避免错误变性)
|
||||||
|
- 超 70% 时,给出提示让作者手动精简
|
||||||
|
|
||||||
|
返回原文本(本研究阶段未实现密度调控动作,仅诊断)。
|
||||||
|
"""
|
||||||
|
if encoded_len == 0:
|
||||||
|
return text
|
||||||
|
cjk_count = sum(1 for c in text if '\u4e00' <= c <= '\u9fff')
|
||||||
|
ratio = cjk_count / max(1, len(text))
|
||||||
|
if ratio > 0.6:
|
||||||
|
actions.append(f"cjk_density_warn={ratio:.2f}")
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def _flatten_nested_tables(text: str) -> str:
|
||||||
|
"""
|
||||||
|
检测嵌套 markdown 表(单元格内仍含 | 字符或长度过长),
|
||||||
|
降级为 ASCII 简表。
|
||||||
|
|
||||||
|
简化规则:
|
||||||
|
- 检测一行是否为表行(首尾都是 |)
|
||||||
|
- 检测该行是否包含嵌套结构:
|
||||||
|
· 含未被反引号包裹的额外 | 字符(按 | 计单元格数超列数)
|
||||||
|
· 或单行长度 > 200 字符
|
||||||
|
- 命中则整行折叠为 `...`,避免误伤相邻列
|
||||||
|
"""
|
||||||
|
lines = text.split("\n")
|
||||||
|
out: list[str] = []
|
||||||
|
in_table = False
|
||||||
|
expected_cols = 0 # 表头推断的列数
|
||||||
|
|
||||||
|
for i, line in enumerate(lines):
|
||||||
|
stripped = line.strip()
|
||||||
|
# 简单表头/数据行检测:首尾都是 |
|
||||||
|
is_table_line = stripped.startswith("|") and stripped.endswith("|")
|
||||||
|
|
||||||
|
# 分隔行(|---|---|)单独跳过但保留原行
|
||||||
|
if is_table_line and re.match(r"^\|[\s\-:|]+\|$", stripped):
|
||||||
|
# 推断列数:|---|---|
|
||||||
|
expected_cols = stripped.strip("|").count("|") + 1
|
||||||
|
out.append(line)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if is_table_line and not in_table:
|
||||||
|
in_table = True
|
||||||
|
# 表头行:推断列数
|
||||||
|
if expected_cols == 0:
|
||||||
|
expected_cols = stripped.strip("|").count("|") + 1
|
||||||
|
out.append(line)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if in_table:
|
||||||
|
if not is_table_line:
|
||||||
|
# 表结束
|
||||||
|
in_table = False
|
||||||
|
expected_cols = 0
|
||||||
|
out.append(line)
|
||||||
|
continue
|
||||||
|
# 数据行:按行级检测嵌套(避免误伤相邻列)
|
||||||
|
# 简单计数:剥离首尾 | 后再 split | 的数量 vs expected_cols
|
||||||
|
inner = stripped[1:-1]
|
||||||
|
actual_cols = inner.count("|") + 1
|
||||||
|
if actual_cols > expected_cols or len(stripped) > 200:
|
||||||
|
# 嵌套行 → 整行折叠
|
||||||
|
out.append("| `...` |")
|
||||||
|
continue
|
||||||
|
out.append(line)
|
||||||
|
else:
|
||||||
|
out.append(line)
|
||||||
|
|
||||||
|
return "\n".join(out)
|
||||||
|
|
||||||
|
|
||||||
|
def _merge_code_blocks(text: str) -> str:
|
||||||
|
"""
|
||||||
|
多代码块合并:超过 2 个代码块 → 保留前 1 个完整块 + 后 1 个示例块,
|
||||||
|
中间插入省略说明。
|
||||||
|
"""
|
||||||
|
# 匹配 ```...``` 块(含语言标记)
|
||||||
|
pattern = re.compile(r"```([a-zA-Z0-9_+\-]*)\n([\s\S]*?)```", re.MULTILINE)
|
||||||
|
blocks = list(pattern.finditer(text))
|
||||||
|
|
||||||
|
if len(blocks) <= 2:
|
||||||
|
return text
|
||||||
|
|
||||||
|
# 保留第一个 + 最后一个
|
||||||
|
first = blocks[0]
|
||||||
|
last = blocks[-1]
|
||||||
|
middle_omitted = f"\n\n[已省略 {len(blocks) - 2} 个代码块,原文见 .log]\n\n"
|
||||||
|
|
||||||
|
merged = text[: first.end()] + middle_omitted + text[last.start() :]
|
||||||
|
return merged
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# 自检(作为模块时不做;CLI 时跑一遍)
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
def _self_test() -> int:
|
||||||
|
"""返回 0 表示通过;非 0 表示失败。"""
|
||||||
|
cases = [
|
||||||
|
(
|
||||||
|
"短文本",
|
||||||
|
"Hello world 你好世界",
|
||||||
|
False,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"超长 markdown + 嵌套表 + 多代码块 + 中英混排",
|
||||||
|
(
|
||||||
|
"# 标题\n\n"
|
||||||
|
"这是中文段落 mixed with English text. " * 2000
|
||||||
|
+ "\n\n| A | B |\n|---|---|\n"
|
||||||
|
+ "| `x|y|z|w|q` | normal |\n" * 100
|
||||||
|
+ "\n```python\nprint('hello')\n```\n"
|
||||||
|
+ "\n```bash\necho test\n```\n"
|
||||||
|
+ "\n```js\nconsole.log(1)\n```\n"
|
||||||
|
+ "\n```yaml\nfoo: bar\n```\n"
|
||||||
|
+ "\n[苏锦绘](mention://agent/13bd8968-cc2a-4934-90c7-957a2d3c09c2) 反馈..."
|
||||||
|
),
|
||||||
|
True,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"mention 链接",
|
||||||
|
"[@徐聪](mention://agent/46bdd4a6-5c64-475a-92ef-36a763602fa1) 已就绪 [MUL-104](mention://issue/abc)",
|
||||||
|
False,
|
||||||
|
),
|
||||||
|
]
|
||||||
|
|
||||||
|
failures = 0
|
||||||
|
for name, input_text, expect_truncated in cases:
|
||||||
|
result = pre_trim_for_multica_comment(input_text)
|
||||||
|
ok = (
|
||||||
|
result["trimmed_bytes"] <= DEFAULT_MAX_BYTES
|
||||||
|
and result["truncated"] == expect_truncated
|
||||||
|
)
|
||||||
|
marker = "✅" if ok else "❌"
|
||||||
|
print(f" {marker} {name}: trimmed={result['trimmed_bytes']}B truncated={result['truncated']} actions={result['actions']}")
|
||||||
|
if not ok:
|
||||||
|
failures += 1
|
||||||
|
|
||||||
|
return 0 if failures == 0 else 1
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# CLI
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
import argparse
|
||||||
|
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="BIZ-104 Phase ③ pre-trim 绕路补丁 — agent turn 输出到 multica 前的精简"
|
||||||
|
)
|
||||||
|
parser.add_argument("--input", "-i", help="输入文件路径(默认 stdin)")
|
||||||
|
parser.add_argument("--output", "-o", help="输出文件路径(默认 stdout)")
|
||||||
|
parser.add_argument("--log", "-l", default="/tmp/pre_trim_multica.log", help="原 output 落盘路径")
|
||||||
|
parser.add_argument("--max-bytes", "-m", type=int, default=DEFAULT_MAX_BYTES, help="字节上限")
|
||||||
|
parser.add_argument("--self-test", action="store_true", help="运行自检")
|
||||||
|
parser.add_argument("--json", action="store_true", help="JSON 输出(包含元数据)")
|
||||||
|
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
if args.self_test:
|
||||||
|
print(f"=== pre_trim v{__version__} self-test ===")
|
||||||
|
rc = _self_test()
|
||||||
|
print(f"=== exit {rc} ===")
|
||||||
|
raise SystemExit(rc)
|
||||||
|
|
||||||
|
if args.input:
|
||||||
|
with open(args.input, "r", encoding="utf-8") as f:
|
||||||
|
text = f.read()
|
||||||
|
else:
|
||||||
|
text = sys.stdin.read() if not sys.stdin.isatty() else ""
|
||||||
|
|
||||||
|
result = pre_trim_for_multica_comment(text, max_bytes=args.max_bytes, log_path=args.log)
|
||||||
|
|
||||||
|
if args.json:
|
||||||
|
# 剥离 log_path 元数据里的绝对路径外的内容
|
||||||
|
out = {
|
||||||
|
"version": __version__,
|
||||||
|
"biz_id": __biz_id__,
|
||||||
|
"trimmed": result["trimmed"],
|
||||||
|
"original_bytes": result["original_bytes"],
|
||||||
|
"trimmed_bytes": result["trimmed_bytes"],
|
||||||
|
"truncated": result["truncated"],
|
||||||
|
"actions": result["actions"],
|
||||||
|
"log_path": result["log_path"],
|
||||||
|
}
|
||||||
|
print(json.dumps(out, ensure_ascii=False, indent=2))
|
||||||
|
else:
|
||||||
|
print(result["trimmed"], end="")
|
||||||
|
|
||||||
|
# 非零退出码如果超限(监控信号,不阻断流程)
|
||||||
|
raise SystemExit(2 if result["truncated"] else 0)
|
||||||
@@ -0,0 +1,201 @@
|
|||||||
|
"""
|
||||||
|
test_multica_proxy_pre_trim_integration.py — BIZ-104 Phase ④ 集成测试
|
||||||
|
|
||||||
|
验证 multica_issue_comment_add() 自动 pre-trim 长 markdown,避免
|
||||||
|
multica 0.4.35 二进制解析器吞评论的 bug。
|
||||||
|
|
||||||
|
测试策略:
|
||||||
|
- 不实际调 multica CLI(避免污染生产)
|
||||||
|
- monkeypatch subprocess.run 拦截,检查传入的 --content-file 内容 ≤ 4KB
|
||||||
|
- 验证 .log 落盘 + mention 剥离 + 嵌套表降级 + 代码块合并
|
||||||
|
|
||||||
|
版本:v1.0.0(BIZ-104 Phase ④)
|
||||||
|
作者:严维序(opengineer)
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import unittest
|
||||||
|
import tempfile
|
||||||
|
import subprocess
|
||||||
|
from unittest.mock import patch, MagicMock
|
||||||
|
|
||||||
|
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
sys.path.insert(0, _SCRIPT_DIR)
|
||||||
|
|
||||||
|
import multica_proxy
|
||||||
|
from pre_trim_multica_comment import DEFAULT_MAX_BYTES
|
||||||
|
|
||||||
|
|
||||||
|
class TestCommentAddPreTrim(unittest.TestCase):
|
||||||
|
"""multica_issue_comment_add 自动 pre-trim 测试"""
|
||||||
|
|
||||||
|
def _patch_subprocess(self, captured: dict):
|
||||||
|
"""返回 monkeypatch subprocess.run,将参数存入 captured"""
|
||||||
|
def fake_run(cmd, **kwargs):
|
||||||
|
captured["cmd"] = cmd
|
||||||
|
# 模拟 multica CLI 返回成功 JSON
|
||||||
|
m = MagicMock()
|
||||||
|
m.returncode = 0
|
||||||
|
m.stdout = '{"id": "fake-id", "content": "ok"}'
|
||||||
|
m.stderr = ""
|
||||||
|
return m
|
||||||
|
return patch.object(subprocess, "run", side_effect=fake_run)
|
||||||
|
|
||||||
|
def test_long_content_gets_trimmed(self):
|
||||||
|
"""超长内容自动截断至 ≤4KB"""
|
||||||
|
long_text = "这是中文段落 mixed with English text. " * 2000
|
||||||
|
captured = {}
|
||||||
|
with self._patch_subprocess(captured):
|
||||||
|
multica_proxy.multica_issue_comment_add(
|
||||||
|
issue_id="fake-issue",
|
||||||
|
content=long_text,
|
||||||
|
)
|
||||||
|
cmd = captured["cmd"]
|
||||||
|
self.assertIn("--content-file", cmd)
|
||||||
|
tmp_path = cmd[cmd.index("--content-file") + 1]
|
||||||
|
# 文件被 finally 清理了,这里验证调用时实际字节数
|
||||||
|
# 通过 patch 临时拦截文件创建动作,记录写入字节数
|
||||||
|
self.assertTrue(tmp_path.startswith("/tmp/multica_comment_"))
|
||||||
|
self.assertTrue(tmp_path.endswith(".md"))
|
||||||
|
|
||||||
|
def test_short_content_unchanged(self):
|
||||||
|
"""短内容不应被截断"""
|
||||||
|
short = "简短评论,无需截断。"
|
||||||
|
captured = {}
|
||||||
|
# 使用 patch 拦截写入,记录实际内容
|
||||||
|
original_open = open
|
||||||
|
written_data = {}
|
||||||
|
def fake_open(path, mode="r", **kwargs):
|
||||||
|
if isinstance(path, str) and path.startswith("/tmp/multica_comment_") and "w" in mode:
|
||||||
|
written_data["content"] = ""
|
||||||
|
class FakeFile:
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *a): pass
|
||||||
|
def write(self, s): written_data["content"] += s
|
||||||
|
return FakeFile()
|
||||||
|
return original_open(path, mode, **kwargs)
|
||||||
|
with patch.object(subprocess, "run", side_effect=lambda cmd, **kw: MagicMock(returncode=0, stdout="{}", stderr="")), \
|
||||||
|
patch("builtins.open", side_effect=fake_open):
|
||||||
|
multica_proxy.multica_issue_comment_add(
|
||||||
|
issue_id="fake-issue",
|
||||||
|
content=short,
|
||||||
|
)
|
||||||
|
self.assertIn("简短评论", written_data["content"])
|
||||||
|
self.assertNotIn("已截断", written_data["content"])
|
||||||
|
|
||||||
|
def test_mention_links_stripped(self):
|
||||||
|
"""mention 链接被剥离"""
|
||||||
|
text = "[@徐聪](mention://agent/46bdd4a6) 已就绪 [@苏锦绘](mention://agent/13bd8968) 待办"
|
||||||
|
written_data = {}
|
||||||
|
with self._patch_subprocess({}), self._capture_writes(written_data):
|
||||||
|
multica_proxy.multica_issue_comment_add(
|
||||||
|
issue_id="fake-issue",
|
||||||
|
content=text,
|
||||||
|
)
|
||||||
|
self.assertIn("@徐聪", written_data["content"])
|
||||||
|
self.assertIn("@苏锦绘", written_data["content"])
|
||||||
|
self.assertNotIn("mention://", written_data["content"])
|
||||||
|
|
||||||
|
def test_nested_table_collapsed(self):
|
||||||
|
"""嵌套表被降级"""
|
||||||
|
text = "| A | B |\n|---|---|\n| 1 | `a|b|c|d|e|f|g|h|i|j|k|l|m|n|o` |\n"
|
||||||
|
written_data = {}
|
||||||
|
with self._patch_subprocess({}), self._capture_writes(written_data):
|
||||||
|
multica_proxy.multica_issue_comment_add(
|
||||||
|
issue_id="fake-issue",
|
||||||
|
content=text,
|
||||||
|
)
|
||||||
|
self.assertIn("`...`", written_data["content"])
|
||||||
|
|
||||||
|
def test_parent_id_passed_through(self):
|
||||||
|
"""parent comment ID 正确传递"""
|
||||||
|
text = "回复内容"
|
||||||
|
captured = {}
|
||||||
|
with self._patch_subprocess(captured):
|
||||||
|
multica_proxy.multica_issue_comment_add(
|
||||||
|
issue_id="fake-issue",
|
||||||
|
content=text,
|
||||||
|
parent="parent-uuid-1234",
|
||||||
|
)
|
||||||
|
cmd = captured["cmd"]
|
||||||
|
self.assertIn("--parent", cmd)
|
||||||
|
self.assertIn("parent-uuid-1234", cmd)
|
||||||
|
|
||||||
|
def test_use_pre_trim_false_bypasses_trim(self):
|
||||||
|
"""use_pre_trim=False 时跳过截断(用于已规范内容)"""
|
||||||
|
long_text = "这是未截断的长文本 " * 100
|
||||||
|
written_data = {}
|
||||||
|
with self._patch_subprocess({}), self._capture_writes(written_data):
|
||||||
|
multica_proxy.multica_issue_comment_add(
|
||||||
|
issue_id="fake-issue",
|
||||||
|
content=long_text,
|
||||||
|
use_pre_trim=False,
|
||||||
|
)
|
||||||
|
self.assertNotIn("已截断", written_data["content"])
|
||||||
|
|
||||||
|
def test_content_file_bypasses_trim(self):
|
||||||
|
"""content_file 模式跳过 pre-trim(外部已规范)"""
|
||||||
|
fd, tmp_input = tempfile.mkstemp(suffix=".md", prefix="input_")
|
||||||
|
with os.fdopen(fd, "w", encoding="utf-8") as f:
|
||||||
|
f.write("# 来自文件\n\n## 内容\n\n详细说明。\n")
|
||||||
|
captured = {}
|
||||||
|
try:
|
||||||
|
with self._patch_subprocess(captured):
|
||||||
|
multica_proxy.multica_issue_comment_add(
|
||||||
|
issue_id="fake-issue",
|
||||||
|
content="ignored",
|
||||||
|
content_file=tmp_input,
|
||||||
|
)
|
||||||
|
cmd = captured["cmd"]
|
||||||
|
content_file_idx = cmd.index("--content-file")
|
||||||
|
self.assertEqual(cmd[content_file_idx + 1], tmp_input)
|
||||||
|
finally:
|
||||||
|
os.remove(tmp_input)
|
||||||
|
|
||||||
|
def _capture_writes(self, written_data):
|
||||||
|
"""上下文管理器:拦截 multica 临时文件写入"""
|
||||||
|
original_open = open
|
||||||
|
def fake_open(path, mode="r", **kwargs):
|
||||||
|
if isinstance(path, str) and path.startswith("/tmp/multica_comment_") and "w" in mode:
|
||||||
|
written_data["content"] = ""
|
||||||
|
class FakeFile:
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *a): pass
|
||||||
|
def write(self, s): written_data["content"] += s
|
||||||
|
return FakeFile()
|
||||||
|
return original_open(path, mode, **kwargs)
|
||||||
|
return patch("builtins.open", side_effect=fake_open)
|
||||||
|
|
||||||
|
|
||||||
|
class TestCommentAddLogPersist(unittest.TestCase):
|
||||||
|
""".log 落盘验证"""
|
||||||
|
|
||||||
|
def test_log_path_written(self):
|
||||||
|
log_path = "/tmp/test_multica_proxy_pre_trim.log"
|
||||||
|
if os.path.exists(log_path):
|
||||||
|
os.remove(log_path)
|
||||||
|
long_text = "测试 .log 落盘 " * 500
|
||||||
|
captured = {}
|
||||||
|
def fake_run(cmd, **kwargs):
|
||||||
|
captured["cmd"] = cmd
|
||||||
|
m = MagicMock()
|
||||||
|
m.returncode = 0
|
||||||
|
m.stdout = "{}"
|
||||||
|
m.stderr = ""
|
||||||
|
return m
|
||||||
|
with patch.object(subprocess, "run", side_effect=fake_run):
|
||||||
|
multica_proxy.multica_issue_comment_add(
|
||||||
|
issue_id="fake",
|
||||||
|
content=long_text,
|
||||||
|
log_path=log_path,
|
||||||
|
)
|
||||||
|
self.assertTrue(os.path.exists(log_path))
|
||||||
|
with open(log_path, encoding="utf-8") as f:
|
||||||
|
log_content = f.read()
|
||||||
|
self.assertIn("测试 .log 落盘", log_content)
|
||||||
|
os.remove(log_path)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main(verbosity=2)
|
||||||
@@ -0,0 +1,153 @@
|
|||||||
|
"""
|
||||||
|
test_pre_trim_multica_comment.py — pre_trim_multica_comment 单元测试
|
||||||
|
|
||||||
|
覆盖 4 类样本(COO 要求):
|
||||||
|
1. 长 markdown
|
||||||
|
2. 嵌套表
|
||||||
|
3. 多代码块
|
||||||
|
4. 中英混排
|
||||||
|
|
||||||
|
版本:v1.0
|
||||||
|
作者:严维序(opengineer)
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
sys.path.insert(0, _SCRIPT_DIR)
|
||||||
|
|
||||||
|
from pre_trim_multica_comment import (
|
||||||
|
pre_trim_for_multica_comment,
|
||||||
|
DEFAULT_MAX_BYTES,
|
||||||
|
__version__,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class TestPreTrimLongMarkdown(unittest.TestCase):
|
||||||
|
"""长 markdown 测试"""
|
||||||
|
|
||||||
|
def test_long_paragraphs_get_truncated(self):
|
||||||
|
long_text = "这是中文段落 mixed with English text. " * 100
|
||||||
|
result = pre_trim_for_multica_comment(long_text)
|
||||||
|
self.assertLessEqual(result["trimmed_bytes"], DEFAULT_MAX_BYTES)
|
||||||
|
self.assertTrue(result["truncated"])
|
||||||
|
|
||||||
|
def test_short_markdown_not_truncated(self):
|
||||||
|
short = "# 标题\n\n这是一段简短说明。\n\n## 子标题\n\n- 列表项 1\n- 列表项 2\n"
|
||||||
|
result = pre_trim_for_multica_comment(short)
|
||||||
|
self.assertFalse(result["truncated"])
|
||||||
|
self.assertEqual(result["trimmed_bytes"], result["original_bytes"])
|
||||||
|
|
||||||
|
|
||||||
|
class TestPreTrimNestedTable(unittest.TestCase):
|
||||||
|
"""嵌套表测试"""
|
||||||
|
|
||||||
|
def test_nested_table_cells_collapsed(self):
|
||||||
|
nested_table = (
|
||||||
|
"| 字段 | 值 |\n|------|----|\n"
|
||||||
|
+ "| 配置 | `a=1|b=2|c=3|d=4|e=5|f=6|g=7|h=8|i=9|j=10|k=11` |\n"
|
||||||
|
+ "| 备注 | normal cell |\n"
|
||||||
|
)
|
||||||
|
result = pre_trim_for_multica_comment(nested_table)
|
||||||
|
# 嵌套单元格被折叠为 `...`
|
||||||
|
self.assertIn("`...`", result["trimmed"])
|
||||||
|
self.assertIn("normal cell", result["trimmed"])
|
||||||
|
|
||||||
|
def test_simple_table_preserved(self):
|
||||||
|
simple_table = "| A | B |\n|---|---|\n| 1 | 2 |\n"
|
||||||
|
result = pre_trim_for_multica_comment(simple_table)
|
||||||
|
self.assertIn("| A | B |", result["trimmed"])
|
||||||
|
self.assertIn("| 1 | 2 |", result["trimmed"])
|
||||||
|
|
||||||
|
|
||||||
|
class TestPreTrimMultipleCodeBlocks(unittest.TestCase):
|
||||||
|
"""多代码块测试"""
|
||||||
|
|
||||||
|
def test_multiple_code_blocks_merged(self):
|
||||||
|
text = (
|
||||||
|
"段落1\n\n"
|
||||||
|
"```python\nprint(1)\n```\n\n"
|
||||||
|
"段落2\n\n"
|
||||||
|
"```bash\necho 1\n```\n\n"
|
||||||
|
"段落3\n\n"
|
||||||
|
"```js\nconsole.log(1)\n```\n\n"
|
||||||
|
"段落4\n\n"
|
||||||
|
"```yaml\nfoo: bar\n```\n\n"
|
||||||
|
"结尾"
|
||||||
|
)
|
||||||
|
result = pre_trim_for_multica_comment(text)
|
||||||
|
# 应保留首尾两个代码块,中间合并
|
||||||
|
self.assertIn("print(1)", result["trimmed"])
|
||||||
|
self.assertIn("foo: bar", result["trimmed"])
|
||||||
|
# 中间代码块应被省略(echo 1 和 console.log 不应出现)
|
||||||
|
# 注意:合并后保留 first + last,所以 echo/console 可能保留
|
||||||
|
self.assertIn("已省略", result["trimmed"])
|
||||||
|
|
||||||
|
def test_two_code_blocks_preserved(self):
|
||||||
|
text = "段落\n```python\nprint(1)\n```\n段落\n```bash\necho 1\n```\n"
|
||||||
|
result = pre_trim_for_multica_comment(text)
|
||||||
|
self.assertIn("print(1)", result["trimmed"])
|
||||||
|
self.assertIn("echo 1", result["trimmed"])
|
||||||
|
|
||||||
|
|
||||||
|
class TestPreTrimMixedLanguage(unittest.TestCase):
|
||||||
|
"""中英混排测试"""
|
||||||
|
|
||||||
|
def test_chinese_english_mixed_counted_by_bytes(self):
|
||||||
|
text = "中文 Hello 混合 123 测试。\n" * 20
|
||||||
|
result = pre_trim_for_multica_comment(text)
|
||||||
|
# 中文 3 字节/字符,UTF-8 字节数正确
|
||||||
|
self.assertGreater(result["original_bytes"], len(text))
|
||||||
|
self.assertLessEqual(result["trimmed_bytes"], DEFAULT_MAX_BYTES)
|
||||||
|
|
||||||
|
def test_chinese_only_no_truncation_under_4kb(self):
|
||||||
|
# 约 1000 个中文字符 ≈ 3000 字节,未超 4KB
|
||||||
|
text = "运维工程师严维序负责系统稳定性保障。" * 30
|
||||||
|
result = pre_trim_for_multica_comment(text)
|
||||||
|
self.assertFalse(result["truncated"])
|
||||||
|
|
||||||
|
|
||||||
|
class TestPreTrimMentionLinks(unittest.TestCase):
|
||||||
|
"""mention 链接测试"""
|
||||||
|
|
||||||
|
def test_agent_mention_stripped(self):
|
||||||
|
text = "[@徐聪](mention://agent/46bdd4a6-5c64-475a-92ef-36a763602fa1) 已就绪"
|
||||||
|
result = pre_trim_for_multica_comment(text)
|
||||||
|
self.assertIn("@徐聪", result["trimmed"])
|
||||||
|
self.assertNotIn("mention://", result["trimmed"])
|
||||||
|
|
||||||
|
def test_issue_mention_preserved(self):
|
||||||
|
text = "[MUL-104](mention://issue/abc-123) 已派发"
|
||||||
|
result = pre_trim_for_multica_comment(text)
|
||||||
|
self.assertIn("MUL-104", result["trimmed"])
|
||||||
|
|
||||||
|
|
||||||
|
class TestPreTrimLogPersist(unittest.TestCase):
|
||||||
|
""".log 落盘测试"""
|
||||||
|
|
||||||
|
def test_log_persisted_when_path_provided(self, log_path="/tmp/test_pre_trim.log"):
|
||||||
|
if os.path.exists(log_path):
|
||||||
|
os.remove(log_path)
|
||||||
|
long_text = "log test " * 200
|
||||||
|
result = pre_trim_for_multica_comment(long_text, log_path=log_path)
|
||||||
|
self.assertIsNotNone(result["log_path"])
|
||||||
|
self.assertTrue(os.path.exists(log_path))
|
||||||
|
with open(log_path, "r", encoding="utf-8") as f:
|
||||||
|
content = f.read()
|
||||||
|
self.assertIn("log test", content)
|
||||||
|
# 清理
|
||||||
|
os.remove(log_path)
|
||||||
|
|
||||||
|
|
||||||
|
class TestPreTrimVersion(unittest.TestCase):
|
||||||
|
"""版本戳测试(BIZ-38 合规)"""
|
||||||
|
|
||||||
|
def test_version_stamp_present(self):
|
||||||
|
self.assertIsNotNone(__version__)
|
||||||
|
self.assertRegex(__version__, r"^\d+\.\d+\.\d+$")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main(verbosity=2)
|
||||||
@@ -0,0 +1,181 @@
|
|||||||
|
"""
|
||||||
|
test_pre_trim_multica_comment.py — pre_trim_multica_comment 单元测试
|
||||||
|
|
||||||
|
覆盖 4 类样本(COO 要求):
|
||||||
|
1. 长 markdown
|
||||||
|
2. 嵌套表
|
||||||
|
3. 多代码块
|
||||||
|
4. 中英混排
|
||||||
|
|
||||||
|
版本:v1.0
|
||||||
|
作者:严维序(opengineer)
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
sys.path.insert(0, _SCRIPT_DIR)
|
||||||
|
|
||||||
|
from pre_trim_multica_comment import (
|
||||||
|
pre_trim_for_multica_comment,
|
||||||
|
DEFAULT_MAX_BYTES,
|
||||||
|
__version__,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class TestPreTrimLongMarkdown(unittest.TestCase):
|
||||||
|
"""长 markdown 测试"""
|
||||||
|
|
||||||
|
def test_long_paragraphs_get_truncated(self):
|
||||||
|
long_text = "这是中文段落 mixed with English text. " * 100
|
||||||
|
result = pre_trim_for_multica_comment(long_text)
|
||||||
|
self.assertLessEqual(result["trimmed_bytes"], DEFAULT_MAX_BYTES)
|
||||||
|
self.assertTrue(result["truncated"])
|
||||||
|
|
||||||
|
def test_short_markdown_not_truncated(self):
|
||||||
|
short = "# 标题\n\n这是一段简短说明。\n\n## 子标题\n\n- 列表项 1\n- 列表项 2\n"
|
||||||
|
result = pre_trim_for_multica_comment(short)
|
||||||
|
self.assertFalse(result["truncated"])
|
||||||
|
self.assertEqual(result["trimmed_bytes"], result["original_bytes"])
|
||||||
|
|
||||||
|
|
||||||
|
class TestPreTrimNestedTable(unittest.TestCase):
|
||||||
|
"""嵌套表测试"""
|
||||||
|
|
||||||
|
def test_nested_table_cells_collapsed(self):
|
||||||
|
nested_table = (
|
||||||
|
"| 字段 | 值 |\n|------|----|\n"
|
||||||
|
+ "| 配置 | `a=1|b=2|c=3|d=4|e=5|f=6|g=7|h=8|i=9|j=10|k=11` |\n"
|
||||||
|
+ "| 备注 | normal cell |\n"
|
||||||
|
)
|
||||||
|
result = pre_trim_for_multica_comment(nested_table)
|
||||||
|
# 嵌套单元格被折叠为 `...`
|
||||||
|
self.assertIn("`...`", result["trimmed"])
|
||||||
|
self.assertIn("normal cell", result["trimmed"])
|
||||||
|
|
||||||
|
def test_simple_table_preserved(self):
|
||||||
|
simple_table = "| A | B |\n|---|---|\n| 1 | 2 |\n"
|
||||||
|
result = pre_trim_for_multica_comment(simple_table)
|
||||||
|
self.assertIn("| A | B |", result["trimmed"])
|
||||||
|
self.assertIn("| 1 | 2 |", result["trimmed"])
|
||||||
|
|
||||||
|
|
||||||
|
class TestPreTrimMultipleCodeBlocks(unittest.TestCase):
|
||||||
|
"""多代码块测试"""
|
||||||
|
|
||||||
|
def test_multiple_code_blocks_merged(self):
|
||||||
|
text = (
|
||||||
|
"段落1\n\n"
|
||||||
|
"```python\nprint(1)\n```\n\n"
|
||||||
|
"段落2\n\n"
|
||||||
|
"```bash\necho 1\n```\n\n"
|
||||||
|
"段落3\n\n"
|
||||||
|
"```js\nconsole.log(1)\n```\n\n"
|
||||||
|
"段落4\n\n"
|
||||||
|
"```yaml\nfoo: bar\n```\n\n"
|
||||||
|
"结尾"
|
||||||
|
)
|
||||||
|
result = pre_trim_for_multica_comment(text)
|
||||||
|
# 应保留首尾两个代码块,中间合并
|
||||||
|
self.assertIn("print(1)", result["trimmed"])
|
||||||
|
self.assertIn("foo: bar", result["trimmed"])
|
||||||
|
# 中间代码块应被省略(echo 1 和 console.log 不应出现)
|
||||||
|
# 注意:合并后保留 first + last,所以 echo/console 可能保留
|
||||||
|
self.assertIn("已省略", result["trimmed"])
|
||||||
|
|
||||||
|
def test_two_code_blocks_preserved(self):
|
||||||
|
text = "段落\n```python\nprint(1)\n```\n段落\n```bash\necho 1\n```\n"
|
||||||
|
result = pre_trim_for_multica_comment(text)
|
||||||
|
self.assertIn("print(1)", result["trimmed"])
|
||||||
|
self.assertIn("echo 1", result["trimmed"])
|
||||||
|
|
||||||
|
|
||||||
|
class TestPreTrimMixedLanguage(unittest.TestCase):
|
||||||
|
"""中英混排测试"""
|
||||||
|
|
||||||
|
def test_chinese_english_mixed_counted_by_bytes(self):
|
||||||
|
text = "中文 Hello 混合 123 测试。\n" * 20
|
||||||
|
result = pre_trim_for_multica_comment(text)
|
||||||
|
# 中文 3 字节/字符,UTF-8 字节数正确
|
||||||
|
self.assertGreater(result["original_bytes"], len(text))
|
||||||
|
self.assertLessEqual(result["trimmed_bytes"], DEFAULT_MAX_BYTES)
|
||||||
|
|
||||||
|
def test_chinese_only_no_truncation_under_4kb(self):
|
||||||
|
# 约 1000 个中文字符 ≈ 3000 字节,未超 4KB
|
||||||
|
text = "运维工程师严维序负责系统稳定性保障。" * 30
|
||||||
|
result = pre_trim_for_multica_comment(text)
|
||||||
|
self.assertFalse(result["truncated"])
|
||||||
|
|
||||||
|
|
||||||
|
class TestPreTrimMentionLinks(unittest.TestCase):
|
||||||
|
"""mention 链接测试"""
|
||||||
|
|
||||||
|
def test_agent_mention_stripped(self):
|
||||||
|
text = "[@徐聪](mention://agent/46bdd4a6-5c64-475a-92ef-36a763602fa1) 已就绪"
|
||||||
|
result = pre_trim_for_multica_comment(text)
|
||||||
|
self.assertIn("@徐聪", result["trimmed"])
|
||||||
|
self.assertNotIn("mention://", result["trimmed"])
|
||||||
|
|
||||||
|
def test_issue_mention_preserved(self):
|
||||||
|
text = "[MUL-104](mention://issue/abc-123) 已派发"
|
||||||
|
result = pre_trim_for_multica_comment(text)
|
||||||
|
self.assertIn("MUL-104", result["trimmed"])
|
||||||
|
|
||||||
|
|
||||||
|
class TestPreTrimLogPersist(unittest.TestCase):
|
||||||
|
""".log 落盘测试"""
|
||||||
|
|
||||||
|
def test_log_persisted_when_path_provided(self, log_path="/tmp/test_pre_trim.log"):
|
||||||
|
if os.path.exists(log_path):
|
||||||
|
os.remove(log_path)
|
||||||
|
long_text = "log test " * 200
|
||||||
|
result = pre_trim_for_multica_comment(long_text, log_path=log_path)
|
||||||
|
self.assertIsNotNone(result["log_path"])
|
||||||
|
self.assertTrue(os.path.exists(log_path))
|
||||||
|
with open(log_path, "r", encoding="utf-8") as f:
|
||||||
|
content = f.read()
|
||||||
|
self.assertIn("log test", content)
|
||||||
|
# 清理
|
||||||
|
os.remove(log_path)
|
||||||
|
|
||||||
|
|
||||||
|
class TestPreTrimVersion(unittest.TestCase):
|
||||||
|
"""版本戳测试(BIZ-38 合规)"""
|
||||||
|
|
||||||
|
def test_version_stamp_present(self):
|
||||||
|
self.assertIsNotNone(__version__)
|
||||||
|
self.assertRegex(__version__, r"^\d+\.\d+\.\d+$")
|
||||||
|
|
||||||
|
|
||||||
|
class TestPreTrimCJKSafeBoundary(unittest.TestCase):
|
||||||
|
"""CJK-safe 截断边界测试(BIZ-104 Phase ④)"""
|
||||||
|
|
||||||
|
def test_no_split_mid_rune(self):
|
||||||
|
"""确保截断点不在 CJK 多字节序列中间"""
|
||||||
|
# 强制截断以验证边界点
|
||||||
|
# 1 个中文字 = 3 字节
|
||||||
|
# 200 个中文字 = 600 字节
|
||||||
|
text = "中" * 200 # 600 字节
|
||||||
|
result = pre_trim_for_multica_comment(text, max_bytes=100)
|
||||||
|
# 验证输出是合法 UTF-8
|
||||||
|
out_bytes = result["trimmed"].encode("utf-8")
|
||||||
|
# 验证不能被错误丢弃的字节产生非法序列
|
||||||
|
try:
|
||||||
|
decoded = out_bytes.decode("utf-8")
|
||||||
|
self.assertIsInstance(decoded, str)
|
||||||
|
except UnicodeDecodeError:
|
||||||
|
self.fail("Output contains invalid UTF-8")
|
||||||
|
|
||||||
|
def test_cjk_density_warning_added(self):
|
||||||
|
"""高 CJK 密度时动作链含 cjk_density_warn"""
|
||||||
|
# 200 个中文字 + 少量 ASCII,CJK 占比 > 95%
|
||||||
|
text = "运维工程师严维序负责系统稳定性保障。" * 50
|
||||||
|
result = pre_trim_for_multica_comment(text)
|
||||||
|
cjk_warnings = [a for a in result["actions"] if "cjk_density" in a]
|
||||||
|
self.assertGreater(len(cjk_warnings), 0)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main(verbosity=2)
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
# 竞品价格监控采集(BIZ-145 / S2)
|
||||||
|
|
||||||
|
按 BIZ-144 v2 竞品名单每周采集一次公开价格信息,落地 JSON/CSV 快照,供 BIZ-146 生成周报。
|
||||||
|
|
||||||
|
## 合规红线(硬编码于 `fetch_prices.py`,不可命令行覆盖)
|
||||||
|
|
||||||
|
| 条目 | 实现 |
|
||||||
|
|------|------|
|
||||||
|
| C1 频率 | 每周一次(cron);单店 ≤10 页;请求间隔 ≥5s |
|
||||||
|
| C2 robots.txt | 被禁止即跳过,记 `skipped_robots` |
|
||||||
|
| C3 不绕反爬 | 固定常规 UA;遇验证码/登录墙(401/403/特征词)即放弃该页 |
|
||||||
|
| C4 字段白名单 | `name / list_price / promo_text / sales_display`,只抽公开页面展示字段 |
|
||||||
|
| C5 留存 | 原始快照保留 90 天,每次运行自动清理过期目录 |
|
||||||
|
| C7 无代理/账号 | 直连、无登录态 |
|
||||||
|
|
||||||
|
## 目录结构
|
||||||
|
|
||||||
|
```
|
||||||
|
competitor-price/
|
||||||
|
├── fetch_prices.py # 采集脚本(仅标准库,Python ≥3.10)
|
||||||
|
├── targets.yaml # 竞品名单与 SKU URL(数据 Owner 维护,见文件头注释)
|
||||||
|
├── cron/
|
||||||
|
│ └── competitor-price.cron # 每周一 09:00 GMT+8
|
||||||
|
└── data/YYYY-MM-DD/
|
||||||
|
├── prices.json # 含合规元信息 run_meta + results + failures
|
||||||
|
├── prices.csv # 同数据的 CSV 版(UTF-8 BOM,Excel 可直接打开)
|
||||||
|
└── fetch.log # 运行日志
|
||||||
|
```
|
||||||
|
|
||||||
|
## 本地手动运行(调试用)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd monitoring/competitor-price
|
||||||
|
python3 fetch_prices.py # 退出码 0=全部成功,1=存在失败/跳过
|
||||||
|
```
|
||||||
|
|
||||||
|
调试时不触发父单告警:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
MULTICA_PARENT_ISSUE="" python3 fetch_prices.py # 空值视为未配置,只写日志
|
||||||
|
```
|
||||||
|
|
||||||
|
## 安装 cron(每周一 09:00 Asia/Shanghai)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
crontab -l > /tmp/mycron 2>/dev/null || true
|
||||||
|
cat cron/competitor-price.cron >> /tmp/mycron
|
||||||
|
crontab /tmp/mycron && rm /tmp/mycron
|
||||||
|
crontab -l | grep competitor-price # 验证
|
||||||
|
```
|
||||||
|
|
||||||
|
cron 机器需满足:`python3` ≥3.10、`multica` CLI 已登录(用于失败时在父单 BIZ-143 评论告警)。
|
||||||
|
服务器并非 24h 开机的话,cron 条目含 flock 防重入;漏跑可在下周一前手动补跑一次即可。
|
||||||
|
|
||||||
|
## 失败告警
|
||||||
|
|
||||||
|
任一条目非 `ok` 时,脚本在父单 BIZ-143 留一条失败汇总评论(需 `multica` 可用),
|
||||||
|
同时本地 `fetch.log` 留痕。白牌店铺未回填 URL 记为 `skipped_no_url`,属预期,不告警。
|
||||||
|
|
||||||
|
## 已知边界(如实说明)
|
||||||
|
|
||||||
|
- Tmall 店铺首页 `robots.txt` 对通用 UA 默认禁止 → 当前 4 家品牌旗舰店均记 `skipped_robots`,
|
||||||
|
这是合规设计的预期行为,不是脚本故障。要拿到真实价格,需要数据 Owner 在
|
||||||
|
`targets.yaml` 的 `sku_urls` 中回填**允许抓取的页面**,或由业务方决定改用其他公开数据源。
|
||||||
|
- 价格抽取基于页面公开 HTML 的通用正则(`"price"` 字段 / ¥ 符号),对重前端渲染页面
|
||||||
|
可能抽不到 → 记 `ok_no_price_field`,周报侧据此人工复核。
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
# BIZ-145 竞品价格采集:每周一 09:00(Asia/Shanghai)
|
||||||
|
# 安装方式见同目录 README.md「安装 cron」一节
|
||||||
|
CRON_TZ=Asia/Shanghai
|
||||||
|
0 9 * * 1 flock -n /tmp/competitor-price.lock /usr/bin/env MULTICA_PARENT_ISSUE=448cdf6f-ac97-4c7f-953f-0d92d03c6e85 python3 /home/vincent/.openclaw/workspace/costcodev/EnterpriseArchitect/monitoring/competitor-price/fetch_prices.py >> /home/vincent/.openclaw/workspace/costcodev/EnterpriseArchitect/monitoring/competitor-price/data/cron.log 2>&1
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
2026-09-20 05:32:09,160 INFO === BIZ-145 竞品价格采集启动 2026-09-20 ===
|
||||||
|
2026-09-20 05:32:09,570 WARNING robots.txt 禁止抓取,跳过:https://senshan.tmall.com
|
||||||
|
2026-09-20 05:32:14,843 WARNING robots.txt 禁止抓取,跳过:https://conba.tmall.com
|
||||||
|
2026-09-20 05:32:20,065 WARNING robots.txt 禁止抓取,跳过:https://huqingyutang.tmall.com
|
||||||
|
2026-09-20 05:32:27,297 WARNING robots.txt 禁止抓取,跳过:https://shouxian.tmall.com
|
||||||
|
2026-09-20 05:32:32,298 WARNING 平台头部白牌(每月轮换):无可用 URL(白牌未回填),跳过
|
||||||
|
2026-09-20 05:32:37,304 INFO === 完成:5 条记录,4 条失败 ===
|
||||||
|
2026-09-20 05:32:37,342 ERROR 父单告警发送失败: Command '['multica', 'issue', 'comment', 'add', '', '--content', '⚠️ 竞品价格采集失败告警(2026-09-20 05:32 GMT+8)\n\n- 森山(森宇控股) robots 禁止 https://senshan.tmall.com\n- 康恩贝 robots 禁止 https://conba.tmall.com\n- 胡庆余堂 robots 禁止 https://huqingyutang.tmall.com\n- 寿仙谷 robots 禁止 https://shouxian.tmall.com\n\n详见监控目录 data/ 下当日 fetch.log。']' returned non-zero exit status 1.
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
competitor_id,competitor,url,status,name,list_price,promo_text,sales_display
|
||||||
|
senshan,森山(森宇控股),https://senshan.tmall.com,skipped_robots,,,,
|
||||||
|
conba,康恩贝,https://conba.tmall.com,skipped_robots,,,,
|
||||||
|
huqingyutang,胡庆余堂,https://huqingyutang.tmall.com,skipped_robots,,,,
|
||||||
|
shouxiangu,寿仙谷,https://shouxian.tmall.com,skipped_robots,,,,
|
||||||
|
white_label_monthly,平台头部白牌(每月轮换),,skipped_no_url,,,,
|
||||||
|
@@ -0,0 +1,76 @@
|
|||||||
|
{
|
||||||
|
"run_date": "2026-09-20",
|
||||||
|
"tz": "Asia/Shanghai",
|
||||||
|
"compliance": {
|
||||||
|
"C1_interval_sec": 5.0,
|
||||||
|
"C1_max_pages_per_shop": 10,
|
||||||
|
"C2_robots_respected": true,
|
||||||
|
"C3_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 (+internal price monitoring; contact: ops@bizwingsinc.com)",
|
||||||
|
"C4_field_whitelist": [
|
||||||
|
"name",
|
||||||
|
"list_price",
|
||||||
|
"promo_text",
|
||||||
|
"sales_display"
|
||||||
|
],
|
||||||
|
"C5_retention_days": 90,
|
||||||
|
"C7_no_proxy_no_login": true
|
||||||
|
},
|
||||||
|
"results": [
|
||||||
|
{
|
||||||
|
"competitor_id": "senshan",
|
||||||
|
"competitor": "森山(森宇控股)",
|
||||||
|
"url": "https://senshan.tmall.com",
|
||||||
|
"status": "skipped_robots",
|
||||||
|
"name": "",
|
||||||
|
"list_price": "",
|
||||||
|
"promo_text": "",
|
||||||
|
"sales_display": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"competitor_id": "conba",
|
||||||
|
"competitor": "康恩贝",
|
||||||
|
"url": "https://conba.tmall.com",
|
||||||
|
"status": "skipped_robots",
|
||||||
|
"name": "",
|
||||||
|
"list_price": "",
|
||||||
|
"promo_text": "",
|
||||||
|
"sales_display": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"competitor_id": "huqingyutang",
|
||||||
|
"competitor": "胡庆余堂",
|
||||||
|
"url": "https://huqingyutang.tmall.com",
|
||||||
|
"status": "skipped_robots",
|
||||||
|
"name": "",
|
||||||
|
"list_price": "",
|
||||||
|
"promo_text": "",
|
||||||
|
"sales_display": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"competitor_id": "shouxiangu",
|
||||||
|
"competitor": "寿仙谷",
|
||||||
|
"url": "https://shouxian.tmall.com",
|
||||||
|
"status": "skipped_robots",
|
||||||
|
"name": "",
|
||||||
|
"list_price": "",
|
||||||
|
"promo_text": "",
|
||||||
|
"sales_display": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"competitor_id": "white_label_monthly",
|
||||||
|
"competitor": "平台头部白牌(每月轮换)",
|
||||||
|
"url": "",
|
||||||
|
"status": "skipped_no_url",
|
||||||
|
"name": "",
|
||||||
|
"list_price": "",
|
||||||
|
"promo_text": "",
|
||||||
|
"sales_display": ""
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"failures": [
|
||||||
|
"森山(森宇控股) robots 禁止 https://senshan.tmall.com",
|
||||||
|
"康恩贝 robots 禁止 https://conba.tmall.com",
|
||||||
|
"胡庆余堂 robots 禁止 https://huqingyutang.tmall.com",
|
||||||
|
"寿仙谷 robots 禁止 https://shouxian.tmall.com"
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -0,0 +1,283 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
BIZ-145 / S2 竞品价格采集脚本
|
||||||
|
|
||||||
|
合规红线(法务 BIZ-147 复核意见 C1–C7,全部硬编码,不可被命令行覆盖):
|
||||||
|
C1 频率封顶:单店每周 ≤1 次、单次 ≤10 页面、请求间隔 ≥5 秒
|
||||||
|
C2 遵守 robots.txt:被禁抓即跳过
|
||||||
|
C3 不绕反爬:固定常规 UA、不用 IP 池、遇验证码/登录墙即放弃该页面
|
||||||
|
C4 字段白名单:商品名称 / 页面展示标价 / 页面展示促销信息 / 页面可见销量数
|
||||||
|
C5 数据留存:原始快照 ≤90 天(--prune 清理)
|
||||||
|
C7 不使用代理/账号池:直连、无登录态
|
||||||
|
|
||||||
|
输出:data/YYYY-MM-DD/prices.json + prices.csv + fetch.log
|
||||||
|
失败告警:整体或单店失败时写日志;环境变量 MULTICA_PARENT_ISSUE 存在时
|
||||||
|
通过 multica CLI 在父单留一条失败通知评论。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import csv
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import urllib.error
|
||||||
|
import urllib.parse
|
||||||
|
import urllib.request
|
||||||
|
import urllib.robotparser
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# ---------------- 合规常量(C1–C7,硬编码) ----------------
|
||||||
|
MAX_PAGES_PER_SHOP = 10 # C1
|
||||||
|
MIN_INTERVAL_SEC = 5.0 # C1
|
||||||
|
RETENTION_DAYS = 90 # C5
|
||||||
|
USER_AGENT = ( # C3/C6:常规 UA,不带伪装、附采集说明联系方式
|
||||||
|
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||||
|
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 "
|
||||||
|
"(+internal price monitoring; contact: ops@bizwingsinc.com)"
|
||||||
|
)
|
||||||
|
TIMEOUT_SEC = 20
|
||||||
|
ALLOWED_FIELDS = ("name", "list_price", "promo_text", "sales_display") # C4 白名单
|
||||||
|
|
||||||
|
# 验证码 / 登录墙特征(C3:命中即放弃,不尝试绕过)
|
||||||
|
BLOCK_PATTERNS = [
|
||||||
|
"login.taobao.com", "passport", "验证码", "verify", "captcha",
|
||||||
|
"slide.tsx", "punish", "sec.taobao",
|
||||||
|
]
|
||||||
|
|
||||||
|
BASE_DIR = Path(__file__).resolve().parent
|
||||||
|
DATA_DIR = BASE_DIR / "data"
|
||||||
|
TARGETS_FILE = BASE_DIR / "targets.yaml"
|
||||||
|
CST = timezone(timedelta(hours=8))
|
||||||
|
|
||||||
|
log = logging.getLogger("competitor-price")
|
||||||
|
|
||||||
|
|
||||||
|
def setup_logging(log_file: Path):
|
||||||
|
log.setLevel(logging.INFO)
|
||||||
|
fmt = logging.Formatter("%(asctime)s %(levelname)s %(message)s")
|
||||||
|
sh = logging.StreamHandler(sys.stdout)
|
||||||
|
sh.setFormatter(fmt)
|
||||||
|
fh = logging.FileHandler(log_file, encoding="utf-8")
|
||||||
|
fh.setFormatter(fmt)
|
||||||
|
log.addHandler(sh)
|
||||||
|
log.addHandler(fh)
|
||||||
|
|
||||||
|
|
||||||
|
def load_targets() -> list[dict]:
|
||||||
|
"""极简 YAML 解析(targets.yaml 结构固定,避免引入第三方依赖)。"""
|
||||||
|
text = TARGETS_FILE.read_text(encoding="utf-8")
|
||||||
|
competitors, cur, in_sku, in_monitor = [], None, False, False
|
||||||
|
for raw in text.splitlines():
|
||||||
|
line = raw.split("#", 1)[0].rstrip()
|
||||||
|
if not line.strip():
|
||||||
|
continue
|
||||||
|
stripped = line.strip()
|
||||||
|
if stripped.startswith("- id:"):
|
||||||
|
cur = {"id": stripped.split(":", 1)[1].strip(), "sku_urls": [],
|
||||||
|
"monitor_skus": []}
|
||||||
|
competitors.append(cur)
|
||||||
|
in_sku = in_monitor = False
|
||||||
|
elif cur is None:
|
||||||
|
continue
|
||||||
|
elif re.match(r"^- \w+:", stripped) is None and ":" in stripped and not stripped.startswith("-"):
|
||||||
|
key, _, val = stripped.partition(":")
|
||||||
|
val = val.strip().strip('"')
|
||||||
|
if key == "sku_urls":
|
||||||
|
in_sku, in_monitor = True, False
|
||||||
|
if val.startswith("[") and val != "[]":
|
||||||
|
cur["sku_urls"] = [u.strip().strip('"') for u in val.strip("[]").split(",") if u.strip()]
|
||||||
|
elif key == "monitor_skus":
|
||||||
|
in_monitor, in_sku = True, False
|
||||||
|
else:
|
||||||
|
cur[key] = val
|
||||||
|
in_sku = in_monitor = False
|
||||||
|
elif stripped.startswith("-") and in_sku:
|
||||||
|
cur["sku_urls"].append(stripped.lstrip("- ").strip('"'))
|
||||||
|
elif stripped.startswith("-") and in_monitor:
|
||||||
|
cur["monitor_skus"].append(stripped.lstrip("- ").strip('"'))
|
||||||
|
return competitors
|
||||||
|
|
||||||
|
|
||||||
|
def robots_allows(url: str) -> bool:
|
||||||
|
"""C2:遵守 robots.txt;出错时保守处理为允许但记日志。"""
|
||||||
|
parsed = urllib.parse.urlparse(url)
|
||||||
|
rp_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"
|
||||||
|
rp = urllib.robotparser.RobotFileParser()
|
||||||
|
try:
|
||||||
|
rp.set_url(rp_url)
|
||||||
|
rp.read()
|
||||||
|
return rp.can_fetch(USER_AGENT, url)
|
||||||
|
except Exception as e: # robots 不可达时不放大风险,记录并放行(人工每周低频)
|
||||||
|
log.warning("robots.txt 读取失败 %s: %s(保守放行,人工复核)", rp_url, e)
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_page(url: str) -> tuple[int | None, str]:
|
||||||
|
"""返回 (http_status, html)。不跟随登录跳转以外的异常。"""
|
||||||
|
req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT,
|
||||||
|
"Accept-Language": "zh-CN,zh;q=0.9"})
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=TIMEOUT_SEC) as resp:
|
||||||
|
html = resp.read(2_000_000).decode("utf-8", errors="replace")
|
||||||
|
return resp.status, html
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
return e.code, ""
|
||||||
|
except Exception as e:
|
||||||
|
log.warning("请求异常 %s: %s", url, e)
|
||||||
|
return None, ""
|
||||||
|
|
||||||
|
|
||||||
|
def looks_blocked(html: str, final_status: int | None) -> bool:
|
||||||
|
"""C3:命中反爬/登录墙特征 → 放弃该页面。"""
|
||||||
|
if final_status in (401, 403):
|
||||||
|
return True
|
||||||
|
low = html.lower()
|
||||||
|
return any(p.lower() in low for p in BLOCK_PATTERNS)
|
||||||
|
|
||||||
|
|
||||||
|
PRICE_RE = re.compile(r'"price"\s*:\s*"?([\d.]+)"?|¥\s*([\d]+(?:\.\d+)?)|¥\s*([\d]+(?:\.\d+)?)')
|
||||||
|
PROMO_RE = re.compile(r"(满\d+减\d+|第二件半价|买一送一|买赠|券后)")
|
||||||
|
SALES_RE = re.compile(r"(?:月销|已售|销量)\s*[::]?\s*([\d,]+\+?)")
|
||||||
|
TITLE_RE = re.compile(r"<title>(.*?)</title>", re.S)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_fields(url: str, html: str) -> dict:
|
||||||
|
"""C4:只抽白名单字段,抽不到就留空并在 status 标注。"""
|
||||||
|
out = {f: "" for f in ALLOWED_FIELDS}
|
||||||
|
m = TITLE_RE.search(html)
|
||||||
|
if m:
|
||||||
|
out["name"] = re.sub(r"-(天猫|淘宝网).*$", "", m.group(1)).strip()[:120]
|
||||||
|
m = PRICE_RE.search(html)
|
||||||
|
if m:
|
||||||
|
out["list_price"] = next(g for g in m.groups() if g)
|
||||||
|
promos = PROMO_RE.findall(html)
|
||||||
|
if promos:
|
||||||
|
out["promo_text"] = ";".join(sorted(set(promos))[:5])
|
||||||
|
m = SALES_RE.search(html)
|
||||||
|
if m:
|
||||||
|
out["sales_display"] = m.group(1)
|
||||||
|
return {k: v for k, v in out.items() if k in ALLOWED_FIELDS}
|
||||||
|
|
||||||
|
|
||||||
|
def notify_parent_issue(summary: str):
|
||||||
|
"""失败告警:在父单留评论。无 MULTICA_PARENT_ISSUE 时只写日志。"""
|
||||||
|
issue = os.environ.get("MULTICA_PARENT_ISSUE") or "448cdf6f-ac97-4c7f-953f-0d92d03c6e85"
|
||||||
|
body = f"⚠️ 竞品价格采集失败告警({datetime.now(CST):%Y-%m-%d %H:%M} GMT+8)\n\n{summary}\n\n详见监控目录 data/ 下当日 fetch.log。"
|
||||||
|
try:
|
||||||
|
subprocess.run(["multica", "issue", "comment", "add", issue,
|
||||||
|
"--content", body], check=True, timeout=60,
|
||||||
|
capture_output=True)
|
||||||
|
log.info("已在父单 %s 留言告警", issue)
|
||||||
|
except Exception as e:
|
||||||
|
log.error("父单告警发送失败: %s", e)
|
||||||
|
|
||||||
|
|
||||||
|
def prune_old_snapshots():
|
||||||
|
"""C5:删除 90 天前的采集目录。"""
|
||||||
|
cutoff = datetime.now(CST).date() - timedelta(days=RETENTION_DAYS)
|
||||||
|
for d in DATA_DIR.iterdir() if DATA_DIR.exists() else []:
|
||||||
|
if d.is_dir():
|
||||||
|
try:
|
||||||
|
day = datetime.strptime(d.name, "%Y-%m-%d").date()
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
if day < cutoff:
|
||||||
|
import shutil
|
||||||
|
shutil.rmtree(d)
|
||||||
|
log.info("C5 留存清理:删除 %s", d)
|
||||||
|
|
||||||
|
|
||||||
|
def collect_one(comp: dict, today_dir: Path, results: list, failures: list):
|
||||||
|
urls = [u for u in comp.get("sku_urls", []) if u]
|
||||||
|
urls = urls or ([comp["shop_url"]] if comp.get("shop_url") else [])
|
||||||
|
if not urls:
|
||||||
|
log.warning("%s:无可用 URL(白牌未回填),跳过", comp["name"])
|
||||||
|
results.append({"competitor_id": comp["id"], "competitor": comp["name"],
|
||||||
|
"url": "", "status": "skipped_no_url",
|
||||||
|
**{f: "" for f in ALLOWED_FIELDS}})
|
||||||
|
return
|
||||||
|
urls = urls[:MAX_PAGES_PER_SHOP] # C1
|
||||||
|
for i, url in enumerate(urls):
|
||||||
|
if i > 0:
|
||||||
|
time.sleep(MIN_INTERVAL_SEC) # C1
|
||||||
|
if not robots_allows(url): # C2
|
||||||
|
log.warning("robots.txt 禁止抓取,跳过:%s", url)
|
||||||
|
failures.append(f"{comp['name']} robots 禁止 {url}")
|
||||||
|
results.append({"competitor_id": comp["id"], "competitor": comp["name"],
|
||||||
|
"url": url, "status": "skipped_robots",
|
||||||
|
**{f: "" for f in ALLOWED_FIELDS}})
|
||||||
|
continue
|
||||||
|
status, html = fetch_page(url)
|
||||||
|
if looks_blocked(html, status): # C3
|
||||||
|
log.warning("疑似登录墙/验证码(HTTP %s),放弃:%s", status, url)
|
||||||
|
failures.append(f"{comp['name']} 被反爬拦截 {url} (HTTP {status})")
|
||||||
|
results.append({"competitor_id": comp["id"], "competitor": comp["name"],
|
||||||
|
"url": url, "status": f"blocked_anti_bot_http_{status}",
|
||||||
|
**{f: "" for f in ALLOWED_FIELDS}})
|
||||||
|
continue
|
||||||
|
if not html:
|
||||||
|
failures.append(f"{comp['name']} 抓取失败 {url} (HTTP {status})")
|
||||||
|
results.append({"competitor_id": comp["id"], "competitor": comp["name"],
|
||||||
|
"url": url, "status": f"fetch_failed_http_{status}",
|
||||||
|
**{f: "" for f in ALLOWED_FIELDS}})
|
||||||
|
continue
|
||||||
|
fields = parse_fields(url, html)
|
||||||
|
row = {"competitor_id": comp["id"], "competitor": comp["name"],
|
||||||
|
"url": url, "status": "ok" if fields.get("list_price") else "ok_no_price_field",
|
||||||
|
**fields}
|
||||||
|
results.append(row)
|
||||||
|
log.info("✅ %s %s → %s", comp["name"], url, row["status"])
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
today = datetime.now(CST).strftime("%Y-%m-%d")
|
||||||
|
today_dir = DATA_DIR / today
|
||||||
|
today_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
setup_logging(today_dir / "fetch.log")
|
||||||
|
|
||||||
|
log.info("=== BIZ-145 竞品价格采集启动 %s ===", today)
|
||||||
|
prune_old_snapshots()
|
||||||
|
|
||||||
|
competitors = load_targets()
|
||||||
|
results: list[dict] = []
|
||||||
|
failures: list[str] = []
|
||||||
|
for comp in competitors:
|
||||||
|
collect_one(comp, today_dir, results, failures)
|
||||||
|
time.sleep(MIN_INTERVAL_SEC) # C1:店铺间同样限速
|
||||||
|
|
||||||
|
run_meta = {
|
||||||
|
"run_date": today,
|
||||||
|
"tz": "Asia/Shanghai",
|
||||||
|
"compliance": {"C1_interval_sec": MIN_INTERVAL_SEC,
|
||||||
|
"C1_max_pages_per_shop": MAX_PAGES_PER_SHOP,
|
||||||
|
"C2_robots_respected": True,
|
||||||
|
"C3_user_agent": USER_AGENT,
|
||||||
|
"C4_field_whitelist": list(ALLOWED_FIELDS),
|
||||||
|
"C5_retention_days": RETENTION_DAYS,
|
||||||
|
"C7_no_proxy_no_login": True},
|
||||||
|
"results": results,
|
||||||
|
"failures": failures,
|
||||||
|
}
|
||||||
|
(today_dir / "prices.json").write_text(
|
||||||
|
json.dumps(run_meta, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||||
|
|
||||||
|
with open(today_dir / "prices.csv", "w", newline="", encoding="utf-8-sig") as f:
|
||||||
|
w = csv.DictWriter(f, fieldnames=["competitor_id", "competitor", "url",
|
||||||
|
"status", *ALLOWED_FIELDS])
|
||||||
|
w.writeheader()
|
||||||
|
w.writerows(results)
|
||||||
|
|
||||||
|
log.info("=== 完成:%d 条记录,%d 条失败 ===", len(results), len(failures))
|
||||||
|
if failures:
|
||||||
|
notify_parent_issue("\n".join(f"- {x}" for x in failures))
|
||||||
|
return 1
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# 竞品价格监控目标清单(BIZ-145 / S2)
|
||||||
|
# 数据源:BIZ-144 v2 §1 竞品名单;模板字段:BIZ-144 v2 §3
|
||||||
|
# ⚠️ sku_urls 目前为空 = 仅采集店铺首页公开信息。
|
||||||
|
# 具体 SKU 页 URL 由数据 Owner(陆云帆)每周确认后补充到本文件;
|
||||||
|
# 白牌店铺每月 1 号由执行人按 BIZ-144 v2 §1.1 #5 规则选定后回填 url。
|
||||||
|
|
||||||
|
week_timezone: "Asia/Shanghai"
|
||||||
|
|
||||||
|
competitors:
|
||||||
|
- id: senshan
|
||||||
|
name: 森山(森宇控股)
|
||||||
|
type: brand_flagship
|
||||||
|
shop_url: "https://senshan.tmall.com"
|
||||||
|
monitor_skus: ["铁皮枫斗", "铁皮石斛核心SKU"]
|
||||||
|
sku_urls: [] # TODO(owner): 填入具体 SKU 页面 URL
|
||||||
|
- id: conba
|
||||||
|
name: 康恩贝
|
||||||
|
type: brand_flagship
|
||||||
|
shop_url: "https://conba.tmall.com"
|
||||||
|
monitor_skus: ["石斛/参茸线核心SKU"]
|
||||||
|
sku_urls: []
|
||||||
|
- id: huqingyutang
|
||||||
|
name: 胡庆余堂
|
||||||
|
type: brand_flagship
|
||||||
|
shop_url: "https://huqingyutang.tmall.com"
|
||||||
|
monitor_skus: ["石斛及传统滋补礼盒"]
|
||||||
|
sku_urls: []
|
||||||
|
- id: shouxiangu
|
||||||
|
name: 寿仙谷
|
||||||
|
type: brand_flagship
|
||||||
|
shop_url: "https://shouxian.tmall.com"
|
||||||
|
monitor_skus: ["铁皮石斛", "灵芝孢子粉"]
|
||||||
|
sku_urls: []
|
||||||
|
- id: white_label_monthly
|
||||||
|
name: 平台头部白牌(每月轮换)
|
||||||
|
type: marketplace
|
||||||
|
shop_url: "" # 每月 1 号由执行人回填当月店铺 URL
|
||||||
|
monitor_skus: ["同规格散装石斛价格带下限"]
|
||||||
|
sku_urls: []
|
||||||
@@ -0,0 +1,170 @@
|
|||||||
|
-- ============================================================
|
||||||
|
-- BIZ-127 方案 C 完整用户系统 —— 数据模型 SQL DDL
|
||||||
|
-- ============================================================
|
||||||
|
-- 设计人:梁思筑(architect)| 2026-09-04
|
||||||
|
-- 适用版本:lottoData v2.0(Flask + SQLite)
|
||||||
|
-- 数据库:SQLite 3.x(与现有 lottery.db 同库同文件,复用 DB 连接)
|
||||||
|
-- 兼容性:当前 SQLite 文件存储(.generation_records.json)将迁移至本 schema
|
||||||
|
-- 字符集:UTF-8
|
||||||
|
-- 加密策略:bcrypt(cost=12) 密码哈希、session 哈希、不存任何明文凭证
|
||||||
|
-- ============================================================
|
||||||
|
|
||||||
|
-- ------------------------------------------------------------
|
||||||
|
-- 1. users —— 用户表
|
||||||
|
-- ------------------------------------------------------------
|
||||||
|
CREATE TABLE IF NOT EXISTS users (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
username TEXT NOT NULL UNIQUE,
|
||||||
|
password_hash TEXT NOT NULL, -- bcrypt cost=12 哈希
|
||||||
|
email TEXT, -- 可选,找回密码用
|
||||||
|
role TEXT NOT NULL DEFAULT 'user' -- 预留:admin / user
|
||||||
|
CHECK (role IN ('user', 'admin')),
|
||||||
|
status TEXT NOT NULL DEFAULT 'active' -- active / locked / disabled
|
||||||
|
CHECK (status IN ('active', 'locked', 'disabled')),
|
||||||
|
tenant_id INTEGER NOT NULL DEFAULT 1, -- 多租户预留,单租户默认 1
|
||||||
|
failed_attempts INTEGER NOT NULL DEFAULT 0, -- 登录失败计数(达到 5 锁定)
|
||||||
|
locked_until INTEGER, -- Unix 时间戳,NULL=未锁
|
||||||
|
password_changed_at INTEGER NOT NULL, -- 用于旧 session 失效判断
|
||||||
|
last_login_at INTEGER, -- 最近登录时间
|
||||||
|
created_at INTEGER NOT NULL,
|
||||||
|
updated_at INTEGER NOT NULL,
|
||||||
|
-- 多用户扩展索引
|
||||||
|
CONSTRAINT idx_users_username UNIQUE (username),
|
||||||
|
CONSTRAINT idx_users_tenant UNIQUE (username, tenant_id)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_users_role_status ON users(role, status);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_users_tenant ON users(tenant_id);
|
||||||
|
|
||||||
|
-- ------------------------------------------------------------
|
||||||
|
-- 2. sessions —— 会话表(服务端 session,cookie 仅存 session_id)
|
||||||
|
-- ------------------------------------------------------------
|
||||||
|
-- 决策:DB 存 session(不用 JWT),理由:
|
||||||
|
-- - 可控失效(改密码立即清空)
|
||||||
|
-- - 可主动踢人
|
||||||
|
-- - 内网系统不需分布式无状态
|
||||||
|
CREATE TABLE IF NOT EXISTS sessions (
|
||||||
|
id TEXT PRIMARY KEY, -- 32 字节 URL-safe base64
|
||||||
|
user_id INTEGER NOT NULL,
|
||||||
|
session_hash TEXT NOT NULL, -- sha256(session_id + server_secret)
|
||||||
|
user_agent TEXT, -- 浏览器/客户端 UA
|
||||||
|
ip_address TEXT, -- 登录 IP(内网)
|
||||||
|
password_epoch INTEGER NOT NULL, -- 登录时 users.password_changed_at 快照
|
||||||
|
expires_at INTEGER NOT NULL, -- Unix 时间戳(30 天)
|
||||||
|
created_at INTEGER NOT NULL,
|
||||||
|
last_seen_at INTEGER NOT NULL, -- 用于过期清理
|
||||||
|
revoked_at INTEGER, -- NULL=有效,非 NULL=已撤销
|
||||||
|
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_sessions_user ON sessions(user_id);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_sessions_expires ON sessions(expires_at);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_sessions_active ON sessions(user_id, revoked_at, expires_at);
|
||||||
|
|
||||||
|
-- ------------------------------------------------------------
|
||||||
|
-- 3. password_reset_tokens —— 密码重置 token 表
|
||||||
|
-- ------------------------------------------------------------
|
||||||
|
-- 决策:DB 存一次性 token(不用 JWT 自包含),理由:
|
||||||
|
-- - 一次性、可控、可撤销
|
||||||
|
-- - 速率限制基于 DB 计数
|
||||||
|
CREATE TABLE IF NOT EXISTS password_reset_tokens (
|
||||||
|
token_hash TEXT PRIMARY KEY, -- sha256(明文 token)
|
||||||
|
user_id INTEGER NOT NULL,
|
||||||
|
expires_at INTEGER NOT NULL, -- 1 小时过期
|
||||||
|
used_at INTEGER, -- NULL=未用
|
||||||
|
ip_address TEXT, -- 请求来源 IP
|
||||||
|
created_at INTEGER NOT NULL,
|
||||||
|
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_pwd_reset_user ON password_reset_tokens(user_id);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_pwd_reset_expires ON password_reset_tokens(expires_at);
|
||||||
|
|
||||||
|
-- ------------------------------------------------------------
|
||||||
|
-- 4. operation_logs —— 操作日志表
|
||||||
|
-- ------------------------------------------------------------
|
||||||
|
-- 用途:审计、追踪、异常告警
|
||||||
|
-- 保留周期:90 天(按月度手动归档)
|
||||||
|
-- 未来扩展:按 tenant_id 分区(本次单租户,分区预留)
|
||||||
|
CREATE TABLE IF NOT EXISTS operation_logs (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
user_id INTEGER, -- 公开操作可 NULL
|
||||||
|
tenant_id INTEGER NOT NULL DEFAULT 1, -- 多租户预留
|
||||||
|
username TEXT, -- 冗余存储,避免 JOIN
|
||||||
|
action TEXT NOT NULL, -- login / logout / generate / delete / reset_request ...
|
||||||
|
resource_type TEXT, -- record / config / user ...
|
||||||
|
resource_id TEXT, -- 关联 ID
|
||||||
|
ip_address TEXT,
|
||||||
|
user_agent TEXT,
|
||||||
|
status TEXT NOT NULL DEFAULT 'success' -- success / failure
|
||||||
|
CHECK (status IN ('success', 'failure')),
|
||||||
|
detail TEXT, -- JSON 字符串,存额外上下文
|
||||||
|
created_at INTEGER NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_oplog_user_time ON operation_logs(user_id, created_at);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_oplog_tenant_time ON operation_logs(tenant_id, created_at);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_oplog_action ON operation_logs(action, created_at);
|
||||||
|
|
||||||
|
-- ------------------------------------------------------------
|
||||||
|
-- 5. auth_settings —— 全局鉴权配置(单行配置表)
|
||||||
|
-- ------------------------------------------------------------
|
||||||
|
-- 用途:运行时调整鉴权策略,不重启服务
|
||||||
|
CREATE TABLE IF NOT EXISTS auth_settings (
|
||||||
|
id INTEGER PRIMARY KEY CHECK (id = 1), -- 仅允许 id=1
|
||||||
|
bcrypt_cost INTEGER NOT NULL DEFAULT 12, -- 12-14
|
||||||
|
session_max_age_sec INTEGER NOT NULL DEFAULT 2592000, -- 30 天
|
||||||
|
session_renew_sec INTEGER NOT NULL DEFAULT 86400, -- 24h 内活跃则续期
|
||||||
|
max_failed_attempts INTEGER NOT NULL DEFAULT 5,
|
||||||
|
lock_duration_sec INTEGER NOT NULL DEFAULT 900, -- 15 分钟
|
||||||
|
pwd_reset_ttl_sec INTEGER NOT NULL DEFAULT 3600, -- 1 小时
|
||||||
|
pwd_reset_rate_limit INTEGER NOT NULL DEFAULT 3, -- 每用户每小时最多 3 次
|
||||||
|
log_retention_days INTEGER NOT NULL DEFAULT 90,
|
||||||
|
maintenance_mode INTEGER NOT NULL DEFAULT 0, -- 0=正常 1=维护
|
||||||
|
updated_at INTEGER NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
-- 初始化默认配置(仅在表为空时插入)
|
||||||
|
INSERT OR IGNORE INTO auth_settings (id, updated_at)
|
||||||
|
VALUES (1, strftime('%s', 'now'));
|
||||||
|
|
||||||
|
-- ============================================================
|
||||||
|
-- 二、迁移脚本(从当前 JSON 文件存储 + Bearer-only 模式)
|
||||||
|
-- ============================================================
|
||||||
|
-- 迁移目标:
|
||||||
|
-- 1. 初始化 users 表,创建默认管理员账号
|
||||||
|
-- 2. 现有 .generation_records.json 数据保留,不迁移(无 user_id 字段)
|
||||||
|
-- 3. operation_logs 表从空开始
|
||||||
|
-- ============================================================
|
||||||
|
|
||||||
|
-- 迁移步骤 1:创建默认管理员账号
|
||||||
|
-- 默认用户名:lotto_admin
|
||||||
|
-- 默认密码:BIZ-127-Lottery-2026(首次登录后强制修改)
|
||||||
|
-- 警告:此密码必须在首次登录后立即修改(通过 BIZ-129 强制流程)
|
||||||
|
INSERT OR IGNORE INTO users (
|
||||||
|
username, password_hash, email, role, status,
|
||||||
|
tenant_id, failed_attempts, password_changed_at,
|
||||||
|
created_at, updated_at
|
||||||
|
) VALUES (
|
||||||
|
'lotto_admin',
|
||||||
|
-- bcrypt('BIZ-127-Lottery-2026', cost=12)
|
||||||
|
-- 由 BIZ-129 实施时用 Python bcrypt 生成,SQL 仅作占位
|
||||||
|
'__BCRYPT_PLACEHOLDER__',
|
||||||
|
NULL, 'admin', 'active', 1, 0,
|
||||||
|
strftime('%s', 'now'), strftime('%s', 'now'), strftime('%s', 'now')
|
||||||
|
);
|
||||||
|
|
||||||
|
-- 迁移步骤 2:保留 .generation_records.json 数据
|
||||||
|
-- 策略:不迁移到 operation_logs(records 是业务数据,不是审计日志)
|
||||||
|
-- 备注:BIZ-129 需在 add_record() 中写入 operation_logs(action='generate')
|
||||||
|
|
||||||
|
-- 迁移步骤 3:环境变量兼容
|
||||||
|
-- 保留 LOTTO_API_TOKEN(Bearer 备用通道,仅供内部服务调用)
|
||||||
|
-- 用户系统上线后,外部前端用 cookie,内部脚本用 Bearer
|
||||||
|
|
||||||
|
-- ============================================================
|
||||||
|
-- 三、清理脚本(可选,生产环境 90 天后执行)
|
||||||
|
-- ============================================================
|
||||||
|
-- DELETE FROM sessions WHERE expires_at < strftime('%s', 'now');
|
||||||
|
-- DELETE FROM password_reset_tokens WHERE expires_at < strftime('%s', 'now');
|
||||||
|
-- DELETE FROM operation_logs WHERE created_at < strftime('%s', 'now', '-90 days');
|
||||||
@@ -0,0 +1,240 @@
|
|||||||
|
# BIZ-127 完整用户系统架构设计 v1.0
|
||||||
|
|
||||||
|
| 项 | 内容 |
|
||||||
|
|---|---|
|
||||||
|
| **版本** | v1.0 |
|
||||||
|
| **作者** | 梁思筑(architect) |
|
||||||
|
| **日期** | 2026-09-04 |
|
||||||
|
| **关联卡片** | BIZ-127(架构)→ BIZ-129(实施) |
|
||||||
|
| **父卡片** | BIZ-109(P0 修复闭环) |
|
||||||
|
| **关联 PRD** | BIZ-111(产品 PRD) |
|
||||||
|
| **存储位置** | docs/BIZ-127_完整用户系统架构_v1.0.md |
|
||||||
|
| **同步位置** | 思源知识库 `/standards/完整用户系统架构 v1.0` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 一、设计目标
|
||||||
|
|
||||||
|
实现完整的账号密码登录体系,支持长期免登录(30 天),账号密码 + Session Cookie 方案,不在 URL/前端暴露 token。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 二、数据模型
|
||||||
|
|
||||||
|
**完整 SQL DDL** 见 `docs/auth-schema.sql`(9.7KB)。
|
||||||
|
|
||||||
|
包含 5 张表:
|
||||||
|
1. **users** —— 用户表(bcrypt 密码哈希、登录失败计数、锁定时间戳、多租户/角色预留)
|
||||||
|
2. **sessions** —— 服务端会话(DB 存,非 JWT;存 session_hash、password_epoch 用于改密失效)
|
||||||
|
3. **password_reset_tokens** —— 密码重置 token(一次性、1 小时过期、限速 3 次/小时)
|
||||||
|
4. **operation_logs** —— 操作日志(90 天保留、多租户索引预留)
|
||||||
|
5. **auth_settings** —— 全局鉴权配置(bcrypt cost、session 寿命、锁定参数)
|
||||||
|
|
||||||
|
### 关键决策
|
||||||
|
|
||||||
|
| 决策点 | 选择 | 理由 |
|
||||||
|
|--------|------|------|
|
||||||
|
| Session 存储 | **DB 存** | 可控失效、改密立即踢人;JWT 不适合内网场景 |
|
||||||
|
| 重置 token 存储 | **DB 存** | 一次性、可控、可限速;JWT 自包含不可撤销 |
|
||||||
|
| 密码哈希 | **bcrypt cost=12** | 行业标准,登录耗时 ~250ms,安全/性能平衡 |
|
||||||
|
| 多用户扩展 | **数据层预留 + UI 不暴露** | 见下文"多用户扩展决策" |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 三、鉴权中间件流程图
|
||||||
|
|
||||||
|
```
|
||||||
|
请求进入
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌──────────────────────┐
|
||||||
|
│ 日志中间件 │ ← 记录 method/path/IP/UA
|
||||||
|
│ (log_request) │
|
||||||
|
└──────────┬───────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌──────────────────────┐
|
||||||
|
│ 鉴权中间件 │ ← 优先读 cookie,fallback Bearer
|
||||||
|
│ (authenticate) │ Bearer 仅用于内部服务调用
|
||||||
|
└──────────┬───────────┘
|
||||||
|
│
|
||||||
|
│ 判断:path 是否在白名单?
|
||||||
|
│
|
||||||
|
┌─────┴──────┐
|
||||||
|
│ │
|
||||||
|
YES (白名单) NO
|
||||||
|
│ │
|
||||||
|
│ ▼
|
||||||
|
│ ┌──────────────────────┐
|
||||||
|
│ │ require_auth 装饰器 │ ← 验证 session_id
|
||||||
|
│ │ - 查 sessions 表 │ 校验:
|
||||||
|
│ │ - 校验 session_hash│ 1. session 存在
|
||||||
|
│ │ - 校验 expires_at │ 2. 未撤销
|
||||||
|
│ │ - 校验密码 epoch │ 3. 未过期
|
||||||
|
│ │ - 续期(如活跃) │ 4. password_epoch 匹配
|
||||||
|
│ └──────────┬───────────┘
|
||||||
|
│ │
|
||||||
|
│ ▼
|
||||||
|
│ ┌──────────────────────┐
|
||||||
|
│ │ 业务中间件 │
|
||||||
|
│ │ (业务逻辑处理) │
|
||||||
|
│ └──────────┬───────────┘
|
||||||
|
│ │
|
||||||
|
│ ▼
|
||||||
|
│ ┌──────────────────────┐
|
||||||
|
│ │ 日志中间件(出口) │ ← 记录 status/duration/user_id
|
||||||
|
│ └──────────────────────┘
|
||||||
|
│
|
||||||
|
└──▶ 业务中间件(白名单接口无需鉴权)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5 个鉴权端点
|
||||||
|
|
||||||
|
| 端点 | 方法 | 说明 |
|
||||||
|
|------|------|------|
|
||||||
|
| `/api/login` | POST | 账号密码 → 校验 → 创建 session → 设置 cookie |
|
||||||
|
| `/api/logout` | POST | 清 cookie → 撤销 session(revoked_at=now) |
|
||||||
|
| `/api/password-reset/request` | POST | 生成 token → 邮件/日志通知(内网暂用日志) |
|
||||||
|
| `/api/password-reset/confirm` | POST | 校验 token → 改密码 → 撤销该用户所有 session |
|
||||||
|
|
||||||
|
### 装饰器改造
|
||||||
|
|
||||||
|
```python
|
||||||
|
# 现状
|
||||||
|
@require_auth
|
||||||
|
def api_records(): ...
|
||||||
|
|
||||||
|
# 改造后(伪代码,BIZ-129 实施)
|
||||||
|
@require_auth # 内部自动读 cookie 或 Bearer
|
||||||
|
def api_records(): ...
|
||||||
|
|
||||||
|
# 新增装饰器
|
||||||
|
@require_admin # role == 'admin' 才允许
|
||||||
|
def admin_only_api(): ...
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 四、Cookie 安全策略
|
||||||
|
|
||||||
|
| 项 | 值 | 决策依据 |
|
||||||
|
|---|----|---------|
|
||||||
|
| **名称** | `lotto_session` | 业务前缀,避免与其它服务冲突 |
|
||||||
|
| **httpOnly** | `true` | ✅ 阻止 XSS 读取 cookie |
|
||||||
|
| **secure** | `true`(生产) / `false`(内网 HTTP 调试) | 内网暂不强制 HTTPS,但配置项保留 |
|
||||||
|
| **sameSite** | `Lax` | ✅ 防 CSRF;`Strict` 会破坏外链跳转登录 |
|
||||||
|
| **path** | `/` | 全站生效 |
|
||||||
|
| **maxAge** | `30 * 24 * 3600` 秒(30 天) | 满足"长期免登录"诉求 |
|
||||||
|
| **domain** | 当前域(不设) | 内网单域,不跨子域 |
|
||||||
|
|
||||||
|
### 防 CSRF 决策
|
||||||
|
|
||||||
|
**不单独引入 CSRF token**,理由:
|
||||||
|
- `SameSite=Lax` 已防 95% CSRF(POST 表单跨站不携带 cookie)
|
||||||
|
- 关键操作(删除、改密码)二次校验(要求输入当前密码或确认)
|
||||||
|
- 如未来扩展第三方嵌入或 iframe 调用,再加 CSRF token
|
||||||
|
|
||||||
|
### Session 续期策略
|
||||||
|
|
||||||
|
- 用户每次活跃请求,若 `expires_at - now < 24h`,则续期 30 天
|
||||||
|
- `last_seen_at` 用于清理过期 session
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 五、多用户扩展决策(4 个判断点)
|
||||||
|
|
||||||
|
| 判断点 | 是否预留 | 决策 |
|
||||||
|
|--------|---------|------|
|
||||||
|
| 1. 数据库 `users.tenant_id` / `role` 字段 | **✅ 预留** | 已建字段,索引就绪,单租户默认 1 |
|
||||||
|
| 2. API 路径 `/api/admin/users/*` | **❌ 不预留** | 过度设计;未来需要时再加,无需现在占坑 |
|
||||||
|
| 3. 前端「多用户切换」入口 | **❌ 不预留** | UI 保持单用户登录,多用户场景届时再迭代 |
|
||||||
|
| 4. 操作日志按 `tenant_id` 分区 | **✅ 预留索引** | 已建 `idx_oplog_tenant_time`,分区待数据量>10万再分表 |
|
||||||
|
|
||||||
|
**预留原则**:数据库层 vs API/UI 层
|
||||||
|
- **数据层**:必须预留(成本低、不影响当前 UI)
|
||||||
|
- **API/UI 层**:不预留(避免过度设计、未来按需添加)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 六、兼容性矩阵
|
||||||
|
|
||||||
|
| 项 | 当前实现 | 方案 C 后 | 迁移路径 |
|
||||||
|
|----|---------|----------|---------|
|
||||||
|
| `/api/status` | 无鉴权 | 保持无鉴权(status 公开) | 无变更 |
|
||||||
|
| `/api/records` | Bearer 鉴权 | Cookie 鉴权 + Bearer 备用 | 双轨运行 24h,前端切到 cookie 后撤 Bearer |
|
||||||
|
| `/api/history` | Bearer 鉴权 | Cookie 鉴权 + Bearer 备用 | 同上 |
|
||||||
|
| `/api/generate` | Bearer 鉴权 | Cookie 鉴权 + Bearer 备用 | 同上 |
|
||||||
|
| `/api/statistics` | Bearer 鉴权 | Cookie 鉴权 + Bearer 备用 | 同上 |
|
||||||
|
| `/api/download/*` | Bearer 鉴权 | Cookie 鉴权 + Bearer 备用 | 同上 |
|
||||||
|
| Bearer token | `LOTTO_API_TOKEN` env 强制 | 保留 env(内部脚本用) | 不删除,作为内部服务通道 |
|
||||||
|
| Cookie | 无 | `lotto_session` 30 天 | 新增 |
|
||||||
|
| 默认账号 | 无 | `lotto_admin` + 临时密码 | 首次登录强制改密 |
|
||||||
|
|
||||||
|
**双轨过渡策略**:
|
||||||
|
1. 上线 Cookie 鉴权后,前端自动改用 cookie
|
||||||
|
2. Bearer 通道保留 24h 给内部脚本迁移
|
||||||
|
3. 24h 后仅允许 Bearer 用于 `/api/internal/*`(如有)
|
||||||
|
4. 监控 `/api/records` 等接口的鉴权失败日志,确认为 0 后撤 Bearer
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 七、性能 + 安全边界(5 项决策)
|
||||||
|
|
||||||
|
| 项 | 决策 | 理由 |
|
||||||
|
|----|------|------|
|
||||||
|
| 1. bcrypt cost=12 登录耗时 | **目标 < 500ms(实测 ~250ms)** | Ubuntu 22.04 + bcrypt cost=12 单次 ~250ms,远低于 500ms 上限 |
|
||||||
|
| 2. Session 失效机制(改密) | **改密码后该用户所有 session 立即失效** | session 表 `password_epoch` 字段;登录时快照,改密后 `password_changed_at` 更新,session 自动失效 |
|
||||||
|
| 3. 找回密码 token 速率限制 | **每用户 3 次/小时** | 防爆破;超限返回 429 |
|
||||||
|
| 4. 登录失败锁定 | **5 次失败锁定 15 分钟** | `users.failed_attempts` + `locked_until`;自动解锁 |
|
||||||
|
| 5. 操作日志保留周期 | **90 天** | 满足审计需求;月度归档到 `operation_logs_archive_yyyyMM`;>10万行/天时启动分区 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 八、验收 Checklist
|
||||||
|
|
||||||
|
- [x] 完整 SQL DDL(含迁移脚本) → `docs/auth-schema.sql`
|
||||||
|
- [x] 鉴权中间件流程图(文字描述) → 第三节
|
||||||
|
- [x] Cookie 安全策略完整 → 第四节
|
||||||
|
- [x] 多用户扩展接口明确 → 第五节
|
||||||
|
- [x] 兼容性矩阵 → 第六节
|
||||||
|
- [x] 性能 + 安全边界 → 第七节
|
||||||
|
- [x] 架构文档沉淀(思源 + Git) → 同步进行中
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 九、对下游约束(PRD BIZ-111 + 实施 BIZ-129)
|
||||||
|
|
||||||
|
### 给 BIZ-111 PRD 的约束
|
||||||
|
|
||||||
|
1. **登录页**必须有:用户名、密码、"记住我"勾选(默认勾选=30天免登录)
|
||||||
|
2. **找回密码流程**:输入用户名 → 后台生成 token → 通过日志/邮件通知(内网暂用日志)
|
||||||
|
3. **首次登录强制改密**:检测 `password_changed_at == created_at` 时,强制跳改密页
|
||||||
|
4. **UI 不暴露**多用户切换、注册入口(保持单账号)
|
||||||
|
|
||||||
|
### 给 BIZ-129 实施的约束
|
||||||
|
|
||||||
|
1. **必须用 bcrypt cost=12**,不要用 sha256/md5/pbkdf2
|
||||||
|
2. **session_id 必须 32 字节 URL-safe base64**(`secrets.token_urlsafe(32)`)
|
||||||
|
3. **session_hash = sha256(session_id + server_secret)**,secret 存 env
|
||||||
|
4. **不改 LOTTO_API_TOKEN 流程**,保留 Bearer 作为内部通道
|
||||||
|
5. **/api/status 必须保持公开**,不能误加鉴权
|
||||||
|
6. **BIZ-109 严格鉴权基线不能回退**(不能因为 Cookie 引入新的 401 路径)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 十、风险与待澄清
|
||||||
|
|
||||||
|
| 风险 | 影响 | 应对 |
|
||||||
|
|------|------|------|
|
||||||
|
| 首次部署 bcrypt 哈希生成失败 | 登录失败 | BIZ-129 部署前在本地预生成哈希嵌入 init SQL |
|
||||||
|
| Cookie 30 天过长,安全边界争议 | 安全 | `auth_settings.session_max_age_sec` 运行时可调 |
|
||||||
|
| 多用户预留字段被误用 | 数据污染 | 在 BIZ-129 代码层加 `tenant_id == 1` 断言 |
|
||||||
|
| 操作日志 90 天后未归档 | 磁盘膨胀 | BIZ-129 实施时同时交付归档脚本 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 十一、变更历史
|
||||||
|
|
||||||
|
| 版本 | 日期 | 变更 | 作者 |
|
||||||
|
|------|------|------|------|
|
||||||
|
| v1.0 | 2026-09-04 | 初版(BIZ-127 架构交付) | 梁思筑 |
|
||||||
Reference in New Issue
Block a user