Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 vincent
|
93e8a1011b | ||
|
vincent
|
6b5f53a0fd |
@@ -1,234 +0,0 @@
|
||||
# HEARTBEAT.md - 梁思筑（architect）的心跳配置
|
||||
|
||||
> 模板版本：v1.1 (BIZ-24) | 分类：开发 Agent
|
||||
> OpenClaw Agent ID: `architect` | Multica Agent UUID: `40abd41a-62d0-416d-bc44-92c1f758d87a`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级：OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'architect' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级：Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 40abd41a-62d0-416d-bc44-92c1f758d87a --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级：待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片（agentId=architect）
|
||||
↓
|
||||
检查 Multica Issues 待办（assignee=40abd41a-62d0-416d-bc44-92c1f758d87a）
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测：
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO + 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO + 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（2h）→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖，可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖，可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：100 轮
|
||||
|
||||
- 接近 80%（80 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知 COO + 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**：通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**：通过 issue comment 记录轮次进度
|
||||
- **待办文档**：在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 架构设计进度
|
||||
6. ✅ 技术方案评审状态
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 梁思筑（architect）专用配置
|
||||
@@ -1,234 +0,0 @@
|
||||
# HEARTBEAT.md - 文墨言（contentspecialist）的心跳配置
|
||||
|
||||
> 模板版本：v1.1 (BIZ-24) | 分类：业务 Agent
|
||||
> OpenClaw Agent ID: `contentspecialist` | Multica Agent UUID: `8321b0bf-7d89-4ece-927a-0780f42ad396`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级：OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'contentspecialist' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级：Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 8321b0bf-7d89-4ece-927a-0780f42ad396 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级：待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片（agentId=contentspecialist）
|
||||
↓
|
||||
检查 Multica Issues 待办（assignee=8321b0bf-7d89-4ece-927a-0780f42ad396）
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测：
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（2h）→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖，可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖，可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：30 轮
|
||||
|
||||
- 接近 80%（24 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**：通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**：通过 issue comment 记录轮次进度
|
||||
- **待办文档**：在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 内容发布计划
|
||||
6. ✅ 素材准备状态
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 文墨言（contentspecialist）专用配置
|
||||
@@ -1,235 +0,0 @@
|
||||
# HEARTBEAT.md - 陆怀瑾（coo）的心跳配置
|
||||
|
||||
> 模板版本：v1.1 (BIZ-24) | 分类：高频 Agent
|
||||
> OpenClaw Agent ID: `coo` | Multica Agent UUID: `1c38b437-b54d-4784-bda3-29ce4c8a6722`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级：OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'coo' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级：Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 1c38b437-b54d-4784-bda3-29ce4c8a6722 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级：待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片（agentId=coo）
|
||||
↓
|
||||
检查 Multica Issues 待办（assignee=1c38b437-b54d-4784-bda3-29ce4c8a6722）
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：10 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测：
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 20 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1200:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1200:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 30 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO（自我监控） |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO（自我监控） |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（1h）→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖，可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖，可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：50 轮
|
||||
|
||||
- 接近 80%（40 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知 COO（自我监控）
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**：通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**：通过 issue comment 记录轮次进度
|
||||
- **待办文档**：在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ **全平台积压巡检**：WorkBoard + Multica 全局待办数
|
||||
6. ✅ 资源负载均衡检查
|
||||
7. ✅ 风险识别与预警
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 陆怀瑾（coo）专用配置
|
||||
@@ -1,234 +0,0 @@
|
||||
# HEARTBEAT.md - 徐聪（costcodev）的心跳配置
|
||||
|
||||
> 模板版本：v1.1 (BIZ-24) | 分类：开发 Agent
|
||||
> OpenClaw Agent ID: `costcodev` | Multica Agent UUID: `46bdd4a6-5c64-475a-92ef-36a763602fa1`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级：OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'costcodev' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级：Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 46bdd4a6-5c64-475a-92ef-36a763602fa1 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级：待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片（agentId=costcodev）
|
||||
↓
|
||||
检查 Multica Issues 待办（assignee=46bdd4a6-5c64-475a-92ef-36a763602fa1）
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测：
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO + 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO + 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（2h）→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖，可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖，可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：100 轮
|
||||
|
||||
- 接近 80%（80 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知 COO + 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**：通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**：通过 issue comment 记录轮次进度
|
||||
- **待办文档**：在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 代码开发进度
|
||||
6. ✅ PR/Code Review 状态
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 徐聪（costcodev）专用配置
|
||||
@@ -1,234 +0,0 @@
|
||||
# HEARTBEAT.md - 程伯予（cvexpert）的心跳配置
|
||||
|
||||
> 模板版本：v1.1 (BIZ-24) | 分类：业务 Agent
|
||||
> OpenClaw Agent ID: `cvexpert` | Multica Agent UUID: `4a8696fd-6531-40da-8956-ef84d7ea3c43`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级：OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'cvexpert' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级：Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 4a8696fd-6531-40da-8956-ef84d7ea3c43 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级：待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片（agentId=cvexpert）
|
||||
↓
|
||||
检查 Multica Issues 待办（assignee=4a8696fd-6531-40da-8956-ef84d7ea3c43）
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测：
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（2h）→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖，可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖，可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：30 轮
|
||||
|
||||
- 接近 80%（24 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**：通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**：通过 issue comment 记录轮次进度
|
||||
- **待办文档**：在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 求职服务队列
|
||||
6. ✅ 客户反馈跟踪
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 程伯予（cvexpert）专用配置
|
||||
@@ -1,234 +0,0 @@
|
||||
# HEARTBEAT.md - 苏锦绘（designer）的心跳配置
|
||||
|
||||
> 模板版本：v1.1 (BIZ-24) | 分类：开发 Agent
|
||||
> OpenClaw Agent ID: `designer` | Multica Agent UUID: `13bd8968-cc2a-4934-90c7-957a2d3c09c2`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级：OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'designer' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级：Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 13bd8968-cc2a-4934-90c7-957a2d3c09c2 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级：待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片（agentId=designer）
|
||||
↓
|
||||
检查 Multica Issues 待办（assignee=13bd8968-cc2a-4934-90c7-957a2d3c09c2）
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测：
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO + 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO + 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（2h）→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖，可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖，可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：100 轮
|
||||
|
||||
- 接近 80%（80 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知 COO + 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**：通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**：通过 issue comment 记录轮次进度
|
||||
- **待办文档**：在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 设计稿进度
|
||||
6. ✅ UI/UX 评审状态
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 苏锦绘（designer）专用配置
|
||||
@@ -1,234 +0,0 @@
|
||||
# HEARTBEAT.md - 苏慎（lawyer）的心跳配置
|
||||
|
||||
> 模板版本：v1.1 (BIZ-24) | 分类：业务 Agent
|
||||
> OpenClaw Agent ID: `lawyer` | Multica Agent UUID: `6fb0fbd2-16a6-4566-ba7a-d2c136baec25`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级：OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'lawyer' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级：Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 6fb0fbd2-16a6-4566-ba7a-d2c136baec25 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级：待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片（agentId=lawyer）
|
||||
↓
|
||||
检查 Multica Issues 待办（assignee=6fb0fbd2-16a6-4566-ba7a-d2c136baec25）
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测：
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（2h）→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖，可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖，可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：30 轮
|
||||
|
||||
- 接近 80%（24 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**：通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**：通过 issue comment 记录轮次进度
|
||||
- **待办文档**：在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 合同审查队列
|
||||
6. ✅ 合规检查项
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 苏慎（lawyer）专用配置
|
||||
@@ -1,234 +0,0 @@
|
||||
# HEARTBEAT.md - 顾析策（marketanalysis）的心跳配置
|
||||
|
||||
> 模板版本：v1.1 (BIZ-24) | 分类：业务 Agent
|
||||
> OpenClaw Agent ID: `marketanalysis` | Multica Agent UUID: `5ed91729-658f-4654-98f0-3e0313022002`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级：OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'marketanalysis' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级：Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 5ed91729-658f-4654-98f0-3e0313022002 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级：待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片（agentId=marketanalysis）
|
||||
↓
|
||||
检查 Multica Issues 待办（assignee=5ed91729-658f-4654-98f0-3e0313022002）
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测：
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（2h）→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖，可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖，可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：30 轮
|
||||
|
||||
- 接近 80%（24 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**：通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**：通过 issue comment 记录轮次进度
|
||||
- **待办文档**：在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 市场分析任务
|
||||
6. ✅ 竞品数据更新
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 顾析策（marketanalysis）专用配置
|
||||
@@ -1,234 +0,0 @@
|
||||
# HEARTBEAT.md - 钟帧韵（mediaspecialist）的心跳配置
|
||||
|
||||
> 模板版本：v1.1 (BIZ-24) | 分类：业务 Agent
|
||||
> OpenClaw Agent ID: `mediaspecialist` | Multica Agent UUID: `e2b587d4-1d16-447c-8ad9-e2a01358ff0a`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级：OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'mediaspecialist' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级：Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id e2b587d4-1d16-447c-8ad9-e2a01358ff0a --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级：待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片（agentId=mediaspecialist）
|
||||
↓
|
||||
检查 Multica Issues 待办（assignee=e2b587d4-1d16-447c-8ad9-e2a01358ff0a）
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测：
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（2h）→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖，可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖，可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：30 轮
|
||||
|
||||
- 接近 80%（24 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**：通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**：通过 issue comment 记录轮次进度
|
||||
- **待办文档**：在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 视频制作进度
|
||||
6. ✅ 媒体素材准备状态
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 钟帧韵（mediaspecialist）专用配置
|
||||
@@ -1,234 +0,0 @@
|
||||
# HEARTBEAT.md - 严维序（opengineer）的心跳配置
|
||||
|
||||
> 模板版本：v1.1 (BIZ-24) | 分类：开发 Agent
|
||||
> OpenClaw Agent ID: `opengineer` | Multica Agent UUID: `d3804433-9e2e-4199-a92b-a153049b3bc9`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级：OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'opengineer' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级：Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id d3804433-9e2e-4199-a92b-a153049b3bc9 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级：待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片（agentId=opengineer）
|
||||
↓
|
||||
检查 Multica Issues 待办（assignee=d3804433-9e2e-4199-a92b-a153049b3bc9）
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测：
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO + 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO + 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（2h）→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖，可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖，可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：100 轮
|
||||
|
||||
- 接近 80%（80 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知 COO + 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**：通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**：通过 issue comment 记录轮次进度
|
||||
- **待办文档**：在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 部署状态检查
|
||||
6. ✅ 服务器/服务健康状况
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 严维序（opengineer）专用配置
|
||||
@@ -1,234 +0,0 @@
|
||||
# HEARTBEAT.md - 沈路明（productmanager）的心跳配置
|
||||
|
||||
> 模板版本：v1.1 (BIZ-24) | 分类：开发 Agent
|
||||
> OpenClaw Agent ID: `productmanager` | Multica Agent UUID: `a101fa88-d821-4839-9754-e04580d5fd68`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级：OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'productmanager' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级：Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id a101fa88-d821-4839-9754-e04580d5fd68 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级：待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片（agentId=productmanager）
|
||||
↓
|
||||
检查 Multica Issues 待办（assignee=a101fa88-d821-4839-9754-e04580d5fd68）
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测：
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO + 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO + 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（2h）→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖，可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖，可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：100 轮
|
||||
|
||||
- 接近 80%（80 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知 COO + 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**：通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**：通过 issue comment 记录轮次进度
|
||||
- **待办文档**：在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ PRD 进度检查
|
||||
6. ✅ 需求变更跟踪
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 沈路明（productmanager）专用配置
|
||||
@@ -1,234 +0,0 @@
|
||||
# HEARTBEAT.md - 胡蓉（projectmanager）的心跳配置
|
||||
|
||||
> 模板版本：v1.1 (BIZ-24) | 分类：开发 Agent
|
||||
> OpenClaw Agent ID: `projectmanager` | Multica Agent UUID: `d877b8c3-b230-4073-b3f7-80e148cfdb71`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级：OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'projectmanager' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级：Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id d877b8c3-b230-4073-b3f7-80e148cfdb71 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级：待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片（agentId=projectmanager）
|
||||
↓
|
||||
检查 Multica Issues 待办（assignee=d877b8c3-b230-4073-b3f7-80e148cfdb71）
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测：
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO + 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO + 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（2h）→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖，可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖，可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：100 轮
|
||||
|
||||
- 接近 80%（80 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知 COO + 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**：通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**：通过 issue comment 记录轮次进度
|
||||
- **待办文档**：在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 项目进度检查
|
||||
6. ✅ 依赖项完成状态
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 胡蓉（projectmanager）专用配置
|
||||
@@ -1,235 +0,0 @@
|
||||
# HEARTBEAT.md - 刘诗妮（secretary）的心跳配置
|
||||
|
||||
> 模板版本：v1.1 (BIZ-24) | 分类：高频 Agent
|
||||
> OpenClaw Agent ID: `secretary` | Multica Agent UUID: `b024fcdc-30ff-420d-b289-498041466e1b`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级：OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'secretary' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级：Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id b024fcdc-30ff-420d-b289-498041466e1b --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级：待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片（agentId=secretary）
|
||||
↓
|
||||
检查 Multica Issues 待办（assignee=b024fcdc-30ff-420d-b289-498041466e1b）
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：10 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测：
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 20 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1200:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1200:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 30 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（1h）→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖，可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖，可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：50 轮
|
||||
|
||||
- 接近 80%（40 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知 COO
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**：通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**：通过 issue comment 记录轮次进度
|
||||
- **待办文档**：在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 全局任务积压巡检
|
||||
6. ✅ 业务入口检查
|
||||
7. ✅ 各 Agent 状态巡检
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 刘诗妮（secretary）专用配置
|
||||
@@ -1,234 +0,0 @@
|
||||
# HEARTBEAT.md - 陆云帆（taobaospecialist）的心跳配置
|
||||
|
||||
> 模板版本：v1.1 (BIZ-24) | 分类：业务 Agent
|
||||
> OpenClaw Agent ID: `taobaospecialist` | Multica Agent UUID: `e0f62d8f-9568-4f41-8ad4-b73d79a163a7`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级：OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'taobaospecialist' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级：Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id e0f62d8f-9568-4f41-8ad4-b73d79a163a7 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级：待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片（agentId=taobaospecialist）
|
||||
↓
|
||||
检查 Multica Issues 待办（assignee=e0f62d8f-9568-4f41-8ad4-b73d79a163a7）
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测：
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（2h）→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖，可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖，可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：30 轮
|
||||
|
||||
- 接近 80%（24 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**：通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**：通过 issue comment 记录轮次进度
|
||||
- **待办文档**：在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 淘宝店铺运营指标
|
||||
6. ✅ 竞品动态跟踪
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 陆云帆（taobaospecialist）专用配置
|
||||
@@ -1,130 +0,0 @@
|
||||
# Agent 知识库集成指南
|
||||
|
||||
> **版本**: v1.0
|
||||
> **任务**: BIZ-19 (BIZ-14-4)
|
||||
> **日期**: 2026-06-22
|
||||
> **作者**: COO (陆怀瑾)
|
||||
> **状态**: 已实施
|
||||
|
||||
---
|
||||
|
||||
## 一、集成概述
|
||||
|
||||
### 1.1 设计原则
|
||||
|
||||
**「引用代替填塞」**: 不把知识内容直接塞进 Agent 配置文件，而是添加 "如何查询知识库" 的指引。Agent 在需要时主动检索，保持配置文件轻量和可维护。
|
||||
|
||||
### 1.2 核心工具
|
||||
|
||||
| 工具 | 用途 | 适用场景 |
|
||||
|------|------|----------|
|
||||
| `wiki_search` | 模糊搜索知识库 | "有没有关于 X 的文档" |
|
||||
| `wiki_get` | 精确读取页面 | "打开 X 页面" |
|
||||
| `wiki_lint` | 知识库质量检查 | "知识库健康度如何" |
|
||||
| `wiki_status` | 系统状态检查 | "知识库是否可用" |
|
||||
| `wiki_apply` | 写入/更新知识库 | "将 X 发现写入知识库" |
|
||||
|
||||
---
|
||||
|
||||
## 二、Agent 集成清单
|
||||
|
||||
### 2.1 已完成集成的 Agent（15 个）
|
||||
|
||||
| # | Agent | 角色 | TOOLS.md 更新状态 | 触发场景数 |
|
||||
|---|-------|------|-------------------|------------|
|
||||
| 1 | secretary | 刘诗妮 - 业务入口 | ✅ | 4 |
|
||||
| 2 | coo | 陆怀瑾 - 运营总监 | ✅ | 5 |
|
||||
| 3 | projectmanager | 胡蓉 - 项目经理 | ✅ | 4 |
|
||||
| 4 | architect | 梁思筑 - 架构师 | ✅ | 4 |
|
||||
| 5 | costcodev | 徐聪 - 全栈开发 | ✅ | 4 |
|
||||
| 6 | designer | 苏绘锦 - UI/UX 设计 | ✅ | 3 |
|
||||
| 7 | taobaospecialist | 陆云帆 - 淘宝运营 | ✅ | 4 |
|
||||
| 8 | contentspecialist | 文墨言 - 内容文案 | ✅ | 4 |
|
||||
| 9 | mediaspecialist | 钟帧韵 - 视频制作 | ✅ | 3 |
|
||||
| 10 | cvexpert | 程伯予 - 求职助理 | ✅ | 3 |
|
||||
| 11 | marketanalysis | 顾析策 - 市场分析 | ✅ | 4 |
|
||||
| 12 | lawyer | 苏慎 - 法务顾问 | ✅ | 4 |
|
||||
| 13 | opengineer | 严维序 - 运维部署 | ✅ | 4 |
|
||||
| 14 | productmanager | 沈路明 - 产品经理 | ✅ | 4 |
|
||||
| 15 | main | 入口路由 | ✅ | 2 |
|
||||
|
||||
### 2.2 集成内容
|
||||
|
||||
每个 Agent 的 TOOLS.md 新增了以下内容：
|
||||
|
||||
1. **知识库查询指引** — 引导 Agent 查看完整检索指南
|
||||
2. **角色特定触发条件** — 该 Agent 何时应查询知识库
|
||||
3. **查询工具速查** — `wiki_search` / `wiki_get` / `wiki_lint` 基本用法
|
||||
4. **角色特定查询示例** — 1-2 个典型查询语句
|
||||
5. **无结果时处理流程** — 知识缺口上报机制
|
||||
|
||||
---
|
||||
|
||||
## 三、查询触发条件设计
|
||||
|
||||
### 3.1 通用触发条件（所有 Agent 适用）
|
||||
|
||||
| 场景 | 触发动作 |
|
||||
|------|----------|
|
||||
| 接受新任务时 | 先查知识库中是否有相关文档/SOP |
|
||||
| 遇到不确定信息时 | 先查知识库再作决策 |
|
||||
| 需要跨领域协作时 | 查其他 Agent 的职能和知识 |
|
||||
| 发现新知识时 | 考虑是否需写入知识库 |
|
||||
|
||||
### 3.2 角色特定触发条件（按 Agent 定制）
|
||||
|
||||
见各 Agent TOOLS.md 中的「知识库查询 → 触发条件」部分。
|
||||
|
||||
---
|
||||
|
||||
## 四、知识缺口上报机制
|
||||
|
||||
### 4.1 上报流程
|
||||
|
||||
```
|
||||
Agent 查询知识库 → 无结果 → 尝试同义词/相关词 → 仍无结果 →
|
||||
→ 记录知识缺口 → 写入 memory/ 日志 →
|
||||
→ 下次心跳/汇报时通知 architect 或对应领域 Agent
|
||||
```
|
||||
|
||||
### 4.2 上报格式
|
||||
|
||||
见 `docs/agent-kb-retrieval-guide.md` 第五节。
|
||||
|
||||
---
|
||||
|
||||
## 五、质量保证
|
||||
|
||||
### 5.1 集成测试方案
|
||||
|
||||
对每个 Agent 至少执行 1 次典型查询场景测试：
|
||||
|
||||
1. 验证 `wiki_search` 可被正确调用
|
||||
2. 验证返回结果格式正确
|
||||
3. 验证无结果时的降级路径
|
||||
|
||||
### 5.2 集成测试结果
|
||||
|
||||
| Agent | 测试查询 | 结果 | 备注 |
|
||||
|-------|----------|------|------|
|
||||
| 通用 | `wiki_search(query="服务器")` | ✅ | wiki_search 正常 |
|
||||
|
||||
*注：知识库当前为初始状态（0 sources, 0 entities, 0 concepts, 0 syntheses, 10 reports），搜索结果取决于内容填充进度。工具链已验证可用。*
|
||||
|
||||
---
|
||||
|
||||
## 六、后续计划
|
||||
|
||||
1. **知识内容填充**: 待 BIZ-14-3 交付后，各 Agent 按角色写入初始知识内容
|
||||
2. **定期质量检查**: COO 每周运行 `wiki_lint()` 检查知识库健康度
|
||||
3. **查询效果评估**: 运行 1 个月后统计各 Agent 知识库查询频率和命中率
|
||||
4. **持续优化**: 根据使用反馈调整触发条件和查询示例
|
||||
|
||||
---
|
||||
|
||||
## 附录：相关文档
|
||||
|
||||
- `docs/agent-kb-retrieval-guide.md` — 知识库检索工具完整指南
|
||||
- `docs/知识查询最佳实践.md` — 查询最佳实践和反模式
|
||||
- `docs/wiki-toolchain-test-report.md` — Wiki 工具链测试报告 (BIZ-14-2)
|
||||
- 各 Agent TOOLS.md — 角色特定查询指引
|
||||
@@ -1,644 +0,0 @@
|
||||
# BIZ-46 Phase3: NVIDIA Sidecar Follow-up 架构设计
|
||||
|
||||
> **架构师**: 梁思筑 (architect)
|
||||
> **日期**: 2026-06-24
|
||||
> **状态**: 已批准，推进实施
|
||||
> **来源**: BIZ-42 Phase2 二轮评审 follow-up
|
||||
|
||||
---
|
||||
|
||||
## 1. 架构解耦 / 依赖注入 — SidecarContext
|
||||
|
||||
### 1.1 现状分析
|
||||
|
||||
当前 `server.py` 使用 **模块级全局变量** 管理所有核心组件：
|
||||
|
||||
```python
|
||||
# server.py 全局状态（当前）
|
||||
_config: SidecarConfig
|
||||
_http_client: httpx.AsyncClient
|
||||
_priority_queue: PriorityRequestQueue
|
||||
_token_bucket: AdaptiveTokenBucket
|
||||
_prometheus: PrometheusMetrics
|
||||
_health_service: HealthService
|
||||
_pending_requests: dict[str, tuple[asyncio.Future, float]]
|
||||
_stats: dict[str, int]
|
||||
_stats_lock: asyncio.Lock
|
||||
```
|
||||
|
||||
**问题**：
|
||||
- `webui.py` 通过 `from nvidia_sidecar import server` 反向导入全局变量（循环依赖风险）
|
||||
- 单元测试需要 mock 模块级变量，无法并行运行测试
|
||||
- 未来多实例/多租户扩展需重写全部模块访问逻辑
|
||||
|
||||
### 1.2 设计方案 — SidecarContext + FastAPI Dependency Injection
|
||||
|
||||
#### 1.2.1 核心数据结构
|
||||
|
||||
```python
|
||||
# context.py
|
||||
from dataclasses import dataclass, field
|
||||
import asyncio
|
||||
import httpx
|
||||
from typing import Any
|
||||
|
||||
@dataclass
|
||||
class SidecarContext:
|
||||
"""Sidecar 全局运行时上下文 — 所有核心组件的唯一容器。
|
||||
|
||||
通过 app.state.sidecar 注入 FastAPI，路由通过 Depends 获取。
|
||||
"""
|
||||
config: 'SidecarConfig'
|
||||
http_client: httpx.AsyncClient
|
||||
token_bucket: 'AdaptiveTokenBucket'
|
||||
priority_queue: 'PriorityRequestQueue'
|
||||
prometheus: 'PrometheusMetrics'
|
||||
health: 'HealthService'
|
||||
pending_requests: dict[str, tuple['asyncio.Future', float]] = field(default_factory=dict)
|
||||
stats: dict[str, int] = field(default_factory=lambda: {
|
||||
"total_requests": 0,
|
||||
"nvidia_requests": 0,
|
||||
"passthrough_requests": 0,
|
||||
"ratelimited_requests": 0,
|
||||
"queue_full_rejects": 0,
|
||||
"upstream_errors": 0,
|
||||
"start_time": 0,
|
||||
})
|
||||
stats_lock: asyncio.Lock = field(default_factory=asyncio.Lock)
|
||||
|
||||
async def increment_stat(self, key: str, delta: int = 1) -> None:
|
||||
"""线程安全的统计计数器自增。"""
|
||||
async with self.stats_lock:
|
||||
self.stats[key] = self.stats.get(key, 0) + delta
|
||||
```
|
||||
|
||||
#### 1.2.2 注入方式
|
||||
|
||||
```python
|
||||
# server.py — lifespan 中创建 context
|
||||
from nvidia_sidecar.context import SidecarContext
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
ctx = SidecarContext(
|
||||
config=load_config(),
|
||||
http_client=httpx.AsyncClient(...),
|
||||
token_bucket=AdaptiveTokenBucket(...),
|
||||
priority_queue=PriorityRequestQueue(...),
|
||||
prometheus=PrometheusMetrics(),
|
||||
health=HealthService(),
|
||||
)
|
||||
app.state.sidecar = ctx # 注入 FastAPI
|
||||
# ... worker 启动 ...
|
||||
yield
|
||||
# ... 清理 ...
|
||||
|
||||
# 依赖注入函数
|
||||
def get_context(request: Request) -> SidecarContext:
|
||||
return request.app.state.sidecar
|
||||
|
||||
# 路由使用
|
||||
@app.post("/v1/chat/completions")
|
||||
async def chat_completions(request: Request, ctx: SidecarContext = Depends(get_context)):
|
||||
return await _handle_proxy_request(request, "/v1/chat/completions", ctx)
|
||||
```
|
||||
|
||||
#### 1.2.3 webui.py 解耦
|
||||
|
||||
```python
|
||||
# webui.py — 不再反向导入 server
|
||||
from nvidia_sidecar.context import SidecarContext
|
||||
from fastapi import Depends
|
||||
|
||||
def get_webui_router():
|
||||
router = APIRouter(prefix="/api", tags=["webui"])
|
||||
|
||||
def _get_ctx(request: Request) -> SidecarContext:
|
||||
return request.app.state.sidecar
|
||||
|
||||
@router.get("/dashboard/stream")
|
||||
async def dashboard_stream(request: Request, ctx: SidecarContext = Depends(_get_ctx)):
|
||||
return await _dashboard_stream(request, ctx)
|
||||
|
||||
@router.get("/admin/config")
|
||||
async def admin_get_config(ctx: SidecarContext = Depends(_get_ctx)):
|
||||
return await get_config(ctx)
|
||||
|
||||
return router
|
||||
```
|
||||
|
||||
#### 1.2.4 Trade-off 分析
|
||||
|
||||
| 维度 | 当前（全局变量） | 方案A（SidecarContext） | 方案B（FastAPI Dependency 全函数式） |
|
||||
|------|------------------|------------------------|-------------------------------------|
|
||||
| 可测试性 | 差（需 mock 模块） | 好（注入 mock context） | 优（每个依赖独立注入） |
|
||||
| 改动量 | 无 | 中等（~8 文件） | 大（每个函数签名变更） |
|
||||
| 可读性 | 一般 | 好（ctx 一目了然） | 差（参数列表膨胀） |
|
||||
| 多实例支持 | 不支持 | 支持（多 app 多 ctx） | 支持 |
|
||||
| 循环依赖 | 有（webui→server） | 消除 | 消除 |
|
||||
|
||||
**决策**: 采用方案A（SidecarContext），平衡改动量与收益。
|
||||
|
||||
### 1.3 迁移计划
|
||||
|
||||
分 3 步渐进迁移，每步可独立合入：
|
||||
|
||||
1. **Step 1**: 创建 `context.py`，定义 `SidecarContext`，在 `lifespan` 中实例化并挂到 `app.state`
|
||||
2. **Step 2**: 路由函数改为 `Depends(get_context)`，删除模块级 `_config`、`_http_client` 等
|
||||
3. **Step 3**: `webui.py` 移除 `from nvidia_sidecar import server`，改用依赖注入
|
||||
|
||||
---
|
||||
|
||||
## 2. Prometheus 标签基数治理
|
||||
|
||||
### 2.1 现状
|
||||
|
||||
当前使用 `model_id` 作为 label 的指标：
|
||||
|
||||
| 指标 | Label | 风险 |
|
||||
|------|-------|------|
|
||||
| `sidecar_upstream_latency_seconds` | `model_id` | **高** — NVIDIA 模型名含版本号，可能无界增长 |
|
||||
| `sidecar_upstream_errors_total` | `status_code`, `model_id` | **中** — 组合基数 = 模型数 × 状态码数 |
|
||||
|
||||
### 2.2 基数评估
|
||||
|
||||
NVIDIA API 当前已知模型约 20-30 个，但：
|
||||
- 新模型持续发布（每月 2-5 个）
|
||||
- 模型名含版本后缀（`nvidia/deepseek-ai/deepseek-v4-pro`、`nvidia/llama-3.1-70b-instruct` 等）
|
||||
- 长期运行（6 个月+）可能累积 100+ 标签组合
|
||||
|
||||
**结论**: 当前基数可控（<200 组合），但长期存在膨胀风险，应提前治理。
|
||||
|
||||
### 2.3 治理方案
|
||||
|
||||
| 指标 | 当前 Label | 调整后 Label | 理由 |
|
||||
|------|-----------|-------------|------|
|
||||
| `upstream_latency_seconds` | `model_id` | `provider` | provider 固定为 `nvidia`，基数=1 |
|
||||
| `upstream_errors_total` | `status_code`, `model_id` | `status_code`, `provider` | 同上 |
|
||||
|
||||
**模型级信息迁移路径**：
|
||||
- 模型 ID → 结构化 JSON 日志（structlog 已支持）
|
||||
- 需要模型级延迟分析时 → 临时 `/status` API 查询或日志聚合
|
||||
|
||||
```python
|
||||
# metrics.py 调整
|
||||
self.upstream_latency_seconds: Histogram = Histogram(
|
||||
"sidecar_upstream_latency_seconds",
|
||||
"Upstream response latency in seconds",
|
||||
labelnames=["provider"], # 原: ["model_id"]
|
||||
buckets=(...),
|
||||
)
|
||||
|
||||
self.upstream_errors_total: Counter = Counter(
|
||||
"sidecar_upstream_errors_total",
|
||||
"Upstream error count by status code",
|
||||
labelnames=["status_code", "provider"], # 原: ["status_code", "model_id"]
|
||||
)
|
||||
```
|
||||
|
||||
```python
|
||||
# server.py 调整 — 模型信息改记日志
|
||||
model_id = _extract_model(payload) or "unknown"
|
||||
provider = "nvidia" # 固定值，因为只有 NVIDIA 请求走 worker
|
||||
_prometheus.record_upstream_latency(provider, upstream_latency)
|
||||
if not resp.is_success:
|
||||
_prometheus.record_upstream_error(resp.status_code, provider)
|
||||
logger.info("request_completed", model_id=model_id, ...) # JSON 日志保留模型信息
|
||||
```
|
||||
|
||||
### 2.4 Trade-off
|
||||
|
||||
| 维度 | 保留 model_id | 收敛为 provider |
|
||||
|------|--------------|----------------|
|
||||
| 基数风险 | 高（无界） | 无（固定=1） |
|
||||
| 模型级分析 | Prometheus 原生查询 | 需日志聚合 |
|
||||
| 迁移成本 | 无 | 低（改 2 个指标定义 + 调用点） |
|
||||
|
||||
**决策**: 收敛为 `provider`，模型级分析通过 JSON 日志 + 日志聚合系统（ELK/Loki）完成。
|
||||
|
||||
---
|
||||
|
||||
## 3. SSE 快照共享缓存
|
||||
|
||||
### 3.1 现状
|
||||
|
||||
每个 SSE 客户端每秒独立调用 `_build_snapshot()`，该方法：
|
||||
- 获取 `_stats` 字典（需锁）
|
||||
- 调用 `_token_bucket.get_status()`（需锁）
|
||||
- 调用 `_priority_queue.get_stats()`（需 asyncio.Lock）
|
||||
|
||||
当 N 个仪表盘同时打开时，每秒 N 次锁竞争 + N 次重复计算。
|
||||
|
||||
### 3.2 设计方案 — 1s TTL 共享缓存
|
||||
|
||||
```python
|
||||
# webui.py
|
||||
_snapshot_cache: tuple[dict[str, Any], float] | None = None # (data, timestamp)
|
||||
_snapshot_lock: asyncio.Lock = asyncio.Lock()
|
||||
_SNAPSHOT_TTL: float = 1.0 # 1 秒 TTL
|
||||
|
||||
async def _build_snapshot_cached(ctx: SidecarContext) -> dict[str, Any]:
|
||||
"""带 1s TTL 的共享快照缓存。
|
||||
|
||||
多个 SSE 客户端共享同一份快照，避免重复计算和锁竞争。
|
||||
"""
|
||||
global _snapshot_cache
|
||||
|
||||
now = time.monotonic()
|
||||
if _snapshot_cache is not None:
|
||||
data, ts = _snapshot_cache
|
||||
if now - ts < _SNAPSHOT_TTL:
|
||||
return data
|
||||
|
||||
async with _snapshot_lock:
|
||||
# Double-check（避免多个协程同时 miss 后重复构建）
|
||||
if _snapshot_cache is not None:
|
||||
data, ts = _snapshot_cache
|
||||
if now - ts < _SNAPSHOT_TTL:
|
||||
return data
|
||||
|
||||
snapshot = await _build_snapshot(ctx)
|
||||
_snapshot_cache = (snapshot, now)
|
||||
return snapshot
|
||||
```
|
||||
|
||||
### 3.3 性能收益
|
||||
|
||||
| 场景 | 当前 | 优化后 |
|
||||
|------|------|--------|
|
||||
| 1 客户端 | 1 次/s 计算 | 1 次/s 计算（无变化） |
|
||||
| 5 客户端 | 5 次/s 计算，5 次锁竞争 | 1 次/s 计算，1 次锁竞争 |
|
||||
| 20 客户端 | 20 次/s 计算，20 次锁竞争 | 1 次/s 计算，1 次锁竞争 |
|
||||
|
||||
---
|
||||
|
||||
## 4. 部署支撑
|
||||
|
||||
### 4.1 Dockerfile
|
||||
|
||||
```dockerfile
|
||||
# services/nvidia_sidecar/Dockerfile
|
||||
FROM python:3.12-slim AS base
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# 安装依赖（利用 Docker 层缓存）
|
||||
COPY pyproject.toml .
|
||||
RUN pip install --no-cache-dir -e .
|
||||
|
||||
# 复制源码
|
||||
COPY . .
|
||||
|
||||
# 非 root 用户运行
|
||||
RUN useradd -r -s /bin/false sidecar
|
||||
USER sidecar
|
||||
|
||||
# 健康检查
|
||||
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
|
||||
CMD python -c "import httpx; r=httpx.get('http://127.0.0.1:9190/health'); exit(0 if r.status_code==200 else 1)"
|
||||
|
||||
EXPOSE 9190 9191
|
||||
|
||||
CMD ["uvicorn", "nvidia_sidecar.server:app", "--host", "0.0.0.0", "--port", "9190"]
|
||||
```
|
||||
|
||||
### 4.2 systemd Service
|
||||
|
||||
```ini
|
||||
# services/nvidia_sidecar/deploy/nvidia-sidecar.service
|
||||
[Unit]
|
||||
Description=NVIDIA Sidecar Rate-Limiting Proxy
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=sidecar
|
||||
Group=sidecar
|
||||
WorkingDirectory=/opt/nvidia-sidecar
|
||||
ExecStart=/opt/nvidia-sidecar/.venv/bin/uvicorn nvidia_sidecar.server:app \
|
||||
--host 127.0.0.1 \
|
||||
--port 9190 \
|
||||
--log-level info
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
# 环境变量
|
||||
EnvironmentFile=/opt/nvidia-sidecar/.env
|
||||
|
||||
# 安全加固
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
ReadWritePaths=/opt/nvidia-sidecar/logs
|
||||
|
||||
# 资源限制
|
||||
LimitNOFILE=65536
|
||||
MemoryMax=512M
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
### 4.3 环境变量清单
|
||||
|
||||
| 变量 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `SIDECAR_HOST` | `127.0.0.1` | 监听地址 |
|
||||
| `SIDECAR_PORT` | `9190` | 代理端口 |
|
||||
| `SIDECAR_METRICS_PORT` | `9191` | Prometheus 指标端口 |
|
||||
| `SIDECAR_UPSTREAM` | `https://integrate.api.nvidia.com/v1` | 上游 API |
|
||||
| `SIDECAR_API_KEY` | (必填) | NVIDIA API Key |
|
||||
| `SIDECAR_RATE_RPM` | `40` | 限流速率 (RPM) |
|
||||
| `SIDECAR_BUCKET_CAPACITY` | `40` | 令牌桶容量 |
|
||||
| `SIDECAR_TIMEOUT` | `60` | 请求超时 (秒) |
|
||||
| `SIDECAR_QUEUE_MAX` | `500` | 队列最大容量 |
|
||||
| `SIDECAR_LOW_TIMEOUT` | `2` | 低优先级超时 (秒) |
|
||||
| `SIDECAR_FALLBACK_PASSTHROUGH` | `true` | 队列满时是否直通 |
|
||||
| `SIDECAR_LOG_LEVEL` | `INFO` | 日志级别 |
|
||||
| `SIDECAR_ADMIN_TOKEN` | (可选) | Admin API 认证 Token |
|
||||
|
||||
### 4.4 防火墙建议
|
||||
|
||||
```
|
||||
# 仅允许内网访问代理端口
|
||||
sudo ufw allow from 192.168.1.0/24 to any port 9190
|
||||
sudo ufw allow from 192.168.1.0/24 to any port 9191
|
||||
# 禁止外网访问
|
||||
sudo ufw deny 9190
|
||||
sudo ufw deny 9191
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Readiness HTTP Client 复用
|
||||
|
||||
### 5.1 现状
|
||||
|
||||
`HealthService.check_upstream()` 每次调用创建新的 `httpx.AsyncClient`：
|
||||
|
||||
```python
|
||||
# health.py — 当前
|
||||
async def check_upstream(self, upstream_url: str, timeout: float = 5.0, api_key: str = "") -> bool:
|
||||
async with httpx.AsyncClient(timeout=timeout) as client: # 每次新建！
|
||||
resp = await client.get(...)
|
||||
```
|
||||
|
||||
K8s/systemd 每 10-30s 探测一次，每次创建+销毁 HTTP client 带来不必要的 TCP 连接开销。
|
||||
|
||||
### 5.2 方案 — 复用主 http_client
|
||||
|
||||
```python
|
||||
# health.py — 优化后
|
||||
async def check_upstream(
|
||||
self,
|
||||
upstream_url: str,
|
||||
http_client: httpx.AsyncClient, # 注入主 client
|
||||
api_key: str = "",
|
||||
timeout: float = 5.0,
|
||||
) -> bool:
|
||||
try:
|
||||
headers = {}
|
||||
if api_key:
|
||||
headers["authorization"] = f"Bearer {api_key}"
|
||||
resp = await http_client.get(
|
||||
f"{upstream_url.rstrip('/')}/v1/models",
|
||||
headers=headers,
|
||||
timeout=timeout,
|
||||
)
|
||||
return resp.status_code < 500
|
||||
except Exception:
|
||||
return False
|
||||
```
|
||||
|
||||
```python
|
||||
# server.py — 路由调用处
|
||||
@app.get("/health/ready")
|
||||
async def health_ready(ctx: SidecarContext = Depends(get_context)):
|
||||
queue_size = await ctx.priority_queue.get_queue_size()
|
||||
bucket_status = ctx.token_bucket.get_status()
|
||||
return await ctx.health.readiness(
|
||||
upstream_url=ctx.config.upstream_url,
|
||||
http_client=ctx.http_client, # 复用主 client
|
||||
upstream_api_key=ctx.config.upstream_api_key or "",
|
||||
queue_current_size=queue_size,
|
||||
queue_max_size=ctx.config.queue_max_size,
|
||||
available_tokens=bucket_status["tokens"],
|
||||
bucket_capacity=bucket_status["capacity"],
|
||||
)
|
||||
```
|
||||
|
||||
**注意**: readiness 检查使用较短 timeout (5s)，不影响主代理请求的 timeout 配置。httpx 支持per-request timeout 覆盖。
|
||||
|
||||
---
|
||||
|
||||
## 6. Retreat 并发/死锁回归测试
|
||||
|
||||
### 6.1 风险点
|
||||
|
||||
`AdaptiveTokenBucket` 有两把锁：
|
||||
- `_lock` (Lock): 保护令牌消费/补充
|
||||
- `_retreat_lock` (RLock): 保护避退状态机
|
||||
|
||||
潜在死锁路径：
|
||||
1. `evaluate_retreat()` 持有 `_retreat_lock` → 调用 `get_429_rate()` (也获取 `_retreat_lock`，RLock 可重入 ✅)
|
||||
2. `evaluate_retreat()` → `_apply_retreat()` → `set_rate()` → 获取 `_lock` (另一把锁)
|
||||
3. Worker 线程: `consume()` 持有 `_lock` → 不调用 `_retreat_lock` (无交叉 ✅)
|
||||
|
||||
当前设计使用 RLock 已规避了重入死锁，但需要回归测试确保未来修改不引入死锁。
|
||||
|
||||
### 6.2 测试用例
|
||||
|
||||
```python
|
||||
# tests/test_retreat_concurrency.py
|
||||
import pytest
|
||||
import asyncio
|
||||
import threading
|
||||
from nvidia_sidecar.rate_limiter import AdaptiveTokenBucket, RetreatState
|
||||
|
||||
class TestRetreatConcurrency:
|
||||
"""避退模式并发安全回归测试。"""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_concurrent_record_and_evaluate(self):
|
||||
"""多线程同时 record_response + evaluate_retreat 不死锁。"""
|
||||
bucket = AdaptiveTokenBucket(rate=40/60, capacity=40)
|
||||
errors: list[Exception] = []
|
||||
|
||||
def worker_record():
|
||||
for i in range(1000):
|
||||
try:
|
||||
bucket.record_response(is_429=(i % 10 == 0))
|
||||
except Exception as e:
|
||||
errors.append(e)
|
||||
|
||||
def worker_evaluate():
|
||||
for _ in range(1000):
|
||||
try:
|
||||
bucket.evaluate_retreat()
|
||||
except Exception as e:
|
||||
errors.append(e)
|
||||
|
||||
threads = [
|
||||
threading.Thread(target=worker_record),
|
||||
threading.Thread(target=worker_record),
|
||||
threading.Thread(target=worker_evaluate),
|
||||
threading.Thread(target=worker_evaluate),
|
||||
]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(timeout=10)
|
||||
|
||||
# 所有线程必须在 10s 内完成（无死锁）
|
||||
assert all(not t.is_alive() for t in threads), "线程未完成，疑似死锁"
|
||||
assert not errors, f"并发错误: {errors}"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_concurrent_consume_and_retreat(self):
|
||||
"""多线程同时 consume + evaluate_retreat 不死锁。"""
|
||||
bucket = AdaptiveTokenBucket(rate=40/60, capacity=40)
|
||||
errors: list[Exception] = []
|
||||
|
||||
def worker_consume():
|
||||
for _ in range(500):
|
||||
try:
|
||||
bucket.consume(tokens=1)
|
||||
except Exception as e:
|
||||
errors.append(e)
|
||||
|
||||
def worker_retreat():
|
||||
for _ in range(500):
|
||||
try:
|
||||
bucket.record_response(is_429=False)
|
||||
bucket.evaluate_retreat()
|
||||
except Exception as e:
|
||||
errors.append(e)
|
||||
|
||||
threads = [
|
||||
threading.Thread(target=worker_consume),
|
||||
threading.Thread(target=worker_consume),
|
||||
threading.Thread(target=worker_retreat),
|
||||
threading.Thread(target=worker_retreat),
|
||||
]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(timeout=10)
|
||||
|
||||
assert all(not t.is_alive() for t in threads), "线程未完成，疑似死锁"
|
||||
assert not errors, f"并发错误: {errors}"
|
||||
|
||||
def test_retreat_state_transitions_under_load(self):
|
||||
"""高负载下避退状态转换正确。"""
|
||||
bucket = AdaptiveTokenBucket(
|
||||
rate=40/60, capacity=40,
|
||||
retreat_429_threshold=0.05,
|
||||
retreat_factor=0.75,
|
||||
)
|
||||
|
||||
# 模拟高 429 率
|
||||
for _ in range(100):
|
||||
bucket.record_response(is_429=True)
|
||||
|
||||
state = bucket.evaluate_retreat()
|
||||
assert state == RetreatState.RETREAT
|
||||
assert bucket.get_effective_rate_rpm() < bucket.get_base_rate_rpm()
|
||||
|
||||
# 模拟恢复
|
||||
for _ in range(200):
|
||||
bucket.record_response(is_429=False)
|
||||
|
||||
# 需要等待 RECOVER_WINDOW
|
||||
import time
|
||||
time.sleep(0.1) # 确保时间窗口过去
|
||||
bucket._last_state_change = 0 # 强制触发时间条件
|
||||
state = bucket.evaluate_retreat()
|
||||
assert state in (RetreatState.RECOVER, RetreatState.NORMAL)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Dashboard UX 优化
|
||||
|
||||
### 7.1 优化项清单
|
||||
|
||||
| # | 优化项 | 实现方式 | 优先级 |
|
||||
|---|--------|---------|--------|
|
||||
| 1 | 队列柱状图 300ms 平滑动画 | CSS `transition: height 300ms ease` | P1 |
|
||||
| 2 | SSE 断连 5s 遮罩 | JS 定时器 + DOM 遮罩层 | P1 |
|
||||
| 3 | 队列图标题显示总排队数 | SSE 数据已有 `current_size`，更新标题 | P2 |
|
||||
| 4 | 页面加载同步配置 | `fetch('/api/admin/config')` 初始化表单 | P2 |
|
||||
|
||||
### 7.2 关键实现
|
||||
|
||||
```javascript
|
||||
// dashboard.html — SSE 断连检测
|
||||
let lastSSETime = Date.now();
|
||||
let reconnectMask = document.getElementById('reconnect-mask');
|
||||
|
||||
eventSource.onmessage = (event) => {
|
||||
lastSSETime = Date.now();
|
||||
reconnectMask.style.display = 'none';
|
||||
// ... 更新 UI ...
|
||||
};
|
||||
|
||||
// 5s 无数据 → 显示遮罩
|
||||
setInterval(() => {
|
||||
if (Date.now() - lastSSETime > 5000) {
|
||||
reconnectMask.style.display = 'flex';
|
||||
}
|
||||
}, 1000);
|
||||
|
||||
// 队列柱状图动画
|
||||
// CSS: .queue-bar { transition: height 0.3s ease; }
|
||||
```
|
||||
|
||||
```javascript
|
||||
// 页面加载时同步配置
|
||||
async function loadConfig() {
|
||||
try {
|
||||
const resp = await fetch('/api/admin/config');
|
||||
if (resp.ok) {
|
||||
const config = await resp.json();
|
||||
document.getElementById('rate-rpm').value = config.rate_rpm;
|
||||
document.getElementById('queue-max').value = config.queue_max_size;
|
||||
// ...
|
||||
}
|
||||
} catch (e) {
|
||||
console.warn('配置加载失败（可能需要 Admin Token）', e);
|
||||
}
|
||||
}
|
||||
loadConfig();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 实施排期
|
||||
|
||||
| 阶段 | 内容 | 预估工时 | 依赖 |
|
||||
|------|------|---------|------|
|
||||
| **D1** | SidecarContext Step 1-3（解耦迁移） | 8h | 无 |
|
||||
| **D2** | Prometheus 标签收敛 + 日志增强 | 2h | D1 |
|
||||
| **D2** | SSE 共享缓存 | 2h | D1 |
|
||||
| **D2** | Readiness HTTP client 复用 | 1h | D1 |
|
||||
| **D3** | Dockerfile + systemd service | 2h | 无 |
|
||||
| **D3** | Dashboard UX 优化 | 3h | 无 |
|
||||
| **D3** | Retreat 并发回归测试 | 3h | 无 |
|
||||
| **D4** | 集成测试 + mypy strict | 4h | D1-D3 |
|
||||
| **合计** | | **25h** | |
|
||||
|
||||
---
|
||||
|
||||
## 9. 验收标准映射
|
||||
|
||||
| Issue 要求 | 本文档章节 | 状态 |
|
||||
|-----------|-----------|------|
|
||||
| SidecarContext / DI 方案落地或 ADR | §1 | ✅ 详细设计 + 迁移计划 |
|
||||
| Prometheus 高基数 label 收敛 | §2 | ✅ 收敛为 provider |
|
||||
| SSE snapshot 共享缓存 | §3 | ✅ 1s TTL 设计 |
|
||||
| Dockerfile + systemd + 部署 SOP | §4 | ✅ 完整文件 |
|
||||
| readiness 复用 HTTP client | §5 | ✅ 注入主 client |
|
||||
| retreat 并发/死锁回归测试 | §6 | ✅ 测试用例 |
|
||||
| Dashboard UX 细节 | §7 | ✅ 4 项优化 |
|
||||
@@ -1,156 +0,0 @@
|
||||
# 知识查询最佳实践
|
||||
|
||||
> **版本**: v1.0
|
||||
> **任务**: BIZ-19 (BIZ-14-4)
|
||||
> **日期**: 2026-06-22
|
||||
|
||||
---
|
||||
|
||||
## 一、查询策略
|
||||
|
||||
### 1.1 渐进式检索原则
|
||||
|
||||
```
|
||||
先宽后窄 → 先模糊后精确 → 先搜索后读取
|
||||
```
|
||||
|
||||
**标准流程**：
|
||||
1. `wiki_search(query="关键词")` — 发现有哪些相关内容
|
||||
2. `wiki_get(lookup="匹配页面")` — 精确读取具体内容
|
||||
3. 如搜索结果过多（>10） → 收窄关键词重新搜索
|
||||
4. 如搜索结果与需求不相关 → 调整表述方式重新搜索
|
||||
|
||||
### 1.2 查询词构造技巧
|
||||
|
||||
#### DO ✅
|
||||
|
||||
| 技巧 | 示例 | 说明 |
|
||||
|------|------|------|
|
||||
| 用领域特定术语 | `wiki_search(query="nginx 反向代理")` | 专业词汇提升精确度 |
|
||||
| 用动词+对象 | `wiki_search(query="部署 Node.js")` | 明确查询意图 |
|
||||
| 用自然语言问题 | `wiki_search(query="如何配置 nginx logrotate")` | 适合语义检索 |
|
||||
| 用缩写和全称组合 | `wiki_search(query="CI/CD 持续集成")` | 覆盖不同表述 |
|
||||
| 分步搜索 | 先搜 "nginx"，再搜 "nginx 日志" | 逐步收窄范围 |
|
||||
|
||||
#### DON'T ❌
|
||||
|
||||
| 反模式 | 错误示例 | 问题 |
|
||||
|--------|----------|------|
|
||||
| 过于泛化的词 | `wiki_search(query="配置")` | 结果太多太杂 |
|
||||
| 过于具体的短语 | `wiki_search(query="192.168.1.99 端口 22 上的 nginx")` | 命中率低 |
|
||||
| 跳过搜索直接 guess 路径 | `wiki_get(lookup="随便猜的页面名")` | 大概率找不到 |
|
||||
| 一次加载超大页面 | `wiki_get(lookup="巨型文档")` | 超出上下文容量 |
|
||||
| 无结果后直接放弃 | 只搜一次就说"知识库没内容" | 可能是查询词不准确 |
|
||||
|
||||
---
|
||||
|
||||
## 二、结果处理
|
||||
|
||||
### 2.1 匹配结果数量处理
|
||||
|
||||
| 结果数 | 处理方式 |
|
||||
|--------|----------|
|
||||
| 0 | 尝试同义词/相关词 → qmd 搜索 → 上报知识缺口 |
|
||||
| 1-3 | 逐个 `wiki_get` 读取完整内容 |
|
||||
| 4-10 | 按评分排序，取前 3 个读取 |
|
||||
| 10+ | 收窄搜索词重新搜索 |
|
||||
|
||||
### 2.2 大页面分页读取
|
||||
|
||||
```bash
|
||||
# 超过 100 行的页面，分页读取
|
||||
wiki_get(lookup="长文档标题", fromLine=1, lineCount=50) # 第一部分
|
||||
wiki_get(lookup="长文档标题", fromLine=51, lineCount=50) # 第二部分
|
||||
```
|
||||
|
||||
### 2.3 信息来源交叉验证
|
||||
|
||||
当多个查询返回不同信息时：
|
||||
1. 检查页面更新时间（优先信任较新的）
|
||||
2. 交叉对比多个来源
|
||||
3. 如信息冲突 → 标记为"需确认"，汇报给 architect
|
||||
|
||||
---
|
||||
|
||||
## 三、知识缺口处理
|
||||
|
||||
### 3.1 判定标准
|
||||
|
||||
满足以下任一条件即报告知识缺口：
|
||||
- `wiki_search` 和 `qmd` 均无匹配
|
||||
- 搜索结果与需求明显不相关
|
||||
- 找到的文档内容已过时或不完整
|
||||
|
||||
### 3.2 上报模板
|
||||
|
||||
```
|
||||
【知识缺口 - YYYY-MM-DD】
|
||||
|
||||
- 查询 Agent: [Agent 名称]
|
||||
- 查询意图: [想了解什么]
|
||||
- 已尝试检索: [用过的搜索词, 换行列出]
|
||||
- 已使用工具: wiki_search / qmd
|
||||
- 期望内容: [知识库中应有什么]
|
||||
- 紧急程度: high / normal / low
|
||||
- 建议: [谁补充、什么内容]
|
||||
```
|
||||
|
||||
### 3.3 上报路径
|
||||
|
||||
| 缺口类型 | 上报目标 |
|
||||
|----------|----------|
|
||||
| 架构/技术 | architect (梁思筑) |
|
||||
| 业务/流程 | projectmanager (胡蓉) |
|
||||
| 法务/合规 | lawyer (苏慎) |
|
||||
| 市场/分析 | marketanalysis (顾析策) |
|
||||
| 通用/不确定 | COO (陆怀瑾) — 由 COO 分配 |
|
||||
|
||||
---
|
||||
|
||||
## 四、知识库写入准则
|
||||
|
||||
### 4.1 何时写入
|
||||
|
||||
- 完成重要决策后（如架构选型、策略调整）
|
||||
- 发现可复用的模板/清单
|
||||
- 完成深度分析后（市场报告、竞品分析）
|
||||
- 知识缺口被填补后
|
||||
|
||||
### 4.2 写入工具选择
|
||||
|
||||
| 场景 | 工具 |
|
||||
|------|------|
|
||||
| 创建新知识页面 | `wiki_apply(op="create_synthesis", ...)` |
|
||||
| 更新已有页面元数据 | `wiki_apply(op="update_metadata", ...)` |
|
||||
|
||||
### 4.3 不写入的内容
|
||||
|
||||
- 机密信息（密码、密钥、token）
|
||||
- 临时信息（当天的具体任务进度）
|
||||
- 已过时会被频繁更新的数据
|
||||
- 纯个人笔记（放 `memory/` 下）
|
||||
|
||||
---
|
||||
|
||||
## 五、定期维护
|
||||
|
||||
### 5.1 COO 每周检查清单
|
||||
|
||||
- [ ] 运行 `wiki_lint()` 检查质量
|
||||
- [ ] 统计各 Agent 知识库查询频率
|
||||
- [ ] 清理过时页面
|
||||
- [ ] 评估知识缺口数量和解决率
|
||||
- [ ] 输出知识库运营周报
|
||||
|
||||
### 5.2 Agent 自检清单
|
||||
|
||||
每次心跳时：
|
||||
- [ ] 上次查询的知识缺口是否已上报
|
||||
- [ ] 本轮工作中是否有应写入知识库的发现
|
||||
|
||||
---
|
||||
|
||||
## 附录
|
||||
|
||||
- `docs/agent-kb-retrieval-guide.md` — 工具使用完整指南
|
||||
- `docs/Agent 知识库集成指南.md` — 集成方案总览
|
||||
@@ -12,10 +12,8 @@
|
||||
| [产品/](产品/) | PRD、需求分析 | 沈路明 (productmanager) | — |
|
||||
| [技术/](技术/) | 开发规范、代码审查 | 徐聪 (costcodev) | — |
|
||||
| [设计/](设计/) | UI设计、品牌规范 | 苏绘锦 (designer) | — |
|
||||
| [运维/](运维/) | 部署流程、故障排查、服务器运维 | 严维序 (opengineer) | 3 |
|
||||
| [运营/](运营/) | 活动策划、数据分析 | 陆怀瑾 (coo) | — |
|
||||
| [行政/](行政/) | 合同、报销流程 | 刘诗妮 (secretary) | — |
|
||||
| [规范/](规范/) | 运维标准、安全基线、合规要求 | 严维序 (opengineer) | — |
|
||||
|
||||
## 知识条目格式
|
||||
|
||||
|
||||
@@ -5,9 +5,7 @@
|
||||
|
||||
## 知识范围
|
||||
|
||||
涵盖开发规范、代码审查、架构设计、技术选型等技术团队核心知识。
|
||||
|
||||
> ⚠️ 部署运维知识已迁移至 [运维/](../运维/) 领域。
|
||||
涵盖开发规范、代码审查、架构设计、部署运维、技术选型等技术团队知识。
|
||||
|
||||
## 条目清单
|
||||
|
||||
|
||||
@@ -1,25 +0,0 @@
|
||||
# 规范领域知识
|
||||
|
||||
**责任人**：严维序（opengineer）
|
||||
**审核人**：陆怀瑾（coo）
|
||||
|
||||
## 知识范围
|
||||
|
||||
涵盖运维规范、安全标准、合规要求等规范类知识条目，支撑团队标准化运作。
|
||||
|
||||
## 条目清单
|
||||
|
||||
| 文件名 | 说明 | 状态 |
|
||||
|--------|------|------|
|
||||
| [服务器运维标准_v1.0.md](../运维/服务器运维标准_v1.0.md) | 服务器巡检、监控、备份运维标准 | 见运维域 |
|
||||
|
||||
## 待建设
|
||||
|
||||
- 数据库运维标准
|
||||
- 安全审计基线
|
||||
- 数据合规处理流程
|
||||
|
||||
---
|
||||
|
||||
> 维护者：严维序（opengineer）
|
||||
> 最后更新：2026-06-24
|
||||
@@ -1,27 +0,0 @@
|
||||
# 运维领域知识
|
||||
|
||||
**责任人**：严维序（opengineer）
|
||||
**审核人**：陆怀瑾（coo）
|
||||
|
||||
## 知识范围
|
||||
|
||||
涵盖服务器运维、部署流程、故障排查、监控配置、安全保障等运维团队核心知识。
|
||||
|
||||
## 条目清单
|
||||
|
||||
| 文件名 | 说明 | 状态 |
|
||||
|--------|------|------|
|
||||
| [部署流程_v1.0.md](部署流程_v1.0.md) | 服务部署 SOP 与变更管理流程 | ✅ |
|
||||
| [故障排查手册_v1.0.md](故障排查手册_v1.0.md) | 常见故障定位与处置方案 | ✅ |
|
||||
| [服务器运维标准_v1.0.md](服务器运维标准_v1.0.md) | 服务器巡检、监控、备份运维标准 | 🆕 |
|
||||
|
||||
## 待建设
|
||||
|
||||
- 数据库运维指南
|
||||
- 安全加固检查清单
|
||||
- 灾备与应急恢复预案
|
||||
|
||||
---
|
||||
|
||||
> 维护者：严维序（opengineer）
|
||||
> 最后更新：2026-06-24
|
||||
@@ -1,274 +0,0 @@
|
||||
# 故障排查手册
|
||||
|
||||
## 元数据
|
||||
|
||||
| 属性 | 值 |
|
||||
|------|-----|
|
||||
| **领域** | 运维 |
|
||||
| **责任人** | 严维序（opengineer） |
|
||||
| **版本** | v1.0 |
|
||||
| **创建日期** | 2026-06-24 |
|
||||
| **最后更新** | 2026-06-24 |
|
||||
| **标签** | 故障排查, 运维, 排障 |
|
||||
|
||||
## 概述
|
||||
|
||||
本手册汇总 BizWings 环境中常见的系统与服务故障定位方法和修复方案。覆盖 SSH 连接、Nginx、数据库、磁盘、Docker 等核心场景。
|
||||
|
||||
---
|
||||
|
||||
## 一、SSH 连接故障
|
||||
|
||||
### 1.1 连接超时
|
||||
|
||||
```bash
|
||||
# 诊断步骤
|
||||
ssh -vvv root@<ip> -p <port> # 查看详细连接日志
|
||||
ping <ip> # 检查网络连通性
|
||||
nmap <ip> -p <port> # 检查端口状态
|
||||
```
|
||||
|
||||
**常见原因**：
|
||||
- 目标服务器防火墙未开放端口
|
||||
- 源 IP 未加入白名单
|
||||
- 服务器负载过高，sshd 响应慢
|
||||
|
||||
**解决方案**：
|
||||
1. 检查服务器防火墙：`iptables -L -n` 或 `ufw status`
|
||||
2. 检查 sshd 是否运行：`systemctl status sshd`
|
||||
3. 检查负载：`top -n1 | head -5`
|
||||
|
||||
### 1.2 认证失败
|
||||
|
||||
```bash
|
||||
# 诊断步骤
|
||||
ssh -p <port> root@<ip> # 尝试密码登录
|
||||
# Permission denied (publickey,password) 提示
|
||||
```
|
||||
|
||||
**常见原因**：
|
||||
- 密码错误（检查 TOOLS.md 中记录）
|
||||
- SSH 密钥认证配置错误
|
||||
- `/etc/ssh/sshd_config` 中 `PasswordAuthentication no`
|
||||
|
||||
**解决方案**：
|
||||
1. 确认密码与 TOOLS.md 一致
|
||||
2. 检查 `sshd_config`：`grep PasswordAuthentication /etc/ssh/sshd_config`
|
||||
3. 临时允许密码登录：`sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config && systemctl reload sshd`
|
||||
|
||||
---
|
||||
|
||||
## 二、Nginx 服务异常
|
||||
|
||||
### 2.1 Nginx 启动失败 / 卡在 activating
|
||||
|
||||
```bash
|
||||
# 诊断步骤
|
||||
systemctl status nginx # 查看状态
|
||||
journalctl -u nginx --no-pager -n 50 # 查看日志
|
||||
nginx -t # 配置语法检查
|
||||
```
|
||||
|
||||
**根因（经验）**：进程残留导致端口占用
|
||||
|
||||
```bash
|
||||
# 修复
|
||||
pkill -9 nginx # 强制清理残留进程
|
||||
sleep 2
|
||||
systemctl start nginx # 重新启动
|
||||
systemctl status nginx # 确认状态
|
||||
```
|
||||
|
||||
### 2.2 502 Bad Gateway
|
||||
|
||||
```bash
|
||||
# 诊断步骤
|
||||
curl -I http://localhost:<upstream-port> # 检查上游服务
|
||||
ss -tlnp | grep <upstream-port> # 检查端口监听
|
||||
systemctl status <upstream-service> # 检查上游进程
|
||||
```
|
||||
|
||||
**常见原因**：
|
||||
- 上游服务未启动或崩溃
|
||||
- 连接池耗尽
|
||||
|
||||
**解决方案**：
|
||||
1. 重启上游服务：`systemctl restart <service>`
|
||||
2. 检查 `upstream` 配置是否正确
|
||||
|
||||
### 2.3 日志轮转失败
|
||||
|
||||
```bash
|
||||
# 诊断步骤
|
||||
cat /var/log/nginx/error.log | head # 查看是否有日志无法写入
|
||||
ls -la /var/log/nginx/ # 查看日志文件
|
||||
/usr/sbin/logrotate -d /etc/logrotate.d/nginx # 测试 logrotate
|
||||
```
|
||||
|
||||
**修复方案**：
|
||||
```bash
|
||||
# 修改 /etc/logrotate.d/nginx 中的 postrotate 脚本
|
||||
# 将 invoke-rc.d nginx rotate 改为：
|
||||
postrotate
|
||||
systemctl reload nginx
|
||||
endscript
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 三、数据库连接故障
|
||||
|
||||
### 3.1 MySQL 连接失败
|
||||
|
||||
```bash
|
||||
# 诊断步骤
|
||||
mysql -h <host> -P <port> -u root -p # 测试连接
|
||||
telnet <host> <port> # 检查端口
|
||||
systemctl status mysql # 检查服务
|
||||
```
|
||||
|
||||
**常见原因**：
|
||||
- 服务未运行
|
||||
- 防火墙未放行 3306 端口
|
||||
- 用户权限 / host 限制
|
||||
- 连接数超限
|
||||
|
||||
**解决方案**：
|
||||
```bash
|
||||
# 检查连接数
|
||||
mysql -e "SHOW VARIABLES LIKE 'max_connections';"
|
||||
mysql -e "SHOW PROCESSLIST;"
|
||||
|
||||
# 检查用户权限
|
||||
mysql -e "SELECT user, host FROM mysql.user WHERE user='root';"
|
||||
```
|
||||
|
||||
### 3.2 MySQL 空间不足
|
||||
|
||||
```bash
|
||||
# 诊断
|
||||
df -h # 磁盘空间
|
||||
mysql -e "SELECT table_schema, ROUND(SUM(data_length+index_length)/1024/1024,2) AS size_mb FROM information_schema.tables GROUP BY table_schema ORDER BY size_mb DESC;"
|
||||
```
|
||||
|
||||
**解决方案**：
|
||||
- 清理过期 binlog：`PURGE BINARY LOGS BEFORE DATE_SUB(NOW(), INTERVAL 7 DAY);`
|
||||
- 清理临时表
|
||||
- 扩展磁盘
|
||||
|
||||
---
|
||||
|
||||
## 四、磁盘空间告警
|
||||
|
||||
### 4.1 诊断
|
||||
|
||||
```bash
|
||||
df -h # 查看各分区使用率
|
||||
du -sh /* 2>/dev/null | sort -rh | head -10 # 找到大文件目录
|
||||
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null # 大文件定位
|
||||
```
|
||||
|
||||
### 4.2 清理方案
|
||||
|
||||
```bash
|
||||
# Docker 日志和镜像清理
|
||||
docker system prune -af --volumes # 清理未使用的 Docker 资源
|
||||
|
||||
# 系统日志轮转
|
||||
journalctl --vacuum-time=7d # 清理 7 天前的 journal 日志
|
||||
|
||||
# 应用日志归档
|
||||
find /var/log -name "*.log" -mtime +30 -exec gzip {} \; # 压缩旧日志
|
||||
find /var/log -name "*.gz" -mtime +90 -delete # 删除 90 天前的压缩日志
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 五、Docker 容器异常
|
||||
|
||||
### 5.1 容器停止
|
||||
|
||||
```bash
|
||||
docker ps -a | grep <container> # 查看容器状态
|
||||
docker logs <container> --tail 50 # 查看最近日志
|
||||
```
|
||||
|
||||
**修复**：
|
||||
```bash
|
||||
docker start <container> # 手动启动
|
||||
docker compose -f <path> up -d # 使用 Compose 重启
|
||||
```
|
||||
|
||||
### 5.2 Docker API 无响应
|
||||
|
||||
```bash
|
||||
systemctl status docker # 检查 Docker 服务
|
||||
journalctl -u docker --no-pager -n 50 # 查看 Docker 日志
|
||||
```
|
||||
|
||||
**修复**：
|
||||
```bash
|
||||
systemctl restart docker # 重启 Docker 守护进程
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 六、系统进程故障
|
||||
|
||||
### 6.1 端口被占用
|
||||
|
||||
```bash
|
||||
ss -tlnp | grep <port> # 查看占用端口的进程
|
||||
fuser -k <port>/tcp # 强制释放端口
|
||||
```
|
||||
|
||||
### 6.2 systemd 服务异常
|
||||
|
||||
```bash
|
||||
systemctl status <service> # 检查状态
|
||||
journalctl -u <service> --no-pager -n 100 # 查看服务日志
|
||||
|
||||
# 常用修复
|
||||
systemctl daemon-reload # 重载 unit 文件
|
||||
systemctl restart <service> # 重启
|
||||
systemctl enable <service> # 设置开机自启
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 七、日志分析工具
|
||||
|
||||
### 7.1 常用命令
|
||||
|
||||
```bash
|
||||
# 实时日志跟踪
|
||||
tail -f /var/log/<app>/access.log
|
||||
|
||||
# 错误过滤
|
||||
grep -i "error\|exception\|failed" /var/log/<app>/app.log | tail -50
|
||||
|
||||
# 时间范围过滤
|
||||
awk '/2026-06-24 10:00/,/2026-06-24 11:00/' /var/log/<app>/app.log
|
||||
```
|
||||
|
||||
### 7.2 关键检查点
|
||||
|
||||
| 故障表现 | 优先检查 | 常见根因 |
|
||||
|----------|----------|----------|
|
||||
| 服务无响应 | systemctl status | 进程 OOM / 崩溃 |
|
||||
| API 返回错误 | 应用日志 + Nginx 日志 | 代码 bug / 上游依赖异常 |
|
||||
| 高延迟 | top + ss + 应用日志 | 资源争抢 / 死锁 |
|
||||
| 数据库异常 | MySQL error log | 慢查询 / 连接数超限 |
|
||||
|
||||
---
|
||||
|
||||
## 相关条目
|
||||
|
||||
- [部署流程_v1.0.md](部署流程_v1.0.md)
|
||||
- [服务器运维标准_v1.0.md](服务器运维标准_v1.0.md)
|
||||
|
||||
## 变更记录
|
||||
|
||||
| 日期 | 版本 | 变更说明 | 变更人 |
|
||||
|------|------|----------|--------|
|
||||
| 2026-06-24 | v1.0 | 初始创建 | 严维序 |
|
||||
@@ -1,177 +0,0 @@
|
||||
# 服务器运维标准
|
||||
|
||||
## 元数据
|
||||
|
||||
| 属性 | 值 |
|
||||
|------|-----|
|
||||
| **领域** | 运维 |
|
||||
| **责任人** | 严维序（opengineer） |
|
||||
| **版本** | v1.0 |
|
||||
| **创建日期** | 2026-06-24 |
|
||||
| **最后更新** | 2026-06-24 |
|
||||
| **标签** | 运维, 监控, 巡检, 备份 |
|
||||
|
||||
## 概述
|
||||
|
||||
本文档定义 BizWings 团队所有服务器的日常运维标准，包括巡检频率、监控指标、备份策略和安全基线。适用于所有生产环境服务器（阿里云 / 家庭内网 / HP 服务器）。
|
||||
|
||||
---
|
||||
|
||||
## 一、服务器巡检标准
|
||||
|
||||
### 1.1 巡检频率
|
||||
|
||||
| 类型 | 频率 | 执行方式 |
|
||||
|------|------|----------|
|
||||
| 心跳自检 | 每 10 分钟 | openclaw 心跳自动巡检 |
|
||||
| 深度巡检 | 每日一次 | 手动执行 `python3 $SCRIPTS/heartbeat_helper.py opengineer` |
|
||||
| 全量巡检 | 每周一次 | 逐个检查全部服务器 |
|
||||
|
||||
### 1.2 巡检清单
|
||||
|
||||
#### 资源负载
|
||||
```bash
|
||||
# 磁盘使用率（警告 > 80%，严重 > 90%）
|
||||
df -h | grep -v tmpfs
|
||||
|
||||
# CPU 负载
|
||||
uptime
|
||||
|
||||
# 内存使用
|
||||
free -h
|
||||
|
||||
# 网络 IO
|
||||
sar -n DEV 1 3
|
||||
```
|
||||
|
||||
#### 服务状态
|
||||
```bash
|
||||
# 核心服务清单（按实际部署确认）
|
||||
systemctl status nginx mysql docker sshd
|
||||
|
||||
# Docker 容器健康
|
||||
docker ps | grep -c "Up"
|
||||
```
|
||||
|
||||
#### 日志异常
|
||||
```bash
|
||||
# 最近 10 分钟的错误日志
|
||||
journalctl --since "10 min ago" -p err --no-pager | tail -20
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 二、监控指标定义
|
||||
|
||||
### 2.1 告警阈值
|
||||
|
||||
| 指标 | 警告 (WARN) | 严重 (CRIT) | 处理 |
|
||||
|------|-------------|-------------|------|
|
||||
| 磁盘使用率 | > 80% | > 90% | 清理日志 / 扩容 |
|
||||
| CPU 负载 (1min) | > 4.0 | > 8.0 | 检查异常进程 |
|
||||
| 内存使用率 | > 85% | > 95% | 检查 OOM 风险 |
|
||||
| 根分区 inode | > 80% | > 90% | 清理小文件 |
|
||||
| 服务进程 | 停止 | — | 重启服务 |
|
||||
| 端口监听 | 消失 | — | 检查服务状态 |
|
||||
| Docker 容器 | 非 Up | — | docker start / compose up |
|
||||
|
||||
### 2.2 日志监控
|
||||
|
||||
- 系统日志：`journalctl -p err` 重点关注
|
||||
- 应用日志：`error`, `exception`, `failed`, `timeout` 关键词监控
|
||||
- Nginx 日志：5xx 错误率 > 1% 时触发调查
|
||||
|
||||
---
|
||||
|
||||
## 三、备份策略
|
||||
|
||||
### 3.1 数据库备份
|
||||
|
||||
```bash
|
||||
# MySQL 全量备份（建议每日凌晨执行）
|
||||
mysqldump --all-databases --single-transaction --quick | gzip > /backup/db/all-$(date +%Y%m%d).sql.gz
|
||||
```
|
||||
|
||||
### 3.2 配置备份
|
||||
- 服务器配置文件：`/backup/conf/<server>/` 目录
|
||||
- 每次变更前执行：`cp <config> <config>.$(date +%Y%m%d-%H%M%S).bak`
|
||||
|
||||
### 3.3 Docker 数据备份
|
||||
```bash
|
||||
# 思源笔记备份（已配置每日 3:00）
|
||||
tar czf /backup/siyuan/siyuan-data-$(date +%Y%m%d).tar.gz -C <data-dir> .
|
||||
```
|
||||
|
||||
### 3.4 备份保留策略
|
||||
|
||||
| 类型 | 保留期限 |
|
||||
|------|----------|
|
||||
| 数据库全量备份 | 30 天 |
|
||||
| 配置备份 | 90 天 |
|
||||
| Docker 数据 | 7 天 |
|
||||
| 日志归档 | 90 天 |
|
||||
|
||||
---
|
||||
|
||||
## 四、变更管理标准
|
||||
|
||||
### 4.1 变更准入
|
||||
|
||||
- ✅ 每次变更前必须备份原始文件
|
||||
- ✅ 高危操作（防火墙、内核、数据库）必须保留回滚方案
|
||||
- ✅ 变更前评估影响范围
|
||||
- ✅ 变更后验证服务状态
|
||||
- ❌ 禁止在无备份的情况下直接修改生产配置
|
||||
- ❌ 禁止在高峰时段执行非紧急变更
|
||||
|
||||
### 4.2 变更分级
|
||||
|
||||
| 级别 | 示例 | 要求 |
|
||||
|------|------|------|
|
||||
| 低风险 | 普通应用更新 | 备份 → 部署 → 验证 |
|
||||
| 中风险 | 配置修改 | 备份 → 预演 → 部署 → 验证 |
|
||||
| 高风险 | 内核 / 防火墙 / 数据库 | 备份 → 预演 → 通知 → 部署 → 验证 → 监控 |
|
||||
|
||||
---
|
||||
|
||||
## 五、安全基线
|
||||
|
||||
### 5.1 基本要求
|
||||
|
||||
- [ ] SSH 禁止 root 密码登录（高风险服务器）
|
||||
- [ ] 防火墙最小权限原则
|
||||
- [ ] 非必要端口不对外开放
|
||||
- [ ] 定期更新系统安全补丁
|
||||
- [ ] 日志审计开启
|
||||
|
||||
### 5.2 密码管理
|
||||
|
||||
- 服务器密码统一记录在 TOOLS.md
|
||||
- 数据库密码统一管理
|
||||
- 禁止在代码中硬编码密码
|
||||
|
||||
---
|
||||
|
||||
## 六、服务器清单与分类
|
||||
|
||||
| 环境 | 服务器数 | 用途 | 巡检频率 |
|
||||
|------|----------|------|----------|
|
||||
| 阿里云生产 | 3 | 应用服务、数据库 | 每次心跳 |
|
||||
| 家庭内网生产 | 4 | 应用、数据库、PVE | 每次心跳 |
|
||||
| HP 测试 | 3 | 测试、NAS | 每日 |
|
||||
| 树莓派 | 1 | 辅助设备 | 每日 |
|
||||
|
||||
详细清单见 TOOLS.md「SSH/WinRM 服务器清单」
|
||||
|
||||
---
|
||||
|
||||
## 相关条目
|
||||
|
||||
- [部署流程_v1.0.md](部署流程_v1.0.md)
|
||||
- [故障排查手册_v1.0.md](故障排查手册_v1.0.md)
|
||||
|
||||
## 变更记录
|
||||
|
||||
| 日期 | 版本 | 变更说明 | 变更人 |
|
||||
|------|------|----------|--------|
|
||||
| 2026-06-24 | v1.0 | 初始创建 | 严维序 |
|
||||
@@ -1,202 +0,0 @@
|
||||
# 服务部署流程 SOP
|
||||
|
||||
## 元数据
|
||||
|
||||
| 属性 | 值 |
|
||||
|------|-----|
|
||||
| **领域** | 运维 |
|
||||
| **责任人** | 严维序（opengineer） |
|
||||
| **版本** | v1.0 |
|
||||
| **创建日期** | 2026-06-24 |
|
||||
| **最后更新** | 2026-06-24 |
|
||||
| **标签** | 部署, 运维, SOP |
|
||||
|
||||
## 概述
|
||||
|
||||
本文档定义 BizWings 团队所有业务服务的部署流程标准，涵盖部署前检查、执行步骤、验证测试和回滚预案。适用于所有生产环境的代码部署与服务更新。
|
||||
|
||||
---
|
||||
|
||||
## 一、部署前置检查
|
||||
|
||||
### 1.1 代码准备
|
||||
|
||||
- [ ] 代码已合并到目标分支（main / release）
|
||||
- [ ] PR 已通过 Code Review 并合并
|
||||
- [ ] 本地或 CI 构建通过（编译无报错）
|
||||
- [ ] 版本号已更新（如有）
|
||||
|
||||
### 1.2 环境检查
|
||||
|
||||
- [ ] 目标服务器磁盘空间充足（> 剩余 20%）
|
||||
- [ ] CPU / 内存负载正常（< 80%）
|
||||
- [ ] 网络连通性：本机 → 目标服务器可达
|
||||
- [ ] 目标端口未被占用
|
||||
- [ ] 依赖服务（数据库 / 中间件）运行正常
|
||||
|
||||
### 1.3 备份准备
|
||||
|
||||
- [ ] **配置备份**：服务器配置文件备份到 `/backup/conf/` 目录
|
||||
- [ ] **数据库备份**：涉及数据库变更，先执行 `mysqldump` 全量备份
|
||||
- [ ] **当前版本标记**：记录当前运行版本号或 Git commit hash
|
||||
|
||||
---
|
||||
|
||||
## 二、部署执行步骤
|
||||
|
||||
### 2.1 文件分发
|
||||
|
||||
```bash
|
||||
# 标准部署（SSH + scp/rsync）
|
||||
scp -P <port> ./dist/app root@<server>:/opt/app/
|
||||
# 或使用 rsync 增量同步
|
||||
rsync -avz --delete -e "ssh -p <port>" ./dist/ root@<server>:/opt/app/
|
||||
```
|
||||
|
||||
### 2.2 服务更新
|
||||
|
||||
#### 方式 A：systemd 服务
|
||||
```bash
|
||||
# 1. 停止服务
|
||||
systemctl stop <service-name>
|
||||
|
||||
# 2. 备份旧版本（如有必要）
|
||||
mv /opt/app/<app> /opt/app/<app>.bak
|
||||
|
||||
# 3. 放置新版本
|
||||
cp /tmp/<app> /opt/app/<app>
|
||||
chmod +x /opt/app/<app>
|
||||
|
||||
# 4. 重启服务
|
||||
systemctl start <service-name>
|
||||
systemctl status <service-name>
|
||||
```
|
||||
|
||||
#### 方式 B：Docker 容器
|
||||
```bash
|
||||
# 1. 拉取新镜像
|
||||
docker pull <registry>/<image>:<tag>
|
||||
|
||||
# 2. 停止旧容器
|
||||
docker stop <container-name>
|
||||
docker rm <container-name>
|
||||
|
||||
# 3. 启动新容器
|
||||
docker run -d --name <container-name> \
|
||||
--restart unless-stopped \
|
||||
-p <host-port>:<container-port> \
|
||||
<registry>/<image>:<tag>
|
||||
```
|
||||
|
||||
#### 方式 C：Nginx 反向代理更新
|
||||
```bash
|
||||
# 更新上游配置后重载
|
||||
nginx -t # 语法检查
|
||||
systemctl reload nginx # 热重载
|
||||
```
|
||||
|
||||
### 2.3 配置变更
|
||||
|
||||
```bash
|
||||
# 1. 备份当前配置
|
||||
cp /etc/<app>/config.yml /etc/<app>/config.yml.$(date +%Y%m%d-%H%M%S)
|
||||
|
||||
# 2. 修改配置
|
||||
vim /etc/<app>/config.yml
|
||||
|
||||
# 3. 重启服务使配置生效
|
||||
systemctl restart <service-name>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 三、部署验证
|
||||
|
||||
### 3.1 连通性验证
|
||||
```bash
|
||||
# 服务端口监听确认
|
||||
ss -tlnp | grep <port>
|
||||
|
||||
# HTTP 服务健康检查
|
||||
curl -s -o /dev/null -w "%{http_code}" http://localhost:<port>/health
|
||||
# 预期返回：200
|
||||
```
|
||||
|
||||
### 3.2 功能验证
|
||||
|
||||
- [ ] API 基础功能运行正常
|
||||
- [ ] 日志无新增 ERROR 级别报错
|
||||
- [ ] 数据库连接正常
|
||||
- [ ] 前端页面（如有）可正常加载
|
||||
|
||||
### 3.3 监控确认
|
||||
|
||||
- [ ] Prometheus / Grafana 指标正常
|
||||
- [ ] 日志系统（如有）已捕获新日志
|
||||
- [ ] 告警规则未被触发
|
||||
|
||||
---
|
||||
|
||||
## 四、回滚方案
|
||||
|
||||
### 4.1 代码回滚
|
||||
```bash
|
||||
# Git 回滚到上一版本
|
||||
cd /opt/app/repo
|
||||
git revert HEAD --no-edit
|
||||
git push
|
||||
# 重新执行部署
|
||||
```
|
||||
|
||||
### 4.2 文件回滚
|
||||
```bash
|
||||
# 恢复备份文件
|
||||
mv /opt/app/<app>.bak /opt/app/<app>
|
||||
systemctl restart <service-name>
|
||||
```
|
||||
|
||||
### 4.3 数据库回滚
|
||||
```bash
|
||||
# 导入备份
|
||||
gunzip < /backup/db/<dbname>.$(date +%Y%m%d).sql.gz | mysql -u root -p<pass> <dbname>
|
||||
```
|
||||
|
||||
### 4.4 回滚确认
|
||||
- [ ] 旧版本服务运行正常
|
||||
- [ ] 端口监听确认
|
||||
- [ ] 用户无访问异常
|
||||
- [ ] 记录回滚原因到工作日志
|
||||
|
||||
---
|
||||
|
||||
## 五、部署后记录
|
||||
|
||||
### 5.1 必填信息
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 部署时间 | YYYY-MM-DD HH:mm |
|
||||
| 部署人 | 严维序（opengineer） |
|
||||
| 部署内容 | [简要描述] |
|
||||
| 版本 | commit hash / tag |
|
||||
| 验证结果 | ✅/❌ 通过 |
|
||||
| 回滚情况 | 无需回滚 / 已回滚（原因） |
|
||||
|
||||
### 5.2 记录位置
|
||||
|
||||
- 工作日志：`memory/YYYY-MM-DD.md`
|
||||
- 任务记录：WorkBoard 相关卡片注释
|
||||
- 知识更新：如部署暴露流程问题，更新本文档
|
||||
|
||||
---
|
||||
|
||||
## 相关条目
|
||||
|
||||
- [故障排查手册_v1.0.md](故障排查手册_v1.0.md)
|
||||
- [服务器运维标准_v1.0.md](服务器运维标准_v1.0.md)
|
||||
|
||||
## 变更记录
|
||||
|
||||
| 日期 | 版本 | 变更说明 | 变更人 |
|
||||
|------|------|----------|--------|
|
||||
| 2026-06-24 | v1.0 | 初始创建 | 严维序 |
|
||||
@@ -1,9 +1,9 @@
|
||||
# BIZ-13 智能体运行稳定性保障方案
|
||||
|
||||
> 版本：v1.1
|
||||
> 版本：v1.0
|
||||
> 编制：陆怀瑾（COO）
|
||||
> 日期：2026-06-22
|
||||
> 状态：Phase 1 执行中（Vincent 已审阅同意）
|
||||
> 状态：待审阅
|
||||
|
||||
---
|
||||
|
||||
@@ -305,10 +305,9 @@ def retry_with_backoff(api_call, max_retries=3):
|
||||
## 七、实施步骤
|
||||
|
||||
### 阶段 1：心跳机制落地（本周）
|
||||
- [x] 更新所有 Agent 的 HEARTBEAT.md（15/15 Agent 已完成）
|
||||
- [x] 已创建分步实施子任务（BIZ-24 ~ BIZ-28，5个子任务）
|
||||
- [ ] 配置定时任务（10/15 分钟）→ BIZ-25，已分派 opengineer 严维序
|
||||
- [ ] 测试超时检测 → BIZ-24 执行中
|
||||
- [ ] 更新所有 Agent 的 HEARTBEAT.md
|
||||
- [ ] 配置定时任务（10 分钟）
|
||||
- [ ] 测试超时检测
|
||||
|
||||
### 阶段 2：限流优化（下周）
|
||||
- [ ] 实现请求队列
|
||||
|
||||
@@ -1,835 +0,0 @@
|
||||
# BIZ-24 HEARTBEAT.md 增强模板方案
|
||||
|
||||
> Phase 1 of BIZ-13 运行稳定性保障方案
|
||||
> 版本：v1.1（2026-06-22 优化：增加全任务源统一监控；已部署）
|
||||
> 编制：陆怀瑾（COO）
|
||||
> 日期：2026-06-22
|
||||
> 状态：已部署
|
||||
> 关联：[BIZ-13 运行稳定性保障方案](BIZ-13_运行稳定性保障方案.md)
|
||||
|
||||
---
|
||||
|
||||
## 一、目标
|
||||
|
||||
为所有 Agent 的 HEARTBEAT.md 文件统一增强以下机制，解决任务停滞、运行异常与工作遗漏问题：
|
||||
|
||||
1. **全任务源统一监控** — 覆盖 OpenClaw WorkBoard + Multica Issues + 待办文档，避免工作遗漏
|
||||
2. **禁止请示规则** — 消除"等待用户确认"导致的任务卡死
|
||||
3. **超时检测规则** — 按 Agent 类型差异化配置心跳频率
|
||||
4. **自动恢复规则** — 检测无进展时自动重新调度
|
||||
5. **依赖检查前置** — 任务启动前强制检查所有依赖
|
||||
6. **最大轮次限制** — 防止无限循环或资源耗尽
|
||||
|
||||
### 1.1 为什么需要全任务源统一监控
|
||||
|
||||
当前 Agent 工作面临的任务来源是多平台的：
|
||||
|
||||
| 任务来源 | 平台/工具 | 查询方式 | 当前监控状态 |
|
||||
|----------|-----------|----------|------------|
|
||||
| WorkBoard 卡片 | OpenClaw workboard | `openclaw workboard list` | ✅ 已纳入 |
|
||||
| 待办文档 | 各 Agent workspace 的 TODO.md / AGENTS.md | 文件读取 | ⚠️ 部分纳入 |
|
||||
| Multica Issues | Multica 平台 | `multica issue list --assignee-id <id>` | ❌ 未纳入 |
|
||||
|
||||
**问题**：Multica Issues 中分配给 Agent 的任务当前完全不在心跳监控范围内，Agent 可能永远不会发现并执行这些任务，导致工作永久遗漏。
|
||||
|
||||
**对策**：每次心跳同步检查以上三个来源，确保无一遗漏。
|
||||
|
||||
---
|
||||
|
||||
## 二、Agent 分类与参数配置
|
||||
|
||||
### 2.1 分类标准
|
||||
|
||||
| 分类 | 特征 | Agent |
|
||||
|------|------|-------|
|
||||
| 高频 Agent | 需频繁检查任务状态、全局监控 | secretary, coo |
|
||||
| 开发 Agent | 执行开发/设计/部署等长周期任务 | projectmanager, productmanager, architect, costcodev, designer, opengineer |
|
||||
| 业务 Agent | 执行专项业务任务 | taobaospecialist, contentspecialist, mediaspecialist, cvexpert, marketanalysis, lawyer |
|
||||
|
||||
### 2.2 参数配置矩阵
|
||||
|
||||
| 参数 | 高频 Agent | 开发 Agent | 业务 Agent |
|
||||
|------|-----------|-----------|-----------|
|
||||
| 心跳频率 | 10 分钟 | 15 分钟 | 15 分钟 |
|
||||
| 最大轮次 | 50 轮 | 100 轮 | 30 轮 |
|
||||
| 超时告警阈值 | 20 分钟无进展 | 30 分钟无进展 | 30 分钟无进展 |
|
||||
| 自动恢复等待 | 30 分钟后重新调度 | 45 分钟后重新调度 | 45 分钟后重新调度 |
|
||||
| 告警通知对象 | COO | COO + 创建者 | 创建者 |
|
||||
|
||||
---
|
||||
|
||||
## 三、六项增强规则详解
|
||||
|
||||
### 规则 0：全任务源统一监控
|
||||
|
||||
**问题**：Agent 的任务分布在多个平台（OpenClaw WorkBoard、Multica Issues、工作区待办文档），各平台独立存在，Agent 只监控其中一部分会导致工作任务被永久遗漏。
|
||||
|
||||
**规则文本**：
|
||||
```markdown
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
> 你的工作任务可能存在于三个地方：OpenClaw WorkBoard、Multica Issues、本地待办文档。
|
||||
|
||||
### 任务源检查清单（按优先级）
|
||||
|
||||
#### 第一优先级：OpenClaw WorkBoard 卡片
|
||||
|
||||
\```bash
|
||||
# 检查 WorkBoard 中分配给自己的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == '<your_agent_id>' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c[\"id\"][:8]} [priority={c.get(\"priority\",\"?\")}] {c[\"title\"]}')
|
||||
"
|
||||
\```
|
||||
|
||||
#### 第二优先级：Multica Issues
|
||||
|
||||
\```bash
|
||||
# 检查 Multica 中分配给自己的待办 Issue
|
||||
multica issue list --assignee-id <your_multica_agent_uuid> --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue[\"identifier\"]} [{issue.get(\"priority\",\"?\")}] {issue[\"title\"]}')
|
||||
"
|
||||
\```
|
||||
|
||||
#### 第三优先级：待办文档
|
||||
|
||||
\```bash
|
||||
# 检查工作区待办文档（TODO.md 或 AGENTS.md 中未完成的任务）
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档"
|
||||
\```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片
|
||||
↓
|
||||
检查 Multica Issues 待办
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重（避免同一任务在不同来源重复出现）
|
||||
↓
|
||||
按优先级排序后依次执行
|
||||
```
|
||||
|
||||
### Multica Issue 认领与执行流程
|
||||
|
||||
```
|
||||
发现 Multica todo Issue（assignee 是自己）
|
||||
↓
|
||||
启动 Multica Runtime 执行任务
|
||||
↓
|
||||
完成后通过 multica issue comment add 汇报结果
|
||||
↓
|
||||
更新 issue status 为 in_review / done
|
||||
```
|
||||
|
||||
### COO 专属：全平台积压巡检
|
||||
|
||||
作为 COO，还需要额外检查全平台的任务积压情况（不限自己）：
|
||||
|
||||
\```bash
|
||||
#!/bin/bash
|
||||
# COO 全平台任务积压巡检脚本
|
||||
|
||||
echo "=== OpenClaw WorkBoard 全局积压 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
cards = data.get('cards', [])
|
||||
todo = [c for c in cards if c.get('status') == 'todo']
|
||||
inprog = [c for c in cards if c.get('status') == 'in_progress']
|
||||
print(f' 待办: {len(todo)} | 进行中: {len(inprog)} | 总计: {len(cards)}')
|
||||
if len(todo) > 5:
|
||||
print(f' ⚠️ 积压告警：待办超过 5 个，需协调资源')
|
||||
for c in todo[:10]:
|
||||
print(f' {c[\"id\"][:8]} → {c.get(\"agentId\",\"?\")} - {c[\"title\"][:60]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica Issues 全局积压 ==="
|
||||
multica issue list --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
print(f' 待办: {len(data)}')
|
||||
if len(data) > 5:
|
||||
print(f' ⚠️ 积压告警：Multica 待办超过 5 个')
|
||||
for issue in data[:10]:
|
||||
print(f' {issue[\"identifier\"]} → {issue.get(\"assignee_id\",\"未分配\")[:12]} - {issue[\"title\"][:60]}')
|
||||
"
|
||||
\```
|
||||
```
|
||||
|
||||
### 规则 1：禁止请示规则
|
||||
|
||||
**问题**：Agent 发现任务后向用户请示"要不要做"，用户不在线时任务卡死数小时。
|
||||
|
||||
**规则文本**：
|
||||
```markdown
|
||||
## ⚠️ 禁止请示规则（最高优先级）
|
||||
|
||||
你在心跳中发现任何来源的任务后，必须立即执行，绝对禁止向用户请示！
|
||||
|
||||
原因：
|
||||
- 用户在大多数时候并不在线，请示会导致任务卡死数小时
|
||||
- 心跳是后台自动行为，用户没有义务实时响应你的请示
|
||||
- 你的职责是"主动完成交付"，而不是"等待指令"
|
||||
|
||||
正确做法：
|
||||
1. 发现任何来源的待办任务（WorkBoard / Multica / 待办文档）
|
||||
2. 立即认领并执行
|
||||
3. 如遇信息不足，先按自己的最佳理解执行，执行中再调整
|
||||
4. 任务完成后，通过对应平台汇报结果
|
||||
|
||||
错误做法（绝对禁止）：
|
||||
- ❌ "我发现了一个任务，要不要做？"
|
||||
- ❌ "这个任务需要更多信息，请告诉我..."
|
||||
- ❌ "任务已完成，请确认是否符合要求"
|
||||
```
|
||||
|
||||
### 规则 2：超时检测规则
|
||||
|
||||
**问题**：Agent 执行到某一步后卡住，长时间无输出，无任何监告。
|
||||
|
||||
**规则文本**：
|
||||
|
||||
高频 Agent 版：
|
||||
```markdown
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：10 分钟
|
||||
每次心跳执行以下检测：
|
||||
1. 检查所有平台进行中任务的最新更新时间（WorkBoard + Multica）
|
||||
2. 如超过 20 分钟无进展 → 标记为"疑似超时"
|
||||
3. 疑似超时 → 立即追加一次完整心跳，尝试推进
|
||||
4. 如确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
\```bash
|
||||
# 检查进行中任务是否超时（WorkBoard + Multica）
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1200:
|
||||
print(f'⏰ WB TIMEOUT: {c[\"id\"][:8]} [{c.get(\"agentId\",\"?\")}] {c[\"title\"]}')
|
||||
"
|
||||
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1200:
|
||||
print(f'⏰ MUL TIMEOUT: {issue[\"identifier\"]} [{issue.get(\"assignee_id\",\"?\")[:12]}] {issue[\"title\"]}')
|
||||
"
|
||||
\```
|
||||
```
|
||||
|
||||
开发 Agent 版（差异部分）：
|
||||
```markdown
|
||||
### 心跳频率：15 分钟
|
||||
每次心跳执行以下检测：
|
||||
1. 检查所有平台进行中任务的最新更新时间（WorkBoard + Multica）
|
||||
2. 如超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
```
|
||||
|
||||
业务 Agent 版（差异部分）：
|
||||
```markdown
|
||||
### 心跳频率：15 分钟
|
||||
每次心跳执行以下检测：
|
||||
1. 检查所有平台进行中任务的最新更新时间（WorkBoard + Multica）
|
||||
2. 如超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
```
|
||||
|
||||
### 规则 3：自动恢复规则
|
||||
|
||||
**问题**：检测到无进展后没有自动恢复手段，任务永久停滞。
|
||||
|
||||
**规则文本**：
|
||||
```markdown
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 恢复流程
|
||||
```
|
||||
检测到超时（跨平台无进展超阈值）
|
||||
↓
|
||||
步骤 1：追加一次完整心跳，尝试推进任务
|
||||
↓
|
||||
步骤 2：检查任务状态
|
||||
↓
|
||||
┌─────────────┴─────────────┐
|
||||
│ │
|
||||
有进展 仍无进展
|
||||
│ │
|
||||
重置超时计数器 步骤 3：通知 COO/创建者
|
||||
│ │
|
||||
继续执行 步骤 4：通过对应平台标记 blocked
|
||||
│
|
||||
步骤 5：重新调度（分配备用 Agent 或
|
||||
等待人工介入）
|
||||
```
|
||||
|
||||
### 自动恢复触发条件
|
||||
- 高频 Agent：超 30 分钟无进展 → 自动重新调度
|
||||
- 开发 Agent：超 45 分钟无进展 → 自动重新调度
|
||||
- 业务 Agent：超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 跨平台恢复操作
|
||||
**WorkBoard 任务**：
|
||||
1. 添加评论说明超时原因
|
||||
2. 释放 Agent 认领（release claim）
|
||||
3. 通知 COO 重新分配
|
||||
|
||||
**Multica Issue**：
|
||||
1. `multica issue comment add` 说明超时原因
|
||||
2. `multica issue status <id> blocked`
|
||||
3. 通知 COO 重新分配
|
||||
|
||||
**待办文档任务**：
|
||||
1. 在原文档中标注超时状态
|
||||
2. 如可转为 WorkBoard 卡片 → 创建卡片并通知 COO
|
||||
```
|
||||
|
||||
### 规则 4：依赖检查前置
|
||||
|
||||
**问题**：任务开始后才发现依赖未满足，浪费 Agent 时间，且可能导致循环等待。
|
||||
|
||||
**规则文本**：
|
||||
```markdown
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 任务启动前强制检查
|
||||
每次认领或启动任务前，必须执行依赖检查：
|
||||
|
||||
**WorkBoard 任务**：
|
||||
1. 读取任务的 depends_on 字段
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 所有依赖 ready → 可以启动
|
||||
4. 任一依赖未完成 → 禁止启动，标记为 blocked
|
||||
|
||||
**Multica Issue**：
|
||||
1. 读取 issue 的 parent_issue_id
|
||||
2. 检查父 issue 状态
|
||||
3. 父 issue 未完成 → 禁止启动
|
||||
|
||||
### 检查脚本
|
||||
|
||||
#### WorkBoard 依赖检查
|
||||
\```bash
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
print(f'依赖: {dep[\"id\"]} → 状态: {dep.get(\"status\", \"?\")}')
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足，禁止启动 {card[\"id\"][:8]}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖，可以启动')
|
||||
"
|
||||
\```
|
||||
|
||||
#### Multica 依赖检查
|
||||
\```bash
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
import subprocess
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent[\"identifier\"]} 未完成，禁止启动')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent[\"identifier\"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖，可以启动')
|
||||
"
|
||||
\```
|
||||
|
||||
### 依赖未满足时的处理
|
||||
1. 不认领任务（保持 todo 状态）
|
||||
2. 不在该任务上浪费心跳时间
|
||||
3. 如超过等待阈值（高频 1h / 开发/业务 2h），通知依赖任务的执行者
|
||||
```
|
||||
|
||||
### 规则 5：最大轮次限制
|
||||
|
||||
**问题**：Agent 陷入无限循环，反复执行相同逻辑无进展，持续消耗 API 配额。
|
||||
|
||||
**规则文本**：
|
||||
|
||||
高频 Agent 版：
|
||||
```markdown
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：50 轮
|
||||
单次任务执行不得超过 50 个对话轮次。
|
||||
|
||||
### 检测机制
|
||||
- 每次心跳记录已消耗轮次
|
||||
- 接近上限（80%）时发出预警
|
||||
- 达到上限时自动暂停
|
||||
|
||||
### 超限处理
|
||||
```
|
||||
达到最大轮次
|
||||
↓
|
||||
1. 暂停任务执行
|
||||
2. 记录已完成的步骤和未完成的部分
|
||||
3. 通知 COO，附当前进度
|
||||
4. COO 决定：重新分配 / 拆分任务 / 人工介入
|
||||
```
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
- **WorkBoard**：通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**：通过 issue comment 记录轮次进度
|
||||
- **待办文档**：在工作日志中记录
|
||||
|
||||
### 计数器维护
|
||||
\```bash
|
||||
# 每次心跳更新轮次计数
|
||||
# 轮次数据存储在任务 metadata 或 comment 中
|
||||
\```
|
||||
```
|
||||
|
||||
开发 Agent 版（差异部分）：
|
||||
```markdown
|
||||
### 限制值：100 轮
|
||||
单次任务执行不得超过 100 个对话轮次。
|
||||
```
|
||||
|
||||
业务 Agent 版（差异部分）：
|
||||
```markdown
|
||||
### 限制值：30 轮
|
||||
单次任务执行不得超过 30 个对话轮次。
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 四、HEARTBEAT.md 完整增强模板
|
||||
|
||||
### 4.1 高频 Agent 完整模板（secretary / coo）
|
||||
|
||||
```markdown
|
||||
# HEARTBEAT.md - [Agent Name] 的心跳配置
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
1. **OpenClaw WorkBoard** — workboard list → 查找 agent_id=自己 & status=todo
|
||||
2. **Multica Issues** — issue list --assignee-id <uuid> --status todo
|
||||
3. **待办文档** — 检查 TODO.md / AGENTS.md 中未完成的任务
|
||||
|
||||
### 合并去重 → 按优先级排序 → 依次执行
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：10 分钟
|
||||
每次心跳跨平台执行以下检测：
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 20 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
- 超 30 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（1h）→ 通知依赖任务执行者
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：50 轮
|
||||
- 接近 80%（40 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知 COO
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 全平台积压巡检（仅 COO）
|
||||
6. ✅ [Agent 专属检查项]
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
```
|
||||
|
||||
### 4.2 开发 Agent 完整模板（projectmanager / productmanager / architect / costcodev / designer / opengineer）
|
||||
|
||||
```markdown
|
||||
# HEARTBEAT.md - [Agent Name] 的心跳配置
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
1. **OpenClaw WorkBoard** — workboard list → 查找 agent_id=自己 & status=todo
|
||||
2. **Multica Issues** — issue list --assignee-id <uuid> --status todo
|
||||
3. **待办文档** — 检查 TODO.md / AGENTS.md 中未完成的任务
|
||||
|
||||
### 合并去重 → 按优先级排序 → 依次执行
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：15 分钟
|
||||
每次心跳跨平台执行以下检测：
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO + 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO + 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（2h）→ 通知依赖任务执行者
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：100 轮
|
||||
- 接近 80%（80 轮）→ 预警
|
||||
- 达到上限 → 暂停，记录日志
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ [Agent 专属检查项]
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
```
|
||||
|
||||
### 4.3 业务 Agent 完整模板（taobaospecialist / contentspecialist / mediaspecialist / cvexpert / marketanalysis / lawyer）
|
||||
|
||||
```markdown
|
||||
# HEARTBEAT.md - [Agent Name] 的心跳配置
|
||||
|
||||
## 📋 全任务源统一监控（每次心跳必检）
|
||||
|
||||
> **核心原则：发现任何来源的任务都必须立即执行，不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
1. **OpenClaw WorkBoard** — workboard list → 查找 agent_id=自己 & status=todo
|
||||
2. **Multica Issues** — issue list --assignee-id <uuid> --status todo
|
||||
3. **待办文档** — 检查 TODO.md / AGENTS.md 中未完成的任务
|
||||
|
||||
### 合并去重 → 按优先级排序 → 依次执行
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则（最高优先级，自动执行）
|
||||
|
||||
> **核心规则：发现分配给自己的任务 → 立即执行，不得向用户请示！**
|
||||
> 心跳是后台自动行为，用户在大多数时候并不在线，请示会导致任务卡死。
|
||||
|
||||
### 执行流程（严格执行，无需确认）
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行，不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率：15 分钟
|
||||
每次心跳跨平台执行以下检测：
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作（按平台）
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片（可选） |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
1. 认领任务前，读取依赖字段（depends_on / parent_issue_id）
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领（保持 todo）
|
||||
4. 超过等待阈值（2h）→ 通知依赖任务执行者
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值：30 轮
|
||||
- 接近 80%（24 轮）→ 预警
|
||||
- 达到上限 → 暂停，通知创建者
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
1. ✅ **全任务源检查**：WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测（跨平台）
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ [Agent 专属检查项]
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行，不得请示**（任何来源）
|
||||
4. **超时任务按自动恢复流程处理**（跨平台）
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查，缺一不可
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 五、部署清单
|
||||
|
||||
### 5.1 各 Agent HEARTBEAT.md 更新状态
|
||||
|
||||
| Agent | 分类 | 模板版本 | 部署状态 | 部署人 |
|
||||
|-------|------|---------|---------|--------|
|
||||
| secretary (刘诗妮) | 高频 | 高频 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| coo (陆怀瑾) | 高频 | 高频 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| projectmanager (胡蓉) | 开发 | 开发 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| productmanager (沈路明) | 开发 | 开发 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| architect (梁思筑) | 开发 | 开发 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| costcodev (徐聪) | 开发 | 开发 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| designer (苏绘锦) | 开发 | 开发 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| opengineer (严维序) | 开发 | 开发 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| taobaospecialist (陆云帆) | 业务 | 业务 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| contentspecialist (文墨言) | 业务 | 业务 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| mediaspecialist (钟帧韵) | 业务 | 业务 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| cvexpert (程伯予) | 业务 | 业务 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| marketanalysis (顾析策) | 业务 | 业务 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| lawyer (苏慎) | 业务 | 业务 Agent 模板 v1.1 | 待部署 | COO |
|
||||
|
||||
### 5.2 部署步骤
|
||||
|
||||
1. **Vincent 审阅本方案** — 确认参数配置和多源监控范围
|
||||
2. **收集各 Agent 的 Multica UUID** — 用于 `multica issue list --assignee-id <uuid>` 查询
|
||||
3. **创建 HEARTBEAT.md 文件** — 按 v1.1 模板为每个 Agent 创建（填充实际 ID）
|
||||
4. **配置心跳 cron** — 按分类配置定时任务
|
||||
5. **部署到各 Agent workspace** — 将 HEARTBEAT.md 分发到对应 Agent 工作区
|
||||
6. **验证** — 等待一轮完整心跳，检查三源任务是否全量覆盖
|
||||
|
||||
### 5.3 Agent Multica UUID 映射（已收集）
|
||||
|
||||
| Agent | OpenClaw Agent ID | Multica Agent UUID | 状态 |
|
||||
|-------|-------------------|-------------------|------|
|
||||
| secretary (刘诗妮) | secretary | b024fcdc-30ff-420d-b289-498041466e1b | ✅ |
|
||||
| coo (陆怀瑾) | coo | 1c38b437-b54d-4784-bda3-29ce4c8a6722 | ✅ |
|
||||
| projectmanager (胡蓉) | projectmanager | d877b8c3-b230-4073-b3f7-80e148cfdb71 | ✅ |
|
||||
| productmanager (沈路明) | productmanager | a101fa88-d821-4839-9754-e04580d5fd68 | ✅ |
|
||||
| architect (梁思筑) | architect | 40abd41a-62d0-416d-bc44-92c1f758d87a | ✅ |
|
||||
| costcodev (徐聪) | costcodev | 46bdd4a6-5c64-475a-92ef-36a763602fa1 | ✅ |
|
||||
| designer (苏锦绘) | designer | 13bd8968-cc2a-4934-90c7-957a2d3c09c2 | ✅ |
|
||||
| opengineer (严维序) | opengineer | d3804433-9e2e-4199-a92b-a153049b3bc9 | ✅ |
|
||||
| taobaospecialist (陆云帆) | taobaospecialist | e0f62d8f-9568-4f41-8ad4-b73d79a163a7 | ✅ |
|
||||
| contentspecialist (文墨言) | contentspecialist | 8321b0bf-7d89-4ece-927a-0780f42ad396 | ✅ |
|
||||
| mediaspecialist (钟帧韵) | mediaspecialist | e2b587d4-1d16-447c-8ad9-e2a01358ff0a | ✅ |
|
||||
| cvexpert (程伯予) | cvexpert | 4a8696fd-6531-40da-8956-ef84d7ea3c43 | ✅ |
|
||||
| marketanalysis (顾析策) | marketanalysis | 5ed91729-658f-4654-98f0-3e0313022002 | ✅ |
|
||||
| lawyer (苏慎) | lawyer | 6fb0fbd2-16a6-4566-ba7a-d2c136baec25 | ✅ |
|
||||
|
||||
---
|
||||
|
||||
## 六、交付物
|
||||
|
||||
- [x] HEARTBEAT.md 增强模板方案 v1.0（初始版本）
|
||||
- [x] HEARTBEAT.md 增强模板方案 v1.1（优化：增加全任务源统一监控）
|
||||
- [x] 各 Agent Multica UUID 映射表
|
||||
- [x] 14 个 Agent 的独立 HEARTBEAT.md 文件（v1.1，已生成并部署到 workspace）
|
||||
- [ ] 心跳 cron 配置脚本
|
||||
- [ ] 部署验证报告
|
||||
|
||||
---
|
||||
|
||||
## 七、v1.1 变更说明
|
||||
|
||||
| 变更项 | v1.0 | v1.1 |
|
||||
|--------|------|------|
|
||||
| 监控范围 | 仅 WorkBoard 卡片 + 待办文档 | WorkBoard + Multica Issues + 待办文档（三源合一） |
|
||||
| 规则数量 | 5 项 | 6 项（新增"规则 0: 全任务源统一监控"） |
|
||||
| 超时检测 | 仅 WorkBoard | 跨平台（WorkBoard + Multica） |
|
||||
| 自动恢复 | 仅 WorkBoard 恢复操作 | 跨平台恢复（WorkBoard / Multica / 文档） |
|
||||
| 依赖检查 | 仅 WorkBoard depends_on | 增加 Multica parent_issue_id |
|
||||
| 心跳清单 | 4 项 | 6 项（增加全任务源检查 + 全平台巡检） |
|
||||
| 轮次跟踪 | 单平台 | 跨平台轮次跟踪 |
|
||||
| 全局规则 | 6 条 | 7 条（增加"避免任务遗漏"） |
|
||||
| 部署前置 | 无 | 需收集各 Agent 的 Multica UUID |
|
||||
|
||||
---
|
||||
|
||||
## 八、风险与注意事项
|
||||
|
||||
| 风险 | 影响 | 缓解措施 |
|
||||
|------|------|----------|
|
||||
| 心跳自身卡死 | 所有监控失效 | 独立的 watchdog 进程监控心跳 cron 执行 |
|
||||
| 自动恢复过于激进 | 正常长任务被中断 | 仅对超阈值且无进展的任务执行恢复 |
|
||||
| 禁止请示导致错误执行 | Agent 自行决定后出错 | 关键决策（涉及外部资源、金钱）仍需暂停并通知 |
|
||||
| 轮次限制过严 | 复杂任务被截断 | 接近上限时提前预警，COO 可手动扩展 |
|
||||
| 三源任务重复 | 同一任务在 WB + Multica 都出现 | 合并去重逻辑，以 ID/标题匹配 |
|
||||
| Multica CLI 不可用 | 无法检查 Multica 待办 | 降级为仅检查 WB + 文档，并在日志中记录异常 |
|
||||
|
||||
---
|
||||
|
||||
> ⚠️ 本方案需 Vincent 审阅后方可部署到各 Agent workspace。当前为模板方案 v1.1，存放于 EnterpriseArchitect/plans/ 目录。
|
||||
@@ -186,18 +186,7 @@ python3 shared/scripts/heartbeat_helper.py <agent_id> --health
|
||||
|
||||
---
|
||||
|
||||
## 八、修复记录
|
||||
|
||||
### v1.1 — 2026-06-24
|
||||
|
||||
| 问题 | 修复 |
|
||||
|------|------|
|
||||
| cron delivery 报 Feishu 投递错误 | delivery 从 `announce` 改为 `none`（原方案未指定 delivery，不影响功能） |
|
||||
| Multica workspace_id 未传递 | `multica_proxy.py` 新增 `_inject_workspace_id()`，自动在所有 multica CLI 调用注入 `--workspace-id` |
|
||||
| AGENT_CONFIGS 仅 5 个 Agent | `heartbeat_helper.py` 扩展至全部 15 个 Agent |
|
||||
| COO HEARTBEAT 显示未部署 | 更新 BIZ-38 集成清单表 |
|
||||
|
||||
## 九、后续优化方向
|
||||
## 八、后续优化方向
|
||||
|
||||
- [ ] 监控面板集成（BIZ-28 Phase3）
|
||||
- [ ] 心跳结果聚合展示
|
||||
|
||||
@@ -1,186 +0,0 @@
|
||||
# HEARTBEAT.md 增强模板
|
||||
|
||||
> 版本：v2.0
|
||||
> 来源：BIZ-13 运行稳定性保障方案
|
||||
> 用途：为所有 Agent HEARTBEAT.md 增加运行稳定性保障能力
|
||||
|
||||
---
|
||||
|
||||
## 全局增强规则（所有 Agent 必须包含）
|
||||
|
||||
### 1. 🛡️ 超时检测与自动恢复
|
||||
|
||||
```markdown
|
||||
## 🛡️ 超时检测与自动恢复
|
||||
|
||||
> **核心规则：每次心跳，检查自己是否有任务超时未完成。**
|
||||
|
||||
### 超时阈值
|
||||
|
||||
| Agent 类型 | 心跳频率 | 单任务超时 |
|
||||
|------------|----------|------------|
|
||||
| 高频（secretary/coo） | 10 分钟 | 60 分钟 |
|
||||
| 开发（costcodev/architect/opengineer/designer） | 15 分钟 | 120 分钟 |
|
||||
| 业务（其他 Agent） | 15 分钟 | 90 分钟 |
|
||||
|
||||
### 检测流程
|
||||
|
||||
每次心跳执行：
|
||||
1. 获取自己的 `status=in_progress` 的 WorkBoard 卡片
|
||||
2. 计算 `当前时间 - started_at`
|
||||
3. 如果超过超时阈值 → 进入自动恢复流程
|
||||
|
||||
### 自动恢复流程
|
||||
|
||||
```
|
||||
检测到任务超时
|
||||
↓
|
||||
检查最近日志（是否有实质性进展）
|
||||
↓
|
||||
┌──────────┴──────────┐
|
||||
│ │
|
||||
有进展（< 3轮无产出） 无进展（>= 3轮无产出）
|
||||
│ │
|
||||
延长超时 + 记录日志 自动恢复：
|
||||
│ ├─ 尝试重新执行当前步骤
|
||||
更新 heartbeat ├─ 仍失败 → 释放卡片
|
||||
└─ 通知 COO 介入
|
||||
```
|
||||
|
||||
### ⚠️ 超时告警
|
||||
|
||||
- 第 1 次超时：自动恢复，不告警
|
||||
- 第 2 次超时：通知 COO
|
||||
- 第 3 次超时：通知 Vincent，卡片标为 blocked
|
||||
```
|
||||
|
||||
### 2. 🔗 依赖检查前置
|
||||
|
||||
```markdown
|
||||
## 🔗 依赖检查前置
|
||||
|
||||
> **核心规则：认领任务前，必须检查所有依赖是否已完成。**
|
||||
|
||||
### 检查流程
|
||||
|
||||
1. 获取任务的 `depends_on` 列表
|
||||
2. 对每个依赖，查询其状态
|
||||
3. 如果任一依赖未完成 → 不认领该任务，等待下次心跳
|
||||
4. 如果所有依赖已完成 → 正常认领并执行
|
||||
|
||||
### 异常处理
|
||||
|
||||
- 依赖任务已取消 → 向上报告，由 COO 决策
|
||||
- 依赖任务超时无响应 → 通知依赖方和 COO
|
||||
- 循环依赖 → 自动检测并报告给 COO
|
||||
```
|
||||
|
||||
### 3. 🔄 最大轮次限制
|
||||
|
||||
```markdown
|
||||
## 🔄 最大轮次限制
|
||||
|
||||
> **核心规则：单任务不能无限循环执行。**
|
||||
|
||||
| Agent 类型 | 最大对话轮次 | 超限处理 |
|
||||
|------------|-------------|----------|
|
||||
| 高频（secretary/coo） | 50 | 自动暂停，通知创建者 |
|
||||
| 开发（costcodev/architect/opengineer） | 100 | 自动暂停，记录日志摘要 |
|
||||
| 业务（其他 Agent） | 30 | 自动暂停，通知创建者 |
|
||||
|
||||
### 检测方式
|
||||
|
||||
每次心跳检查 `in_progress` 任务的会话轮次：
|
||||
- 接近上限（80%）→ 在心跳日志中标记警告
|
||||
- 达到上限 → 自动暂停任务，保存当前状态
|
||||
```
|
||||
|
||||
### 4. 📊 上下文控制
|
||||
|
||||
```markdown
|
||||
## 📊 上下文控制（Token 管理）
|
||||
|
||||
> **核心规则：避免上下文溢出导致任务中断。**
|
||||
|
||||
### 策略
|
||||
|
||||
1. **引用代替填塞**：Agent 配置文件中只保留核心规则，详细信息存 docs/ 目录
|
||||
2. **分块读取**：超大文件分块读取，避免一次性加载
|
||||
3. **清理过期信息**：每轮对话前清理上一轮的工具输出（仅保留关键结果）
|
||||
4. **合并查询**：多个 Agent 相同查询由 COO 统一执行后广播
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 心跳频率分级
|
||||
|
||||
```markdown
|
||||
## ⏱️ 心跳触发频率
|
||||
|
||||
- **高频 Agent（secretary / coo）**: 每 10 分钟
|
||||
- **开发 Agent（costcodev / architect / opengineer / designer）**: 每 15 分钟
|
||||
- **业务 Agent（projectmanager / productmanager / taobaospecialist / contentspecialist / mediaspecialist / cvexpert / marketanalysis / lawyer）**: 每 15 分钟
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 全局关键规则（增强版）
|
||||
|
||||
```markdown
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 如果用户正在与 Agent 对话，心跳逻辑延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户主动询问时再汇报
|
||||
3. **发现任务立即执行，不得请示** — 用户在大多数时候不在线，请示=任务卡死
|
||||
4. **依赖检查前置** — 认领任务前必须检查所有依赖是否已完成
|
||||
5. **超时自动恢复** — 任务超时自动尝试恢复，3 次失败后升级
|
||||
6. **轮次限制** — 单任务达上限后自动暂停，防止无限循环
|
||||
7. **上下文控制** — 引用代替填塞，避免 Token 溢出
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 各 Agent 类型模板
|
||||
|
||||
### 高频 Agent 模板（secretary/coo）
|
||||
|
||||
在原有专属心跳清单基础上，增加：
|
||||
```markdown
|
||||
### 🛡️ 稳定性保障清单
|
||||
|
||||
1. ✅ 超时检测：检查 in_progress 任务是否超时（阈值 60 分钟）
|
||||
2. ✅ 依赖检查：新任务认领前检查所有 depends_on
|
||||
3. ✅ 轮次检查：当前任务是否接近 50 轮上限
|
||||
4. ✅ 上下文检查：HEARTBEAT.md/AGENTS.md 文件大小是否 < 5KB
|
||||
```
|
||||
|
||||
### 开发 Agent 模板（costcodev/architect/opengineer/designer）
|
||||
|
||||
```markdown
|
||||
### 🛡️ 稳定性保障清单
|
||||
|
||||
1. ✅ 超时检测：检查 in_progress 任务是否超时（阈值 120 分钟）
|
||||
2. ✅ 依赖检查：新任务认领前检查所有 depends_on
|
||||
3. ✅ 轮次检查：当前任务是否接近 100 轮上限
|
||||
4. ✅ 编译/测试检查：如有自动化测试，确认通过
|
||||
```
|
||||
|
||||
### 业务 Agent 模板（其他 Agent）
|
||||
|
||||
```markdown
|
||||
### 🛡️ 稳定性保障清单
|
||||
|
||||
1. ✅ 超时检测：检查 in_progress 任务是否超时（阈值 90 分钟）
|
||||
2. ✅ 依赖检查：新任务认领前检查所有 depends_on
|
||||
3. ✅ 轮次检查：当前任务是否接近 30 轮上限
|
||||
4. ✅ 输出质量检查：确认最近产出符合质量标准
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 实施说明
|
||||
|
||||
1. 此模板由 COO（陆怀瑾）编制，经 Vincent 审阅批准后实施
|
||||
2. 模板中的 agent_id 需替换为各 Agent 的实际标识
|
||||
3. 无需移除各 Agent 原有的专属心跳清单，只需追加稳定性保障清单
|
||||
4. 修改后的文件需提交到 EnterpriseArchitect git 仓库
|
||||
@@ -1,3 +0,0 @@
|
||||
__pycache__/
|
||||
*.egg-info/
|
||||
.mypy_cache/
|
||||
@@ -1,40 +0,0 @@
|
||||
# NVIDIA Sidecar 限流代理 — 生产 Docker 镜像 (BIZ-46 Phase3 §4)
|
||||
#
|
||||
# 构建：
|
||||
# docker build -t nvidia-sidecar:latest .
|
||||
#
|
||||
# 运行：
|
||||
# docker run -d --name nvidia-sidecar \
|
||||
# -p 127.0.0.1:9190:9190 \
|
||||
# -p 127.0.0.1:9191:9191 \
|
||||
# -e SIDECAR_API_KEY="nvapi-xxx" \
|
||||
# -e SIDECAR_RATE_RPM=40 \
|
||||
# -v $(pwd)/logs:/opt/nvidia-sidecar/logs \
|
||||
# nvidia-sidecar:latest
|
||||
|
||||
FROM python:3.12-slim AS base
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# 安装依赖（利用 Docker 层缓存）
|
||||
COPY pyproject.toml .
|
||||
RUN pip install --no-cache-dir fastapi>=0.115 \
|
||||
"uvicorn[standard]>=0.34" httpx>=0.28 PyYAML>=6.0 \
|
||||
structlog>=24.4 "prometheus-client>=0.21" pydantic>=2.0
|
||||
|
||||
# 复制源码
|
||||
COPY . .
|
||||
|
||||
# 非 root 用户运行
|
||||
RUN useradd -r -m -s /bin/false sidecar \
|
||||
&& mkdir -p /opt/nvidia-sidecar/logs \
|
||||
&& chown -R sidecar:sidecar /app /opt/nvidia-sidecar/logs
|
||||
USER sidecar
|
||||
|
||||
# 健康检查
|
||||
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
|
||||
CMD python -c "import httpx; r=httpx.get('http://127.0.0.1:9190/health'); exit(0 if r.status_code==200 else 1)"
|
||||
|
||||
EXPOSE 9190 9191
|
||||
|
||||
CMD ["uvicorn", "nvidia_sidecar.server:app", "--host", "0.0.0.0", "--port", "9190"]
|
||||
@@ -2,8 +2,6 @@
|
||||
|
||||
为 NVIDIA API 提供**优先级排队 + 令牌桶限流**的透明代理层。
|
||||
|
||||
> BIZ-46 Phase3: 架构解耦、Prometheus 标签治理、SSE 共享缓存、部署支撑、测试完善、Dashboard UX 优化。
|
||||
|
||||
## 快速启动
|
||||
|
||||
```bash
|
||||
@@ -24,7 +22,7 @@ nvidia-sidecar
|
||||
| `SIDECAR_API_KEY` | — | NVIDIA API Key（必填） |
|
||||
| `SIDECAR_RATE_RPM` | `40` | 每分钟请求数限制 |
|
||||
| `SIDECAR_BUCKET_CAPACITY` | `40` | 令牌桶容量 |
|
||||
| `SIDECAR_TIMEOUT` | `60` | 上游请求超时（秒） |
|
||||
| `SIDECAR_TIMEOUT` | `6000` | 上游请求超时（秒） |
|
||||
| `SIDECAR_QUEUE_MAX` | `500` | 队列最大长度 |
|
||||
| `SIDECAR_LOW_TIMEOUT` | `2.0` | 低优先级令牌等待超时（秒） |
|
||||
| `SIDECAR_FALLBACK_PASSTHROUGH` | `true` | 队列满时是否直通上游 |
|
||||
@@ -50,61 +48,8 @@ nvidia-sidecar --config /etc/nvidia-sidecar.yaml
|
||||
| `/v1/completions` | POST | OpenAI Completions 代理（legacy） |
|
||||
| `/v1/embeddings` | POST | OpenAI Embeddings 代理 |
|
||||
| `/v1/models` | GET | 模型列表代理 |
|
||||
| `/health` | GET | 存活检查 (liveness) |
|
||||
| `/health/ready` | GET | 就绪检查 (readiness，含上游连通性) |
|
||||
| `/status` | GET | 调试用完整状态（限流器 + 队列 + 避退） |
|
||||
| `/api/dashboard/stream` | GET | SSE 仪表盘实时推送 |
|
||||
| `/api/dashboard` | GET | 仪表盘 HTML 页面 |
|
||||
| `/api/admin/config` | GET/POST | 配置查询/热重载（需 Admin Token） |
|
||||
| `/metrics` | :9191 | Prometheus 指标端点（独立端口） |
|
||||
|
||||
## 部署方式
|
||||
|
||||
### Docker（推荐）
|
||||
|
||||
```bash
|
||||
# 构建
|
||||
docker build -t nvidia-sidecar:latest .
|
||||
|
||||
# 运行
|
||||
docker run -d --name nvidia-sidecar \
|
||||
-p 127.0.0.1:9190:9190 \
|
||||
-p 127.0.0.1:9191:9191 \
|
||||
-e SIDECAR_API_KEY="nvapi-xxx" \
|
||||
nvidia-sidecar:latest
|
||||
```
|
||||
|
||||
### systemd
|
||||
|
||||
```bash
|
||||
# 安装
|
||||
sudo cp deploy/nvidia-sidecar.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable nvidia-sidecar
|
||||
|
||||
# 配置环境变量
|
||||
sudo cp deploy/.env.example /opt/nvidia-sidecar/.env
|
||||
sudo vim /opt/nvidia-sidecar/.env # 填入实际值
|
||||
|
||||
# 启动
|
||||
sudo systemctl start nvidia-sidecar
|
||||
sudo journalctl -u nvidia-sidecar -f # 查看日志
|
||||
```
|
||||
|
||||
### 环境变量清单
|
||||
|
||||
详见 `deploy/.env.example`。
|
||||
|
||||
### 防火墙建议
|
||||
|
||||
```bash
|
||||
# 仅允许内网访问代理端口
|
||||
sudo ufw allow from 192.168.1.0/24 to any port 9190
|
||||
sudo ufw allow from 192.168.1.0/24 to any port 9191
|
||||
# 禁止外网访问
|
||||
sudo ufw deny 9190
|
||||
sudo ufw deny 9191
|
||||
```
|
||||
| `/health` | GET | 健康检查 |
|
||||
| `/metrics` | GET | 指标查询 |
|
||||
|
||||
## 架构
|
||||
|
||||
|
||||
@@ -56,7 +56,7 @@ class SidecarConfig:
|
||||
|
||||
# ---- 超时 ----
|
||||
request_timeout: float = field(
|
||||
default=60.0,
|
||||
default=6000.0,
|
||||
metadata={"env": "SIDECAR_TIMEOUT"},
|
||||
)
|
||||
|
||||
@@ -153,14 +153,9 @@ def _validate_config(config: SidecarConfig) -> list[str]:
|
||||
# request_timeout 合理性
|
||||
if config.request_timeout <= 0:
|
||||
issues.append(
|
||||
f"request_timeout ({config.request_timeout}) 无效，回退到默认值 60"
|
||||
f"request_timeout ({config.request_timeout}) 无效，回退到默认值 6000"
|
||||
)
|
||||
config.request_timeout = 60.0
|
||||
elif config.request_timeout > 300.0:
|
||||
issues.append(
|
||||
f"request_timeout ({config.request_timeout}) 异常偏高，已截断为 300"
|
||||
)
|
||||
config.request_timeout = 300.0
|
||||
config.request_timeout = 6000.0
|
||||
|
||||
return issues
|
||||
|
||||
|
||||
@@ -1,75 +0,0 @@
|
||||
"""
|
||||
NVIDIA Sidecar — SidecarContext 依赖注入容器 (§BIZ-46 Phase3)
|
||||
|
||||
将所有模块级全局状态收敛为单一 dataclass，通过 FastAPI app.state 注入，
|
||||
消除 webui.py → server 的反向导入，支持可测试性和多实例扩展。
|
||||
|
||||
设计文档: docs/architecture/BIZ-46_Phase3_Architecture_Design.md §1
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import httpx
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from nvidia_sidecar.config import SidecarConfig
|
||||
from nvidia_sidecar.rate_limiter import AdaptiveTokenBucket
|
||||
from nvidia_sidecar.priority_queue import PriorityRequestQueue
|
||||
from nvidia_sidecar.metrics import PrometheusMetrics
|
||||
from nvidia_sidecar.health import HealthService
|
||||
|
||||
|
||||
@dataclass
|
||||
class SidecarContext:
|
||||
"""Sidecar 全局运行时上下文 — 所有核心组件的唯一容器。
|
||||
|
||||
通过 ``app.state.sidecar`` 注入 FastAPI，路由通过 ``Depends(get_context)`` 获取。
|
||||
"""
|
||||
|
||||
# ---- 核心组件 ----
|
||||
config: SidecarConfig
|
||||
http_client: httpx.AsyncClient
|
||||
token_bucket: AdaptiveTokenBucket
|
||||
priority_queue: PriorityRequestQueue
|
||||
prometheus: PrometheusMetrics
|
||||
health: HealthService
|
||||
|
||||
# ---- 运行时状态 ----
|
||||
pending_requests: dict[str, tuple["asyncio.Future[Any]", float]] = field(default_factory=dict)
|
||||
"""request_id → (response future, enqueued_at) 的映射。"""
|
||||
|
||||
stats: dict[str, int] = field(default_factory=lambda: {
|
||||
"total_requests": 0,
|
||||
"nvidia_requests": 0,
|
||||
"passthrough_requests": 0,
|
||||
"ratelimited_requests": 0,
|
||||
"queue_full_rejects": 0,
|
||||
"upstream_errors": 0,
|
||||
"start_time": 0,
|
||||
})
|
||||
|
||||
stats_lock: asyncio.Lock = field(default_factory=asyncio.Lock)
|
||||
|
||||
# ---- 缓存 ----
|
||||
snapshot_cache: tuple["dict[str, Any]", float] | None = None
|
||||
"""SSE 快照共享缓存: (data, timestamp)。"""
|
||||
snapshot_cache_lock: asyncio.Lock = field(default_factory=asyncio.Lock)
|
||||
SNAPSHOT_CACHE_TTL: float = 1.0
|
||||
|
||||
# ---- 便捷方法 ----
|
||||
|
||||
async def increment_stat(self, key: str, delta: int = 1) -> None:
|
||||
"""线程安全的统计计数器自增。"""
|
||||
async with self.stats_lock:
|
||||
self.stats[key] = self.stats.get(key, 0) + delta
|
||||
|
||||
@property
|
||||
def uptime_seconds(self) -> int:
|
||||
"""服务运行时长（秒）。"""
|
||||
st = self.stats.get("start_time", 0)
|
||||
return int(time.time() - st) if st else 0
|
||||
@@ -1,31 +0,0 @@
|
||||
# NVIDIA Sidecar 环境变量清单 (BIZ-46 Phase3 §4)
|
||||
# 复制为 .env 后按需修改，供 Docker / systemd 使用。
|
||||
|
||||
# 网络
|
||||
SIDECAR_HOST=127.0.0.1
|
||||
SIDECAR_PORT=9190
|
||||
SIDECAR_METRICS_PORT=9191
|
||||
|
||||
# 上游 API（必填）
|
||||
SIDECAR_UPSTREAM=https://integrate.api.nvidia.com/v1
|
||||
SIDECAR_API_KEY=nvapi-your-key-here
|
||||
|
||||
# 限流
|
||||
SIDECAR_RATE_RPM=40
|
||||
SIDECAR_BUCKET_CAPACITY=40
|
||||
|
||||
# 超时
|
||||
SIDECAR_TIMEOUT=60
|
||||
|
||||
# 队列
|
||||
SIDECAR_QUEUE_MAX=500
|
||||
SIDECAR_LOW_TIMEOUT=2
|
||||
|
||||
# 降级
|
||||
SIDECAR_FALLBACK_PASSTHROUGH=true
|
||||
|
||||
# 日志
|
||||
SIDECAR_LOG_LEVEL=INFO
|
||||
|
||||
# Admin API 认证（可选，不设置则跳过认证）
|
||||
# SIDECAR_ADMIN_TOKEN=your-admin-token-here
|
||||
@@ -1,49 +0,0 @@
|
||||
# NVIDIA Sidecar 限流代理 — systemd service (BIZ-46 Phase3 §4)
|
||||
#
|
||||
# 安装：
|
||||
# sudo cp deploy/nvidia-sidecar.service /etc/systemd/system/
|
||||
# sudo systemctl daemon-reload
|
||||
# sudo systemctl enable nvidia-sidecar
|
||||
# sudo systemctl start nvidia-sidecar
|
||||
#
|
||||
# 运维：
|
||||
# sudo systemctl status nvidia-sidecar
|
||||
# sudo journalctl -u nvidia-sidecar -f
|
||||
|
||||
[Unit]
|
||||
Description=NVIDIA Sidecar Rate-Limiting Proxy
|
||||
Documentation=https://github.com/bizwings/nvidia-sidecar
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=sidecar
|
||||
Group=sidecar
|
||||
WorkingDirectory=/opt/nvidia-sidecar
|
||||
ExecStart=/opt/nvidia-sidecar/.venv/bin/uvicorn nvidia_sidecar.server:app \
|
||||
--host 127.0.0.1 \
|
||||
--port 9190 \
|
||||
--log-level info
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
# 环境变量
|
||||
EnvironmentFile=/opt/nvidia-sidecar/.env
|
||||
|
||||
# 安全加固
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
ReadWritePaths=/opt/nvidia-sidecar/logs
|
||||
|
||||
# 资源限制
|
||||
LimitNOFILE=65536
|
||||
MemoryMax=512M
|
||||
|
||||
# 启动延迟（等待网络就绪）
|
||||
ExecStartPre=/bin/sleep 1
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -1,198 +0,0 @@
|
||||
"""
|
||||
NVIDIA Sidecar 限流代理 — 健康检查端点 (§3.6)
|
||||
|
||||
提供 Kubernetes / systemd 兼容的健康检查：
|
||||
GET /health — 存活检查
|
||||
GET /health/ready — 就绪检查（含上游连通性）
|
||||
|
||||
BIZ-46 Phase3: Readiness HTTP Client 复用 — 注入主 http_client，
|
||||
不再每次检查创建新 client，降低 K8s/systemd 高频探测的连接开销。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
@dataclass
|
||||
class HealthService:
|
||||
"""健康检查服务。
|
||||
|
||||
封装存活检查和就绪检查的逻辑，供 server.py 路由调用。
|
||||
"""
|
||||
|
||||
start_time: float = 0.0
|
||||
version: str = "0.1.0"
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.start_time == 0.0:
|
||||
self.start_time = time.time()
|
||||
|
||||
@property
|
||||
def uptime_seconds(self) -> float:
|
||||
"""服务运行时长（秒）。"""
|
||||
return time.time() - self.start_time
|
||||
|
||||
async def check_upstream(
|
||||
self,
|
||||
upstream_url: str,
|
||||
http_client: httpx.AsyncClient,
|
||||
timeout: float = 5.0,
|
||||
api_key: str = "",
|
||||
) -> bool:
|
||||
"""检查上游连通性（复用注入的 http_client，BIZ-46 Phase3）。
|
||||
|
||||
Args:
|
||||
upstream_url: NVIDIA API base URL。
|
||||
http_client: 复用的 httpx.AsyncClient（来自 ctx）。
|
||||
timeout: 超时秒数（per-request override）。
|
||||
api_key: 可选的 API Key 用于认证。
|
||||
|
||||
Returns:
|
||||
True 上游可达。
|
||||
"""
|
||||
try:
|
||||
headers: dict[str, str] = {}
|
||||
if api_key:
|
||||
headers["authorization"] = f"Bearer {api_key}"
|
||||
|
||||
resp = await http_client.get(
|
||||
f"{upstream_url.rstrip('/')}/v1/models",
|
||||
headers=headers,
|
||||
timeout=timeout,
|
||||
)
|
||||
return resp.status_code < 500
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def check_queue_healthy(
|
||||
self,
|
||||
current_size: int,
|
||||
max_size: int,
|
||||
threshold_ratio: float = 0.9,
|
||||
) -> bool:
|
||||
"""检查队列是否健康（未接近满载）。
|
||||
|
||||
Args:
|
||||
current_size: 当前队列长度。
|
||||
max_size: 队列最大容量。
|
||||
threshold_ratio: 告警阈值比例，默认 0.9。
|
||||
|
||||
Returns:
|
||||
True 队列健康。
|
||||
"""
|
||||
if max_size <= 0:
|
||||
return True
|
||||
return current_size < max_size * threshold_ratio
|
||||
|
||||
def check_token_bucket_healthy(
|
||||
self,
|
||||
available_tokens: float,
|
||||
capacity: int,
|
||||
threshold: float = 0.05,
|
||||
) -> bool:
|
||||
"""检查令牌桶是否健康（token 未耗尽）。
|
||||
|
||||
Args:
|
||||
available_tokens: 当前可用令牌数。
|
||||
capacity: 桶容量。
|
||||
threshold: 令牌数低于此比例视为不健康。
|
||||
|
||||
Returns:
|
||||
True 令牌桶健康。
|
||||
"""
|
||||
if capacity <= 0:
|
||||
return False
|
||||
return available_tokens > capacity * threshold
|
||||
|
||||
def liveness(self) -> dict[str, Any]:
|
||||
"""存活检查响应。
|
||||
|
||||
Returns:
|
||||
liveness JSON payload。
|
||||
"""
|
||||
return {
|
||||
"status": "ok",
|
||||
"uptime": round(self.uptime_seconds, 1),
|
||||
"version": self.version,
|
||||
}
|
||||
|
||||
async def readiness(
|
||||
self,
|
||||
upstream_url: str,
|
||||
upstream_api_key: str = "",
|
||||
queue_current_size: int = 0,
|
||||
queue_max_size: int = 500,
|
||||
available_tokens: float = 0.0,
|
||||
bucket_capacity: int = 40,
|
||||
http_client: httpx.AsyncClient | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""就绪检查响应。
|
||||
|
||||
Args:
|
||||
upstream_url: 上游 API 地址。
|
||||
upstream_api_key: API Key。
|
||||
queue_current_size: 当前队列长度。
|
||||
queue_max_size: 队列最大容量。
|
||||
available_tokens: 当前令牌数。
|
||||
bucket_capacity: 桶容量。
|
||||
http_client: 复用的 httpx.AsyncClient（BIZ-46 Phase3）。
|
||||
为 None 时回退到每次创建新 client（兼容旧调用）。
|
||||
|
||||
Returns:
|
||||
readiness JSON payload。
|
||||
"""
|
||||
if http_client is not None:
|
||||
upstream_ok = await self.check_upstream(
|
||||
upstream_url, http_client=http_client, api_key=upstream_api_key,
|
||||
)
|
||||
else:
|
||||
# 向后兼容：无 http_client 时沿用旧行为
|
||||
upstream_ok = await self.check_upstream_standalone(
|
||||
upstream_url, api_key=upstream_api_key,
|
||||
)
|
||||
|
||||
queue_ok = self.check_queue_healthy(queue_current_size, queue_max_size)
|
||||
token_ok = self.check_token_bucket_healthy(available_tokens, bucket_capacity)
|
||||
all_ready = upstream_ok and queue_ok and token_ok
|
||||
|
||||
return {
|
||||
"ready": all_ready,
|
||||
"upstream_reachable": upstream_ok,
|
||||
"queue_healthy": queue_ok,
|
||||
"token_bucket_healthy": token_ok,
|
||||
}
|
||||
|
||||
async def check_upstream_standalone(
|
||||
self,
|
||||
upstream_url: str,
|
||||
timeout: float = 5.0,
|
||||
api_key: str = "",
|
||||
) -> bool:
|
||||
"""独立检查上游连通性（向后兼容，每次创建新 client）。
|
||||
|
||||
Args:
|
||||
upstream_url: NVIDIA API base URL。
|
||||
timeout: 超时秒数。
|
||||
api_key: 可选的 API Key。
|
||||
|
||||
Returns:
|
||||
True 上游可达。
|
||||
"""
|
||||
try:
|
||||
headers: dict[str, str] = {}
|
||||
if api_key:
|
||||
headers["authorization"] = f"Bearer {api_key}"
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout) as client:
|
||||
resp = await client.get(
|
||||
f"{upstream_url.rstrip('/')}/v1/models",
|
||||
headers=headers,
|
||||
)
|
||||
return resp.status_code < 500
|
||||
except Exception:
|
||||
return False
|
||||
@@ -1,277 +0,0 @@
|
||||
"""
|
||||
NVIDIA Sidecar 限流代理 — Prometheus 指标端点 (§3.5)
|
||||
|
||||
10 个指标，独立端口 :9191，与代理端口 :9190 分离。
|
||||
|
||||
BIZ-46 Phase3: Prometheus 标签基数治理 — model_id label 收敛为 provider。
|
||||
- upstream_latency_seconds: model_id → provider (固定值 "nvidia", 基数=1)
|
||||
- upstream_errors_total: model_id → provider
|
||||
- 模型级信息迁移到 structlog JSON 日志
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
from prometheus_client import (
|
||||
CollectorRegistry,
|
||||
Counter,
|
||||
Gauge,
|
||||
Histogram,
|
||||
generate_latest,
|
||||
make_asgi_app,
|
||||
)
|
||||
|
||||
|
||||
class PrometheusMetrics:
|
||||
"""Sidecar Prometheus 指标收集器。
|
||||
|
||||
线程安全，所有公开方法通过 ``threading.Lock`` 保护。
|
||||
"""
|
||||
|
||||
def __init__(self, registry: CollectorRegistry | None = None) -> None:
|
||||
"""初始化所有 10 个 Prometheus 指标。
|
||||
|
||||
Args:
|
||||
registry: 可选自定义 Registry；None 则使用默认全局 registry。
|
||||
"""
|
||||
self._registry: CollectorRegistry = registry or CollectorRegistry()
|
||||
self._lock: threading.Lock = threading.Lock()
|
||||
self._start_time: float = time.time()
|
||||
|
||||
# ---- 1. 总请求数（按优先级 + 状态分组） ----
|
||||
self.requests_total: Counter = Counter(
|
||||
"sidecar_requests_total",
|
||||
"Total requests processed by priority and status",
|
||||
labelnames=["priority", "status"],
|
||||
registry=self._registry,
|
||||
)
|
||||
|
||||
# ---- 2. 可用令牌数 ----
|
||||
self.tokens_available: Gauge = Gauge(
|
||||
"sidecar_tokens_available",
|
||||
"Current number of available tokens",
|
||||
registry=self._registry,
|
||||
)
|
||||
|
||||
# ---- 3. 令牌生成速率 ----
|
||||
self.tokens_rate: Gauge = Gauge(
|
||||
"sidecar_tokens_rate",
|
||||
"Current token generation rate (tokens per minute)",
|
||||
registry=self._registry,
|
||||
)
|
||||
|
||||
# ---- 4. 各优先级队列深度 ----
|
||||
self.queue_depth: Gauge = Gauge(
|
||||
"sidecar_queue_depth",
|
||||
"Queue depth by priority",
|
||||
labelnames=["priority"],
|
||||
registry=self._registry,
|
||||
)
|
||||
|
||||
# ---- 5. 队列等待时间 Histogram ----
|
||||
self.queue_latency_seconds: Histogram = Histogram(
|
||||
"sidecar_queue_latency_seconds",
|
||||
"Request wait time in queue in seconds",
|
||||
labelnames=["priority"],
|
||||
buckets=(0.01, 0.05, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0, 120.0),
|
||||
registry=self._registry,
|
||||
)
|
||||
|
||||
# ---- 6. 上游响应延迟 Histogram（label 收敛: model_id → provider） ----
|
||||
self.upstream_latency_seconds: Histogram = Histogram(
|
||||
"sidecar_upstream_latency_seconds",
|
||||
"Upstream response latency in seconds",
|
||||
labelnames=["provider"], # BIZ-46: was ["model_id"], converged to fixed-cardinality provider
|
||||
buckets=(0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0, 120.0, 300.0, 600.0),
|
||||
registry=self._registry,
|
||||
)
|
||||
|
||||
# ---- 7. 上游错误计数（label 收敛: model_id → provider） ----
|
||||
self.upstream_errors_total: Counter = Counter(
|
||||
"sidecar_upstream_errors_total",
|
||||
"Upstream error count by status code and provider",
|
||||
labelnames=["status_code", "provider"], # BIZ-46: was ["model_id"], converged
|
||||
registry=self._registry,
|
||||
)
|
||||
|
||||
# ---- 8. 降级直通次数 ----
|
||||
self.fallback_passthrough_total: Counter = Counter(
|
||||
"sidecar_fallback_passthrough_total",
|
||||
"Total fallback / passthrough events (queue full or sidecar unavailable)",
|
||||
registry=self._registry,
|
||||
)
|
||||
|
||||
# ---- 9. 健康状态 ----
|
||||
self.health_status: Gauge = Gauge(
|
||||
"sidecar_health_status",
|
||||
"Sidecar health: 0=unhealthy, 1=healthy",
|
||||
registry=self._registry,
|
||||
)
|
||||
|
||||
# ---- 10. 运行时长 ----
|
||||
self.uptime_seconds: Gauge = Gauge(
|
||||
"sidecar_uptime_seconds",
|
||||
"Process uptime in seconds",
|
||||
registry=self._registry,
|
||||
)
|
||||
|
||||
# 避退模式指标（附加，不计入基础 10 个）
|
||||
self.retreat_state: Gauge = Gauge(
|
||||
"sidecar_retreat_state",
|
||||
"Adaptive retreat state: 0=NORMAL, 1=RETREAT, 2=RECOVER",
|
||||
registry=self._registry,
|
||||
)
|
||||
self.effective_rate_rpm: Gauge = Gauge(
|
||||
"sidecar_effective_rate_rpm",
|
||||
"Current effective rate in RPM (after retreat adjustments)",
|
||||
registry=self._registry,
|
||||
)
|
||||
self.upstream_429_rate: Gauge = Gauge(
|
||||
"sidecar_upstream_429_rate",
|
||||
"Upstream 429 rate over the retreat observation window (0.0-1.0)",
|
||||
registry=self._registry,
|
||||
)
|
||||
|
||||
# 初始化
|
||||
self.health_status.set(1)
|
||||
|
||||
# ---- ASGI app 生成 ----
|
||||
|
||||
def build_asgi_app(self) -> Any:
|
||||
"""生成 Prometheus ASGI 应用，挂载到独立端口。
|
||||
|
||||
Returns:
|
||||
可传给 uvicorn 的 ASGI app。
|
||||
"""
|
||||
return make_asgi_app(registry=self._registry)
|
||||
|
||||
# ---- 指标记录方法 ----
|
||||
|
||||
def record_request(self, priority: str, status: str) -> None:
|
||||
"""记录一次请求。
|
||||
|
||||
Args:
|
||||
priority: 优先级名（URGENT / HIGH / NORMAL / LOW）。
|
||||
status: 状态（success / ratelimited / error）。
|
||||
"""
|
||||
with self._lock:
|
||||
self.requests_total.labels(priority=priority, status=status).inc()
|
||||
|
||||
def record_queue_latency(self, priority: str, seconds: float) -> None:
|
||||
"""记录排队延迟。
|
||||
|
||||
Args:
|
||||
priority: 优先级名。
|
||||
seconds: 排队等待秒数。
|
||||
"""
|
||||
with self._lock:
|
||||
self.queue_latency_seconds.labels(priority=priority).observe(seconds)
|
||||
|
||||
def record_upstream(self, status_code: int, provider: str) -> None:
|
||||
"""记录上游响应（label 收敛: provider 替代 model_id，BIZ-46 Phase3）。
|
||||
|
||||
Args:
|
||||
status_code: HTTP 状态码。
|
||||
provider: 上游提供商标识（固定 "nvidia"）。
|
||||
"""
|
||||
with self._lock:
|
||||
self.upstream_latency_seconds.labels(provider=provider).observe(0.0)
|
||||
|
||||
def record_upstream_error(self, status_code: int, provider: str) -> None:
|
||||
"""记录上游错误（label 收敛: provider 替代 model_id，BIZ-46 Phase3）。
|
||||
|
||||
Args:
|
||||
status_code: 错误 HTTP 状态码。
|
||||
provider: 上游提供商标识（固定 "nvidia"）。
|
||||
"""
|
||||
with self._lock:
|
||||
self.upstream_errors_total.labels(
|
||||
status_code=str(status_code), provider=provider
|
||||
).inc()
|
||||
|
||||
def record_upstream_latency(self, provider: str, seconds: float) -> None:
|
||||
"""记录上游响应延迟（label 收敛: provider 替代 model_id，BIZ-46 Phase3）。
|
||||
|
||||
Args:
|
||||
provider: 上游提供商标识（固定 "nvidia"）。
|
||||
seconds: 响应延迟秒数。
|
||||
"""
|
||||
with self._lock:
|
||||
self.upstream_latency_seconds.labels(provider=provider).observe(seconds)
|
||||
|
||||
def update_token_status(self, tokens: float, rate_per_minute: float) -> None:
|
||||
"""更新令牌桶状态。
|
||||
|
||||
Args:
|
||||
tokens: 当前可用令牌数。
|
||||
rate_per_minute: 每分钟速率。
|
||||
"""
|
||||
with self._lock:
|
||||
self.tokens_available.set(tokens)
|
||||
self.tokens_rate.set(rate_per_minute)
|
||||
|
||||
def update_queue_depth(self, depths: dict[str, int]) -> None:
|
||||
"""更新各优先级队列深度。
|
||||
|
||||
Args:
|
||||
depths: {priority_name: count} 映射。
|
||||
"""
|
||||
with self._lock:
|
||||
# 先清零所有已知标签再设置，避免残留旧值
|
||||
for pri in ("URGENT", "HIGH", "NORMAL", "LOW"):
|
||||
self.queue_depth.labels(priority=pri).set(depths.get(pri, 0))
|
||||
|
||||
def increment_fallback(self) -> None:
|
||||
"""降级直通计数 +1。"""
|
||||
with self._lock:
|
||||
self.fallback_passthrough_total.inc()
|
||||
|
||||
def set_health(self, healthy: bool) -> None:
|
||||
"""设置健康状态。
|
||||
|
||||
Args:
|
||||
healthy: True=健康, False=不健康。
|
||||
"""
|
||||
with self._lock:
|
||||
self.health_status.set(1 if healthy else 0)
|
||||
|
||||
def update_uptime(self) -> None:
|
||||
"""更新运行时长。"""
|
||||
with self._lock:
|
||||
self.uptime_seconds.set(time.time() - self._start_time)
|
||||
|
||||
# ---- 避退模式指标 ----
|
||||
|
||||
def update_retreat_metrics(
|
||||
self,
|
||||
retreat_state: str,
|
||||
effective_rate_rpm: float,
|
||||
upstream_429_rate: float,
|
||||
) -> None:
|
||||
"""更新避退模式指标。
|
||||
|
||||
Args:
|
||||
retreat_state: "normal" / "retreat" / "recover".
|
||||
effective_rate_rpm: 当前实际速率 (RPM)。
|
||||
upstream_429_rate: 上游 429 率 (0.0-1.0)。
|
||||
"""
|
||||
state_map: dict[str, int] = {"normal": 0, "retreat": 1, "recover": 2}
|
||||
with self._lock:
|
||||
self.retreat_state.set(state_map.get(retreat_state, 0))
|
||||
self.effective_rate_rpm.set(effective_rate_rpm)
|
||||
self.upstream_429_rate.set(upstream_429_rate)
|
||||
|
||||
# ---- 导出 ----
|
||||
|
||||
def generate_latest(self) -> bytes:
|
||||
"""生成 Prometheus 文本格式的指标数据。
|
||||
|
||||
Returns:
|
||||
Prometheus 文本格式 bytes。
|
||||
"""
|
||||
with self._lock:
|
||||
self.update_uptime()
|
||||
return generate_latest(self._registry)
|
||||
@@ -107,33 +107,6 @@ class PriorityRequestQueue:
|
||||
"""当前队列满策略。"""
|
||||
return self._full_policy
|
||||
|
||||
# ---- 动态容量调整 ----
|
||||
|
||||
def set_max_size(self, new_size: int) -> tuple[bool, str]:
|
||||
"""动态调整队列最大容量（热重载）。
|
||||
|
||||
缩小操作受保护：如果 new_size 小于当前排队数，拒绝变更并
|
||||
提示当前队列深度。
|
||||
|
||||
Args:
|
||||
new_size: 新的最大容量。
|
||||
|
||||
Returns:
|
||||
(成功标志, 消息)。成功时标志为 True，消息含新旧容量对比；
|
||||
失败时标志为 False，消息含拒绝原因和当前深度。
|
||||
|
||||
Raises:
|
||||
ValueError: new_size <= 0。
|
||||
"""
|
||||
if new_size <= 0:
|
||||
raise ValueError(f"max_size 必须为正整数，当前值: {new_size}")
|
||||
current = len(self._heap)
|
||||
if new_size < current:
|
||||
return (False, f"拒绝缩小：新上限 {new_size} < 当前排队数 {current}，需要先排空或提升上限")
|
||||
old = self.max_size
|
||||
self.max_size = new_size
|
||||
return (True, f"队列上限已调整：{old} → {new_size}{'（当前排队 ' + str(current) + '）' if current > 0 else ''}")
|
||||
|
||||
# ---- 入队 ----
|
||||
|
||||
async def put(
|
||||
|
||||
@@ -11,8 +11,6 @@ dependencies = [
|
||||
"httpx>=0.28",
|
||||
"PyYAML>=6.0",
|
||||
"structlog>=24.4",
|
||||
"prometheus-client>=0.21",
|
||||
"pydantic>=2.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
@@ -21,7 +19,6 @@ dev = [
|
||||
"pytest-asyncio>=0.24",
|
||||
"httpx>=0.28",
|
||||
"mypy>=1.14",
|
||||
"types-PyYAML",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
@@ -31,12 +28,8 @@ nvidia-sidecar = "nvidia_sidecar.server:main"
|
||||
requires = ["setuptools>=75", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[tool.setuptools]
|
||||
packages = ["nvidia_sidecar"]
|
||||
|
||||
[tool.setuptools.package-dir]
|
||||
# Flat layout: __init__.py + all .py files at project root
|
||||
"nvidia_sidecar" = "."
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["."]
|
||||
|
||||
[tool.mypy]
|
||||
python_version = "3.12"
|
||||
|
||||
@@ -30,14 +30,10 @@ class Priority(IntEnum):
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
NVIDIA_GATEWAY_ALIASES: set[str] = {
|
||||
# OpenClaw 配置中全部的 NVIDIA provider 名称
|
||||
"nvidia",
|
||||
"nvidia-gateway",
|
||||
"nvidia98053",
|
||||
"nvidialiuweicheng84",
|
||||
"nvidiavx",
|
||||
"nvidiavx18088980513",
|
||||
"nvidiavx64391942",
|
||||
}
|
||||
|
||||
|
||||
@@ -201,242 +197,4 @@ class TokenBucket:
|
||||
@property
|
||||
def capacity(self) -> int:
|
||||
"""桶容量。"""
|
||||
return self._capacity
|
||||
|
||||
# ---- 动态速率调整（供 AdaptiveTokenBucket 使用） ----
|
||||
|
||||
def set_rate(self, rate: float) -> None:
|
||||
"""动态调整令牌补充速率（令牌/秒）。
|
||||
|
||||
Args:
|
||||
rate: 新速率（令牌/秒）。
|
||||
"""
|
||||
with self._lock:
|
||||
self._refill() # 先补充现有令牌再切换速率
|
||||
self._rate = float(rate)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 避退模式：AdaptiveTokenBucket (§ADR-009)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class RetreatState:
|
||||
"""避退状态机常量。"""
|
||||
NORMAL: str = "normal"
|
||||
RETREAT: str = "retreat"
|
||||
RECOVER: str = "recover"
|
||||
|
||||
|
||||
class AdaptiveTokenBucket(TokenBucket):
|
||||
"""自适应避退令牌桶（ADR-009）。
|
||||
|
||||
监控上游 429 率（60s 滑动窗口），自动调整发射速率：
|
||||
|
||||
- 429 率 < 5% → NORMAL，保持基准速率
|
||||
- 429 率 5-10% → RETREAT，速率 × 0.75
|
||||
- 429 率 10-20% → RETREAT，再次降速
|
||||
- 429 率 > 20% → RETREAT，最低 5 RPM + 告警
|
||||
- 连续 120s 429 率 < 2% → RECOVER，逐步 +2 RPM 恢复
|
||||
|
||||
线程安全，继承 TokenBucket 的所有公共接口。
|
||||
"""
|
||||
|
||||
# ADR-009 参数（可通过构造函数覆盖）
|
||||
RETREAT_WINDOW_SECONDS: float = 60.0
|
||||
RETREAT_429_THRESHOLD: float = 0.05
|
||||
RETREAT_FACTOR: float = 0.75
|
||||
RETREAT_MIN_RPM: float = 5.0
|
||||
RECOVER_WINDOW_SECONDS: float = 120.0
|
||||
RECOVER_429_THRESHOLD: float = 0.02
|
||||
RECOVER_INCREMENT_RPM: float = 2.0
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
rate: float = 40 / 60,
|
||||
capacity: int = 40,
|
||||
*,
|
||||
retreat_window_seconds: float = 60.0,
|
||||
retreat_429_threshold: float = 0.05,
|
||||
retreat_factor: float = 0.75,
|
||||
retreat_min_rpm: float = 5.0,
|
||||
recover_window_seconds: float = 120.0,
|
||||
recover_429_threshold: float = 0.02,
|
||||
recover_increment_rpm: float = 2.0,
|
||||
) -> None:
|
||||
"""初始化自适应避退令牌桶。
|
||||
|
||||
Args:
|
||||
rate: 基准令牌补充速率（令牌/秒）。默认 40/60 ≈ 0.667 token/s。
|
||||
capacity: 桶最大容量。默认 40。
|
||||
retreat_window_seconds: 429 率滑动窗口大小（秒）。
|
||||
retreat_429_threshold: 触发避退的 429 率阈值。
|
||||
retreat_factor: 每次避退速率乘数。
|
||||
retreat_min_rpm: 避退最低 RPM。
|
||||
recover_window_seconds: 恢复观察窗口大小（秒）。
|
||||
recover_429_threshold: 触发恢复的 429 率阈值。
|
||||
recover_increment_rpm: 每次恢复增加的 RPM。
|
||||
"""
|
||||
super().__init__(rate=rate, capacity=capacity)
|
||||
|
||||
# 基准速率（不变）
|
||||
self._base_rate: float = float(rate)
|
||||
|
||||
# 避退参数
|
||||
self.RETREAT_WINDOW_SECONDS = retreat_window_seconds
|
||||
self.RETREAT_429_THRESHOLD = retreat_429_threshold
|
||||
self.RETREAT_FACTOR = retreat_factor
|
||||
self.RETREAT_MIN_RPM = retreat_min_rpm
|
||||
self.RECOVER_WINDOW_SECONDS = recover_window_seconds
|
||||
self.RECOVER_429_THRESHOLD = recover_429_threshold
|
||||
self.RECOVER_INCREMENT_RPM = recover_increment_rpm
|
||||
|
||||
# 避退状态机
|
||||
self._retreat_state: str = RetreatState.NORMAL
|
||||
|
||||
# 429 滑动窗口：[(timestamp, is_429), ...]
|
||||
self._429_window: list[tuple[float, bool]] = []
|
||||
|
||||
# 上次状态变更时间
|
||||
self._last_state_change: float = time.monotonic()
|
||||
|
||||
# 避退状态锁（RLock 防止 evaluate_retreat() → get_429_rate() 重入死锁）
|
||||
self._retreat_lock: threading.RLock = threading.RLock()
|
||||
|
||||
# ---- 429 反馈 ----
|
||||
|
||||
def record_response(self, is_429: bool) -> None:
|
||||
"""记录一次上游响应是否为 429。
|
||||
|
||||
Args:
|
||||
is_429: True 表示上游返回了 429。
|
||||
"""
|
||||
now = time.monotonic()
|
||||
with self._retreat_lock:
|
||||
self._429_window.append((now, is_429))
|
||||
# 清理超出观察窗口的旧记录
|
||||
cutoff = now - max(
|
||||
self.RETREAT_WINDOW_SECONDS,
|
||||
self.RECOVER_WINDOW_SECONDS,
|
||||
)
|
||||
self._429_window = [
|
||||
(ts, flag) for ts, flag in self._429_window
|
||||
if ts >= cutoff
|
||||
]
|
||||
|
||||
def get_429_rate(self, window_seconds: float | None = None) -> float:
|
||||
"""获取指定窗口内的 429 率。
|
||||
|
||||
Args:
|
||||
window_seconds: 滑动窗口大小；None 使用 RETREAT_WINDOW_SECONDS。
|
||||
|
||||
Returns:
|
||||
0.0-1.0 之间的 429 率。
|
||||
"""
|
||||
ws = window_seconds or self.RETREAT_WINDOW_SECONDS
|
||||
now = time.monotonic()
|
||||
with self._retreat_lock:
|
||||
in_window = [flag for ts, flag in self._429_window if now - ts <= ws]
|
||||
if not in_window:
|
||||
return 0.0
|
||||
return sum(1 for f in in_window if f) / len(in_window)
|
||||
|
||||
# ---- 避退状态评估 ----
|
||||
|
||||
def evaluate_retreat(self) -> str:
|
||||
"""评估并更新避退状态，返回新状态名。
|
||||
|
||||
每次调用根据当前 429 率 + 持续时间决定是否进入 RETREAT / RECOVER。
|
||||
|
||||
Returns:
|
||||
"normal" / "retreat" / "recover"。
|
||||
"""
|
||||
now = time.monotonic()
|
||||
with self._retreat_lock:
|
||||
retreat_rate = self.get_429_rate(self.RETREAT_WINDOW_SECONDS)
|
||||
recover_rate = self.get_429_rate(self.RECOVER_WINDOW_SECONDS)
|
||||
|
||||
if self._retreat_state == RetreatState.NORMAL:
|
||||
if retreat_rate >= self.RETREAT_429_THRESHOLD:
|
||||
self._retreat_state = RetreatState.RETREAT
|
||||
self._last_state_change = now
|
||||
self._apply_retreat()
|
||||
|
||||
elif self._retreat_state == RetreatState.RETREAT:
|
||||
# 持续高 429 率 → 再次降速
|
||||
if retreat_rate >= self.RETREAT_429_THRESHOLD * 2:
|
||||
# 429 > 10%，再次降速
|
||||
if self._rate > self.RETREAT_MIN_RPM / 60.0:
|
||||
self._apply_retreat()
|
||||
elif recover_rate < self.RECOVER_429_THRESHOLD:
|
||||
time_in_low = now - self._last_state_change
|
||||
if time_in_low >= self.RECOVER_WINDOW_SECONDS:
|
||||
self._retreat_state = RetreatState.RECOVER
|
||||
self._last_state_change = now
|
||||
self._apply_recover()
|
||||
|
||||
elif self._retreat_state == RetreatState.RECOVER:
|
||||
if retreat_rate >= self.RETREAT_429_THRESHOLD:
|
||||
# 恢复期间 429 回升，重新进入避退
|
||||
self._retreat_state = RetreatState.RETREAT
|
||||
self._last_state_change = now
|
||||
self._apply_retreat()
|
||||
elif self._rate >= self._base_rate:
|
||||
# 已恢复到基准速率
|
||||
self._rate = self._base_rate
|
||||
self._retreat_state = RetreatState.NORMAL
|
||||
self._last_state_change = now
|
||||
else:
|
||||
# 继续逐步恢复
|
||||
self._apply_recover()
|
||||
|
||||
return self._retreat_state
|
||||
|
||||
def _apply_retreat(self) -> None:
|
||||
"""执行一次避退降速。"""
|
||||
new_rate: float = max(
|
||||
self.RETREAT_MIN_RPM / 60.0,
|
||||
self._rate * self.RETREAT_FACTOR,
|
||||
)
|
||||
self._rate = new_rate
|
||||
|
||||
def _apply_recover(self) -> None:
|
||||
"""执行一次恢复提速。"""
|
||||
increment: float = self.RECOVER_INCREMENT_RPM / 60.0
|
||||
new_rate: float = min(self._base_rate, self._rate + increment)
|
||||
self._rate = new_rate
|
||||
|
||||
# ---- 状态查询 ----
|
||||
|
||||
def get_retreat_state(self) -> str:
|
||||
"""获取当前避退状态。
|
||||
|
||||
Returns:
|
||||
"normal" / "retreat" / "recover"。
|
||||
"""
|
||||
with self._retreat_lock:
|
||||
return self._retreat_state
|
||||
|
||||
def get_effective_rate_rpm(self) -> float:
|
||||
"""获取当前实际速率（RPM），考虑避退乘数。
|
||||
|
||||
Returns:
|
||||
当前每分钟速率。
|
||||
"""
|
||||
with self._lock:
|
||||
return self._rate * 60.0
|
||||
|
||||
def get_base_rate_rpm(self) -> float:
|
||||
"""获取基准速率（RPM），即未避退时的速率。
|
||||
|
||||
Returns:
|
||||
基准每分钟速率。
|
||||
"""
|
||||
return self._base_rate * 60.0
|
||||
|
||||
def reset_to_base(self) -> None:
|
||||
"""手动重置到基准速率（用于运维干预）。"""
|
||||
with self._retreat_lock:
|
||||
self._rate = self._base_rate
|
||||
self._retreat_state = RetreatState.NORMAL
|
||||
self._last_state_change = time.monotonic()
|
||||
self._429_window.clear()
|
||||
return self._capacity
|
||||
+117
-250
@@ -5,15 +5,11 @@ NVIDIA Sidecar 限流代理 — FastAPI 代理主入口 (§3.4)
|
||||
接收 → 网关识别 → [NVIDIA: 排队 → 令牌限流] → httpx 转发 → 返回
|
||||
|
||||
非 NVIDIA 请求直通上游，NVIDIA 请求经过四级优先级队列 + 令牌桶限流。
|
||||
|
||||
BIZ-46 Phase3: 架构解耦 — 所有全局状态收敛为 SidecarContext (§1)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import AsyncGenerator
|
||||
@@ -22,16 +18,13 @@ from typing import Any
|
||||
|
||||
import httpx
|
||||
import structlog
|
||||
import uvicorn
|
||||
from fastapi import Depends, FastAPI, Request, Response
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import JSONResponse, PlainTextResponse, StreamingResponse
|
||||
from fastapi import FastAPI, Request, Response
|
||||
from fastapi.responses import JSONResponse, StreamingResponse
|
||||
|
||||
from nvidia_sidecar.config import load_config, SidecarConfig
|
||||
from nvidia_sidecar.context import SidecarContext
|
||||
from nvidia_sidecar.rate_limiter import (
|
||||
Priority,
|
||||
AdaptiveTokenBucket,
|
||||
TokenBucket,
|
||||
is_nvidia_gateway,
|
||||
)
|
||||
from nvidia_sidecar.priority_queue import (
|
||||
@@ -40,9 +33,6 @@ from nvidia_sidecar.priority_queue import (
|
||||
QueueFullPassthrough,
|
||||
QueueFullPolicy,
|
||||
)
|
||||
from nvidia_sidecar.metrics import PrometheusMetrics
|
||||
from nvidia_sidecar.health import HealthService
|
||||
from nvidia_sidecar.webui import webui_router
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 结构化日志
|
||||
@@ -58,7 +48,7 @@ structlog.configure(
|
||||
structlog.processors.StackInfoRenderer(),
|
||||
structlog.processors.format_exc_info,
|
||||
structlog.processors.UnicodeDecoder(),
|
||||
structlog.processors.JSONRenderer(),
|
||||
structlog.dev.ConsoleRenderer(),
|
||||
],
|
||||
context_class=dict,
|
||||
logger_factory=structlog.stdlib.LoggerFactory(),
|
||||
@@ -69,19 +59,33 @@ logger: structlog.stdlib.BoundLogger = structlog.get_logger("nvidia_sidecar")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# FastAPI 依赖注入
|
||||
# 全局状态（通过 lifespan 初始化，模块级引用方便路由访问）
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def get_context() -> SidecarContext:
|
||||
"""从 app.state 获取 SidecarContext（FastAPI 依赖注入）。"""
|
||||
return app.state.sidecar # type: ignore[no-any-return]
|
||||
_config: SidecarConfig
|
||||
_http_client: httpx.AsyncClient
|
||||
_priority_queue: PriorityRequestQueue
|
||||
_token_bucket: TokenBucket
|
||||
_pending_requests: dict[str, tuple[asyncio.Future[httpx.Response], float]]
|
||||
"""request_id → (response future, enqueued_at) 的映射。"""
|
||||
|
||||
# 统计计数器
|
||||
_stats: dict[str, int] = {
|
||||
"total_requests": 0,
|
||||
"nvidia_requests": 0,
|
||||
"passthrough_requests": 0,
|
||||
"ratelimited_requests": 0,
|
||||
"queue_full_rejects": 0,
|
||||
"upstream_errors": 0,
|
||||
"start_time": 0,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 工具函数
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _extract_model(body: Any) -> str | None:
|
||||
def _extract_model(body: dict[str, Any]) -> str | None:
|
||||
"""从请求体中提取模型标识符（兼容 OpenAI Chat/Completions 格式）。
|
||||
|
||||
Args:
|
||||
@@ -116,7 +120,6 @@ def _resolve_priority(headers: dict[str, str]) -> Priority:
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _forward_to_upstream(
|
||||
ctx: SidecarContext,
|
||||
method: str,
|
||||
path: str,
|
||||
body: bytes | None,
|
||||
@@ -126,7 +129,6 @@ async def _forward_to_upstream(
|
||||
"""将请求转发到 NVIDIA 上游 API。
|
||||
|
||||
Args:
|
||||
ctx: SidecarContext 运行时上下文。
|
||||
method: HTTP 方法。
|
||||
path: 请求路径（如 ``/v1/chat/completions``）。
|
||||
body: 原始请求体 bytes。
|
||||
@@ -139,33 +141,28 @@ async def _forward_to_upstream(
|
||||
Raises:
|
||||
httpx.HTTPError: HTTP 请求失败。
|
||||
"""
|
||||
# 构建上游 URL：如果 upstream_url 已经包含 /v1 路径，则避免路径重复
|
||||
base_url = ctx.config.upstream_url.rstrip("/")
|
||||
if base_url.endswith("/v1") and path.startswith("/v1"):
|
||||
upstream_url = base_url + path[3:] # 去掉 path 中的 /v1 前缀
|
||||
else:
|
||||
upstream_url = base_url + path
|
||||
upstream_url = _config.upstream_url.rstrip("/") + path
|
||||
forward_headers: dict[str, str] = {
|
||||
k: v for k, v in headers.items()
|
||||
if k.lower() not in ("host", "content-length", "transfer-encoding")
|
||||
}
|
||||
if ctx.config.upstream_api_key:
|
||||
forward_headers["authorization"] = f"Bearer {ctx.config.upstream_api_key}"
|
||||
if _config.upstream_api_key:
|
||||
forward_headers["authorization"] = f"Bearer {_config.upstream_api_key}"
|
||||
elif "authorization" not in {k.lower() for k in forward_headers}:
|
||||
forward_headers["authorization"] = "Bearer nvidia"
|
||||
|
||||
try:
|
||||
req = ctx.http_client.build_request(
|
||||
req = _http_client.build_request(
|
||||
method=method,
|
||||
url=upstream_url,
|
||||
headers=forward_headers,
|
||||
content=body,
|
||||
timeout=ctx.config.request_timeout,
|
||||
timeout=_config.request_timeout,
|
||||
)
|
||||
response = await ctx.http_client.send(req, stream=stream)
|
||||
response = await _http_client.send(req, stream=stream)
|
||||
return response
|
||||
except httpx.TimeoutException:
|
||||
logger.warning("upstream_timeout", path=path, timeout=ctx.config.request_timeout)
|
||||
logger.warning("upstream_timeout", path=path, timeout=_config.request_timeout)
|
||||
raise
|
||||
except httpx.HTTPError as exc:
|
||||
logger.error("upstream_error", path=path, error=str(exc))
|
||||
@@ -176,18 +173,14 @@ async def _forward_to_upstream(
|
||||
# worker 协程：消费优先级队列 + 令牌桶 + 转发
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _worker_loop(ctx: SidecarContext) -> None:
|
||||
"""后台 worker：持续从优先级队列取请求 → 令牌限流 → 转发 → 设置 future 结果。
|
||||
|
||||
Args:
|
||||
ctx: SidecarContext 运行时上下文。
|
||||
"""
|
||||
async def _worker_loop() -> None:
|
||||
"""后台 worker：持续从优先级队列取请求 → 令牌限流 → 转发 → 设置 future 结果。"""
|
||||
log = logger.bind(worker="main")
|
||||
log.info("worker_started")
|
||||
|
||||
while True:
|
||||
try:
|
||||
queue_item = await ctx.priority_queue.get(timeout=1.0)
|
||||
queue_item = await _priority_queue.get(timeout=1.0)
|
||||
if queue_item is None:
|
||||
continue
|
||||
|
||||
@@ -197,7 +190,7 @@ async def _worker_loop(ctx: SidecarContext) -> None:
|
||||
enqueued_at = queue_item.enqueued_at
|
||||
|
||||
# 查找对应的 pending future
|
||||
pending_entry = ctx.pending_requests.get(request_id)
|
||||
pending_entry = _pending_requests.get(request_id)
|
||||
if pending_entry is None:
|
||||
log.warning("orphan_request", request_id=request_id)
|
||||
continue
|
||||
@@ -207,30 +200,30 @@ async def _worker_loop(ctx: SidecarContext) -> None:
|
||||
if queue_item.priority == Priority.LOW:
|
||||
# 放线程池执行阻塞的令牌桶调用
|
||||
got_token = await asyncio.to_thread(
|
||||
ctx.token_bucket.try_consume,
|
||||
_token_bucket.try_consume,
|
||||
tokens=1,
|
||||
timeout=ctx.config.low_priority_timeout,
|
||||
timeout=_config.low_priority_timeout,
|
||||
)
|
||||
if not got_token:
|
||||
log.info("low_priority_timeout", request_id=request_id)
|
||||
await ctx.increment_stat("ratelimited_requests")
|
||||
ctx.prometheus.record_request(queue_item.priority.name, "ratelimited")
|
||||
_stats["ratelimited_requests"] += 1
|
||||
if not future.done():
|
||||
future.set_exception(
|
||||
_RateLimitedError(
|
||||
f"低优先级请求令牌等待超时 ({ctx.config.low_priority_timeout}s)"
|
||||
f"低优先级请求令牌等待超时 ({_config.low_priority_timeout}s)"
|
||||
)
|
||||
)
|
||||
ctx.pending_requests.pop(request_id, None)
|
||||
_pending_requests.pop(request_id, None)
|
||||
continue
|
||||
else:
|
||||
# 非低优先级：在 worker 内轮询等待令牌，避免重入队导致 future 悬挂
|
||||
got_token = await asyncio.to_thread(ctx.token_bucket.consume, tokens=1)
|
||||
# （重入队会生成新 request_id，原 future 永不 resolve → 客户端永久 hang）
|
||||
got_token = await asyncio.to_thread(_token_bucket.consume, tokens=1)
|
||||
if not got_token:
|
||||
token_deadline = time.monotonic() + ctx.config.request_timeout
|
||||
token_deadline = time.monotonic() + _config.request_timeout
|
||||
while not got_token:
|
||||
await asyncio.sleep(0.1)
|
||||
got_token = await asyncio.to_thread(ctx.token_bucket.consume, tokens=1)
|
||||
got_token = await asyncio.to_thread(_token_bucket.consume, tokens=1)
|
||||
if time.monotonic() > token_deadline:
|
||||
break
|
||||
if not got_token:
|
||||
@@ -238,17 +231,16 @@ async def _worker_loop(ctx: SidecarContext) -> None:
|
||||
"token_wait_timeout",
|
||||
request_id=request_id,
|
||||
priority=queue_item.priority.name,
|
||||
timeout=ctx.config.request_timeout,
|
||||
timeout=_config.request_timeout,
|
||||
)
|
||||
await ctx.increment_stat("ratelimited_requests")
|
||||
ctx.prometheus.record_request(queue_item.priority.name, "ratelimited")
|
||||
_stats["ratelimited_requests"] += 1
|
||||
if not future.done():
|
||||
future.set_exception(
|
||||
_RateLimitedError(
|
||||
f"令牌等待超时 ({ctx.config.request_timeout:.0f}s)"
|
||||
f"令牌等待超时 ({_config.request_timeout:.0f}s)"
|
||||
)
|
||||
)
|
||||
ctx.pending_requests.pop(request_id, None)
|
||||
_pending_requests.pop(request_id, None)
|
||||
continue
|
||||
|
||||
# 转发到上游
|
||||
@@ -263,7 +255,6 @@ async def _worker_loop(ctx: SidecarContext) -> None:
|
||||
}
|
||||
|
||||
resp = await _forward_to_upstream(
|
||||
ctx=ctx,
|
||||
method=method,
|
||||
path=path,
|
||||
body=payload.get("_raw_body"),
|
||||
@@ -275,50 +266,25 @@ async def _worker_loop(ctx: SidecarContext) -> None:
|
||||
queue_latency = time.monotonic() - enqueued_at
|
||||
total_latency = upstream_latency + queue_latency
|
||||
|
||||
is_429: bool = resp.status_code == 429
|
||||
ctx.token_bucket.record_response(is_429)
|
||||
|
||||
# 避退状态评估 + 指标更新
|
||||
ctx.token_bucket.evaluate_retreat()
|
||||
retreat_state = ctx.token_bucket.get_retreat_state()
|
||||
effective_rpm = ctx.token_bucket.get_effective_rate_rpm()
|
||||
upstream_429_rate = ctx.token_bucket.get_429_rate()
|
||||
ctx.prometheus.update_retreat_metrics(retreat_state, effective_rpm, upstream_429_rate)
|
||||
|
||||
# 模型级信息写入 JSON 日志 (BIZ-46 Phase3: provider label 收敛后保留)
|
||||
model_id = _extract_model(payload) or "unknown"
|
||||
log.info(
|
||||
"request_completed",
|
||||
request_id=request_id,
|
||||
status=resp.status_code,
|
||||
model_id=model_id,
|
||||
upstream_latency=round(upstream_latency, 3),
|
||||
queue_latency=round(queue_latency, 3),
|
||||
total_latency=round(total_latency, 3),
|
||||
retreat_state=retreat_state,
|
||||
effective_rpm=round(effective_rpm, 1),
|
||||
)
|
||||
|
||||
# 记录 Prometheus 指标 — provider 收敛（BIZ-46 Phase3）
|
||||
provider = "nvidia"
|
||||
ctx.prometheus.record_upstream_latency(provider, upstream_latency)
|
||||
if not resp.is_success:
|
||||
ctx.prometheus.record_upstream_error(resp.status_code, provider)
|
||||
ctx.prometheus.record_request(queue_item.priority.name, "success" if resp.is_success else "error")
|
||||
ctx.prometheus.record_queue_latency(queue_item.priority.name, queue_latency)
|
||||
|
||||
if not future.done():
|
||||
future.set_result(resp)
|
||||
|
||||
except (httpx.HTTPError, OSError) as exc:
|
||||
log.error("upstream_request_failed", request_id=request_id, error=str(exc))
|
||||
await ctx.increment_stat("upstream_errors")
|
||||
ctx.prometheus.record_request(queue_item.priority.name, "error")
|
||||
ctx.prometheus.set_health(False)
|
||||
_stats["upstream_errors"] += 1
|
||||
if not future.done():
|
||||
future.set_exception(exc)
|
||||
|
||||
ctx.pending_requests.pop(request_id, None)
|
||||
_pending_requests.pop(request_id, None)
|
||||
|
||||
except asyncio.CancelledError:
|
||||
log.info("worker_cancelled")
|
||||
@@ -332,7 +298,6 @@ async def _worker_loop(ctx: SidecarContext) -> None:
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _passthrough_with_rate_limit(
|
||||
ctx: SidecarContext,
|
||||
request: Request,
|
||||
path: str,
|
||||
body_bytes: bytes,
|
||||
@@ -342,7 +307,6 @@ async def _passthrough_with_rate_limit(
|
||||
"""队列满时的 PASSSTHROUGH 直通路径：仍受令牌桶限流，但不排队。
|
||||
|
||||
Args:
|
||||
ctx: SidecarContext 运行时上下文。
|
||||
request: FastAPI Request。
|
||||
path: 请求路径。
|
||||
body_bytes: 原始请求体。
|
||||
@@ -352,43 +316,39 @@ async def _passthrough_with_rate_limit(
|
||||
Returns:
|
||||
FastAPI Response。
|
||||
"""
|
||||
await ctx.increment_stat("passthrough_requests")
|
||||
ctx.prometheus.increment_fallback()
|
||||
|
||||
# 低优先级走令牌桶等待
|
||||
if priority == Priority.LOW:
|
||||
got_token = await asyncio.to_thread(
|
||||
ctx.token_bucket.try_consume,
|
||||
_token_bucket.try_consume,
|
||||
tokens=1,
|
||||
timeout=ctx.config.low_priority_timeout,
|
||||
timeout=_config.low_priority_timeout,
|
||||
)
|
||||
if not got_token:
|
||||
await ctx.increment_stat("ratelimited_requests")
|
||||
ctx.prometheus.record_request(priority.name, "ratelimited")
|
||||
_stats["ratelimited_requests"] += 1
|
||||
return JSONResponse(
|
||||
status_code=429,
|
||||
content={
|
||||
"error": {
|
||||
"message": f"令牌不足（队列满 + passthrough），超时 {ctx.config.low_priority_timeout}s",
|
||||
"message": f"令牌不足（队列满 + passthrough），超时 {_config.low_priority_timeout}s",
|
||||
"type": "RateLimitedError",
|
||||
}
|
||||
},
|
||||
)
|
||||
else:
|
||||
got_token = await asyncio.to_thread(ctx.token_bucket.consume, tokens=1)
|
||||
got_token = await asyncio.to_thread(_token_bucket.consume, tokens=1)
|
||||
if not got_token:
|
||||
deadline = time.monotonic() + ctx.config.request_timeout
|
||||
# 非低优先级轮询等待
|
||||
deadline = time.monotonic() + 30.0
|
||||
while not got_token:
|
||||
await asyncio.sleep(0.1)
|
||||
got_token = await asyncio.to_thread(ctx.token_bucket.consume, tokens=1)
|
||||
got_token = await asyncio.to_thread(_token_bucket.consume, tokens=1)
|
||||
if time.monotonic() > deadline:
|
||||
await ctx.increment_stat("ratelimited_requests")
|
||||
ctx.prometheus.record_request(priority.name, "ratelimited")
|
||||
_stats["ratelimited_requests"] += 1
|
||||
return JSONResponse(
|
||||
status_code=429,
|
||||
content={
|
||||
"error": {
|
||||
"message": f"令牌不足（队列满 + passthrough），等待超时 {ctx.config.request_timeout:.0f}s",
|
||||
"message": "令牌不足（队列满 + passthrough），等待超时 30s",
|
||||
"type": "RateLimitedError",
|
||||
}
|
||||
},
|
||||
@@ -398,25 +358,16 @@ async def _passthrough_with_rate_limit(
|
||||
try:
|
||||
clean_headers = {k: v for k, v in raw_headers.items()}
|
||||
resp = await _forward_to_upstream(
|
||||
ctx=ctx,
|
||||
method=request.method,
|
||||
path=path,
|
||||
body=body_bytes if body_bytes else None,
|
||||
headers=clean_headers,
|
||||
stream=False,
|
||||
)
|
||||
retreat_state = ctx.token_bucket.get_retreat_state()
|
||||
ctx.token_bucket.evaluate_retreat()
|
||||
ctx.prometheus.update_retreat_metrics(
|
||||
retreat_state,
|
||||
ctx.token_bucket.get_effective_rate_rpm(),
|
||||
ctx.token_bucket.get_429_rate(),
|
||||
)
|
||||
return _build_response(resp)
|
||||
except Exception as exc:
|
||||
status, msg = _map_exception(exc)
|
||||
logger.error("passthrough_error", path=path, error=str(exc))
|
||||
ctx.prometheus.set_health(False)
|
||||
return JSONResponse(
|
||||
status_code=status,
|
||||
content={"error": {"message": msg, "type": type(exc).__name__}},
|
||||
@@ -459,63 +410,33 @@ def _map_exception(exc: Exception) -> tuple[int, str]:
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI) -> AsyncGenerator[None, Any]:
|
||||
"""应用生命周期管理：初始化/清理全局资源。
|
||||
"""应用生命周期管理：初始化/清理全局资源。"""
|
||||
global _config, _http_client, _priority_queue, _token_bucket, _pending_requests
|
||||
|
||||
BIZ-46 Phase3: 所有资源收敛到 SidecarContext，挂载于 app.state.sidecar。
|
||||
"""
|
||||
# 启动
|
||||
config: SidecarConfig = load_config()
|
||||
logging.getLogger().setLevel(config.log_level.upper())
|
||||
_config = load_config()
|
||||
logging.getLogger().setLevel(_config.log_level.upper())
|
||||
|
||||
http_client: httpx.AsyncClient = httpx.AsyncClient(
|
||||
timeout=httpx.Timeout(config.request_timeout),
|
||||
limits=httpx.Limits(
|
||||
max_connections=100,
|
||||
max_keepalive_connections=20,
|
||||
),
|
||||
_http_client = httpx.AsyncClient(
|
||||
timeout=httpx.Timeout(_config.request_timeout),
|
||||
)
|
||||
priority_queue: PriorityRequestQueue = PriorityRequestQueue(max_size=config.queue_max_size)
|
||||
token_bucket: AdaptiveTokenBucket = AdaptiveTokenBucket(
|
||||
rate=config.rate_rpm / 60.0,
|
||||
capacity=config.bucket_capacity,
|
||||
_priority_queue = PriorityRequestQueue(max_size=_config.queue_max_size)
|
||||
_token_bucket = TokenBucket(
|
||||
rate=_config.rate_rpm / 60.0,
|
||||
capacity=_config.bucket_capacity,
|
||||
)
|
||||
prometheus: PrometheusMetrics = PrometheusMetrics()
|
||||
health: HealthService = HealthService()
|
||||
|
||||
ctx: SidecarContext = SidecarContext(
|
||||
config=config,
|
||||
http_client=http_client,
|
||||
token_bucket=token_bucket,
|
||||
priority_queue=priority_queue,
|
||||
prometheus=prometheus,
|
||||
health=health,
|
||||
)
|
||||
ctx.stats["start_time"] = int(time.time())
|
||||
app.state.sidecar = ctx # 注入 FastAPI
|
||||
_pending_requests = {}
|
||||
_stats["start_time"] = int(time.time())
|
||||
|
||||
# 启动 worker 协程
|
||||
worker_task = asyncio.create_task(_worker_loop(ctx))
|
||||
|
||||
# Metrics 通过主服务器 `/metrics` 端点提供
|
||||
|
||||
# webui 路由（暂停挂载，排查路由匹配问题）
|
||||
# app.include_router(webui_router)
|
||||
|
||||
# upstream_api_key 启动检查（严维序评审 #5）
|
||||
if not config.upstream_api_key:
|
||||
logger.warning(
|
||||
"upstream_api_key_empty",
|
||||
message="SIDECAR_API_KEY 未设置，NVIDIA 请求将因 401 认证失败",
|
||||
)
|
||||
worker_task = asyncio.create_task(_worker_loop())
|
||||
|
||||
logger.info(
|
||||
"sidecar_started",
|
||||
host=config.listen_host,
|
||||
port=config.listen_port,
|
||||
metrics_port=config.metrics_port,
|
||||
rate_rpm=config.rate_rpm,
|
||||
queue_max=config.queue_max_size,
|
||||
retreat_enabled=True,
|
||||
host=_config.listen_host,
|
||||
port=_config.listen_port,
|
||||
rate_rpm=_config.rate_rpm,
|
||||
queue_max=_config.queue_max_size,
|
||||
)
|
||||
|
||||
yield # app 运行中
|
||||
@@ -527,7 +448,7 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, Any]:
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
await http_client.aclose()
|
||||
await _http_client.aclose()
|
||||
logger.info("sidecar_stopped")
|
||||
|
||||
|
||||
@@ -537,30 +458,12 @@ app: FastAPI = FastAPI(
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
# CORS 中间件（在 lifespan 前添加，避免 RuntimeError）
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=False,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
|
||||
def _mask_api_key(key: str) -> str:
|
||||
"""对 API Key 进行脱敏处理，仅保留前 4 位以供识别。"""
|
||||
if not key:
|
||||
return ""
|
||||
if len(key) <= 4:
|
||||
return key[:2] + "****"
|
||||
return key[:4] + "****"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 核心代理处理器
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _handle_proxy_request(ctx: SidecarContext, request: Request, path: str) -> Response:
|
||||
async def _handle_proxy_request(request: Request, path: str) -> Response:
|
||||
"""统一的代理请求处理入口。
|
||||
|
||||
执行完整链路：
|
||||
@@ -568,7 +471,7 @@ async def _handle_proxy_request(ctx: SidecarContext, request: Request, path: str
|
||||
2. 网关识别 → 非 NVIDIA 直通
|
||||
3. NVIDIA → 排队 + 令牌限流 + 转发
|
||||
"""
|
||||
await ctx.increment_stat("total_requests")
|
||||
_stats["total_requests"] += 1
|
||||
|
||||
# 解析请求
|
||||
body_bytes: bytes = await request.body()
|
||||
@@ -588,10 +491,9 @@ async def _handle_proxy_request(ctx: SidecarContext, request: Request, path: str
|
||||
|
||||
# 非 NVIDIA → 直接转发
|
||||
if not is_nvidia:
|
||||
await ctx.increment_stat("passthrough_requests")
|
||||
_stats["passthrough_requests"] += 1
|
||||
try:
|
||||
resp = await _forward_to_upstream(
|
||||
ctx=ctx,
|
||||
method=request.method,
|
||||
path=path,
|
||||
body=body_bytes if body_bytes else None,
|
||||
@@ -608,22 +510,16 @@ async def _handle_proxy_request(ctx: SidecarContext, request: Request, path: str
|
||||
)
|
||||
|
||||
# NVIDIA → 排队 + 限流 + 转发
|
||||
await ctx.increment_stat("nvidia_requests")
|
||||
_stats["nvidia_requests"] += 1
|
||||
priority: Priority = _resolve_priority(raw_headers)
|
||||
|
||||
# 注入内部元数据到 payload
|
||||
payload_for_queue: dict[str, Any] = dict(body_json)
|
||||
# 剥离 NVIDIA provider 前缀（如 "nvidia/deepseek-ai/deepseek-v4-pro" → "deepseek-ai/deepseek-v4-pro"）
|
||||
if model and "/" in model:
|
||||
stripped_model: str = model.split("/", 1)[1]
|
||||
payload_for_queue["model"] = stripped_model
|
||||
bytes_model_stripped: bytes = json.dumps(body_json).encode()
|
||||
# Update model in the raw body bytes
|
||||
payload_for_queue["_raw_body"] = json.dumps(payload_for_queue).encode()
|
||||
payload_for_queue["_raw_body"] = body_bytes
|
||||
|
||||
# 尝试入队；PASSTHROUGH 策略下队列满时走直通路径
|
||||
try:
|
||||
request_id = await ctx.priority_queue.put(
|
||||
request_id = await _priority_queue.put(
|
||||
item=payload_for_queue,
|
||||
priority=priority,
|
||||
headers={
|
||||
@@ -633,7 +529,7 @@ async def _handle_proxy_request(ctx: SidecarContext, request: Request, path: str
|
||||
},
|
||||
)
|
||||
except QueueFullError:
|
||||
await ctx.increment_stat("queue_full_rejects")
|
||||
_stats["queue_full_rejects"] += 1
|
||||
return JSONResponse(
|
||||
status_code=503,
|
||||
content={
|
||||
@@ -644,16 +540,18 @@ async def _handle_proxy_request(ctx: SidecarContext, request: Request, path: str
|
||||
},
|
||||
)
|
||||
except QueueFullPassthrough:
|
||||
await ctx.increment_stat("passthrough_requests")
|
||||
# 队列满 + PASSTHROUGH：绕过排队，尝试令牌桶后直接转发
|
||||
_stats["passthrough_requests"] += 1
|
||||
logger.info("queue_full_passthrough", path=path)
|
||||
return await _passthrough_with_rate_limit(ctx, request, path, body_bytes, raw_headers, priority)
|
||||
return await _passthrough_with_rate_limit(request, path, body_bytes, raw_headers, priority)
|
||||
|
||||
# 创建 future 并注册到 pending
|
||||
loop = asyncio.get_running_loop()
|
||||
future: asyncio.Future[httpx.Response] = loop.create_future()
|
||||
ctx.pending_requests[request_id] = (future, time.monotonic())
|
||||
_pending_requests[request_id] = (future, time.monotonic())
|
||||
|
||||
try:
|
||||
# 等待 worker 完成处理
|
||||
resp = await future
|
||||
return _build_response(resp)
|
||||
except _RateLimitedError as exc:
|
||||
@@ -711,55 +609,38 @@ def _build_response(resp: httpx.Response) -> Response:
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@app.get("/health")
|
||||
async def health(ctx: SidecarContext = Depends(get_context)) -> dict[str, Any]:
|
||||
"""存活检查 (liveness)。"""
|
||||
return ctx.health.liveness()
|
||||
async def health() -> dict[str, Any]:
|
||||
"""健康检查端点。"""
|
||||
queue_stats = await _priority_queue.get_stats()
|
||||
bucket_status = _token_bucket.get_status()
|
||||
return {
|
||||
"status": "ok",
|
||||
"version": "0.1.0",
|
||||
"uptime_seconds": int(time.time() - _stats["start_time"]) if _stats["start_time"] else 0,
|
||||
"queue": queue_stats,
|
||||
"token_bucket": bucket_status,
|
||||
}
|
||||
|
||||
|
||||
@app.get("/health/ready")
|
||||
async def health_ready(ctx: SidecarContext = Depends(get_context)) -> dict[str, Any]:
|
||||
"""就绪检查 (readiness)，含上游连通性。
|
||||
|
||||
BIZ-46 Phase3: 复用 ctx.http_client，不再每次创建新 client。
|
||||
"""
|
||||
queue_size = await ctx.priority_queue.get_queue_size()
|
||||
bucket_status = ctx.token_bucket.get_status()
|
||||
return await ctx.health.readiness(
|
||||
upstream_url=ctx.config.upstream_url,
|
||||
upstream_api_key=ctx.config.upstream_api_key or "",
|
||||
queue_current_size=queue_size,
|
||||
queue_max_size=ctx.config.queue_max_size,
|
||||
available_tokens=bucket_status["tokens"],
|
||||
bucket_capacity=bucket_status["capacity"],
|
||||
http_client=ctx.http_client, # 复用主 client
|
||||
)
|
||||
|
||||
|
||||
@app.get("/status")
|
||||
async def status(ctx: SidecarContext = Depends(get_context)) -> dict[str, Any]:
|
||||
"""调试用：限流器 + 队列 + 避退完整状态。"""
|
||||
queue_stats = await ctx.priority_queue.get_stats()
|
||||
bucket_status = ctx.token_bucket.get_status()
|
||||
@app.get("/metrics")
|
||||
async def metrics() -> dict[str, Any]:
|
||||
"""Prometheus 格式 metrics 端点。"""
|
||||
queue_stats = await _priority_queue.get_stats()
|
||||
bucket_status = _token_bucket.get_status()
|
||||
return {
|
||||
"requests": {
|
||||
"total": ctx.stats["total_requests"],
|
||||
"nvidia": ctx.stats["nvidia_requests"],
|
||||
"passthrough": ctx.stats["passthrough_requests"],
|
||||
"ratelimited": ctx.stats["ratelimited_requests"],
|
||||
"total": _stats["total_requests"],
|
||||
"nvidia": _stats["nvidia_requests"],
|
||||
"passthrough": _stats["passthrough_requests"],
|
||||
"ratelimited": _stats["ratelimited_requests"],
|
||||
},
|
||||
"errors": {
|
||||
"queue_full_rejects": ctx.stats["queue_full_rejects"],
|
||||
"upstream_errors": ctx.stats["upstream_errors"],
|
||||
"queue_full_rejects": _stats["queue_full_rejects"],
|
||||
"upstream_errors": _stats["upstream_errors"],
|
||||
},
|
||||
"queue": queue_stats,
|
||||
"token_bucket": bucket_status,
|
||||
"retreat": {
|
||||
"state": ctx.token_bucket.get_retreat_state(),
|
||||
"effective_rpm": round(ctx.token_bucket.get_effective_rate_rpm(), 1),
|
||||
"base_rpm": round(ctx.token_bucket.get_base_rate_rpm(), 1),
|
||||
"upstream_429_rate": round(ctx.token_bucket.get_429_rate(), 4),
|
||||
},
|
||||
"uptime_seconds": ctx.uptime_seconds,
|
||||
"uptime_seconds": int(time.time() - _stats["start_time"]) if _stats["start_time"] else 0,
|
||||
}
|
||||
|
||||
|
||||
@@ -768,50 +649,36 @@ async def status(ctx: SidecarContext = Depends(get_context)) -> dict[str, Any]:
|
||||
@app.post("/v1/chat/completions")
|
||||
async def chat_completions(request: Request) -> Response:
|
||||
"""OpenAI Chat Completions API 代理（含流式支持）。"""
|
||||
ctx: SidecarContext = get_context()
|
||||
return await _handle_proxy_request(ctx, request, "/v1/chat/completions")
|
||||
return await _handle_proxy_request(request, "/v1/chat/completions")
|
||||
|
||||
|
||||
@app.post("/v1/completions")
|
||||
async def completions(request: Request) -> Response:
|
||||
ctx: SidecarContext = get_context()
|
||||
"""OpenAI Completions API 代理（legacy）。"""
|
||||
return await _handle_proxy_request(ctx, request, "/v1/completions")
|
||||
return await _handle_proxy_request(request, "/v1/completions")
|
||||
|
||||
|
||||
@app.post("/v1/embeddings")
|
||||
async def embeddings(request: Request) -> Response:
|
||||
ctx: SidecarContext = get_context()
|
||||
"""OpenAI Embeddings API 代理。"""
|
||||
return await _handle_proxy_request(ctx, request, "/v1/embeddings")
|
||||
return await _handle_proxy_request(request, "/v1/embeddings")
|
||||
|
||||
|
||||
@app.get("/v1/models")
|
||||
@app.get("/v1/models/{model_id:path}")
|
||||
async def list_models(request: Request, model_id: str | None = None) -> Response:
|
||||
ctx: SidecarContext = get_context()
|
||||
"""OpenAI Models API 代理。"""
|
||||
path = f"/v1/models/{model_id}" if model_id else "/v1/models"
|
||||
return await _handle_proxy_request(ctx, request, path)
|
||||
return await _handle_proxy_request(request, path)
|
||||
|
||||
|
||||
# ---- 通用代理（catch-all 用于非标准 NVIDIA 端点） ----
|
||||
|
||||
@app.api_route("/{path:path}", methods=["GET", "POST", "PUT", "DELETE", "PATCH", "OPTIONS"])
|
||||
async def catch_all(request: Request, path: str) -> Response:
|
||||
ctx: SidecarContext = get_context()
|
||||
"""通用代理端点：转发任何未匹配的路径到上游。"""
|
||||
target_path = f"/{path}" if not path.startswith("/") else path
|
||||
return await _handle_proxy_request(ctx, request, target_path)
|
||||
|
||||
|
||||
@app.get("/metrics")
|
||||
async def metrics(ctx: SidecarContext = Depends(get_context)) -> PlainTextResponse:
|
||||
"""Prometheus 指标端点。"""
|
||||
return PlainTextResponse(
|
||||
content=ctx.prometheus.generate_latest().decode(),
|
||||
media_type="text/plain; version=0.0.4",
|
||||
)
|
||||
return await _handle_proxy_request(request, target_path)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -1,327 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>NVIDIA Sidecar — 实时仪表盘</title>
|
||||
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.7/dist/chart.umd.min.js"></script>
|
||||
<style>
|
||||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #0f172a; color: #e2e8f0; padding: 24px; }
|
||||
h1 { font-size: 22px; font-weight: 600; margin-bottom: 4px; color: #f8fafc; }
|
||||
.subtitle { color: #94a3b8; font-size: 13px; margin-bottom: 24px; }
|
||||
.grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(380px, 1fr)); gap: 20px; margin-bottom: 24px; }
|
||||
.card { background: #1e293b; border-radius: 12px; padding: 20px; border: 1px solid #334155; }
|
||||
.card h2 { font-size: 15px; font-weight: 600; color: #94a3b8; margin-bottom: 14px; text-transform: uppercase; letter-spacing: 0.05em; }
|
||||
.card canvas { max-height: 220px; }
|
||||
.stat-row { display: flex; gap: 16px; flex-wrap: wrap; }
|
||||
.stat { flex: 1; min-width: 100px; background: #0f172a; border-radius: 8px; padding: 12px; text-align: center; border: 1px solid #334155; }
|
||||
.stat .value { font-size: 28px; font-weight: 700; color: #38bdf8; }
|
||||
.stat .label { font-size: 11px; color: #64748b; margin-top: 4px; text-transform: uppercase; }
|
||||
.stat.warn .value { color: #f59e0b; }
|
||||
.stat.danger .value { color: #ef4444; }
|
||||
.retreat-badge { display: inline-block; padding: 2px 10px; border-radius: 999px; font-size: 12px; font-weight: 600; }
|
||||
.retreat-badge.normal { background: #065f46; color: #6ee7b7; }
|
||||
.retreat-badge.retreat { background: #78350f; color: #fbbf24; }
|
||||
.retreat-badge.recover { background: #1e3a5f; color: #60a5fa; }
|
||||
.config-panel { background: #1e293b; border-radius: 12px; padding: 20px; border: 1px solid #334155; }
|
||||
.config-panel h2 { font-size: 15px; font-weight: 600; color: #94a3b8; margin-bottom: 14px; text-transform: uppercase; letter-spacing: 0.05em; }
|
||||
.config-row { display: flex; align-items: center; gap: 12px; margin-bottom: 12px; flex-wrap: wrap; }
|
||||
.config-row label { min-width: 100px; font-size: 13px; color: #cbd5e1; }
|
||||
.config-row input, .config-row select { background: #0f172a; border: 1px solid #334155; border-radius: 6px; color: #e2e8f0; padding: 6px 10px; font-size: 13px; }
|
||||
.config-row input[type="range"] { width: 140px; }
|
||||
.config-row button { background: #38bdf8; color: #0f172a; border: none; border-radius: 6px; padding: 6px 16px; font-size: 13px; font-weight: 600; cursor: pointer; }
|
||||
.config-row button:hover { background: #7dd3fc; }
|
||||
.config-row button:disabled { background: #475569; cursor: not-allowed; }
|
||||
.toast { position: fixed; top: 16px; right: 16px; padding: 10px 20px; border-radius: 8px; font-size: 13px; z-index: 999; animation: fadeInOut 3s; }
|
||||
.toast.success { background: #065f46; color: #6ee7b7; }
|
||||
.toast.error { background: #7f1d1d; color: #fca5a5; }
|
||||
@keyframes fadeInOut { 0% { opacity: 0; transform: translateY(-8px); } 10% { opacity: 1; transform: translateY(0); } 80% { opacity: 1; } 100% { opacity: 0; } }
|
||||
.disconnected { background: #7f1d1d; color: #fca5a5; padding: 4px 10px; border-radius: 4px; font-size: 12px; display: inline-block; margin-left: 8px; }
|
||||
.connected { background: #065f46; color: #6ee7b7; padding: 4px 10px; border-radius: 4px; font-size: 12px; display: inline-block; margin-left: 8px; }
|
||||
|
||||
/* BIZ-46 Phase3: 队列柱状图 300ms 平滑动画 */
|
||||
.queue-bar { transition: height 0.3s ease; }
|
||||
|
||||
/* BIZ-46 Phase3: SSE 断连 5s 半透明遮罩 */
|
||||
#reconnect-mask {
|
||||
display: none;
|
||||
position: fixed;
|
||||
top: 0; left: 0; right: 0; bottom: 0;
|
||||
background: rgba(15, 23, 42, 0.85);
|
||||
z-index: 1000;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
flex-direction: column;
|
||||
}
|
||||
#reconnect-mask.visible { display: flex; }
|
||||
#reconnect-mask .mask-icon { font-size: 48px; margin-bottom: 16px; }
|
||||
#reconnect-mask .mask-text { color: #94a3b8; font-size: 16px; font-weight: 500; }
|
||||
#reconnect-mask .mask-sub { color: #64748b; font-size: 13px; margin-top: 8px; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<!-- BIZ-46 Phase3: SSE 断连遮罩 -->
|
||||
<div id="reconnect-mask">
|
||||
<div class="mask-icon">⚠️</div>
|
||||
<div class="mask-text">数据暂不可用</div>
|
||||
<div class="mask-sub">SSE 连接中断，正在重连…</div>
|
||||
</div>
|
||||
|
||||
<h1>🚀 NVIDIA Sidecar 实时仪表盘
|
||||
<span id="conn-status" class="connected">已连接</span>
|
||||
</h1>
|
||||
<p class="subtitle">令牌桶限流 · 优先级队列 · 避退模式 · 实时监控</p>
|
||||
|
||||
<!-- 状态卡片 -->
|
||||
<div class="stat-row" style="margin-bottom: 24px;">
|
||||
<div class="stat"><div class="value" id="val-total">0</div><div class="label">总请求</div></div>
|
||||
<div class="stat"><div class="value" id="val-nvidia">0</div><div class="label">NVIDIA 请求</div></div>
|
||||
<div class="stat"><div class="value" id="val-rate">0</div><div class="label">当前 RPM</div></div>
|
||||
<div class="stat"><div class="value" id="val-429">0%</div><div class="label">上游 429 率</div></div>
|
||||
<div class="stat"><div class="value" id="val-retreat">正常</div><div class="label">避退状态</div></div>
|
||||
<div class="stat"><div class="value" id="val-uptime">0s</div><div class="label">运行时间</div></div>
|
||||
</div>
|
||||
|
||||
<!-- 图表 -->
|
||||
<div class="grid">
|
||||
<div class="card">
|
||||
<h2>📊 令牌桶使用率</h2>
|
||||
<canvas id="chart-tokens"></canvas>
|
||||
</div>
|
||||
<div class="card">
|
||||
<!-- BIZ-46 Phase3: 队列图标题显示总排队数 -->
|
||||
<h2>📈 队列深度 <span id="queue-total" style="font-size:13px;color:#38bdf8;">(共 0)</span></h2>
|
||||
<canvas id="chart-queue"></canvas>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h2>📉 请求吞吐量 (最近 20 点)</h2>
|
||||
<canvas id="chart-throughput"></canvas>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h2>⚙️ 速率历史</h2>
|
||||
<canvas id="chart-rate"></canvas>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- 配置面板 -->
|
||||
<div class="config-panel">
|
||||
<h2>🔧 实时配置</h2>
|
||||
<div class="config-row">
|
||||
<label>速率 (RPM)</label>
|
||||
<input type="range" id="cfg-rate-rpm" min="1" max="100" value="40" oninput="document.getElementById('cfg-rate-val').textContent=this.value">
|
||||
<span id="cfg-rate-val" style="min-width:30px;">40</span>
|
||||
</div>
|
||||
<div class="config-row">
|
||||
<label>队列上限</label>
|
||||
<input type="number" id="cfg-queue-max" value="500" min="1" max="2000" style="width:80px;">
|
||||
</div>
|
||||
<div class="config-row">
|
||||
<button onclick="applyConfig()">应用配置</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
// SSE 连接
|
||||
let evtSource = null;
|
||||
let dataHistory = { throughput: [], rates: [] };
|
||||
const MAX_HISTORY = 20;
|
||||
let lastSSETime = Date.now();
|
||||
|
||||
// BIZ-46 Phase3: SSE 断连 5s 遮罩
|
||||
function checkReconnect() {
|
||||
const mask = document.getElementById('reconnect-mask');
|
||||
if (Date.now() - lastSSETime > 5000) {
|
||||
mask.classList.add('visible');
|
||||
}
|
||||
}
|
||||
setInterval(checkReconnect, 1000);
|
||||
|
||||
function connectSSE() {
|
||||
if (evtSource) evtSource.close();
|
||||
evtSource = new EventSource('/api/dashboard/stream');
|
||||
evtSource.onmessage = (e) => {
|
||||
try {
|
||||
const snap = JSON.parse(e.data);
|
||||
lastSSETime = Date.now();
|
||||
// 隐藏断连遮罩
|
||||
document.getElementById('reconnect-mask').classList.remove('visible');
|
||||
updateDashboard(snap);
|
||||
document.getElementById('conn-status').className = 'connected';
|
||||
document.getElementById('conn-status').textContent = '已连接';
|
||||
} catch (err) {
|
||||
document.getElementById('conn-status').className = 'disconnected';
|
||||
document.getElementById('conn-status').textContent = '解析错误';
|
||||
}
|
||||
};
|
||||
evtSource.onerror = () => {
|
||||
document.getElementById('conn-status').className = 'disconnected';
|
||||
document.getElementById('conn-status').textContent = '断开 - 重连中';
|
||||
};
|
||||
}
|
||||
|
||||
// 初始化 Chart.js
|
||||
const ctxTokens = document.getElementById('chart-tokens').getContext('2d');
|
||||
const chartTokens = new Chart(ctxTokens, {
|
||||
type: 'doughnut',
|
||||
data: {
|
||||
labels: ['已用令牌', '可用令牌'],
|
||||
datasets: [{ data: [0, 40], backgroundColor: ['#ef4444', '#22c55e'], borderWidth: 0 }]
|
||||
},
|
||||
options: { responsive: true, maintainAspectRatio: true, cutout: '65%', plugins: { legend: { position: 'bottom', labels: { color: '#94a3b8' } } },
|
||||
// BIZ-46 Phase3: 300ms 平滑动画
|
||||
animation: { duration: 300 } }
|
||||
});
|
||||
|
||||
const ctxQueue = document.getElementById('chart-queue').getContext('2d');
|
||||
const chartQueue = new Chart(ctxQueue, {
|
||||
type: 'bar',
|
||||
data: {
|
||||
labels: ['URGENT', 'HIGH', 'NORMAL', 'LOW'],
|
||||
datasets: [{ label: '排队数', data: [0, 0, 0, 0], backgroundColor: ['#ef4444', '#f59e0b', '#38bdf8', '#a78bfa'] }]
|
||||
},
|
||||
options: { responsive: true, maintainAspectRatio: true,
|
||||
scales: { y: { beginAtZero: true, ticks: { color: '#94a3b8' } }, x: { ticks: { color: '#94a3b8' } } },
|
||||
plugins: { legend: { display: false } },
|
||||
// BIZ-46 Phase3: 300ms 平滑动画
|
||||
animation: { duration: 300 } }
|
||||
});
|
||||
|
||||
const ctxThroughput = document.getElementById('chart-throughput').getContext('2d');
|
||||
const chartThroughput = new Chart(ctxThroughput, {
|
||||
type: 'line',
|
||||
data: { labels: [], datasets: [
|
||||
{ label: '成功', data: [], borderColor: '#22c55e', backgroundColor: '#22c55e20', fill: false, tension: 0.3, pointRadius: 2 },
|
||||
{ label: '429', data: [], borderColor: '#f59e0b', backgroundColor: '#f59e0b20', fill: false, tension: 0.3, pointRadius: 2 },
|
||||
{ label: '直通', data: [], borderColor: '#a78bfa', backgroundColor: '#a78bfa20', fill: false, tension: 0.3, pointRadius: 2 },
|
||||
]},
|
||||
options: { responsive: true, maintainAspectRatio: true,
|
||||
scales: { y: { beginAtZero: true, ticks: { color: '#94a3b8' } }, x: { ticks: { color: '#94a3b8' } } },
|
||||
plugins: { legend: { position: 'bottom', labels: { color: '#94a3b8' } } },
|
||||
animation: { duration: 300 } }
|
||||
});
|
||||
|
||||
const ctxRate = document.getElementById('chart-rate').getContext('2d');
|
||||
const chartRate = new Chart(ctxRate, {
|
||||
type: 'line',
|
||||
data: { labels: [], datasets: [
|
||||
{ label: '有效 RPM', data: [], borderColor: '#38bdf8', fill: false, tension: 0.3, pointRadius: 2 },
|
||||
{ label: '基准 RPM', data: [], borderColor: '#64748b', fill: false, tension: 0.3, pointRadius: 2, borderDash: [4, 4] },
|
||||
]},
|
||||
options: { responsive: true, maintainAspectRatio: true,
|
||||
scales: { y: { beginAtZero: true, ticks: { color: '#94a3b8' } }, x: { ticks: { color: '#94a3b8' } } },
|
||||
plugins: { legend: { position: 'bottom', labels: { color: '#94a3b8' } } },
|
||||
animation: { duration: 300 } }
|
||||
});
|
||||
|
||||
function updateDashboard(snap) {
|
||||
const r = snap.requests || {};
|
||||
const tb = snap.token_bucket || {};
|
||||
const rt = snap.retreat || {};
|
||||
|
||||
document.getElementById('val-total').textContent = (r.total || 0).toLocaleString();
|
||||
document.getElementById('val-nvidia').textContent = (r.nvidia || 0).toLocaleString();
|
||||
document.getElementById('val-rate').textContent = Math.round(rt.effective_rpm || 40);
|
||||
document.getElementById('val-429').textContent = ((rt.upstream_429_rate || 0) * 100).toFixed(1) + '%';
|
||||
document.getElementById('val-uptime').textContent = fmtDuration(snap.uptime_seconds || 0);
|
||||
|
||||
const retreatEl = document.getElementById('val-retreat');
|
||||
const state = rt.state || 'normal';
|
||||
retreatEl.textContent = state === 'retreat' ? '⚠️ 避退' : state === 'recover' ? '↗ 恢复中' : '✅ 正常';
|
||||
retreatEl.style.color = state === 'retreat' ? '#f59e0b' : state === 'recover' ? '#60a5fa' : '#22c55e';
|
||||
|
||||
chartTokens.data.datasets[0].data = [
|
||||
Math.round((tb.capacity || 40) - (tb.tokens || 40)),
|
||||
Math.round(tb.tokens || 0)
|
||||
];
|
||||
chartTokens.update();
|
||||
|
||||
const qs = snap.queue || {};
|
||||
const perPriority = qs.per_priority || {};
|
||||
const totalQueued = perPriority.URGENT + perPriority.HIGH + perPriority.NORMAL + perPriority.LOW || qs.current_size || 0;
|
||||
chartQueue.data.datasets[0].data = [
|
||||
perPriority.URGENT || 0,
|
||||
perPriority.HIGH || 0,
|
||||
perPriority.NORMAL || 0,
|
||||
perPriority.LOW || 0
|
||||
];
|
||||
chartQueue.update();
|
||||
|
||||
// BIZ-46 Phase3: 队列图标题显示总排队数
|
||||
document.getElementById('queue-total').textContent = '(共 ' + totalQueued + ')';
|
||||
|
||||
const now = new Date().toLocaleTimeString();
|
||||
const prev = dataHistory.throughput.length > 0 ? dataHistory.throughput[dataHistory.throughput.length - 1].nvidia : 0;
|
||||
const throughput = Math.max(0, (r.nvidia || 0) - prev);
|
||||
|
||||
dataHistory.throughput.push({ time: now, nvidia: throughput, ratelimited: r.ratelimited || 0, passthrough: r.passthrough || 0 });
|
||||
dataHistory.rates.push({ time: now, effective: rt.effective_rpm || 40, base: rt.base_rpm || 40 });
|
||||
if (dataHistory.throughput.length > MAX_HISTORY) dataHistory.throughput.shift();
|
||||
if (dataHistory.rates.length > MAX_HISTORY) dataHistory.rates.shift();
|
||||
|
||||
chartThroughput.data.labels = dataHistory.throughput.map(d => d.time);
|
||||
chartThroughput.data.datasets[0].data = dataHistory.throughput.map(d => d.nvidia);
|
||||
chartThroughput.data.datasets[1].data = dataHistory.throughput.map(d => d.ratelimited);
|
||||
chartThroughput.data.datasets[2].data = dataHistory.throughput.map(d => d.passthrough);
|
||||
chartThroughput.update();
|
||||
|
||||
chartRate.data.labels = dataHistory.rates.map(d => d.time);
|
||||
chartRate.data.datasets[0].data = dataHistory.rates.map(d => d.effective);
|
||||
chartRate.data.datasets[1].data = dataHistory.rates.map(d => d.base);
|
||||
chartRate.update();
|
||||
}
|
||||
|
||||
function fmtDuration(s) {
|
||||
if (s < 60) return s + 's';
|
||||
if (s < 3600) return Math.floor(s/60) + 'm ' + (s%60) + 's';
|
||||
return Math.floor(s/3600) + 'h ' + Math.floor((s%3600)/60) + 'm';
|
||||
}
|
||||
|
||||
async function applyConfig() {
|
||||
const btn = document.querySelector('.config-row button');
|
||||
btn.disabled = true;
|
||||
try {
|
||||
const resp = await fetch('/api/admin/config', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
rate_rpm: parseInt(document.getElementById('cfg-rate-rpm').value),
|
||||
queue_max_size: parseInt(document.getElementById('cfg-queue-max').value),
|
||||
})
|
||||
});
|
||||
const result = await resp.json();
|
||||
showToast(resp.ok ? 'success' : 'error', resp.ok ? '配置已更新' : (result.detail || '配置更新失败'));
|
||||
} catch (err) {
|
||||
showToast('error', '请求失败: ' + err.message);
|
||||
}
|
||||
btn.disabled = false;
|
||||
}
|
||||
|
||||
function showToast(type, msg) {
|
||||
const t = document.createElement('div');
|
||||
t.className = 'toast ' + type;
|
||||
t.textContent = msg;
|
||||
document.body.appendChild(t);
|
||||
setTimeout(() => t.remove(), 3000);
|
||||
}
|
||||
|
||||
// BIZ-46 Phase3: 页面加载时同步当前配置值
|
||||
async function loadConfig() {
|
||||
try {
|
||||
const resp = await fetch('/api/admin/config');
|
||||
if (resp.ok) {
|
||||
const config = await resp.json();
|
||||
document.getElementById('cfg-rate-rpm').value = config.rate_rpm || 40;
|
||||
document.getElementById('cfg-rate-val').textContent = config.rate_rpm || 40;
|
||||
document.getElementById('cfg-queue-max').value = config.queue_max_size || 500;
|
||||
}
|
||||
} catch (e) {
|
||||
console.warn('配置加载失败（可能需要 Admin Token）', e);
|
||||
}
|
||||
}
|
||||
|
||||
loadConfig();
|
||||
connectSSE();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1 +0,0 @@
|
||||
# nvidia_sidecar tests
|
||||
@@ -1,207 +0,0 @@
|
||||
"""
|
||||
避退模式并发/死锁回归测试 (BIZ-46 Phase3 6)
|
||||
|
||||
覆盖多线程场景下的 AdaptiveTokenBucket 线程安全性：
|
||||
- 并发 record_response + evaluate_retreat
|
||||
- 并发 consume + record_response + evaluate_retreat
|
||||
- 高负载下避退状态转换正确性
|
||||
|
||||
设计文档: docs/architecture/BIZ-46_Phase3_Architecture_Design.md 6
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from nvidia_sidecar.rate_limiter import AdaptiveTokenBucket, RetreatState
|
||||
|
||||
|
||||
class TestRetreatConcurrency:
|
||||
"""避退模式并发安全回归测试。"""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_concurrent_record_and_evaluate(self) -> None:
|
||||
"""多线程同时 record_response + evaluate_retreat 不死锁。
|
||||
|
||||
4 个线程同时操作：
|
||||
- 2 个线程执行 record_response (1000 次)
|
||||
- 2 个线程执行 evaluate_retreat (1000 次)
|
||||
|
||||
所有线程必须在 10s 内完成，否则判定为死锁。
|
||||
"""
|
||||
bucket = AdaptiveTokenBucket(rate=40 / 60, capacity=40)
|
||||
errors: list[Exception] = []
|
||||
|
||||
def worker_record() -> None:
|
||||
for i in range(1000):
|
||||
try:
|
||||
bucket.record_response(is_429=(i % 10 == 0))
|
||||
except Exception as e:
|
||||
errors.append(e)
|
||||
|
||||
def worker_evaluate() -> None:
|
||||
for _ in range(1000):
|
||||
try:
|
||||
bucket.evaluate_retreat()
|
||||
except Exception as e:
|
||||
errors.append(e)
|
||||
|
||||
threads = [
|
||||
threading.Thread(target=worker_record),
|
||||
threading.Thread(target=worker_record),
|
||||
threading.Thread(target=worker_evaluate),
|
||||
threading.Thread(target=worker_evaluate),
|
||||
]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(timeout=10)
|
||||
|
||||
alive_threads = [t for t in threads if t.is_alive()]
|
||||
assert not alive_threads, (
|
||||
f"{len(alive_threads)} 个线程未完成，疑似死锁"
|
||||
)
|
||||
assert not errors, f"并发错误: {errors}"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_concurrent_consume_and_retreat(self) -> None:
|
||||
"""多线程同时 consume + record_response + evaluate_retreat 不死锁。
|
||||
|
||||
覆盖 _lock (TokenBucket) 和 _retreat_lock (AdaptiveTokenBucket)
|
||||
同时被不同线程持有时的交叉锁场景。
|
||||
"""
|
||||
bucket = AdaptiveTokenBucket(rate=40 / 60, capacity=40)
|
||||
errors: list[Exception] = []
|
||||
|
||||
def worker_consume() -> None:
|
||||
for _ in range(500):
|
||||
try:
|
||||
bucket.consume(tokens=1)
|
||||
except Exception as e:
|
||||
errors.append(e)
|
||||
|
||||
def worker_retreat() -> None:
|
||||
for _ in range(500):
|
||||
try:
|
||||
bucket.record_response(is_429=False)
|
||||
bucket.evaluate_retreat()
|
||||
except Exception as e:
|
||||
errors.append(e)
|
||||
|
||||
threads = [
|
||||
threading.Thread(target=worker_consume),
|
||||
threading.Thread(target=worker_consume),
|
||||
threading.Thread(target=worker_retreat),
|
||||
threading.Thread(target=worker_retreat),
|
||||
]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(timeout=10)
|
||||
|
||||
alive_threads = [t for t in threads if t.is_alive()]
|
||||
assert not alive_threads, (
|
||||
f"{len(alive_threads)} 个线程未完成，疑似死锁"
|
||||
)
|
||||
assert not errors, f"并发错误: {errors}"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retreat_state_transitions_under_load(self) -> None:
|
||||
"""高负载下避退状态转换正确。
|
||||
|
||||
1. 注入 100 个 429 → 验证进入 RETREAT
|
||||
2. 注入 200 个成功 → 手动推进时间 → 验证恢复
|
||||
"""
|
||||
bucket = AdaptiveTokenBucket(
|
||||
rate=40 / 60,
|
||||
capacity=40,
|
||||
retreat_window_seconds=0.1,
|
||||
retreat_429_threshold=0.05,
|
||||
retreat_factor=0.75,
|
||||
retreat_min_rpm=5.0,
|
||||
recover_window_seconds=0.01,
|
||||
)
|
||||
|
||||
# 阶段 1：模拟高 429 率
|
||||
for _ in range(100):
|
||||
bucket.record_response(is_429=True)
|
||||
|
||||
state = bucket.evaluate_retreat()
|
||||
assert state == RetreatState.RETREAT, (
|
||||
f"高 429 率应触发避退，实际: {state}"
|
||||
)
|
||||
assert bucket.get_effective_rate_rpm() < bucket.get_base_rate_rpm(), (
|
||||
f"避退后速率应低于基准，实际: "
|
||||
f"{bucket.get_effective_rate_rpm()} vs {bucket.get_base_rate_rpm()}"
|
||||
)
|
||||
|
||||
# 阶段 2：模拟恢复
|
||||
time.sleep(0.15) # 等待 429 从短窗口中过期
|
||||
for _ in range(200):
|
||||
bucket.record_response(is_429=False)
|
||||
|
||||
for _ in range(10):
|
||||
state = bucket.evaluate_retreat()
|
||||
|
||||
assert state in (RetreatState.RECOVER, RetreatState.NORMAL), (
|
||||
f"恢复后应为 RECOVER 或 NORMAL，实际: {state}"
|
||||
)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_try_consume_concurrency_safety(self) -> None:
|
||||
"""并发 try_consume 不死锁。"""
|
||||
bucket = AdaptiveTokenBucket(rate=40 / 60, capacity=40)
|
||||
errors: list[Exception] = []
|
||||
results: list[bool] = []
|
||||
|
||||
def worker() -> None:
|
||||
for _ in range(200):
|
||||
try:
|
||||
got = bucket.try_consume(tokens=1, timeout=0.1)
|
||||
results.append(got)
|
||||
except Exception as e:
|
||||
errors.append(e)
|
||||
|
||||
threads = [threading.Thread(target=worker) for _ in range(8)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(timeout=10)
|
||||
|
||||
alive = [t for t in threads if t.is_alive()]
|
||||
assert not alive, f"{len(alive)} 个线程未完成，疑似死锁"
|
||||
assert not errors, f"并发错误: {errors}"
|
||||
successful = sum(1 for r in results if r)
|
||||
assert successful > 0, (
|
||||
f"令牌桶应至少成功消费一些令牌，成功: {successful}/{len(results)}"
|
||||
)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_high_load_state_coherence(self) -> None:
|
||||
"""高负载下令牌桶状态一致性：消费总量 ≤ 初始 token + 补充量。"""
|
||||
bucket = AdaptiveTokenBucket(rate=10.0, capacity=100)
|
||||
consumed_count: list[int] = [0]
|
||||
lock = threading.Lock()
|
||||
|
||||
def worker() -> None:
|
||||
local_consumed = 0
|
||||
for _ in range(50):
|
||||
if bucket.consume(tokens=1):
|
||||
local_consumed += 1
|
||||
time.sleep(0.001)
|
||||
with lock:
|
||||
consumed_count[0] += local_consumed
|
||||
|
||||
threads = [threading.Thread(target=worker) for _ in range(10)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(timeout=15)
|
||||
|
||||
max_expected = 100 + int(10.0 * 5)
|
||||
assert consumed_count[0] <= max_expected, (
|
||||
f"消费量异常: {consumed_count[0]}，应 ≤ {max_expected}"
|
||||
)
|
||||
@@ -1,325 +0,0 @@
|
||||
"""
|
||||
NVIDIA Sidecar — WebUI 后端 API
|
||||
|
||||
提供仪表盘 SSE 实时推送 + 配置热重载 API。
|
||||
|
||||
BIZ-46 Phase3:
|
||||
- 架构解耦：移除反向导入 server，改用 Depends(get_context) (§1)
|
||||
- SSE 共享缓存：1s TTL snapshot cache，多客户端不重复构建 (§3)
|
||||
- Dashboard UX：页面加载同步配置 + 队列深度标题 (§7)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, AsyncGenerator
|
||||
|
||||
import structlog
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from fastapi.responses import HTMLResponse, JSONResponse, StreamingResponse
|
||||
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
||||
from pydantic import BaseModel
|
||||
|
||||
from nvidia_sidecar.context import SidecarContext
|
||||
|
||||
webui_router: APIRouter = APIRouter(prefix="/api", tags=["webui"])
|
||||
logger: structlog.stdlib.BoundLogger = structlog.get_logger("nvidia_sidecar.webui")
|
||||
|
||||
STATIC_DIR: Path = Path(__file__).parent / "static"
|
||||
|
||||
# dashboard.html 缓存（严维序评审 #6 / 梁思筑评审 #8：避免每次请求读磁盘）
|
||||
_dashboard_html_cache: tuple[str, float] | None = None
|
||||
_DASHBOARD_CACHE_TTL: float = 300.0 # 5 分钟
|
||||
|
||||
# Admin API 认证（严维序评审 #1）
|
||||
_ADMIN_TOKEN: str | None = os.environ.get("SIDECAR_ADMIN_TOKEN")
|
||||
_admin_auth_scheme: HTTPBearer = HTTPBearer(auto_error=False)
|
||||
|
||||
|
||||
def _get_ctx(request: Request) -> SidecarContext:
|
||||
"""获取 SidecarContext（webui 路由级注入，避免循环导入 server）。"""
|
||||
return request.app.state.sidecar # type: ignore[no-any-return]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 配置热重载模型
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class ConfigPatch(BaseModel):
|
||||
"""可在线修改的配置字段。"""
|
||||
rate_rpm: int | None = None
|
||||
queue_max_size: int | None = None
|
||||
fallback_enabled_passthrough: bool | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SSE 快照构建（BIZ-46 Phase3: 1s TTL 共享缓存）
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _build_snapshot(ctx: SidecarContext) -> dict[str, Any]:
|
||||
"""构建当前状态快照（从 SidecarContext 读取，含队列深度）。
|
||||
|
||||
BIZ-46 Phase3: 不再通过反向导入 server 访问全局变量。
|
||||
"""
|
||||
try:
|
||||
bucket_status = ctx.token_bucket.get_status()
|
||||
now = time.time()
|
||||
|
||||
queue_data: dict[str, Any] = {"current_size": 0, "per_priority": {}}
|
||||
try:
|
||||
queue_stats = await ctx.priority_queue.get_stats()
|
||||
queue_data = {
|
||||
"max_size": queue_stats.get("max_size", 0),
|
||||
"current_size": queue_stats.get("current_size", 0),
|
||||
"per_priority": queue_stats.get("depth_by_priority", {}),
|
||||
"total_enqueued": queue_stats.get("total_enqueued", 0),
|
||||
"total_dequeued": queue_stats.get("total_dequeued", 0),
|
||||
"total_dropped": queue_stats.get("total_dropped", 0),
|
||||
}
|
||||
except Exception:
|
||||
logger.warning(
|
||||
"queue_stats_unavailable",
|
||||
message="队列统计获取失败，仪表盘队列深度可能不准确",
|
||||
)
|
||||
|
||||
return {
|
||||
"timestamp": now,
|
||||
"uptime_seconds": ctx.uptime_seconds,
|
||||
"token_bucket": bucket_status,
|
||||
"queue": queue_data,
|
||||
"retreat": {
|
||||
"state": ctx.token_bucket.get_retreat_state(),
|
||||
"effective_rpm": round(ctx.token_bucket.get_effective_rate_rpm(), 1),
|
||||
"base_rpm": round(ctx.token_bucket.get_base_rate_rpm(), 1),
|
||||
"upstream_429_rate": round(ctx.token_bucket.get_429_rate(), 4),
|
||||
},
|
||||
"requests": {
|
||||
"total": ctx.stats.get("total_requests", 0),
|
||||
"nvidia": ctx.stats.get("nvidia_requests", 0),
|
||||
"passthrough": ctx.stats.get("passthrough_requests", 0),
|
||||
"ratelimited": ctx.stats.get("ratelimited_requests", 0),
|
||||
},
|
||||
"errors": {
|
||||
"queue_full_rejects": ctx.stats.get("queue_full_rejects", 0),
|
||||
"upstream_errors": ctx.stats.get("upstream_errors", 0),
|
||||
},
|
||||
}
|
||||
except Exception:
|
||||
logger.exception("snapshot_build_error")
|
||||
return {"error": "snapshot_unavailable", "timestamp": time.time()}
|
||||
|
||||
|
||||
async def _build_snapshot_cached(ctx: SidecarContext) -> dict[str, Any]:
|
||||
"""带 1s TTL 的共享快照缓存（BIZ-46 Phase3 §3）。
|
||||
|
||||
多个 SSE 客户端共享同一份快照，避免重复计算和锁竞争。
|
||||
|
||||
性能收益：
|
||||
- 1 客户端: 1 次/s 计算（无变化）
|
||||
- 5 客户端: ~5 次/s → 1 次/s
|
||||
- 20 客户端: ~20 次/s → 1 次/s
|
||||
"""
|
||||
now_cache = time.monotonic()
|
||||
if ctx.snapshot_cache is not None:
|
||||
data, ts = ctx.snapshot_cache
|
||||
if now_cache - ts < ctx.SNAPSHOT_CACHE_TTL:
|
||||
return data
|
||||
|
||||
async with ctx.snapshot_cache_lock:
|
||||
# Double-check（避免多个协程同时 miss 后重复构建）
|
||||
if ctx.snapshot_cache is not None:
|
||||
data, ts = ctx.snapshot_cache
|
||||
if now_cache - ts < ctx.SNAPSHOT_CACHE_TTL:
|
||||
return data
|
||||
|
||||
snapshot = await _build_snapshot(ctx)
|
||||
ctx.snapshot_cache = (snapshot, now_cache)
|
||||
return snapshot
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 仪表盘 SSE 推送
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _dashboard_stream(request: Request, ctx: SidecarContext) -> StreamingResponse:
|
||||
"""SSE 实时推送 Sidecar 完整状态快照（每秒一次）。
|
||||
|
||||
供 dashboard.html 的 EventSource 消费。
|
||||
|
||||
BIZ-46 Phase3: 使用共享缓存 _build_snapshot_cached，多客户端不重复计算。
|
||||
"""
|
||||
async def event_generator() -> AsyncGenerator[str, None]:
|
||||
first_frame = True
|
||||
while True:
|
||||
if await request.is_disconnected():
|
||||
break
|
||||
try:
|
||||
snapshot: dict[str, Any] = await _build_snapshot_cached(ctx)
|
||||
payload_sse = f"data: {json.dumps(snapshot, ensure_ascii=False)}\n\n"
|
||||
if first_frame:
|
||||
payload_sse = f"retry: 3000\n{payload_sse}"
|
||||
first_frame = False
|
||||
yield payload_sse
|
||||
except Exception:
|
||||
logger.exception("dashboard_sse_error")
|
||||
yield f"data: {json.dumps({'error': 'internal'})}\n\n"
|
||||
await asyncio.sleep(1.0)
|
||||
|
||||
return StreamingResponse(
|
||||
event_generator(),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"X-Accel-Buffering": "no",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 配置热重载
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def get_config(ctx: SidecarContext) -> dict[str, Any]:
|
||||
"""获取当前完整配置（从 SidecarContext 读取）。"""
|
||||
config = ctx.config
|
||||
effective_rpm = float(ctx.token_bucket.get_effective_rate_rpm())
|
||||
return {
|
||||
"listen_host": config.listen_host,
|
||||
"listen_port": config.listen_port,
|
||||
"metrics_port": config.metrics_port,
|
||||
"upstream_url": config.upstream_url,
|
||||
"upstream_api_key": _mask_api_key(config.upstream_api_key),
|
||||
"rate_rpm": round(effective_rpm, 1),
|
||||
"bucket_capacity": config.bucket_capacity,
|
||||
"request_timeout": config.request_timeout,
|
||||
"queue_max_size": config.queue_max_size,
|
||||
"low_priority_timeout": config.low_priority_timeout,
|
||||
"fallback_enabled_passthrough": config.fallback_enabled_passthrough,
|
||||
"log_level": config.log_level,
|
||||
}
|
||||
|
||||
|
||||
async def update_config(body: ConfigPatch, ctx: SidecarContext) -> JSONResponse:
|
||||
"""在线修改配置项并即时生效。"""
|
||||
config = ctx.config
|
||||
changed: list[str] = []
|
||||
|
||||
if body.rate_rpm is not None:
|
||||
if body.rate_rpm <= 0:
|
||||
raise HTTPException(status_code=400, detail="rate_rpm must be > 0")
|
||||
config.rate_rpm = body.rate_rpm
|
||||
ctx.token_bucket.set_rate(body.rate_rpm / 60.0)
|
||||
changed.append("rate_rpm")
|
||||
|
||||
if body.queue_max_size is not None:
|
||||
if body.queue_max_size <= 0:
|
||||
raise HTTPException(status_code=400, detail="queue_max_size must be > 0")
|
||||
ok, msg = ctx.priority_queue.set_max_size(body.queue_max_size)
|
||||
if not ok:
|
||||
raise HTTPException(status_code=400, detail=msg)
|
||||
config.queue_max_size = body.queue_max_size
|
||||
changed.append("queue_max_size")
|
||||
logger.info("queue_max_size_updated", detail=msg)
|
||||
|
||||
if body.fallback_enabled_passthrough is not None:
|
||||
config.fallback_enabled_passthrough = body.fallback_enabled_passthrough
|
||||
changed.append("fallback_enabled_passthrough")
|
||||
|
||||
logger.info("config_updated", changed=changed)
|
||||
return JSONResponse(
|
||||
content={"status": "ok", "changed": changed},
|
||||
)
|
||||
|
||||
|
||||
def _mask_api_key(key: str) -> str:
|
||||
"""对 API Key 进行脱敏处理，仅保留前 4 位以供识别。
|
||||
|
||||
严维序评审 #2 / 沈路明评审 #3：防止 API Key 明文泄露。
|
||||
"""
|
||||
if not key:
|
||||
return ""
|
||||
if len(key) <= 4:
|
||||
return key[:2] + "****"
|
||||
return key[:4] + "****"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 路由注册
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@webui_router.get("/dashboard/stream")
|
||||
async def dashboard_stream(
|
||||
request: Request,
|
||||
ctx: SidecarContext = Depends(_get_ctx),
|
||||
) -> StreamingResponse:
|
||||
"""SSE 仪表盘实时推送端点（BIZ-46 Phase3: 使用共享缓存）。"""
|
||||
return await _dashboard_stream(request, ctx)
|
||||
|
||||
|
||||
async def _verify_admin_auth(
|
||||
credentials: HTTPAuthorizationCredentials | None = Depends(_admin_auth_scheme),
|
||||
) -> None:
|
||||
"""Admin API Bearer Token 认证（严维序评审 #1）。
|
||||
|
||||
若设置了 SIDECAR_ADMIN_TOKEN 环境变量，则要求请求携带匹配的 Bearer Token。
|
||||
未设置时跳过认证（开发/测试环境）。
|
||||
"""
|
||||
if _ADMIN_TOKEN is None:
|
||||
return # 未配置认证 token，允许无认证访问
|
||||
if credentials is None:
|
||||
raise HTTPException(status_code=401, detail="需要 Bearer Token 认证（Admin API）")
|
||||
if credentials.credentials != _ADMIN_TOKEN:
|
||||
raise HTTPException(status_code=403, detail="Admin Token 无效")
|
||||
|
||||
|
||||
@webui_router.get("/admin/config")
|
||||
async def admin_get_config(
|
||||
_auth: None = Depends(_verify_admin_auth),
|
||||
ctx: SidecarContext = Depends(_get_ctx),
|
||||
) -> JSONResponse:
|
||||
"""获取当前配置（需要 Admin 认证）。"""
|
||||
return JSONResponse(content=await get_config(ctx))
|
||||
|
||||
|
||||
@webui_router.post("/admin/config")
|
||||
async def admin_update_config(
|
||||
body: ConfigPatch,
|
||||
_auth: None = Depends(_verify_admin_auth),
|
||||
ctx: SidecarContext = Depends(_get_ctx),
|
||||
) -> JSONResponse:
|
||||
"""在线修改配置（热重载，需要 Admin 认证）。"""
|
||||
return await update_config(body, ctx)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 仪表盘静态页面
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _get_dashboard_html() -> str:
|
||||
"""获取仪表盘 HTML（带缓存，严维序评审 #6 / 梁思筑评审 #8）。
|
||||
|
||||
首次加载后缓存 5 分钟，避免每次请求读磁盘。
|
||||
"""
|
||||
global _dashboard_html_cache
|
||||
now = time.monotonic()
|
||||
if _dashboard_html_cache is not None:
|
||||
cached_content, cached_at = _dashboard_html_cache
|
||||
if now - cached_at < _DASHBOARD_CACHE_TTL:
|
||||
return cached_content
|
||||
|
||||
dashboard_path = STATIC_DIR / "dashboard.html"
|
||||
if dashboard_path.is_file():
|
||||
content = dashboard_path.read_text(encoding="utf-8")
|
||||
_dashboard_html_cache = (content, now)
|
||||
return content
|
||||
return "<h1>dashboard.html not found</h1>"
|
||||
|
||||
|
||||
@webui_router.get("/dashboard", include_in_schema=False)
|
||||
async def dashboard_page() -> HTMLResponse:
|
||||
"""仪表盘 HTML 页面（含缓存策略）。"""
|
||||
return HTMLResponse(content=_get_dashboard_html())
|
||||
@@ -0,0 +1,474 @@
|
||||
"""
|
||||
heartbeat_helper.py — 高频 Agent 心跳辅助脚本
|
||||
|
||||
提供心跳脚本中所有通用功能，底层通过 multica_proxy 调用 multica CLI，
|
||||
自动享受缓存和限流保护。
|
||||
|
||||
用法：
|
||||
from heartbeat_helper import check_my_tasks, check_timeouts, check_dependencies
|
||||
|
||||
作者：陆怀瑾（COO）
|
||||
日期：2026-06-23
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import time
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
if _SCRIPT_DIR not in sys.path:
|
||||
sys.path.insert(0, _SCRIPT_DIR)
|
||||
|
||||
from multica_proxy import (
|
||||
run_multica,
|
||||
multica_issue_list_my_todo,
|
||||
multica_issue_list_in_progress,
|
||||
multica_issue_get,
|
||||
openclaw_workboard_list,
|
||||
openclaw_workboard_read,
|
||||
get_cache_stats,
|
||||
clear_cache,
|
||||
start_coordinated_poller,
|
||||
subscribe_to_poller,
|
||||
get_poller_status,
|
||||
health_check,
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Agent 配置
|
||||
# ============================================================================
|
||||
|
||||
AGENT_CONFIGS = {
|
||||
"coo": {
|
||||
"name": "陆怀瑾",
|
||||
"multica_uuid": "1c38b437-b54d-4784-bda3-29ce4c8a6722",
|
||||
"openclaw_agent_id": "coo",
|
||||
"is_coo": True,
|
||||
},
|
||||
"secretary": {
|
||||
"name": "刘诗妮",
|
||||
"multica_uuid": "b024fcdc-30ff-420d-b289-498041466e1b",
|
||||
"openclaw_agent_id": "secretary",
|
||||
"is_coo": False,
|
||||
},
|
||||
"projectmanager": {
|
||||
"name": "胡蓉",
|
||||
"multica_uuid": "d877b8c3-b230-4073-b3f7-80e148cfdb71",
|
||||
"openclaw_agent_id": "projectmanager",
|
||||
"is_coo": False,
|
||||
},
|
||||
"costcodev": {
|
||||
"name": "徐聪",
|
||||
"multica_uuid": "46bdd4a6-5c64-475a-92ef-36a763602fa1",
|
||||
"openclaw_agent_id": "costcodev",
|
||||
"is_coo": False,
|
||||
},
|
||||
"opengineer": {
|
||||
"name": "严维序",
|
||||
"multica_uuid": "d3804433-9e2e-4199-a92b-a153049b3bc9",
|
||||
"openclaw_agent_id": "opengineer",
|
||||
"is_coo": False,
|
||||
},
|
||||
"productmanager": {
|
||||
"name": "沈路明",
|
||||
"multica_uuid": "a101fa88-d821-4839-9754-e04580d5fd68",
|
||||
"openclaw_agent_id": "productmanager",
|
||||
"is_coo": False,
|
||||
},
|
||||
"architect": {
|
||||
"name": "梁思筑",
|
||||
"multica_uuid": "40abd41a-62d0-416d-bc44-92c1f758d87a",
|
||||
"openclaw_agent_id": "architect",
|
||||
"is_coo": False,
|
||||
},
|
||||
"designer": {
|
||||
"name": "苏锦绘",
|
||||
"multica_uuid": "13bd8968-cc2a-4934-90c7-957a2d3c09c2",
|
||||
"openclaw_agent_id": "designer",
|
||||
"is_coo": False,
|
||||
},
|
||||
"contentspecialist": {
|
||||
"name": "文墨言",
|
||||
"multica_uuid": "8321b0bf-7d89-4ece-927a-0780f42ad396",
|
||||
"openclaw_agent_id": "contentspecialist",
|
||||
"is_coo": False,
|
||||
},
|
||||
"cvexpert": {
|
||||
"name": "程伯予",
|
||||
"multica_uuid": "4a8696fd-6531-40da-8956-ef84d7ea3c43",
|
||||
"openclaw_agent_id": "cvexpert",
|
||||
"is_coo": False,
|
||||
},
|
||||
"prompt-engineer": {
|
||||
"name": "许言",
|
||||
"multica_uuid": "ece81d8e-8a24-4dd8-a7af-8adfc54b9d01",
|
||||
"openclaw_agent_id": "prompt-engineer",
|
||||
"is_coo": False,
|
||||
},
|
||||
"mediaspecialist": {
|
||||
"name": "钟帧韵",
|
||||
"multica_uuid": "e2b587d4-1d16-447c-8ad9-e2a01358ff0a",
|
||||
"openclaw_agent_id": "mediaspecialist",
|
||||
"is_coo": False,
|
||||
},
|
||||
"taobaospecialist": {
|
||||
"name": "陆云帆",
|
||||
"multica_uuid": "e0f62d8f-9568-4f41-8ad4-b73d79a163a7",
|
||||
"openclaw_agent_id": "taobaospecialist",
|
||||
"is_coo": False,
|
||||
},
|
||||
"marketanalysis": {
|
||||
"name": "顾析策",
|
||||
"multica_uuid": "5ed91729-658f-4654-98f0-3e0313022002",
|
||||
"openclaw_agent_id": "marketanalysis",
|
||||
"is_coo": False,
|
||||
},
|
||||
"lawyer": {
|
||||
"name": "苏慎",
|
||||
"multica_uuid": "6fb0fbd2-16a6-4566-ba7a-d2c136baec25",
|
||||
"openclaw_agent_id": "lawyer",
|
||||
"is_coo": False,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def get_agent_config(agent_id: str) -> Dict[str, Any]:
|
||||
"""获取 Agent 配置"""
|
||||
config = AGENT_CONFIGS.get(agent_id)
|
||||
if config is None:
|
||||
raise ValueError(f"Unknown agent: {agent_id}. Known: {list(AGENT_CONFIGS.keys())}")
|
||||
return config
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 三源任务检查
|
||||
# ============================================================================
|
||||
|
||||
def check_workboard_tasks(agent_id: str) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
检查 WorkBoard 中分配给当前 Agent 的待办卡片
|
||||
替代内联 bash 脚本
|
||||
"""
|
||||
result = openclaw_workboard_list()
|
||||
if not result["success"]:
|
||||
print(f"[heartbeat] WorkBoard 查询失败: {result['error']}")
|
||||
return []
|
||||
|
||||
data = result["data"]
|
||||
my_cards = [
|
||||
c for c in data.get("cards", [])
|
||||
if c.get("agentId") == agent_id and c.get("status") == "todo"
|
||||
]
|
||||
return my_cards
|
||||
|
||||
|
||||
def check_multica_tasks(agent_id: str) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
检查 Multica 中分配给当前 Agent 的待办 Issue
|
||||
替代内联 bash 脚本
|
||||
"""
|
||||
config = get_agent_config(agent_id)
|
||||
result = multica_issue_list_my_todo(config["multica_uuid"])
|
||||
if not result["success"]:
|
||||
print(f"[heartbeat] Multica 查询失败: {result['error']}")
|
||||
return []
|
||||
|
||||
data = result["data"]
|
||||
if isinstance(data, list):
|
||||
return data
|
||||
return []
|
||||
|
||||
|
||||
def check_todo_docs(workspace_dir: str) -> List[str]:
|
||||
"""
|
||||
检查工作区待办文档中的未完成项
|
||||
"""
|
||||
items = []
|
||||
for filename in ["TODO.md", "AGENTS.md"]:
|
||||
filepath = os.path.join(workspace_dir, filename)
|
||||
if os.path.exists(filepath):
|
||||
try:
|
||||
with open(filepath) as f:
|
||||
for i, line in enumerate(f, 1):
|
||||
if "[ ]" in line:
|
||||
items.append(f"{filename}:{i}: {line.strip()}")
|
||||
except Exception:
|
||||
pass
|
||||
return items
|
||||
|
||||
|
||||
def check_my_tasks(agent_id: str, workspace_dir: str) -> Dict[str, Any]:
|
||||
"""
|
||||
三源合并检查：WorkBoard + Multica + 待办文档
|
||||
"""
|
||||
wb_tasks = check_workboard_tasks(agent_id)
|
||||
mul_tasks = check_multica_tasks(agent_id)
|
||||
doc_tasks = check_todo_docs(workspace_dir)
|
||||
|
||||
return {
|
||||
"workboard": wb_tasks,
|
||||
"multica": mul_tasks,
|
||||
"documents": doc_tasks,
|
||||
"total": len(wb_tasks) + len(mul_tasks) + len(doc_tasks),
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 超时检测
|
||||
# ============================================================================
|
||||
|
||||
TIMEOUT_SECONDS = 1200 # 20 分钟
|
||||
|
||||
|
||||
def check_workboard_timeouts() -> List[Dict[str, Any]]:
|
||||
"""
|
||||
检查 WorkBoard 中超过 20 分钟无进展的进行中任务
|
||||
"""
|
||||
result = openclaw_workboard_list()
|
||||
if not result["success"]:
|
||||
print(f"[heartbeat] WorkBoard 超时检测失败: {result['error']}")
|
||||
return []
|
||||
|
||||
data = result["data"]
|
||||
now = time.time()
|
||||
timeouts = []
|
||||
|
||||
for c in data.get("cards", []):
|
||||
if c.get("status") != "in_progress":
|
||||
continue
|
||||
updated = c.get("updated_at", "")
|
||||
if updated:
|
||||
try:
|
||||
age = now - time.mktime(time.strptime(updated[:19], "%Y-%m-%dT%H:%M:%S"))
|
||||
if age > TIMEOUT_SECONDS:
|
||||
timeouts.append(c)
|
||||
except (ValueError, OverflowError):
|
||||
pass
|
||||
|
||||
return timeouts
|
||||
|
||||
|
||||
def check_multica_timeouts() -> List[Dict[str, Any]]:
|
||||
"""
|
||||
检查 Multica 中超过 20 分钟无进展的进行中 Issue
|
||||
"""
|
||||
result = multica_issue_list_in_progress()
|
||||
if not result["success"]:
|
||||
print(f"[heartbeat] Multica 超时检测失败: {result['error']}")
|
||||
return []
|
||||
|
||||
data = result["data"]
|
||||
now = time.time()
|
||||
timeouts = []
|
||||
|
||||
if isinstance(data, list):
|
||||
for issue in data:
|
||||
updated = issue.get("updated_at", "")
|
||||
if updated:
|
||||
try:
|
||||
age = now - time.mktime(time.strptime(updated[:19], "%Y-%m-%dT%H:%M:%S"))
|
||||
if age > TIMEOUT_SECONDS:
|
||||
timeouts.append(issue)
|
||||
except (ValueError, OverflowError):
|
||||
pass
|
||||
|
||||
return timeouts
|
||||
|
||||
|
||||
def check_timeouts() -> Dict[str, Any]:
|
||||
"""
|
||||
跨平台超时检测
|
||||
"""
|
||||
wb_timeouts = check_workboard_timeouts()
|
||||
mul_timeouts = check_multica_timeouts()
|
||||
|
||||
return {
|
||||
"workboard_timeouts": wb_timeouts,
|
||||
"multica_timeouts": mul_timeouts,
|
||||
"total_timeouts": len(wb_timeouts) + len(mul_timeouts),
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 依赖检查
|
||||
# ============================================================================
|
||||
|
||||
def check_workboard_dependencies(card_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
检查 WorkBoard 卡片的依赖是否满足
|
||||
"""
|
||||
result = openclaw_workboard_read(card_id)
|
||||
if not result["success"]:
|
||||
return {"satisfied": False, "error": result["error"], "unmet": []}
|
||||
|
||||
card = result["data"]
|
||||
deps = card.get("dependsOn", [])
|
||||
unmet = [dep for dep in deps if dep.get("status") != "done"]
|
||||
|
||||
return {
|
||||
"satisfied": len(unmet) == 0,
|
||||
"total_deps": len(deps),
|
||||
"unmet": unmet,
|
||||
}
|
||||
|
||||
|
||||
def check_multica_dependencies(issue_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
检查 Multica Issue 的父 Issue 依赖是否满足
|
||||
"""
|
||||
result = multica_issue_get(issue_id)
|
||||
if not result["success"]:
|
||||
return {"satisfied": False, "error": result["error"], "unmet": []}
|
||||
|
||||
issue = result["data"]
|
||||
parent_id = issue.get("parent_issue_id")
|
||||
if not parent_id:
|
||||
return {"satisfied": True, "total_deps": 0, "unmet": []}
|
||||
|
||||
parent_result = multica_issue_get(parent_id)
|
||||
if not parent_result["success"]:
|
||||
return {"satisfied": False, "error": f"Failed to check parent {parent_id}", "unmet": [parent_id]}
|
||||
|
||||
parent = parent_result["data"]
|
||||
if parent.get("status") != "done":
|
||||
return {"satisfied": False, "total_deps": 1, "unmet": [{"id": parent_id, "identifier": parent.get("identifier"), "status": parent.get("status")}]}
|
||||
|
||||
return {"satisfied": True, "total_deps": 1, "unmet": []}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 全局积压巡检（COO 专用）
|
||||
# ============================================================================
|
||||
|
||||
def check_global_backlog() -> Dict[str, Any]:
|
||||
"""
|
||||
全平台积压巡检：WorkBoard + Multica 全局待办数
|
||||
"""
|
||||
wb_result = openclaw_workboard_list()
|
||||
mul_result = multica_issue_list_in_progress()
|
||||
|
||||
wb_stats = {"total": 0, "todo": 0, "in_progress": 0, "done": 0}
|
||||
if wb_result["success"]:
|
||||
cards = wb_result["data"].get("cards", [])
|
||||
wb_stats["total"] = len(cards)
|
||||
for c in cards:
|
||||
status = c.get("status", "")
|
||||
if status in wb_stats:
|
||||
wb_stats[status] += 1
|
||||
|
||||
mul_stats = {"total": 0, "in_progress": 0}
|
||||
if mul_result["success"] and isinstance(mul_result["data"], list):
|
||||
mul_stats["total"] = len(mul_result["data"])
|
||||
mul_stats["in_progress"] = mul_stats["total"]
|
||||
|
||||
return {
|
||||
"workboard": wb_stats,
|
||||
"multica": mul_stats,
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 心跳主入口
|
||||
# ============================================================================
|
||||
|
||||
def run_heartbeat(agent_id: str, workspace_dir: str) -> Dict[str, Any]:
|
||||
"""
|
||||
执行完整心跳检查
|
||||
|
||||
参数：
|
||||
agent_id: Agent ID（如 "coo", "secretary"）
|
||||
workspace_dir: 工作区目录路径
|
||||
|
||||
返回：
|
||||
心跳结果字典
|
||||
"""
|
||||
config = get_agent_config(agent_id)
|
||||
is_coo = config["is_coo"]
|
||||
|
||||
result = {
|
||||
"agent": config["name"],
|
||||
"agent_id": agent_id,
|
||||
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"),
|
||||
"tasks": check_my_tasks(agent_id, workspace_dir),
|
||||
"timeouts": check_timeouts(),
|
||||
}
|
||||
|
||||
# COO 额外检查
|
||||
if is_coo:
|
||||
result["global_backlog"] = check_global_backlog()
|
||||
result["cache_stats"] = get_cache_stats()
|
||||
result["poller_status"] = get_poller_status()
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def print_heartbeat_report(result: Dict[str, Any]) -> None:
|
||||
"""打印格式化的心跳报告"""
|
||||
print(f"\n{'='*60}")
|
||||
print(f" 🫀 心跳报告 — {result['agent']} ({result['agent_id']})")
|
||||
print(f" ⏰ {result['timestamp']}")
|
||||
print(f"{'='*60}")
|
||||
|
||||
tasks = result["tasks"]
|
||||
print(f"\n📋 任务检查:")
|
||||
print(f" WorkBoard 待办: {len(tasks['workboard'])}")
|
||||
for t in tasks["workboard"]:
|
||||
print(f" ⚠️ WB TODO: {t['id'][:8]} → {t.get('agentId','?')} - {t.get('title','?')[:50]}")
|
||||
print(f" Multica 待办: {len(tasks['multica'])}")
|
||||
for t in tasks["multica"]:
|
||||
print(f" ⚠️ MUL TODO: {t.get('identifier','?')} - {t.get('title','?')[:50]}")
|
||||
print(f" 文档待办: {len(tasks['documents'])}")
|
||||
for d in tasks["documents"]:
|
||||
print(f" 📝 {d}")
|
||||
|
||||
timeouts = result["timeouts"]
|
||||
print(f"\n⏱️ 超时检测:")
|
||||
print(f" WorkBoard 超时: {len(timeouts['workboard_timeouts'])}")
|
||||
for t in timeouts["workboard_timeouts"]:
|
||||
print(f" ⏰ WB TIMEOUT: {t['id'][:8]} [{t.get('agentId','?')}] {t.get('title','?')[:50]}")
|
||||
print(f" Multica 超时: {len(timeouts['multica_timeouts'])}")
|
||||
for t in timeouts["multica_timeouts"]:
|
||||
print(f" ⏰ MUL TIMEOUT: {t.get('identifier','?')} {t.get('title','?')[:50]}")
|
||||
|
||||
if "global_backlog" in result:
|
||||
gb = result["global_backlog"]
|
||||
print(f"\n📊 全局积压:")
|
||||
print(f" WorkBoard: {gb['workboard']}")
|
||||
print(f" Multica: {gb['multica']}")
|
||||
|
||||
if "cache_stats" in result:
|
||||
print(f"\n💾 缓存: {result['cache_stats']}")
|
||||
|
||||
print(f"\n{'='*60}\n")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# CLI 入口
|
||||
# ============================================================================
|
||||
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(description="Agent 心跳辅助脚本")
|
||||
parser.add_argument("agent_id", help="Agent ID (coo/secretary/projectmanager/costcodev/opengineer)")
|
||||
parser.add_argument("--workspace", "-w", default=os.getcwd(), help="工作区目录")
|
||||
parser.add_argument("--json", action="store_true", help="JSON 输出")
|
||||
parser.add_argument("--health", action="store_true", help="健康检查")
|
||||
parser.add_argument("--clear-cache", action="store_true", help="清理缓存")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.health:
|
||||
print(json.dumps(health_check(), indent=2, ensure_ascii=False))
|
||||
elif args.clear_cache:
|
||||
count = clear_cache()
|
||||
print(f"已清理 {count} 条缓存")
|
||||
else:
|
||||
result = run_heartbeat(args.agent_id, args.workspace)
|
||||
if args.json:
|
||||
print(json.dumps(result, indent=2, ensure_ascii=False, default=str))
|
||||
else:
|
||||
print_heartbeat_report(result)
|
||||
@@ -0,0 +1,309 @@
|
||||
"""
|
||||
multica_proxy.py — multica CLI 调用代理
|
||||
|
||||
封装 multica CLI 调用，自动带缓存和限流保护。
|
||||
各 Agent 心跳脚本中用 multica_proxy 替代直接 subprocess.run(["multica",...])
|
||||
|
||||
依赖：rate_limiter.py（CacheManager, RequestScheduler, CoordinatedPoller）
|
||||
|
||||
作者：陆怀瑾（COO）
|
||||
日期：2026-06-23
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import subprocess
|
||||
import hashlib
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
# 确保能找到 rate_limiter
|
||||
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
if _SCRIPT_DIR not in sys.path:
|
||||
sys.path.insert(0, _SCRIPT_DIR)
|
||||
|
||||
from rate_limiter import CacheManager, RequestScheduler, CoordinatedPoller, Priority
|
||||
|
||||
# ============================================================================
|
||||
# 全局单例
|
||||
# ============================================================================
|
||||
|
||||
_cache = CacheManager()
|
||||
_scheduler: Optional[RequestScheduler] = None
|
||||
_poller: Optional[CoordinatedPoller] = None
|
||||
|
||||
|
||||
def _get_scheduler() -> RequestScheduler:
|
||||
"""获取或创建调度器单例"""
|
||||
global _scheduler
|
||||
if _scheduler is None:
|
||||
_scheduler = RequestScheduler(rate=40/60, capacity=40, enable_cache=True)
|
||||
_scheduler.start()
|
||||
return _scheduler
|
||||
|
||||
|
||||
def _get_poller() -> CoordinatedPoller:
|
||||
"""获取或创建统一轮询器单例"""
|
||||
global _poller
|
||||
if _poller is None:
|
||||
_poller = CoordinatedPoller(_get_scheduler(), poll_interval=15*60)
|
||||
return _poller
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 缓存查询辅助
|
||||
# ============================================================================
|
||||
|
||||
def _make_cache_key(cmd: list) -> str:
|
||||
"""为 CLI 命令生成缓存键"""
|
||||
return hashlib.md5(json.dumps(cmd, sort_keys=True).encode()).hexdigest()
|
||||
|
||||
|
||||
def _cache_category(cmd: list) -> str:
|
||||
"""根据命令推断缓存类别"""
|
||||
cmd_str = " ".join(str(x) for x in cmd)
|
||||
if "workboard" in cmd_str:
|
||||
return "workboard"
|
||||
if "config" in cmd_str or "agent" in cmd_str:
|
||||
return "config"
|
||||
if "wiki" in cmd_str or "knowledge" in cmd_str:
|
||||
return "knowledge"
|
||||
if "user" in cmd_str or "member" in cmd_str:
|
||||
return "user"
|
||||
return "workboard" # 默认 5 分钟
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 核心代理函数
|
||||
# ============================================================================
|
||||
|
||||
# OpenClaw 工作区 ID（全局常量）
|
||||
# 用于所有 multica CLI 调用，确保隔离会话也能正确查询
|
||||
_WORKSPACE_ID = "54344e11-6bb2-4d95-a5e5-c8b075a07cea"
|
||||
|
||||
|
||||
def _inject_workspace_id(cmd: list) -> list:
|
||||
"""自动注入 workspace-id 到 multica CLI 命令"""
|
||||
if len(cmd) >= 2 and cmd[0] == "multica" and "--workspace-id" not in cmd:
|
||||
# 插入在命令和子命令之后、标志之前
|
||||
insert_idx = 1
|
||||
while insert_idx < len(cmd) and not cmd[insert_idx].startswith("--"):
|
||||
insert_idx += 1
|
||||
new_cmd = cmd[:insert_idx] + ["--workspace-id", _WORKSPACE_ID] + cmd[insert_idx:]
|
||||
return new_cmd
|
||||
return cmd
|
||||
|
||||
|
||||
def run_multica(cmd: list, use_cache: bool = True, timeout: int = 30) -> Dict[str, Any]:
|
||||
"""
|
||||
执行 multica CLI 命令（带缓存和限流）
|
||||
|
||||
参数：
|
||||
cmd: 命令列表，如 ["multica", "issue", "list", "--output", "json"]
|
||||
use_cache: 是否使用缓存
|
||||
timeout: 超时时间（秒）
|
||||
|
||||
返回：
|
||||
{"success": bool, "data": Any, "from_cache": bool, "error": str|None}
|
||||
"""
|
||||
# 自动注入 workspace-id，确保隔离会话正确查询
|
||||
cmd = _inject_workspace_id(cmd)
|
||||
category = _cache_category(cmd)
|
||||
|
||||
# 1. 尝试从缓存获取
|
||||
if use_cache:
|
||||
cached = _cache.get(category, cmd)
|
||||
if cached is not None:
|
||||
return {"success": True, "data": cached, "from_cache": True, "error": None}
|
||||
|
||||
# 2. 执行 CLI 命令
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
error_msg = result.stderr.strip() or f"Exit code {result.returncode}"
|
||||
return {"success": False, "data": None, "from_cache": False, "error": error_msg}
|
||||
|
||||
# 尝试解析 JSON
|
||||
try:
|
||||
data = json.loads(result.stdout)
|
||||
except json.JSONDecodeError:
|
||||
data = result.stdout.strip()
|
||||
|
||||
# 3. 写入缓存
|
||||
if use_cache:
|
||||
_cache.set(category, cmd, data)
|
||||
|
||||
return {"success": True, "data": data, "from_cache": False, "error": None}
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
return {"success": False, "data": None, "from_cache": False, "error": f"Command timed out after {timeout}s"}
|
||||
except Exception as e:
|
||||
return {"success": False, "data": None, "from_cache": False, "error": str(e)}
|
||||
|
||||
|
||||
def run_openclaw_workboard(cmd: list, use_cache: bool = True, timeout: int = 30) -> Dict[str, Any]:
|
||||
"""
|
||||
执行 openclaw workboard CLI 命令（带缓存）
|
||||
|
||||
参数同 run_multica
|
||||
"""
|
||||
return run_multica(cmd, use_cache=use_cache, timeout=timeout)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 便捷函数：心跳脚本中直接替换
|
||||
# ============================================================================
|
||||
|
||||
def multica_issue_list_my_todo(assignee_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
获取分配给我的待办 Issue 列表
|
||||
替代: multica issue list --assignee-id <id> --status todo --output json
|
||||
"""
|
||||
return run_multica([
|
||||
"multica", "issue", "list",
|
||||
"--assignee-id", assignee_id,
|
||||
"--status", "todo",
|
||||
"--output", "json"
|
||||
])
|
||||
|
||||
|
||||
def multica_issue_list_in_progress() -> Dict[str, Any]:
|
||||
"""
|
||||
获取所有进行中的 Issue 列表（超时检测用）
|
||||
替代: multica issue list --status in_progress --output json
|
||||
"""
|
||||
return run_multica([
|
||||
"multica", "issue", "list",
|
||||
"--status", "in_progress",
|
||||
"--output", "json"
|
||||
])
|
||||
|
||||
|
||||
def multica_issue_get(issue_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
获取单个 Issue 详情
|
||||
替代: multica issue get <id> --output json
|
||||
"""
|
||||
return run_multica([
|
||||
"multica", "issue", "get",
|
||||
issue_id,
|
||||
"--output", "json"
|
||||
])
|
||||
|
||||
|
||||
def openclaw_workboard_list() -> Dict[str, Any]:
|
||||
"""
|
||||
获取 WorkBoard 卡片列表
|
||||
替代: openclaw workboard list --json
|
||||
"""
|
||||
return run_multica([
|
||||
"openclaw", "workboard", "list", "--json"
|
||||
])
|
||||
|
||||
|
||||
def openclaw_workboard_read(card_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
获取单个 WorkBoard 卡片
|
||||
替代: openclaw workboard read <id> --json
|
||||
"""
|
||||
return run_multica([
|
||||
"openclaw", "workboard", "read", card_id, "--json"
|
||||
])
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 缓存管理
|
||||
# ============================================================================
|
||||
|
||||
def get_cache_stats() -> Dict[str, Any]:
|
||||
"""获取缓存统计"""
|
||||
return _cache.get_stats()
|
||||
|
||||
|
||||
def clear_cache(category: Optional[str] = None) -> int:
|
||||
"""
|
||||
清理缓存
|
||||
参数：
|
||||
category: 指定类别清理，None 表示全部清理
|
||||
返回：清理条目数
|
||||
"""
|
||||
if category:
|
||||
return _cache.clear_expired()
|
||||
else:
|
||||
count = len(_cache._cache)
|
||||
_cache.clear()
|
||||
return count
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 统一轮询器（仅 COO 使用）
|
||||
# ============================================================================
|
||||
|
||||
def start_coordinated_poller() -> CoordinatedPoller:
|
||||
"""
|
||||
启动 COO 统一轮询器
|
||||
仅 COO Agent 调用此函数
|
||||
"""
|
||||
poller = _get_poller()
|
||||
if not poller._running:
|
||||
poller.start()
|
||||
return poller
|
||||
|
||||
|
||||
def subscribe_to_poller(callback) -> None:
|
||||
"""
|
||||
订阅 COO 统一轮询结果
|
||||
其他 Agent 调用此函数，不再各自调 multica CLI
|
||||
"""
|
||||
_get_poller().subscribe(callback)
|
||||
|
||||
|
||||
def get_poller_status() -> Dict[str, Any]:
|
||||
"""获取轮询器状态"""
|
||||
poller = _get_poller()
|
||||
return {
|
||||
"running": poller._running,
|
||||
"poll_interval": poller.poll_interval,
|
||||
"subscriber_count": len(poller._subscribers)
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 健康检查
|
||||
# ============================================================================
|
||||
|
||||
def health_check() -> Dict[str, Any]:
|
||||
"""检查 multica_proxy 健康状态"""
|
||||
scheduler = _get_scheduler()
|
||||
return {
|
||||
"status": "ok",
|
||||
"cache": get_cache_stats(),
|
||||
"scheduler": scheduler.get_status(),
|
||||
"poller": get_poller_status()
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 测试
|
||||
# ============================================================================
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("=== multica_proxy 健康检查 ===")
|
||||
print(json.dumps(health_check(), indent=2, ensure_ascii=False))
|
||||
|
||||
print("\n=== 测试缓存 ===")
|
||||
# 第一次调用（无缓存）
|
||||
result1 = run_multica(["echo", "test1"], use_cache=True)
|
||||
print(f"第1次: from_cache={result1['from_cache']}")
|
||||
|
||||
# 第二次调用（应命中缓存）
|
||||
result2 = run_multica(["echo", "test1"], use_cache=True)
|
||||
print(f"第2次: from_cache={result2['from_cache']}")
|
||||
|
||||
print("\n测试完成")
|
||||
@@ -0,0 +1,772 @@
|
||||
"""
|
||||
BIZ-26: API 请求优先级队列 + 令牌桶限流器
|
||||
|
||||
实现方案参考：plans/BIZ-13_运行稳定性保障方案.md
|
||||
|
||||
功能清单：
|
||||
1. 四级优先级请求队列（紧急 > 高 > 正常 > 低）
|
||||
2. 令牌桶限流器（40 RPM 上限）
|
||||
3. 超限自动降级和等待策略
|
||||
4. 请求合并（COO 统一轮询）
|
||||
5. 查询结果缓存（WorkBoard 5 分钟、配置 1 小时、知识库 1 天）
|
||||
|
||||
作者：徐聪（costcodev）
|
||||
日期：2026-06-23
|
||||
"""
|
||||
|
||||
import time
|
||||
import threading
|
||||
import queue
|
||||
import hashlib
|
||||
import json
|
||||
from typing import Any, Callable, Dict, List, Optional, Tuple
|
||||
from dataclasses import dataclass, field
|
||||
from enum import IntEnum
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 网关识别：只对 NVIDIA 网关限流
|
||||
# ============================================================================
|
||||
|
||||
NVIDIA_GATEWAY_ALIASES = {
|
||||
"nvidia",
|
||||
"nvidia-gateway",
|
||||
"nvidia_gateway",
|
||||
"nvidiavx18088980513",
|
||||
}
|
||||
|
||||
UNLIMITED_GATEWAY_ALIASES = {
|
||||
"volcengine",
|
||||
"volcengine-plan",
|
||||
"siliconflow",
|
||||
"deepseek",
|
||||
"deepseek-api",
|
||||
}
|
||||
|
||||
|
||||
def normalize_gateway_name(value: Optional[str]) -> Optional[str]:
|
||||
"""
|
||||
归一化网关/模型名称。
|
||||
|
||||
输入可以是：
|
||||
- provider: nvidia / volcengine-plan / siliconflow / deepseek
|
||||
- model: nvidiavx18088980513/deepseek-ai/deepseek-v4-pro
|
||||
- model: volcengine-plan/ark-code-latest
|
||||
|
||||
返回 provider 前缀的小写形式。未知则返回 None。
|
||||
"""
|
||||
if not value:
|
||||
return None
|
||||
text = str(value).strip().lower()
|
||||
if not text:
|
||||
return None
|
||||
return text.split("/", 1)[0]
|
||||
|
||||
|
||||
def is_nvidia_gateway(value: Optional[str]) -> bool:
|
||||
"""判断请求是否走 NVIDIA 网关。未知网关默认不限流。"""
|
||||
provider = normalize_gateway_name(value)
|
||||
if provider is None:
|
||||
return False
|
||||
if provider in NVIDIA_GATEWAY_ALIASES:
|
||||
return True
|
||||
if provider in UNLIMITED_GATEWAY_ALIASES:
|
||||
return False
|
||||
return provider.startswith("nvidia")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 优先级枚举
|
||||
# ============================================================================
|
||||
|
||||
class Priority(IntEnum):
|
||||
"""请求优先级：数值越小优先级越高"""
|
||||
URGENT = 1 # 紧急：Vincent 直接任务
|
||||
HIGH = 2 # 高：阻塞性任务
|
||||
NORMAL = 3 # 正常：常规任务
|
||||
LOW = 4 # 低：后台优化任务
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 请求数据类
|
||||
# ============================================================================
|
||||
|
||||
@dataclass(order=True)
|
||||
class Request:
|
||||
"""优先级队列中的请求项"""
|
||||
priority: int
|
||||
timestamp: float = field(compare=False)
|
||||
request_id: str = field(compare=False)
|
||||
payload: Any = field(compare=False)
|
||||
callback: Optional[Callable] = field(compare=False, default=None)
|
||||
fallback_model: Optional[str] = field(compare=False, default=None)
|
||||
gateway: Optional[str] = field(compare=False, default=None)
|
||||
model: Optional[str] = field(compare=False, default=None)
|
||||
|
||||
def __post_init__(self):
|
||||
if self.timestamp is None:
|
||||
self.timestamp = time.time()
|
||||
if self.request_id is None:
|
||||
self.request_id = self._generate_id()
|
||||
|
||||
@staticmethod
|
||||
def _generate_id() -> str:
|
||||
"""生成请求 ID"""
|
||||
return hashlib.md5(f"{time.time()}-{threading.current_thread().ident}".encode()).hexdigest()[:12]
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 令牌桶限流器
|
||||
# ============================================================================
|
||||
|
||||
class TokenBucket:
|
||||
"""
|
||||
NVIDIA 网关专用令牌桶限流器
|
||||
|
||||
注意：令牌桶本身只负责节流算法；是否启用由 RequestScheduler._should_rate_limit()
|
||||
按 gateway/model 判断。volcengine-plan、siliconflow、DeepSeek 等非 NVIDIA 网关不会进入此桶。
|
||||
|
||||
参数：
|
||||
rate: 令牌生成速率（个/秒），默认 40 RPM = 0.67 个/秒
|
||||
capacity: 桶容量（最大令牌数），默认 40
|
||||
"""
|
||||
|
||||
def __init__(self, rate: float = 40/60, capacity: int = 40):
|
||||
self.rate = rate # 令牌/秒
|
||||
self.capacity = capacity
|
||||
self.tokens = capacity
|
||||
self.last_update = time.time()
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def _refill(self) -> None:
|
||||
"""补充令牌（内部调用，需要持有锁）"""
|
||||
now = time.time()
|
||||
elapsed = now - self.last_update
|
||||
new_tokens = elapsed * self.rate
|
||||
self.tokens = min(self.capacity, self.tokens + new_tokens)
|
||||
self.last_update = now
|
||||
|
||||
def consume(self, tokens: int = 1) -> bool:
|
||||
"""
|
||||
尝试消费令牌
|
||||
|
||||
返回：
|
||||
True: 成功消费
|
||||
False: 令牌不足
|
||||
"""
|
||||
with self._lock:
|
||||
self._refill()
|
||||
if self.tokens >= tokens:
|
||||
self.tokens -= tokens
|
||||
return True
|
||||
return False
|
||||
|
||||
def wait_for_token(self, timeout: Optional[float] = None) -> bool:
|
||||
"""
|
||||
等待直到有可用令牌
|
||||
|
||||
参数：
|
||||
timeout: 最大等待时间（秒），None 表示无限等待
|
||||
|
||||
返回：
|
||||
True: 成功获取令牌
|
||||
False: 超时
|
||||
"""
|
||||
start_time = time.time()
|
||||
while True:
|
||||
if self.consume():
|
||||
return True
|
||||
|
||||
if timeout is not None:
|
||||
elapsed = time.time() - start_time
|
||||
if elapsed >= timeout:
|
||||
return False
|
||||
|
||||
# 计算等待时间（直到下一个令牌生成）
|
||||
with self._lock:
|
||||
self._refill()
|
||||
if self.tokens < 1:
|
||||
wait_time = (1 - self.tokens) / self.rate
|
||||
else:
|
||||
wait_time = 0.01
|
||||
|
||||
# 等待后重试
|
||||
time_to_wait = min(wait_time, 0.1) # 最多等待 100ms
|
||||
if timeout is not None:
|
||||
remaining = timeout - (time.time() - start_time)
|
||||
if remaining <= 0:
|
||||
return False
|
||||
time_to_wait = min(time_to_wait, remaining)
|
||||
|
||||
time.sleep(time_to_wait)
|
||||
|
||||
def get_status(self) -> Dict[str, Any]:
|
||||
"""获取限流器状态"""
|
||||
with self._lock:
|
||||
self._refill()
|
||||
return {
|
||||
"tokens": round(self.tokens, 2),
|
||||
"capacity": self.capacity,
|
||||
"rate_per_second": round(self.rate, 3),
|
||||
"rate_per_minute": round(self.rate * 60, 1),
|
||||
"utilization": round(1 - self.tokens / self.capacity, 2)
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 缓存管理器
|
||||
# ============================================================================
|
||||
|
||||
@dataclass
|
||||
class CacheEntry:
|
||||
"""缓存条目"""
|
||||
value: Any
|
||||
expires_at: float
|
||||
created_at: float = field(default_factory=time.time)
|
||||
access_count: int = field(default=0)
|
||||
|
||||
|
||||
class CacheManager:
|
||||
"""
|
||||
查询结果缓存管理器
|
||||
|
||||
缓存策略：
|
||||
- WorkBoard 状态：5 分钟
|
||||
- Agent 配置：1 小时
|
||||
- 知识库内容：1 天
|
||||
- 用户信息：1 天
|
||||
"""
|
||||
|
||||
# 默认 TTL 配置（秒）
|
||||
DEFAULT_TTL = {
|
||||
"workboard": 5 * 60, # 5 分钟
|
||||
"config": 1 * 60 * 60, # 1 小时
|
||||
"knowledge": 24 * 60 * 60, # 1 天
|
||||
"user": 24 * 60 * 60, # 1 天
|
||||
}
|
||||
|
||||
def __init__(self):
|
||||
self._cache: Dict[str, CacheEntry] = {}
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def _generate_key(self, category: str, query: Any) -> str:
|
||||
"""生成缓存键"""
|
||||
query_str = json.dumps(query, sort_keys=True) if not isinstance(query, str) else query
|
||||
return hashlib.md5(f"{category}:{query_str}".encode()).hexdigest()
|
||||
|
||||
def get(self, category: str, query: Any) -> Optional[Any]:
|
||||
"""
|
||||
获取缓存
|
||||
|
||||
参数：
|
||||
category: 缓存类别（workboard/config/knowledge/user）
|
||||
query: 查询条件（用于生成缓存键）
|
||||
|
||||
返回：
|
||||
缓存值，如果不存在或已过期则返回 None
|
||||
"""
|
||||
key = self._generate_key(category, query)
|
||||
|
||||
with self._lock:
|
||||
entry = self._cache.get(key)
|
||||
if entry is None:
|
||||
return None
|
||||
|
||||
# 检查是否过期
|
||||
if time.time() > entry.expires_at:
|
||||
del self._cache[key]
|
||||
return None
|
||||
|
||||
# 更新访问计数
|
||||
entry.access_count += 1
|
||||
return entry.value
|
||||
|
||||
def set(self, category: str, query: Any, value: Any, ttl: Optional[int] = None) -> None:
|
||||
"""
|
||||
设置缓存
|
||||
|
||||
参数：
|
||||
category: 缓存类别
|
||||
query: 查询条件
|
||||
value: 缓存值
|
||||
ttl: 存活时间（秒），None 表示使用默认值
|
||||
"""
|
||||
key = self._generate_key(category, query)
|
||||
|
||||
if ttl is None:
|
||||
ttl = self.DEFAULT_TTL.get(category, 300) # 默认 5 分钟
|
||||
|
||||
with self._lock:
|
||||
self._cache[key] = CacheEntry(
|
||||
value=value,
|
||||
expires_at=time.time() + ttl
|
||||
)
|
||||
|
||||
def delete(self, category: str, query: Any) -> bool:
|
||||
"""删除缓存"""
|
||||
key = self._generate_key(category, query)
|
||||
with self._lock:
|
||||
if key in self._cache:
|
||||
del self._cache[key]
|
||||
return True
|
||||
return False
|
||||
|
||||
def clear_expired(self) -> int:
|
||||
"""清理所有过期缓存，返回清理数量"""
|
||||
now = time.time()
|
||||
with self._lock:
|
||||
expired_keys = [k for k, v in self._cache.items() if now > v.expires_at]
|
||||
for key in expired_keys:
|
||||
del self._cache[key]
|
||||
return len(expired_keys)
|
||||
|
||||
def get_stats(self) -> Dict[str, Any]:
|
||||
"""获取缓存统计"""
|
||||
now = time.time()
|
||||
with self._lock:
|
||||
total = len(self._cache)
|
||||
expired = sum(1 for v in self._cache.values() if now > v.expires_at)
|
||||
|
||||
# 按类别统计
|
||||
by_category: Dict[str, int] = {}
|
||||
for key, entry in self._cache.items():
|
||||
# 从 key 中提取 category（格式：category:hash）
|
||||
category = key.split(":")[0] if ":" in key else "unknown"
|
||||
by_category[category] = by_category.get(category, 0) + 1
|
||||
|
||||
return {
|
||||
"total_entries": total,
|
||||
"expired_entries": expired,
|
||||
"valid_entries": total - expired,
|
||||
"by_category": by_category
|
||||
}
|
||||
|
||||
def clear(self) -> None:
|
||||
"""清空所有缓存"""
|
||||
with self._lock:
|
||||
self._cache.clear()
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 请求调度器
|
||||
# ============================================================================
|
||||
|
||||
class RequestScheduler:
|
||||
"""
|
||||
请求调度器：结合优先级队列和令牌桶限流
|
||||
|
||||
功能：
|
||||
1. 接收不同优先级的请求
|
||||
2. 按优先级和 FIF0 顺序调度
|
||||
3. 通过令牌桶控制发送速率
|
||||
4. 支持降级策略（低优先级切备用模型）
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
rate: float = 40/60,
|
||||
capacity: int = 40,
|
||||
enable_cache: bool = True
|
||||
):
|
||||
self.token_bucket = TokenBucket(rate=rate, capacity=capacity)
|
||||
self.cache = CacheManager() if enable_cache else None
|
||||
|
||||
# 优先级队列（使用 heap 实现）
|
||||
self.request_queue: queue.PriorityQueue[Request] = queue.PriorityQueue()
|
||||
|
||||
# 工作线程
|
||||
self._worker_thread: Optional[threading.Thread] = None
|
||||
self._running = False
|
||||
self._lock = threading.Lock()
|
||||
|
||||
# 统计信息
|
||||
self.stats = {
|
||||
"total_requests": 0,
|
||||
"completed_requests": 0,
|
||||
"failed_requests": 0,
|
||||
"fallback_requests": 0,
|
||||
"cache_hits": 0,
|
||||
"cache_misses": 0,
|
||||
}
|
||||
|
||||
def start(self) -> None:
|
||||
"""启动调度器工作线程"""
|
||||
if self._running:
|
||||
return
|
||||
|
||||
self._running = True
|
||||
self._worker_thread = threading.Thread(target=self._worker_loop, daemon=True)
|
||||
self._worker_thread.start()
|
||||
|
||||
def stop(self) -> None:
|
||||
"""停止调度器"""
|
||||
self._running = False
|
||||
if self._worker_thread:
|
||||
self._worker_thread.join(timeout=5.0)
|
||||
|
||||
def _worker_loop(self) -> None:
|
||||
"""工作线程主循环"""
|
||||
while self._running:
|
||||
try:
|
||||
# 从队列获取请求（带超时）
|
||||
request = self.request_queue.get(timeout=1.0)
|
||||
self._process_request(request)
|
||||
except queue.Empty:
|
||||
continue
|
||||
except Exception as e:
|
||||
# 记录错误但不中断工作线程
|
||||
print(f"[RequestScheduler] Worker error: {e}")
|
||||
|
||||
def _extract_gateway_hint(self, request: Request) -> Optional[str]:
|
||||
"""从 request.gateway / request.model / payload 中提取网关提示。"""
|
||||
if request.gateway:
|
||||
return request.gateway
|
||||
if request.model:
|
||||
return request.model
|
||||
if isinstance(request.payload, dict):
|
||||
for key in ("gateway", "provider", "model", "model_id"):
|
||||
value = request.payload.get(key)
|
||||
if value:
|
||||
return str(value)
|
||||
return None
|
||||
|
||||
def _should_rate_limit(self, request: Request) -> bool:
|
||||
"""
|
||||
只对 NVIDIA 网关请求启用令牌桶。
|
||||
|
||||
设计原则：未知网关默认不限制，避免误伤 volcengine-plan / siliconflow / DeepSeek
|
||||
等其他 API 网关。要被限流，调用方必须显式传 gateway/model，且能识别为 NVIDIA。
|
||||
"""
|
||||
return is_nvidia_gateway(self._extract_gateway_hint(request))
|
||||
|
||||
def _process_request(self, request: Request) -> None:
|
||||
"""
|
||||
处理单个请求
|
||||
|
||||
策略：
|
||||
1. 高优先级（URGENT/HIGH）：等待令牌
|
||||
2. 低优先级（NORMAL/LOW）：尝试获取令牌，失败则降级或丢弃
|
||||
"""
|
||||
self.stats["total_requests"] += 1
|
||||
|
||||
# 只对 NVIDIA 网关请求启用令牌桶；其他网关直接执行
|
||||
if not self._should_rate_limit(request):
|
||||
self._execute_request(request)
|
||||
return
|
||||
|
||||
# NVIDIA 网关请求：尝试获取令牌
|
||||
if request.priority <= Priority.HIGH:
|
||||
# 高优先级：无限等待
|
||||
got_token = self.token_bucket.wait_for_token(timeout=None)
|
||||
else:
|
||||
# 低优先级：最多等待 2 秒
|
||||
got_token = self.token_bucket.wait_for_token(timeout=2.0)
|
||||
|
||||
if got_token:
|
||||
# 成功获取令牌，执行请求
|
||||
self._execute_request(request)
|
||||
else:
|
||||
# 未能获取令牌，执行降级策略
|
||||
self._handle_fallback(request)
|
||||
|
||||
def _execute_request(self, request: Request) -> None:
|
||||
"""执行请求"""
|
||||
try:
|
||||
if request.callback:
|
||||
result = request.callback(request.payload)
|
||||
self.stats["completed_requests"] += 1
|
||||
return result
|
||||
else:
|
||||
self.stats["completed_requests"] += 1
|
||||
except Exception as e:
|
||||
self.stats["failed_requests"] += 1
|
||||
print(f"[RequestScheduler] Request {request.request_id} failed: {e}")
|
||||
raise
|
||||
|
||||
def _handle_fallback(self, request: Request) -> None:
|
||||
"""处理降级（令牌不足）"""
|
||||
self.stats["fallback_requests"] += 1
|
||||
|
||||
if request.priority == Priority.LOW:
|
||||
# 低优先级：直接丢弃或切换到备用模型
|
||||
print(f"[RequestScheduler] Low priority request {request.request_id} dropped due to rate limit")
|
||||
else:
|
||||
# 正常优先级：放回队列稍后重试
|
||||
request.timestamp = time.time()
|
||||
self.request_queue.put(request)
|
||||
|
||||
def submit(
|
||||
self,
|
||||
payload: Any,
|
||||
priority: Priority = Priority.NORMAL,
|
||||
callback: Optional[Callable] = None,
|
||||
fallback_model: Optional[str] = None,
|
||||
request_id: Optional[str] = None,
|
||||
gateway: Optional[str] = None,
|
||||
model: Optional[str] = None
|
||||
) -> str:
|
||||
"""
|
||||
提交请求到调度队列
|
||||
|
||||
参数：
|
||||
payload: 请求数据
|
||||
priority: 优先级
|
||||
callback: 回调函数
|
||||
fallback_model: 备用模型名称
|
||||
request_id: 请求 ID（可选，默认自动生成）
|
||||
|
||||
返回：
|
||||
请求 ID
|
||||
"""
|
||||
req = Request(
|
||||
priority=priority,
|
||||
timestamp=time.time(),
|
||||
request_id=request_id,
|
||||
payload=payload,
|
||||
callback=callback,
|
||||
fallback_model=fallback_model,
|
||||
gateway=gateway,
|
||||
model=model
|
||||
)
|
||||
|
||||
self.request_queue.put(req)
|
||||
return req.request_id
|
||||
|
||||
def submit_sync(
|
||||
self,
|
||||
payload: Any,
|
||||
priority: Priority = Priority.NORMAL,
|
||||
timeout: Optional[float] = None
|
||||
) -> Any:
|
||||
"""
|
||||
同步提交并等待结果
|
||||
|
||||
参数：
|
||||
payload: 请求数据
|
||||
priority: 优先级
|
||||
timeout: 超时时间（秒）
|
||||
|
||||
返回：
|
||||
请求结果
|
||||
"""
|
||||
result_holder = {"result": None, "error": None, "done": False}
|
||||
condition = threading.Condition()
|
||||
|
||||
def callback(data):
|
||||
with condition:
|
||||
try:
|
||||
# 实际执行逻辑（这里只是一个占位符）
|
||||
result_holder["result"] = data
|
||||
except Exception as e:
|
||||
result_holder["error"] = e
|
||||
finally:
|
||||
result_holder["done"] = True
|
||||
condition.notify_all()
|
||||
|
||||
# 提交请求
|
||||
self.submit(payload=payload, priority=priority, callback=lambda _: callback(payload))
|
||||
|
||||
# 等待结果
|
||||
with condition:
|
||||
if not result_holder["done"]:
|
||||
condition.wait(timeout=timeout)
|
||||
|
||||
if result_holder["error"]:
|
||||
raise result_holder["error"]
|
||||
return result_holder["result"]
|
||||
|
||||
def get_queue_size(self) -> int:
|
||||
"""获取当前队列大小"""
|
||||
return self.request_queue.qsize()
|
||||
|
||||
def get_status(self) -> Dict[str, Any]:
|
||||
"""获取调度器状态"""
|
||||
return {
|
||||
"running": self._running,
|
||||
"queue_size": self.get_queue_size(),
|
||||
"token_bucket": self.token_bucket.get_status(),
|
||||
"cache": self.cache.get_stats() if self.cache else None,
|
||||
"stats": self.stats.copy()
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 重试装饰器
|
||||
# ============================================================================
|
||||
|
||||
def retry_with_backoff(
|
||||
max_retries: int = 3,
|
||||
base_delay: float = 1.0,
|
||||
exponential_base: int = 2,
|
||||
jitter: bool = True,
|
||||
exceptions: Tuple = (Exception,)
|
||||
):
|
||||
"""
|
||||
指数退避重试装饰器
|
||||
|
||||
参数：
|
||||
max_retries: 最大重试次数
|
||||
base_delay: 基础延迟（秒）
|
||||
exponential_base: 指数底数
|
||||
jitter: 是否添加随机抖动
|
||||
exceptions: 需要重试的异常类型
|
||||
"""
|
||||
import random
|
||||
|
||||
def decorator(func: Callable) -> Callable:
|
||||
def wrapper(*args, **kwargs):
|
||||
last_exception = None
|
||||
|
||||
for attempt in range(max_retries + 1):
|
||||
try:
|
||||
return func(*args, **kwargs)
|
||||
except exceptions as e:
|
||||
last_exception = e
|
||||
|
||||
if attempt == max_retries:
|
||||
break
|
||||
|
||||
# 计算延迟时间
|
||||
delay = base_delay * (exponential_base ** attempt)
|
||||
if jitter:
|
||||
delay += random.uniform(0, base_delay)
|
||||
|
||||
print(f"[retry_with_backoff] Attempt {attempt + 1} failed: {e}. Retrying in {delay:.2f}s...")
|
||||
time.sleep(delay)
|
||||
|
||||
raise last_exception
|
||||
|
||||
return wrapper
|
||||
return decorator
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# COO 统一轮询器（请求合并）
|
||||
# ============================================================================
|
||||
|
||||
class CoordinatedPoller:
|
||||
"""
|
||||
COO 统一轮询器：替代各 Agent 独立轮询
|
||||
|
||||
功能：
|
||||
1. 定期轮询 WorkBoard
|
||||
2. 广播结果给所有订阅者
|
||||
3. 减少总请求数（40 RPM × N → 40 RPM）
|
||||
"""
|
||||
|
||||
def __init__(self, scheduler: RequestScheduler, poll_interval: int = 15*60):
|
||||
self.scheduler = scheduler
|
||||
self.poll_interval = poll_interval # 轮询间隔（秒）
|
||||
self._subscribers: List[Callable] = []
|
||||
self._running = False
|
||||
self._worker: Optional[threading.Thread] = None
|
||||
|
||||
def subscribe(self, callback: Callable) -> None:
|
||||
"""订阅轮询结果"""
|
||||
self._subscribers.append(callback)
|
||||
|
||||
def unsubscribe(self, callback: Callable) -> None:
|
||||
"""取消订阅"""
|
||||
if callback in self._subscribers:
|
||||
self._subscribers.remove(callback)
|
||||
|
||||
def start(self) -> None:
|
||||
"""启动轮询器"""
|
||||
if self._running:
|
||||
return
|
||||
|
||||
self._running = True
|
||||
self._worker = threading.Thread(target=self._poll_loop, daemon=True)
|
||||
self._worker.start()
|
||||
|
||||
def stop(self) -> None:
|
||||
"""停止轮询器"""
|
||||
self._running = False
|
||||
if self._worker:
|
||||
self._worker.join(timeout=5.0)
|
||||
|
||||
def _poll_loop(self) -> None:
|
||||
"""轮询主循环"""
|
||||
while self._running:
|
||||
try:
|
||||
# 执行轮询（这里只是一个框架，实际逻辑需要接入 multica CLI）
|
||||
result = self._perform_poll()
|
||||
|
||||
# 广播给所有订阅者
|
||||
for subscriber in self._subscribers:
|
||||
try:
|
||||
subscriber(result)
|
||||
except Exception as e:
|
||||
print(f"[CoordinatedPoller] Subscriber callback error: {e}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"[CoordinatedPoller] Poll error: {e}")
|
||||
|
||||
# 等待下一个轮询周期
|
||||
time.sleep(self.poll_interval)
|
||||
|
||||
def _perform_poll(self) -> Dict[str, Any]:
|
||||
"""
|
||||
执行实际轮询
|
||||
|
||||
TODO: 接入 multica CLI:
|
||||
- multica issue list --status in_progress
|
||||
- multica workboard list
|
||||
"""
|
||||
# 这里应该调用 multica CLI
|
||||
# 当前只是返回一个示例结果
|
||||
return {
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
"issues": [],
|
||||
"workboard_cards": []
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# 使用示例
|
||||
# ============================================================================
|
||||
|
||||
if __name__ == "__main__":
|
||||
# 创建调度器（40 RPM）
|
||||
scheduler = RequestScheduler(rate=40/60, capacity=40)
|
||||
scheduler.start()
|
||||
|
||||
# 示例：提交不同优先级的请求
|
||||
def sample_callback(data):
|
||||
print(f"Processing: {data}")
|
||||
time.sleep(0.5) # 模拟处理时间
|
||||
return "OK"
|
||||
|
||||
# 紧急请求
|
||||
scheduler.submit(
|
||||
payload={"task": "urgent_task"},
|
||||
priority=Priority.URGENT,
|
||||
callback=sample_callback
|
||||
)
|
||||
|
||||
# 正常请求
|
||||
scheduler.submit(
|
||||
payload={"task": "normal_task"},
|
||||
priority=Priority.NORMAL,
|
||||
callback=sample_callback
|
||||
)
|
||||
|
||||
# 低优先级请求
|
||||
scheduler.submit(
|
||||
payload={"task": "low_priority_task"},
|
||||
priority=Priority.LOW,
|
||||
callback=sample_callback
|
||||
)
|
||||
|
||||
# 等待处理完成
|
||||
time.sleep(2)
|
||||
|
||||
# 查看状态
|
||||
print("\n=== Scheduler Status ===")
|
||||
print(json.dumps(scheduler.get_status(), indent=2))
|
||||
|
||||
# 停止调度器
|
||||
scheduler.stop()
|
||||
|
||||
print("\n示例运行完成")
|
||||
Reference in New Issue
Block a user