Compare commits
10 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 474f1eddfd | |||
| 4f415fb500 | |||
| 611ebd11a8 | |||
| 4fd89b038d | |||
| 394f9e2780 | |||
| 1747512117 | |||
| 1561c2eaeb | |||
| ae2fd1032f | |||
| 01640e0617 | |||
| 5942be573b |
@@ -0,0 +1,234 @@
|
||||
# HEARTBEAT.md - 梁思筑(architect)的心跳配置
|
||||
|
||||
> 模板版本:v1.1 (BIZ-24) | 分类:开发 Agent
|
||||
> OpenClaw Agent ID: `architect` | Multica Agent UUID: `40abd41a-62d0-416d-bc44-92c1f758d87a`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级:OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'architect' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级:Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 40abd41a-62d0-416d-bc44-92c1f758d87a --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级:待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片(agentId=architect)
|
||||
↓
|
||||
检查 Multica Issues 待办(assignee=40abd41a-62d0-416d-bc44-92c1f758d87a)
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测:
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO + 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO + 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(2h)→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖,可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖,可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:100 轮
|
||||
|
||||
- 接近 80%(80 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知 COO + 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**:通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**:通过 issue comment 记录轮次进度
|
||||
- **待办文档**:在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 架构设计进度
|
||||
6. ✅ 技术方案评审状态
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 梁思筑(architect)专用配置
|
||||
@@ -0,0 +1,234 @@
|
||||
# HEARTBEAT.md - 文墨言(contentspecialist)的心跳配置
|
||||
|
||||
> 模板版本:v1.1 (BIZ-24) | 分类:业务 Agent
|
||||
> OpenClaw Agent ID: `contentspecialist` | Multica Agent UUID: `8321b0bf-7d89-4ece-927a-0780f42ad396`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级:OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'contentspecialist' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级:Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 8321b0bf-7d89-4ece-927a-0780f42ad396 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级:待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片(agentId=contentspecialist)
|
||||
↓
|
||||
检查 Multica Issues 待办(assignee=8321b0bf-7d89-4ece-927a-0780f42ad396)
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测:
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(2h)→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖,可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖,可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:30 轮
|
||||
|
||||
- 接近 80%(24 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**:通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**:通过 issue comment 记录轮次进度
|
||||
- **待办文档**:在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 内容发布计划
|
||||
6. ✅ 素材准备状态
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 文墨言(contentspecialist)专用配置
|
||||
@@ -0,0 +1,235 @@
|
||||
# HEARTBEAT.md - 陆怀瑾(coo)的心跳配置
|
||||
|
||||
> 模板版本:v1.1 (BIZ-24) | 分类:高频 Agent
|
||||
> OpenClaw Agent ID: `coo` | Multica Agent UUID: `1c38b437-b54d-4784-bda3-29ce4c8a6722`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级:OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'coo' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级:Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 1c38b437-b54d-4784-bda3-29ce4c8a6722 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级:待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片(agentId=coo)
|
||||
↓
|
||||
检查 Multica Issues 待办(assignee=1c38b437-b54d-4784-bda3-29ce4c8a6722)
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:10 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测:
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 20 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1200:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1200:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 30 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO(自我监控) |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO(自我监控) |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(1h)→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖,可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖,可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:50 轮
|
||||
|
||||
- 接近 80%(40 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知 COO(自我监控)
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**:通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**:通过 issue comment 记录轮次进度
|
||||
- **待办文档**:在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ **全平台积压巡检**:WorkBoard + Multica 全局待办数
|
||||
6. ✅ 资源负载均衡检查
|
||||
7. ✅ 风险识别与预警
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 陆怀瑾(coo)专用配置
|
||||
@@ -0,0 +1,234 @@
|
||||
# HEARTBEAT.md - 徐聪(costcodev)的心跳配置
|
||||
|
||||
> 模板版本:v1.1 (BIZ-24) | 分类:开发 Agent
|
||||
> OpenClaw Agent ID: `costcodev` | Multica Agent UUID: `46bdd4a6-5c64-475a-92ef-36a763602fa1`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级:OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'costcodev' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级:Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 46bdd4a6-5c64-475a-92ef-36a763602fa1 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级:待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片(agentId=costcodev)
|
||||
↓
|
||||
检查 Multica Issues 待办(assignee=46bdd4a6-5c64-475a-92ef-36a763602fa1)
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测:
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO + 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO + 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(2h)→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖,可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖,可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:100 轮
|
||||
|
||||
- 接近 80%(80 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知 COO + 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**:通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**:通过 issue comment 记录轮次进度
|
||||
- **待办文档**:在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 代码开发进度
|
||||
6. ✅ PR/Code Review 状态
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 徐聪(costcodev)专用配置
|
||||
@@ -0,0 +1,234 @@
|
||||
# HEARTBEAT.md - 程伯予(cvexpert)的心跳配置
|
||||
|
||||
> 模板版本:v1.1 (BIZ-24) | 分类:业务 Agent
|
||||
> OpenClaw Agent ID: `cvexpert` | Multica Agent UUID: `4a8696fd-6531-40da-8956-ef84d7ea3c43`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级:OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'cvexpert' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级:Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 4a8696fd-6531-40da-8956-ef84d7ea3c43 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级:待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片(agentId=cvexpert)
|
||||
↓
|
||||
检查 Multica Issues 待办(assignee=4a8696fd-6531-40da-8956-ef84d7ea3c43)
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测:
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(2h)→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖,可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖,可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:30 轮
|
||||
|
||||
- 接近 80%(24 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**:通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**:通过 issue comment 记录轮次进度
|
||||
- **待办文档**:在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 求职服务队列
|
||||
6. ✅ 客户反馈跟踪
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 程伯予(cvexpert)专用配置
|
||||
@@ -0,0 +1,234 @@
|
||||
# HEARTBEAT.md - 苏锦绘(designer)的心跳配置
|
||||
|
||||
> 模板版本:v1.1 (BIZ-24) | 分类:开发 Agent
|
||||
> OpenClaw Agent ID: `designer` | Multica Agent UUID: `13bd8968-cc2a-4934-90c7-957a2d3c09c2`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级:OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'designer' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级:Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 13bd8968-cc2a-4934-90c7-957a2d3c09c2 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级:待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片(agentId=designer)
|
||||
↓
|
||||
检查 Multica Issues 待办(assignee=13bd8968-cc2a-4934-90c7-957a2d3c09c2)
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测:
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO + 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO + 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(2h)→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖,可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖,可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:100 轮
|
||||
|
||||
- 接近 80%(80 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知 COO + 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**:通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**:通过 issue comment 记录轮次进度
|
||||
- **待办文档**:在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 设计稿进度
|
||||
6. ✅ UI/UX 评审状态
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 苏锦绘(designer)专用配置
|
||||
@@ -0,0 +1,234 @@
|
||||
# HEARTBEAT.md - 苏慎(lawyer)的心跳配置
|
||||
|
||||
> 模板版本:v1.1 (BIZ-24) | 分类:业务 Agent
|
||||
> OpenClaw Agent ID: `lawyer` | Multica Agent UUID: `6fb0fbd2-16a6-4566-ba7a-d2c136baec25`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级:OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'lawyer' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级:Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 6fb0fbd2-16a6-4566-ba7a-d2c136baec25 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级:待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片(agentId=lawyer)
|
||||
↓
|
||||
检查 Multica Issues 待办(assignee=6fb0fbd2-16a6-4566-ba7a-d2c136baec25)
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测:
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(2h)→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖,可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖,可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:30 轮
|
||||
|
||||
- 接近 80%(24 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**:通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**:通过 issue comment 记录轮次进度
|
||||
- **待办文档**:在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 合同审查队列
|
||||
6. ✅ 合规检查项
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 苏慎(lawyer)专用配置
|
||||
@@ -0,0 +1,234 @@
|
||||
# HEARTBEAT.md - 顾析策(marketanalysis)的心跳配置
|
||||
|
||||
> 模板版本:v1.1 (BIZ-24) | 分类:业务 Agent
|
||||
> OpenClaw Agent ID: `marketanalysis` | Multica Agent UUID: `5ed91729-658f-4654-98f0-3e0313022002`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级:OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'marketanalysis' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级:Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id 5ed91729-658f-4654-98f0-3e0313022002 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级:待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片(agentId=marketanalysis)
|
||||
↓
|
||||
检查 Multica Issues 待办(assignee=5ed91729-658f-4654-98f0-3e0313022002)
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测:
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(2h)→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖,可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖,可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:30 轮
|
||||
|
||||
- 接近 80%(24 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**:通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**:通过 issue comment 记录轮次进度
|
||||
- **待办文档**:在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 市场分析任务
|
||||
6. ✅ 竞品数据更新
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 顾析策(marketanalysis)专用配置
|
||||
@@ -0,0 +1,234 @@
|
||||
# HEARTBEAT.md - 钟帧韵(mediaspecialist)的心跳配置
|
||||
|
||||
> 模板版本:v1.1 (BIZ-24) | 分类:业务 Agent
|
||||
> OpenClaw Agent ID: `mediaspecialist` | Multica Agent UUID: `e2b587d4-1d16-447c-8ad9-e2a01358ff0a`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级:OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'mediaspecialist' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级:Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id e2b587d4-1d16-447c-8ad9-e2a01358ff0a --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级:待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片(agentId=mediaspecialist)
|
||||
↓
|
||||
检查 Multica Issues 待办(assignee=e2b587d4-1d16-447c-8ad9-e2a01358ff0a)
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测:
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(2h)→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖,可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖,可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:30 轮
|
||||
|
||||
- 接近 80%(24 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**:通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**:通过 issue comment 记录轮次进度
|
||||
- **待办文档**:在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 视频制作进度
|
||||
6. ✅ 媒体素材准备状态
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 钟帧韵(mediaspecialist)专用配置
|
||||
@@ -0,0 +1,234 @@
|
||||
# HEARTBEAT.md - 严维序(opengineer)的心跳配置
|
||||
|
||||
> 模板版本:v1.1 (BIZ-24) | 分类:开发 Agent
|
||||
> OpenClaw Agent ID: `opengineer` | Multica Agent UUID: `d3804433-9e2e-4199-a92b-a153049b3bc9`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级:OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'opengineer' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级:Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id d3804433-9e2e-4199-a92b-a153049b3bc9 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级:待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片(agentId=opengineer)
|
||||
↓
|
||||
检查 Multica Issues 待办(assignee=d3804433-9e2e-4199-a92b-a153049b3bc9)
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测:
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO + 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO + 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(2h)→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖,可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖,可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:100 轮
|
||||
|
||||
- 接近 80%(80 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知 COO + 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**:通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**:通过 issue comment 记录轮次进度
|
||||
- **待办文档**:在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 部署状态检查
|
||||
6. ✅ 服务器/服务健康状况
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 严维序(opengineer)专用配置
|
||||
@@ -0,0 +1,234 @@
|
||||
# HEARTBEAT.md - 沈路明(productmanager)的心跳配置
|
||||
|
||||
> 模板版本:v1.1 (BIZ-24) | 分类:开发 Agent
|
||||
> OpenClaw Agent ID: `productmanager` | Multica Agent UUID: `a101fa88-d821-4839-9754-e04580d5fd68`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级:OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'productmanager' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级:Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id a101fa88-d821-4839-9754-e04580d5fd68 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级:待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片(agentId=productmanager)
|
||||
↓
|
||||
检查 Multica Issues 待办(assignee=a101fa88-d821-4839-9754-e04580d5fd68)
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测:
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO + 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO + 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(2h)→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖,可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖,可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:100 轮
|
||||
|
||||
- 接近 80%(80 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知 COO + 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**:通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**:通过 issue comment 记录轮次进度
|
||||
- **待办文档**:在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ PRD 进度检查
|
||||
6. ✅ 需求变更跟踪
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 沈路明(productmanager)专用配置
|
||||
@@ -0,0 +1,234 @@
|
||||
# HEARTBEAT.md - 胡蓉(projectmanager)的心跳配置
|
||||
|
||||
> 模板版本:v1.1 (BIZ-24) | 分类:开发 Agent
|
||||
> OpenClaw Agent ID: `projectmanager` | Multica Agent UUID: `d877b8c3-b230-4073-b3f7-80e148cfdb71`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级:OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'projectmanager' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级:Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id d877b8c3-b230-4073-b3f7-80e148cfdb71 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级:待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片(agentId=projectmanager)
|
||||
↓
|
||||
检查 Multica Issues 待办(assignee=d877b8c3-b230-4073-b3f7-80e148cfdb71)
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测:
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO + 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO + 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(2h)→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖,可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖,可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:100 轮
|
||||
|
||||
- 接近 80%(80 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知 COO + 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**:通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**:通过 issue comment 记录轮次进度
|
||||
- **待办文档**:在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 项目进度检查
|
||||
6. ✅ 依赖项完成状态
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 胡蓉(projectmanager)专用配置
|
||||
@@ -0,0 +1,235 @@
|
||||
# HEARTBEAT.md - 刘诗妮(secretary)的心跳配置
|
||||
|
||||
> 模板版本:v1.1 (BIZ-24) | 分类:高频 Agent
|
||||
> OpenClaw Agent ID: `secretary` | Multica Agent UUID: `b024fcdc-30ff-420d-b289-498041466e1b`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级:OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'secretary' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级:Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id b024fcdc-30ff-420d-b289-498041466e1b --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级:待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片(agentId=secretary)
|
||||
↓
|
||||
检查 Multica Issues 待办(assignee=b024fcdc-30ff-420d-b289-498041466e1b)
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:10 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测:
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 20 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1200:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1200:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 30 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(1h)→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖,可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖,可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:50 轮
|
||||
|
||||
- 接近 80%(40 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知 COO
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**:通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**:通过 issue comment 记录轮次进度
|
||||
- **待办文档**:在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 全局任务积压巡检
|
||||
6. ✅ 业务入口检查
|
||||
7. ✅ 各 Agent 状态巡检
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 刘诗妮(secretary)专用配置
|
||||
@@ -0,0 +1,234 @@
|
||||
# HEARTBEAT.md - 陆云帆(taobaospecialist)的心跳配置
|
||||
|
||||
> 模板版本:v1.1 (BIZ-24) | 分类:业务 Agent
|
||||
> OpenClaw Agent ID: `taobaospecialist` | Multica Agent UUID: `e0f62d8f-9568-4f41-8ad4-b73d79a163a7`
|
||||
|
||||
---
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
|
||||
#### 第一优先级:OpenClaw WorkBoard 卡片
|
||||
|
||||
```bash
|
||||
# 检查 WorkBoard 中分配给我的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == 'taobaospecialist' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c["id"][:8]} [priority={c.get("priority","?")}] {c["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第二优先级:Multica Issues
|
||||
|
||||
```bash
|
||||
# 检查 Multica 中分配给我的待办 Issue
|
||||
multica issue list --assignee-id e0f62d8f-9568-4f41-8ad4-b73d79a163a7 --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue["identifier"]} [{issue.get("priority","?")}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
#### 第三优先级:待办文档
|
||||
|
||||
```bash
|
||||
# 检查工作区待办文档
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档中未完成项"
|
||||
```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片(agentId=taobaospecialist)
|
||||
↓
|
||||
检查 Multica Issues 待办(assignee=e0f62d8f-9568-4f41-8ad4-b73d79a163a7)
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重 → 按优先级排序 → 依次执行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:15 分钟
|
||||
|
||||
每次心跳跨平台执行以下检测:
|
||||
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
|
||||
```bash
|
||||
# WorkBoard 超时检测
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ WB TIMEOUT: {c["id"][:8]} [{c.get("agentId","?")}] {c["title"]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1800:
|
||||
print(f'⏰ MUL TIMEOUT: {issue["identifier"]} [{issue.get("assignee_id","?")[:12]}] {issue["title"]}')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(2h)→ 通知依赖任务执行者
|
||||
|
||||
### 双平台依赖检查
|
||||
|
||||
```bash
|
||||
# WorkBoard 依赖检查
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足: {dep["id"]} → status={dep.get("status","?")}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖,可以启动')
|
||||
"
|
||||
|
||||
# Multica 依赖检查
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, subprocess
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent["identifier"]} 未完成')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent["identifier"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖,可以启动')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:30 轮
|
||||
|
||||
- 接近 80%(24 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知 创建者
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
|
||||
- **WorkBoard**:通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**:通过 issue comment 记录轮次进度
|
||||
- **待办文档**:在工作日志中记录
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
### 每次心跳必须检查
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 淘宝店铺运营指标
|
||||
6. ✅ 竞品动态跟踪
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 基于 BIZ-24 v1.1 模板生成 | 陆云帆(taobaospecialist)专用配置
|
||||
@@ -0,0 +1,130 @@
|
||||
# Agent 知识库集成指南
|
||||
|
||||
> **版本**: v1.0
|
||||
> **任务**: BIZ-19 (BIZ-14-4)
|
||||
> **日期**: 2026-06-22
|
||||
> **作者**: COO (陆怀瑾)
|
||||
> **状态**: 已实施
|
||||
|
||||
---
|
||||
|
||||
## 一、集成概述
|
||||
|
||||
### 1.1 设计原则
|
||||
|
||||
**「引用代替填塞」**: 不把知识内容直接塞进 Agent 配置文件,而是添加 "如何查询知识库" 的指引。Agent 在需要时主动检索,保持配置文件轻量和可维护。
|
||||
|
||||
### 1.2 核心工具
|
||||
|
||||
| 工具 | 用途 | 适用场景 |
|
||||
|------|------|----------|
|
||||
| `wiki_search` | 模糊搜索知识库 | "有没有关于 X 的文档" |
|
||||
| `wiki_get` | 精确读取页面 | "打开 X 页面" |
|
||||
| `wiki_lint` | 知识库质量检查 | "知识库健康度如何" |
|
||||
| `wiki_status` | 系统状态检查 | "知识库是否可用" |
|
||||
| `wiki_apply` | 写入/更新知识库 | "将 X 发现写入知识库" |
|
||||
|
||||
---
|
||||
|
||||
## 二、Agent 集成清单
|
||||
|
||||
### 2.1 已完成集成的 Agent(15 个)
|
||||
|
||||
| # | Agent | 角色 | TOOLS.md 更新状态 | 触发场景数 |
|
||||
|---|-------|------|-------------------|------------|
|
||||
| 1 | secretary | 刘诗妮 - 业务入口 | ✅ | 4 |
|
||||
| 2 | coo | 陆怀瑾 - 运营总监 | ✅ | 5 |
|
||||
| 3 | projectmanager | 胡蓉 - 项目经理 | ✅ | 4 |
|
||||
| 4 | architect | 梁思筑 - 架构师 | ✅ | 4 |
|
||||
| 5 | costcodev | 徐聪 - 全栈开发 | ✅ | 4 |
|
||||
| 6 | designer | 苏绘锦 - UI/UX 设计 | ✅ | 3 |
|
||||
| 7 | taobaospecialist | 陆云帆 - 淘宝运营 | ✅ | 4 |
|
||||
| 8 | contentspecialist | 文墨言 - 内容文案 | ✅ | 4 |
|
||||
| 9 | mediaspecialist | 钟帧韵 - 视频制作 | ✅ | 3 |
|
||||
| 10 | cvexpert | 程伯予 - 求职助理 | ✅ | 3 |
|
||||
| 11 | marketanalysis | 顾析策 - 市场分析 | ✅ | 4 |
|
||||
| 12 | lawyer | 苏慎 - 法务顾问 | ✅ | 4 |
|
||||
| 13 | opengineer | 严维序 - 运维部署 | ✅ | 4 |
|
||||
| 14 | productmanager | 沈路明 - 产品经理 | ✅ | 4 |
|
||||
| 15 | main | 入口路由 | ✅ | 2 |
|
||||
|
||||
### 2.2 集成内容
|
||||
|
||||
每个 Agent 的 TOOLS.md 新增了以下内容:
|
||||
|
||||
1. **知识库查询指引** — 引导 Agent 查看完整检索指南
|
||||
2. **角色特定触发条件** — 该 Agent 何时应查询知识库
|
||||
3. **查询工具速查** — `wiki_search` / `wiki_get` / `wiki_lint` 基本用法
|
||||
4. **角色特定查询示例** — 1-2 个典型查询语句
|
||||
5. **无结果时处理流程** — 知识缺口上报机制
|
||||
|
||||
---
|
||||
|
||||
## 三、查询触发条件设计
|
||||
|
||||
### 3.1 通用触发条件(所有 Agent 适用)
|
||||
|
||||
| 场景 | 触发动作 |
|
||||
|------|----------|
|
||||
| 接受新任务时 | 先查知识库中是否有相关文档/SOP |
|
||||
| 遇到不确定信息时 | 先查知识库再作决策 |
|
||||
| 需要跨领域协作时 | 查其他 Agent 的职能和知识 |
|
||||
| 发现新知识时 | 考虑是否需写入知识库 |
|
||||
|
||||
### 3.2 角色特定触发条件(按 Agent 定制)
|
||||
|
||||
见各 Agent TOOLS.md 中的「知识库查询 → 触发条件」部分。
|
||||
|
||||
---
|
||||
|
||||
## 四、知识缺口上报机制
|
||||
|
||||
### 4.1 上报流程
|
||||
|
||||
```
|
||||
Agent 查询知识库 → 无结果 → 尝试同义词/相关词 → 仍无结果 →
|
||||
→ 记录知识缺口 → 写入 memory/ 日志 →
|
||||
→ 下次心跳/汇报时通知 architect 或对应领域 Agent
|
||||
```
|
||||
|
||||
### 4.2 上报格式
|
||||
|
||||
见 `docs/agent-kb-retrieval-guide.md` 第五节。
|
||||
|
||||
---
|
||||
|
||||
## 五、质量保证
|
||||
|
||||
### 5.1 集成测试方案
|
||||
|
||||
对每个 Agent 至少执行 1 次典型查询场景测试:
|
||||
|
||||
1. 验证 `wiki_search` 可被正确调用
|
||||
2. 验证返回结果格式正确
|
||||
3. 验证无结果时的降级路径
|
||||
|
||||
### 5.2 集成测试结果
|
||||
|
||||
| Agent | 测试查询 | 结果 | 备注 |
|
||||
|-------|----------|------|------|
|
||||
| 通用 | `wiki_search(query="服务器")` | ✅ | wiki_search 正常 |
|
||||
|
||||
*注:知识库当前为初始状态(0 sources, 0 entities, 0 concepts, 0 syntheses, 10 reports),搜索结果取决于内容填充进度。工具链已验证可用。*
|
||||
|
||||
---
|
||||
|
||||
## 六、后续计划
|
||||
|
||||
1. **知识内容填充**: 待 BIZ-14-3 交付后,各 Agent 按角色写入初始知识内容
|
||||
2. **定期质量检查**: COO 每周运行 `wiki_lint()` 检查知识库健康度
|
||||
3. **查询效果评估**: 运行 1 个月后统计各 Agent 知识库查询频率和命中率
|
||||
4. **持续优化**: 根据使用反馈调整触发条件和查询示例
|
||||
|
||||
---
|
||||
|
||||
## 附录:相关文档
|
||||
|
||||
- `docs/agent-kb-retrieval-guide.md` — 知识库检索工具完整指南
|
||||
- `docs/知识查询最佳实践.md` — 查询最佳实践和反模式
|
||||
- `docs/wiki-toolchain-test-report.md` — Wiki 工具链测试报告 (BIZ-14-2)
|
||||
- 各 Agent TOOLS.md — 角色特定查询指引
|
||||
@@ -0,0 +1,156 @@
|
||||
# 知识查询最佳实践
|
||||
|
||||
> **版本**: v1.0
|
||||
> **任务**: BIZ-19 (BIZ-14-4)
|
||||
> **日期**: 2026-06-22
|
||||
|
||||
---
|
||||
|
||||
## 一、查询策略
|
||||
|
||||
### 1.1 渐进式检索原则
|
||||
|
||||
```
|
||||
先宽后窄 → 先模糊后精确 → 先搜索后读取
|
||||
```
|
||||
|
||||
**标准流程**:
|
||||
1. `wiki_search(query="关键词")` — 发现有哪些相关内容
|
||||
2. `wiki_get(lookup="匹配页面")` — 精确读取具体内容
|
||||
3. 如搜索结果过多(>10) → 收窄关键词重新搜索
|
||||
4. 如搜索结果与需求不相关 → 调整表述方式重新搜索
|
||||
|
||||
### 1.2 查询词构造技巧
|
||||
|
||||
#### DO ✅
|
||||
|
||||
| 技巧 | 示例 | 说明 |
|
||||
|------|------|------|
|
||||
| 用领域特定术语 | `wiki_search(query="nginx 反向代理")` | 专业词汇提升精确度 |
|
||||
| 用动词+对象 | `wiki_search(query="部署 Node.js")` | 明确查询意图 |
|
||||
| 用自然语言问题 | `wiki_search(query="如何配置 nginx logrotate")` | 适合语义检索 |
|
||||
| 用缩写和全称组合 | `wiki_search(query="CI/CD 持续集成")` | 覆盖不同表述 |
|
||||
| 分步搜索 | 先搜 "nginx",再搜 "nginx 日志" | 逐步收窄范围 |
|
||||
|
||||
#### DON'T ❌
|
||||
|
||||
| 反模式 | 错误示例 | 问题 |
|
||||
|--------|----------|------|
|
||||
| 过于泛化的词 | `wiki_search(query="配置")` | 结果太多太杂 |
|
||||
| 过于具体的短语 | `wiki_search(query="192.168.1.99 端口 22 上的 nginx")` | 命中率低 |
|
||||
| 跳过搜索直接 guess 路径 | `wiki_get(lookup="随便猜的页面名")` | 大概率找不到 |
|
||||
| 一次加载超大页面 | `wiki_get(lookup="巨型文档")` | 超出上下文容量 |
|
||||
| 无结果后直接放弃 | 只搜一次就说"知识库没内容" | 可能是查询词不准确 |
|
||||
|
||||
---
|
||||
|
||||
## 二、结果处理
|
||||
|
||||
### 2.1 匹配结果数量处理
|
||||
|
||||
| 结果数 | 处理方式 |
|
||||
|--------|----------|
|
||||
| 0 | 尝试同义词/相关词 → qmd 搜索 → 上报知识缺口 |
|
||||
| 1-3 | 逐个 `wiki_get` 读取完整内容 |
|
||||
| 4-10 | 按评分排序,取前 3 个读取 |
|
||||
| 10+ | 收窄搜索词重新搜索 |
|
||||
|
||||
### 2.2 大页面分页读取
|
||||
|
||||
```bash
|
||||
# 超过 100 行的页面,分页读取
|
||||
wiki_get(lookup="长文档标题", fromLine=1, lineCount=50) # 第一部分
|
||||
wiki_get(lookup="长文档标题", fromLine=51, lineCount=50) # 第二部分
|
||||
```
|
||||
|
||||
### 2.3 信息来源交叉验证
|
||||
|
||||
当多个查询返回不同信息时:
|
||||
1. 检查页面更新时间(优先信任较新的)
|
||||
2. 交叉对比多个来源
|
||||
3. 如信息冲突 → 标记为"需确认",汇报给 architect
|
||||
|
||||
---
|
||||
|
||||
## 三、知识缺口处理
|
||||
|
||||
### 3.1 判定标准
|
||||
|
||||
满足以下任一条件即报告知识缺口:
|
||||
- `wiki_search` 和 `qmd` 均无匹配
|
||||
- 搜索结果与需求明显不相关
|
||||
- 找到的文档内容已过时或不完整
|
||||
|
||||
### 3.2 上报模板
|
||||
|
||||
```
|
||||
【知识缺口 - YYYY-MM-DD】
|
||||
|
||||
- 查询 Agent: [Agent 名称]
|
||||
- 查询意图: [想了解什么]
|
||||
- 已尝试检索: [用过的搜索词, 换行列出]
|
||||
- 已使用工具: wiki_search / qmd
|
||||
- 期望内容: [知识库中应有什么]
|
||||
- 紧急程度: high / normal / low
|
||||
- 建议: [谁补充、什么内容]
|
||||
```
|
||||
|
||||
### 3.3 上报路径
|
||||
|
||||
| 缺口类型 | 上报目标 |
|
||||
|----------|----------|
|
||||
| 架构/技术 | architect (梁思筑) |
|
||||
| 业务/流程 | projectmanager (胡蓉) |
|
||||
| 法务/合规 | lawyer (苏慎) |
|
||||
| 市场/分析 | marketanalysis (顾析策) |
|
||||
| 通用/不确定 | COO (陆怀瑾) — 由 COO 分配 |
|
||||
|
||||
---
|
||||
|
||||
## 四、知识库写入准则
|
||||
|
||||
### 4.1 何时写入
|
||||
|
||||
- 完成重要决策后(如架构选型、策略调整)
|
||||
- 发现可复用的模板/清单
|
||||
- 完成深度分析后(市场报告、竞品分析)
|
||||
- 知识缺口被填补后
|
||||
|
||||
### 4.2 写入工具选择
|
||||
|
||||
| 场景 | 工具 |
|
||||
|------|------|
|
||||
| 创建新知识页面 | `wiki_apply(op="create_synthesis", ...)` |
|
||||
| 更新已有页面元数据 | `wiki_apply(op="update_metadata", ...)` |
|
||||
|
||||
### 4.3 不写入的内容
|
||||
|
||||
- 机密信息(密码、密钥、token)
|
||||
- 临时信息(当天的具体任务进度)
|
||||
- 已过时会被频繁更新的数据
|
||||
- 纯个人笔记(放 `memory/` 下)
|
||||
|
||||
---
|
||||
|
||||
## 五、定期维护
|
||||
|
||||
### 5.1 COO 每周检查清单
|
||||
|
||||
- [ ] 运行 `wiki_lint()` 检查质量
|
||||
- [ ] 统计各 Agent 知识库查询频率
|
||||
- [ ] 清理过时页面
|
||||
- [ ] 评估知识缺口数量和解决率
|
||||
- [ ] 输出知识库运营周报
|
||||
|
||||
### 5.2 Agent 自检清单
|
||||
|
||||
每次心跳时:
|
||||
- [ ] 上次查询的知识缺口是否已上报
|
||||
- [ ] 本轮工作中是否有应写入知识库的发现
|
||||
|
||||
---
|
||||
|
||||
## 附录
|
||||
|
||||
- `docs/agent-kb-retrieval-guide.md` — 工具使用完整指南
|
||||
- `docs/Agent 知识库集成指南.md` — 集成方案总览
|
||||
@@ -12,8 +12,10 @@
|
||||
| [产品/](产品/) | PRD、需求分析 | 沈路明 (productmanager) | — |
|
||||
| [技术/](技术/) | 开发规范、代码审查 | 徐聪 (costcodev) | — |
|
||||
| [设计/](设计/) | UI设计、品牌规范 | 苏绘锦 (designer) | — |
|
||||
| [运维/](运维/) | 部署流程、故障排查、服务器运维 | 严维序 (opengineer) | 3 |
|
||||
| [运营/](运营/) | 活动策划、数据分析 | 陆怀瑾 (coo) | — |
|
||||
| [行政/](行政/) | 合同、报销流程 | 刘诗妮 (secretary) | — |
|
||||
| [规范/](规范/) | 运维标准、安全基线、合规要求 | 严维序 (opengineer) | — |
|
||||
|
||||
## 知识条目格式
|
||||
|
||||
|
||||
@@ -5,7 +5,9 @@
|
||||
|
||||
## 知识范围
|
||||
|
||||
涵盖开发规范、代码审查、架构设计、部署运维、技术选型等技术团队知识。
|
||||
涵盖开发规范、代码审查、架构设计、技术选型等技术团队核心知识。
|
||||
|
||||
> ⚠️ 部署运维知识已迁移至 [运维/](../运维/) 领域。
|
||||
|
||||
## 条目清单
|
||||
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
# 规范领域知识
|
||||
|
||||
**责任人**:严维序(opengineer)
|
||||
**审核人**:陆怀瑾(coo)
|
||||
|
||||
## 知识范围
|
||||
|
||||
涵盖运维规范、安全标准、合规要求等规范类知识条目,支撑团队标准化运作。
|
||||
|
||||
## 条目清单
|
||||
|
||||
| 文件名 | 说明 | 状态 |
|
||||
|--------|------|------|
|
||||
| [服务器运维标准_v1.0.md](../运维/服务器运维标准_v1.0.md) | 服务器巡检、监控、备份运维标准 | 见运维域 |
|
||||
|
||||
## 待建设
|
||||
|
||||
- 数据库运维标准
|
||||
- 安全审计基线
|
||||
- 数据合规处理流程
|
||||
|
||||
---
|
||||
|
||||
> 维护者:严维序(opengineer)
|
||||
> 最后更新:2026-06-24
|
||||
@@ -0,0 +1,27 @@
|
||||
# 运维领域知识
|
||||
|
||||
**责任人**:严维序(opengineer)
|
||||
**审核人**:陆怀瑾(coo)
|
||||
|
||||
## 知识范围
|
||||
|
||||
涵盖服务器运维、部署流程、故障排查、监控配置、安全保障等运维团队核心知识。
|
||||
|
||||
## 条目清单
|
||||
|
||||
| 文件名 | 说明 | 状态 |
|
||||
|--------|------|------|
|
||||
| [部署流程_v1.0.md](部署流程_v1.0.md) | 服务部署 SOP 与变更管理流程 | ✅ |
|
||||
| [故障排查手册_v1.0.md](故障排查手册_v1.0.md) | 常见故障定位与处置方案 | ✅ |
|
||||
| [服务器运维标准_v1.0.md](服务器运维标准_v1.0.md) | 服务器巡检、监控、备份运维标准 | 🆕 |
|
||||
|
||||
## 待建设
|
||||
|
||||
- 数据库运维指南
|
||||
- 安全加固检查清单
|
||||
- 灾备与应急恢复预案
|
||||
|
||||
---
|
||||
|
||||
> 维护者:严维序(opengineer)
|
||||
> 最后更新:2026-06-24
|
||||
@@ -0,0 +1,274 @@
|
||||
# 故障排查手册
|
||||
|
||||
## 元数据
|
||||
|
||||
| 属性 | 值 |
|
||||
|------|-----|
|
||||
| **领域** | 运维 |
|
||||
| **责任人** | 严维序(opengineer) |
|
||||
| **版本** | v1.0 |
|
||||
| **创建日期** | 2026-06-24 |
|
||||
| **最后更新** | 2026-06-24 |
|
||||
| **标签** | 故障排查, 运维, 排障 |
|
||||
|
||||
## 概述
|
||||
|
||||
本手册汇总 BizWings 环境中常见的系统与服务故障定位方法和修复方案。覆盖 SSH 连接、Nginx、数据库、磁盘、Docker 等核心场景。
|
||||
|
||||
---
|
||||
|
||||
## 一、SSH 连接故障
|
||||
|
||||
### 1.1 连接超时
|
||||
|
||||
```bash
|
||||
# 诊断步骤
|
||||
ssh -vvv root@<ip> -p <port> # 查看详细连接日志
|
||||
ping <ip> # 检查网络连通性
|
||||
nmap <ip> -p <port> # 检查端口状态
|
||||
```
|
||||
|
||||
**常见原因**:
|
||||
- 目标服务器防火墙未开放端口
|
||||
- 源 IP 未加入白名单
|
||||
- 服务器负载过高,sshd 响应慢
|
||||
|
||||
**解决方案**:
|
||||
1. 检查服务器防火墙:`iptables -L -n` 或 `ufw status`
|
||||
2. 检查 sshd 是否运行:`systemctl status sshd`
|
||||
3. 检查负载:`top -n1 | head -5`
|
||||
|
||||
### 1.2 认证失败
|
||||
|
||||
```bash
|
||||
# 诊断步骤
|
||||
ssh -p <port> root@<ip> # 尝试密码登录
|
||||
# Permission denied (publickey,password) 提示
|
||||
```
|
||||
|
||||
**常见原因**:
|
||||
- 密码错误(检查 TOOLS.md 中记录)
|
||||
- SSH 密钥认证配置错误
|
||||
- `/etc/ssh/sshd_config` 中 `PasswordAuthentication no`
|
||||
|
||||
**解决方案**:
|
||||
1. 确认密码与 TOOLS.md 一致
|
||||
2. 检查 `sshd_config`:`grep PasswordAuthentication /etc/ssh/sshd_config`
|
||||
3. 临时允许密码登录:`sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config && systemctl reload sshd`
|
||||
|
||||
---
|
||||
|
||||
## 二、Nginx 服务异常
|
||||
|
||||
### 2.1 Nginx 启动失败 / 卡在 activating
|
||||
|
||||
```bash
|
||||
# 诊断步骤
|
||||
systemctl status nginx # 查看状态
|
||||
journalctl -u nginx --no-pager -n 50 # 查看日志
|
||||
nginx -t # 配置语法检查
|
||||
```
|
||||
|
||||
**根因(经验)**:进程残留导致端口占用
|
||||
|
||||
```bash
|
||||
# 修复
|
||||
pkill -9 nginx # 强制清理残留进程
|
||||
sleep 2
|
||||
systemctl start nginx # 重新启动
|
||||
systemctl status nginx # 确认状态
|
||||
```
|
||||
|
||||
### 2.2 502 Bad Gateway
|
||||
|
||||
```bash
|
||||
# 诊断步骤
|
||||
curl -I http://localhost:<upstream-port> # 检查上游服务
|
||||
ss -tlnp | grep <upstream-port> # 检查端口监听
|
||||
systemctl status <upstream-service> # 检查上游进程
|
||||
```
|
||||
|
||||
**常见原因**:
|
||||
- 上游服务未启动或崩溃
|
||||
- 连接池耗尽
|
||||
|
||||
**解决方案**:
|
||||
1. 重启上游服务:`systemctl restart <service>`
|
||||
2. 检查 `upstream` 配置是否正确
|
||||
|
||||
### 2.3 日志轮转失败
|
||||
|
||||
```bash
|
||||
# 诊断步骤
|
||||
cat /var/log/nginx/error.log | head # 查看是否有日志无法写入
|
||||
ls -la /var/log/nginx/ # 查看日志文件
|
||||
/usr/sbin/logrotate -d /etc/logrotate.d/nginx # 测试 logrotate
|
||||
```
|
||||
|
||||
**修复方案**:
|
||||
```bash
|
||||
# 修改 /etc/logrotate.d/nginx 中的 postrotate 脚本
|
||||
# 将 invoke-rc.d nginx rotate 改为:
|
||||
postrotate
|
||||
systemctl reload nginx
|
||||
endscript
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 三、数据库连接故障
|
||||
|
||||
### 3.1 MySQL 连接失败
|
||||
|
||||
```bash
|
||||
# 诊断步骤
|
||||
mysql -h <host> -P <port> -u root -p # 测试连接
|
||||
telnet <host> <port> # 检查端口
|
||||
systemctl status mysql # 检查服务
|
||||
```
|
||||
|
||||
**常见原因**:
|
||||
- 服务未运行
|
||||
- 防火墙未放行 3306 端口
|
||||
- 用户权限 / host 限制
|
||||
- 连接数超限
|
||||
|
||||
**解决方案**:
|
||||
```bash
|
||||
# 检查连接数
|
||||
mysql -e "SHOW VARIABLES LIKE 'max_connections';"
|
||||
mysql -e "SHOW PROCESSLIST;"
|
||||
|
||||
# 检查用户权限
|
||||
mysql -e "SELECT user, host FROM mysql.user WHERE user='root';"
|
||||
```
|
||||
|
||||
### 3.2 MySQL 空间不足
|
||||
|
||||
```bash
|
||||
# 诊断
|
||||
df -h # 磁盘空间
|
||||
mysql -e "SELECT table_schema, ROUND(SUM(data_length+index_length)/1024/1024,2) AS size_mb FROM information_schema.tables GROUP BY table_schema ORDER BY size_mb DESC;"
|
||||
```
|
||||
|
||||
**解决方案**:
|
||||
- 清理过期 binlog:`PURGE BINARY LOGS BEFORE DATE_SUB(NOW(), INTERVAL 7 DAY);`
|
||||
- 清理临时表
|
||||
- 扩展磁盘
|
||||
|
||||
---
|
||||
|
||||
## 四、磁盘空间告警
|
||||
|
||||
### 4.1 诊断
|
||||
|
||||
```bash
|
||||
df -h # 查看各分区使用率
|
||||
du -sh /* 2>/dev/null | sort -rh | head -10 # 找到大文件目录
|
||||
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null # 大文件定位
|
||||
```
|
||||
|
||||
### 4.2 清理方案
|
||||
|
||||
```bash
|
||||
# Docker 日志和镜像清理
|
||||
docker system prune -af --volumes # 清理未使用的 Docker 资源
|
||||
|
||||
# 系统日志轮转
|
||||
journalctl --vacuum-time=7d # 清理 7 天前的 journal 日志
|
||||
|
||||
# 应用日志归档
|
||||
find /var/log -name "*.log" -mtime +30 -exec gzip {} \; # 压缩旧日志
|
||||
find /var/log -name "*.gz" -mtime +90 -delete # 删除 90 天前的压缩日志
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 五、Docker 容器异常
|
||||
|
||||
### 5.1 容器停止
|
||||
|
||||
```bash
|
||||
docker ps -a | grep <container> # 查看容器状态
|
||||
docker logs <container> --tail 50 # 查看最近日志
|
||||
```
|
||||
|
||||
**修复**:
|
||||
```bash
|
||||
docker start <container> # 手动启动
|
||||
docker compose -f <path> up -d # 使用 Compose 重启
|
||||
```
|
||||
|
||||
### 5.2 Docker API 无响应
|
||||
|
||||
```bash
|
||||
systemctl status docker # 检查 Docker 服务
|
||||
journalctl -u docker --no-pager -n 50 # 查看 Docker 日志
|
||||
```
|
||||
|
||||
**修复**:
|
||||
```bash
|
||||
systemctl restart docker # 重启 Docker 守护进程
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 六、系统进程故障
|
||||
|
||||
### 6.1 端口被占用
|
||||
|
||||
```bash
|
||||
ss -tlnp | grep <port> # 查看占用端口的进程
|
||||
fuser -k <port>/tcp # 强制释放端口
|
||||
```
|
||||
|
||||
### 6.2 systemd 服务异常
|
||||
|
||||
```bash
|
||||
systemctl status <service> # 检查状态
|
||||
journalctl -u <service> --no-pager -n 100 # 查看服务日志
|
||||
|
||||
# 常用修复
|
||||
systemctl daemon-reload # 重载 unit 文件
|
||||
systemctl restart <service> # 重启
|
||||
systemctl enable <service> # 设置开机自启
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 七、日志分析工具
|
||||
|
||||
### 7.1 常用命令
|
||||
|
||||
```bash
|
||||
# 实时日志跟踪
|
||||
tail -f /var/log/<app>/access.log
|
||||
|
||||
# 错误过滤
|
||||
grep -i "error\|exception\|failed" /var/log/<app>/app.log | tail -50
|
||||
|
||||
# 时间范围过滤
|
||||
awk '/2026-06-24 10:00/,/2026-06-24 11:00/' /var/log/<app>/app.log
|
||||
```
|
||||
|
||||
### 7.2 关键检查点
|
||||
|
||||
| 故障表现 | 优先检查 | 常见根因 |
|
||||
|----------|----------|----------|
|
||||
| 服务无响应 | systemctl status | 进程 OOM / 崩溃 |
|
||||
| API 返回错误 | 应用日志 + Nginx 日志 | 代码 bug / 上游依赖异常 |
|
||||
| 高延迟 | top + ss + 应用日志 | 资源争抢 / 死锁 |
|
||||
| 数据库异常 | MySQL error log | 慢查询 / 连接数超限 |
|
||||
|
||||
---
|
||||
|
||||
## 相关条目
|
||||
|
||||
- [部署流程_v1.0.md](部署流程_v1.0.md)
|
||||
- [服务器运维标准_v1.0.md](服务器运维标准_v1.0.md)
|
||||
|
||||
## 变更记录
|
||||
|
||||
| 日期 | 版本 | 变更说明 | 变更人 |
|
||||
|------|------|----------|--------|
|
||||
| 2026-06-24 | v1.0 | 初始创建 | 严维序 |
|
||||
@@ -0,0 +1,177 @@
|
||||
# 服务器运维标准
|
||||
|
||||
## 元数据
|
||||
|
||||
| 属性 | 值 |
|
||||
|------|-----|
|
||||
| **领域** | 运维 |
|
||||
| **责任人** | 严维序(opengineer) |
|
||||
| **版本** | v1.0 |
|
||||
| **创建日期** | 2026-06-24 |
|
||||
| **最后更新** | 2026-06-24 |
|
||||
| **标签** | 运维, 监控, 巡检, 备份 |
|
||||
|
||||
## 概述
|
||||
|
||||
本文档定义 BizWings 团队所有服务器的日常运维标准,包括巡检频率、监控指标、备份策略和安全基线。适用于所有生产环境服务器(阿里云 / 家庭内网 / HP 服务器)。
|
||||
|
||||
---
|
||||
|
||||
## 一、服务器巡检标准
|
||||
|
||||
### 1.1 巡检频率
|
||||
|
||||
| 类型 | 频率 | 执行方式 |
|
||||
|------|------|----------|
|
||||
| 心跳自检 | 每 10 分钟 | openclaw 心跳自动巡检 |
|
||||
| 深度巡检 | 每日一次 | 手动执行 `python3 $SCRIPTS/heartbeat_helper.py opengineer` |
|
||||
| 全量巡检 | 每周一次 | 逐个检查全部服务器 |
|
||||
|
||||
### 1.2 巡检清单
|
||||
|
||||
#### 资源负载
|
||||
```bash
|
||||
# 磁盘使用率(警告 > 80%,严重 > 90%)
|
||||
df -h | grep -v tmpfs
|
||||
|
||||
# CPU 负载
|
||||
uptime
|
||||
|
||||
# 内存使用
|
||||
free -h
|
||||
|
||||
# 网络 IO
|
||||
sar -n DEV 1 3
|
||||
```
|
||||
|
||||
#### 服务状态
|
||||
```bash
|
||||
# 核心服务清单(按实际部署确认)
|
||||
systemctl status nginx mysql docker sshd
|
||||
|
||||
# Docker 容器健康
|
||||
docker ps | grep -c "Up"
|
||||
```
|
||||
|
||||
#### 日志异常
|
||||
```bash
|
||||
# 最近 10 分钟的错误日志
|
||||
journalctl --since "10 min ago" -p err --no-pager | tail -20
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 二、监控指标定义
|
||||
|
||||
### 2.1 告警阈值
|
||||
|
||||
| 指标 | 警告 (WARN) | 严重 (CRIT) | 处理 |
|
||||
|------|-------------|-------------|------|
|
||||
| 磁盘使用率 | > 80% | > 90% | 清理日志 / 扩容 |
|
||||
| CPU 负载 (1min) | > 4.0 | > 8.0 | 检查异常进程 |
|
||||
| 内存使用率 | > 85% | > 95% | 检查 OOM 风险 |
|
||||
| 根分区 inode | > 80% | > 90% | 清理小文件 |
|
||||
| 服务进程 | 停止 | — | 重启服务 |
|
||||
| 端口监听 | 消失 | — | 检查服务状态 |
|
||||
| Docker 容器 | 非 Up | — | docker start / compose up |
|
||||
|
||||
### 2.2 日志监控
|
||||
|
||||
- 系统日志:`journalctl -p err` 重点关注
|
||||
- 应用日志:`error`, `exception`, `failed`, `timeout` 关键词监控
|
||||
- Nginx 日志:5xx 错误率 > 1% 时触发调查
|
||||
|
||||
---
|
||||
|
||||
## 三、备份策略
|
||||
|
||||
### 3.1 数据库备份
|
||||
|
||||
```bash
|
||||
# MySQL 全量备份(建议每日凌晨执行)
|
||||
mysqldump --all-databases --single-transaction --quick | gzip > /backup/db/all-$(date +%Y%m%d).sql.gz
|
||||
```
|
||||
|
||||
### 3.2 配置备份
|
||||
- 服务器配置文件:`/backup/conf/<server>/` 目录
|
||||
- 每次变更前执行:`cp <config> <config>.$(date +%Y%m%d-%H%M%S).bak`
|
||||
|
||||
### 3.3 Docker 数据备份
|
||||
```bash
|
||||
# 思源笔记备份(已配置每日 3:00)
|
||||
tar czf /backup/siyuan/siyuan-data-$(date +%Y%m%d).tar.gz -C <data-dir> .
|
||||
```
|
||||
|
||||
### 3.4 备份保留策略
|
||||
|
||||
| 类型 | 保留期限 |
|
||||
|------|----------|
|
||||
| 数据库全量备份 | 30 天 |
|
||||
| 配置备份 | 90 天 |
|
||||
| Docker 数据 | 7 天 |
|
||||
| 日志归档 | 90 天 |
|
||||
|
||||
---
|
||||
|
||||
## 四、变更管理标准
|
||||
|
||||
### 4.1 变更准入
|
||||
|
||||
- ✅ 每次变更前必须备份原始文件
|
||||
- ✅ 高危操作(防火墙、内核、数据库)必须保留回滚方案
|
||||
- ✅ 变更前评估影响范围
|
||||
- ✅ 变更后验证服务状态
|
||||
- ❌ 禁止在无备份的情况下直接修改生产配置
|
||||
- ❌ 禁止在高峰时段执行非紧急变更
|
||||
|
||||
### 4.2 变更分级
|
||||
|
||||
| 级别 | 示例 | 要求 |
|
||||
|------|------|------|
|
||||
| 低风险 | 普通应用更新 | 备份 → 部署 → 验证 |
|
||||
| 中风险 | 配置修改 | 备份 → 预演 → 部署 → 验证 |
|
||||
| 高风险 | 内核 / 防火墙 / 数据库 | 备份 → 预演 → 通知 → 部署 → 验证 → 监控 |
|
||||
|
||||
---
|
||||
|
||||
## 五、安全基线
|
||||
|
||||
### 5.1 基本要求
|
||||
|
||||
- [ ] SSH 禁止 root 密码登录(高风险服务器)
|
||||
- [ ] 防火墙最小权限原则
|
||||
- [ ] 非必要端口不对外开放
|
||||
- [ ] 定期更新系统安全补丁
|
||||
- [ ] 日志审计开启
|
||||
|
||||
### 5.2 密码管理
|
||||
|
||||
- 服务器密码统一记录在 TOOLS.md
|
||||
- 数据库密码统一管理
|
||||
- 禁止在代码中硬编码密码
|
||||
|
||||
---
|
||||
|
||||
## 六、服务器清单与分类
|
||||
|
||||
| 环境 | 服务器数 | 用途 | 巡检频率 |
|
||||
|------|----------|------|----------|
|
||||
| 阿里云生产 | 3 | 应用服务、数据库 | 每次心跳 |
|
||||
| 家庭内网生产 | 4 | 应用、数据库、PVE | 每次心跳 |
|
||||
| HP 测试 | 3 | 测试、NAS | 每日 |
|
||||
| 树莓派 | 1 | 辅助设备 | 每日 |
|
||||
|
||||
详细清单见 TOOLS.md「SSH/WinRM 服务器清单」
|
||||
|
||||
---
|
||||
|
||||
## 相关条目
|
||||
|
||||
- [部署流程_v1.0.md](部署流程_v1.0.md)
|
||||
- [故障排查手册_v1.0.md](故障排查手册_v1.0.md)
|
||||
|
||||
## 变更记录
|
||||
|
||||
| 日期 | 版本 | 变更说明 | 变更人 |
|
||||
|------|------|----------|--------|
|
||||
| 2026-06-24 | v1.0 | 初始创建 | 严维序 |
|
||||
@@ -0,0 +1,202 @@
|
||||
# 服务部署流程 SOP
|
||||
|
||||
## 元数据
|
||||
|
||||
| 属性 | 值 |
|
||||
|------|-----|
|
||||
| **领域** | 运维 |
|
||||
| **责任人** | 严维序(opengineer) |
|
||||
| **版本** | v1.0 |
|
||||
| **创建日期** | 2026-06-24 |
|
||||
| **最后更新** | 2026-06-24 |
|
||||
| **标签** | 部署, 运维, SOP |
|
||||
|
||||
## 概述
|
||||
|
||||
本文档定义 BizWings 团队所有业务服务的部署流程标准,涵盖部署前检查、执行步骤、验证测试和回滚预案。适用于所有生产环境的代码部署与服务更新。
|
||||
|
||||
---
|
||||
|
||||
## 一、部署前置检查
|
||||
|
||||
### 1.1 代码准备
|
||||
|
||||
- [ ] 代码已合并到目标分支(main / release)
|
||||
- [ ] PR 已通过 Code Review 并合并
|
||||
- [ ] 本地或 CI 构建通过(编译无报错)
|
||||
- [ ] 版本号已更新(如有)
|
||||
|
||||
### 1.2 环境检查
|
||||
|
||||
- [ ] 目标服务器磁盘空间充足(> 剩余 20%)
|
||||
- [ ] CPU / 内存负载正常(< 80%)
|
||||
- [ ] 网络连通性:本机 → 目标服务器可达
|
||||
- [ ] 目标端口未被占用
|
||||
- [ ] 依赖服务(数据库 / 中间件)运行正常
|
||||
|
||||
### 1.3 备份准备
|
||||
|
||||
- [ ] **配置备份**:服务器配置文件备份到 `/backup/conf/` 目录
|
||||
- [ ] **数据库备份**:涉及数据库变更,先执行 `mysqldump` 全量备份
|
||||
- [ ] **当前版本标记**:记录当前运行版本号或 Git commit hash
|
||||
|
||||
---
|
||||
|
||||
## 二、部署执行步骤
|
||||
|
||||
### 2.1 文件分发
|
||||
|
||||
```bash
|
||||
# 标准部署(SSH + scp/rsync)
|
||||
scp -P <port> ./dist/app root@<server>:/opt/app/
|
||||
# 或使用 rsync 增量同步
|
||||
rsync -avz --delete -e "ssh -p <port>" ./dist/ root@<server>:/opt/app/
|
||||
```
|
||||
|
||||
### 2.2 服务更新
|
||||
|
||||
#### 方式 A:systemd 服务
|
||||
```bash
|
||||
# 1. 停止服务
|
||||
systemctl stop <service-name>
|
||||
|
||||
# 2. 备份旧版本(如有必要)
|
||||
mv /opt/app/<app> /opt/app/<app>.bak
|
||||
|
||||
# 3. 放置新版本
|
||||
cp /tmp/<app> /opt/app/<app>
|
||||
chmod +x /opt/app/<app>
|
||||
|
||||
# 4. 重启服务
|
||||
systemctl start <service-name>
|
||||
systemctl status <service-name>
|
||||
```
|
||||
|
||||
#### 方式 B:Docker 容器
|
||||
```bash
|
||||
# 1. 拉取新镜像
|
||||
docker pull <registry>/<image>:<tag>
|
||||
|
||||
# 2. 停止旧容器
|
||||
docker stop <container-name>
|
||||
docker rm <container-name>
|
||||
|
||||
# 3. 启动新容器
|
||||
docker run -d --name <container-name> \
|
||||
--restart unless-stopped \
|
||||
-p <host-port>:<container-port> \
|
||||
<registry>/<image>:<tag>
|
||||
```
|
||||
|
||||
#### 方式 C:Nginx 反向代理更新
|
||||
```bash
|
||||
# 更新上游配置后重载
|
||||
nginx -t # 语法检查
|
||||
systemctl reload nginx # 热重载
|
||||
```
|
||||
|
||||
### 2.3 配置变更
|
||||
|
||||
```bash
|
||||
# 1. 备份当前配置
|
||||
cp /etc/<app>/config.yml /etc/<app>/config.yml.$(date +%Y%m%d-%H%M%S)
|
||||
|
||||
# 2. 修改配置
|
||||
vim /etc/<app>/config.yml
|
||||
|
||||
# 3. 重启服务使配置生效
|
||||
systemctl restart <service-name>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 三、部署验证
|
||||
|
||||
### 3.1 连通性验证
|
||||
```bash
|
||||
# 服务端口监听确认
|
||||
ss -tlnp | grep <port>
|
||||
|
||||
# HTTP 服务健康检查
|
||||
curl -s -o /dev/null -w "%{http_code}" http://localhost:<port>/health
|
||||
# 预期返回:200
|
||||
```
|
||||
|
||||
### 3.2 功能验证
|
||||
|
||||
- [ ] API 基础功能运行正常
|
||||
- [ ] 日志无新增 ERROR 级别报错
|
||||
- [ ] 数据库连接正常
|
||||
- [ ] 前端页面(如有)可正常加载
|
||||
|
||||
### 3.3 监控确认
|
||||
|
||||
- [ ] Prometheus / Grafana 指标正常
|
||||
- [ ] 日志系统(如有)已捕获新日志
|
||||
- [ ] 告警规则未被触发
|
||||
|
||||
---
|
||||
|
||||
## 四、回滚方案
|
||||
|
||||
### 4.1 代码回滚
|
||||
```bash
|
||||
# Git 回滚到上一版本
|
||||
cd /opt/app/repo
|
||||
git revert HEAD --no-edit
|
||||
git push
|
||||
# 重新执行部署
|
||||
```
|
||||
|
||||
### 4.2 文件回滚
|
||||
```bash
|
||||
# 恢复备份文件
|
||||
mv /opt/app/<app>.bak /opt/app/<app>
|
||||
systemctl restart <service-name>
|
||||
```
|
||||
|
||||
### 4.3 数据库回滚
|
||||
```bash
|
||||
# 导入备份
|
||||
gunzip < /backup/db/<dbname>.$(date +%Y%m%d).sql.gz | mysql -u root -p<pass> <dbname>
|
||||
```
|
||||
|
||||
### 4.4 回滚确认
|
||||
- [ ] 旧版本服务运行正常
|
||||
- [ ] 端口监听确认
|
||||
- [ ] 用户无访问异常
|
||||
- [ ] 记录回滚原因到工作日志
|
||||
|
||||
---
|
||||
|
||||
## 五、部署后记录
|
||||
|
||||
### 5.1 必填信息
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 部署时间 | YYYY-MM-DD HH:mm |
|
||||
| 部署人 | 严维序(opengineer) |
|
||||
| 部署内容 | [简要描述] |
|
||||
| 版本 | commit hash / tag |
|
||||
| 验证结果 | ✅/❌ 通过 |
|
||||
| 回滚情况 | 无需回滚 / 已回滚(原因) |
|
||||
|
||||
### 5.2 记录位置
|
||||
|
||||
- 工作日志:`memory/YYYY-MM-DD.md`
|
||||
- 任务记录:WorkBoard 相关卡片注释
|
||||
- 知识更新:如部署暴露流程问题,更新本文档
|
||||
|
||||
---
|
||||
|
||||
## 相关条目
|
||||
|
||||
- [故障排查手册_v1.0.md](故障排查手册_v1.0.md)
|
||||
- [服务器运维标准_v1.0.md](服务器运维标准_v1.0.md)
|
||||
|
||||
## 变更记录
|
||||
|
||||
| 日期 | 版本 | 变更说明 | 变更人 |
|
||||
|------|------|----------|--------|
|
||||
| 2026-06-24 | v1.0 | 初始创建 | 严维序 |
|
||||
@@ -1,9 +1,9 @@
|
||||
# BIZ-13 智能体运行稳定性保障方案
|
||||
|
||||
> 版本:v1.0
|
||||
> 版本:v1.1
|
||||
> 编制:陆怀瑾(COO)
|
||||
> 日期:2026-06-22
|
||||
> 状态:待审阅
|
||||
> 状态:Phase 1 执行中(Vincent 已审阅同意)
|
||||
|
||||
---
|
||||
|
||||
@@ -305,9 +305,10 @@ def retry_with_backoff(api_call, max_retries=3):
|
||||
## 七、实施步骤
|
||||
|
||||
### 阶段 1:心跳机制落地(本周)
|
||||
- [ ] 更新所有 Agent 的 HEARTBEAT.md
|
||||
- [ ] 配置定时任务(10 分钟)
|
||||
- [ ] 测试超时检测
|
||||
- [x] 更新所有 Agent 的 HEARTBEAT.md(15/15 Agent 已完成)
|
||||
- [x] 已创建分步实施子任务(BIZ-24 ~ BIZ-28,5个子任务)
|
||||
- [ ] 配置定时任务(10/15 分钟)→ BIZ-25,已分派 opengineer 严维序
|
||||
- [ ] 测试超时检测 → BIZ-24 执行中
|
||||
|
||||
### 阶段 2:限流优化(下周)
|
||||
- [ ] 实现请求队列
|
||||
|
||||
@@ -0,0 +1,835 @@
|
||||
# BIZ-24 HEARTBEAT.md 增强模板方案
|
||||
|
||||
> Phase 1 of BIZ-13 运行稳定性保障方案
|
||||
> 版本:v1.1(2026-06-22 优化:增加全任务源统一监控;已部署)
|
||||
> 编制:陆怀瑾(COO)
|
||||
> 日期:2026-06-22
|
||||
> 状态:已部署
|
||||
> 关联:[BIZ-13 运行稳定性保障方案](BIZ-13_运行稳定性保障方案.md)
|
||||
|
||||
---
|
||||
|
||||
## 一、目标
|
||||
|
||||
为所有 Agent 的 HEARTBEAT.md 文件统一增强以下机制,解决任务停滞、运行异常与工作遗漏问题:
|
||||
|
||||
1. **全任务源统一监控** — 覆盖 OpenClaw WorkBoard + Multica Issues + 待办文档,避免工作遗漏
|
||||
2. **禁止请示规则** — 消除"等待用户确认"导致的任务卡死
|
||||
3. **超时检测规则** — 按 Agent 类型差异化配置心跳频率
|
||||
4. **自动恢复规则** — 检测无进展时自动重新调度
|
||||
5. **依赖检查前置** — 任务启动前强制检查所有依赖
|
||||
6. **最大轮次限制** — 防止无限循环或资源耗尽
|
||||
|
||||
### 1.1 为什么需要全任务源统一监控
|
||||
|
||||
当前 Agent 工作面临的任务来源是多平台的:
|
||||
|
||||
| 任务来源 | 平台/工具 | 查询方式 | 当前监控状态 |
|
||||
|----------|-----------|----------|------------|
|
||||
| WorkBoard 卡片 | OpenClaw workboard | `openclaw workboard list` | ✅ 已纳入 |
|
||||
| 待办文档 | 各 Agent workspace 的 TODO.md / AGENTS.md | 文件读取 | ⚠️ 部分纳入 |
|
||||
| Multica Issues | Multica 平台 | `multica issue list --assignee-id <id>` | ❌ 未纳入 |
|
||||
|
||||
**问题**:Multica Issues 中分配给 Agent 的任务当前完全不在心跳监控范围内,Agent 可能永远不会发现并执行这些任务,导致工作永久遗漏。
|
||||
|
||||
**对策**:每次心跳同步检查以上三个来源,确保无一遗漏。
|
||||
|
||||
---
|
||||
|
||||
## 二、Agent 分类与参数配置
|
||||
|
||||
### 2.1 分类标准
|
||||
|
||||
| 分类 | 特征 | Agent |
|
||||
|------|------|-------|
|
||||
| 高频 Agent | 需频繁检查任务状态、全局监控 | secretary, coo |
|
||||
| 开发 Agent | 执行开发/设计/部署等长周期任务 | projectmanager, productmanager, architect, costcodev, designer, opengineer |
|
||||
| 业务 Agent | 执行专项业务任务 | taobaospecialist, contentspecialist, mediaspecialist, cvexpert, marketanalysis, lawyer |
|
||||
|
||||
### 2.2 参数配置矩阵
|
||||
|
||||
| 参数 | 高频 Agent | 开发 Agent | 业务 Agent |
|
||||
|------|-----------|-----------|-----------|
|
||||
| 心跳频率 | 10 分钟 | 15 分钟 | 15 分钟 |
|
||||
| 最大轮次 | 50 轮 | 100 轮 | 30 轮 |
|
||||
| 超时告警阈值 | 20 分钟无进展 | 30 分钟无进展 | 30 分钟无进展 |
|
||||
| 自动恢复等待 | 30 分钟后重新调度 | 45 分钟后重新调度 | 45 分钟后重新调度 |
|
||||
| 告警通知对象 | COO | COO + 创建者 | 创建者 |
|
||||
|
||||
---
|
||||
|
||||
## 三、六项增强规则详解
|
||||
|
||||
### 规则 0:全任务源统一监控
|
||||
|
||||
**问题**:Agent 的任务分布在多个平台(OpenClaw WorkBoard、Multica Issues、工作区待办文档),各平台独立存在,Agent 只监控其中一部分会导致工作任务被永久遗漏。
|
||||
|
||||
**规则文本**:
|
||||
```markdown
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
> 你的工作任务可能存在于三个地方:OpenClaw WorkBoard、Multica Issues、本地待办文档。
|
||||
|
||||
### 任务源检查清单(按优先级)
|
||||
|
||||
#### 第一优先级:OpenClaw WorkBoard 卡片
|
||||
|
||||
\```bash
|
||||
# 检查 WorkBoard 中分配给自己的待办卡片
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
my_cards = [c for c in data.get('cards', [])
|
||||
if c.get('agentId') == '<your_agent_id>' and c.get('status') == 'todo']
|
||||
for c in my_cards:
|
||||
print(f'WORKBOARD TODO: {c[\"id\"][:8]} [priority={c.get(\"priority\",\"?\")}] {c[\"title\"]}')
|
||||
"
|
||||
\```
|
||||
|
||||
#### 第二优先级:Multica Issues
|
||||
|
||||
\```bash
|
||||
# 检查 Multica 中分配给自己的待办 Issue
|
||||
multica issue list --assignee-id <your_multica_agent_uuid> --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
for issue in data:
|
||||
print(f'MULTICA TODO: {issue[\"identifier\"]} [{issue.get(\"priority\",\"?\")}] {issue[\"title\"]}')
|
||||
"
|
||||
\```
|
||||
|
||||
#### 第三优先级:待办文档
|
||||
|
||||
\```bash
|
||||
# 检查工作区待办文档(TODO.md 或 AGENTS.md 中未完成的任务)
|
||||
grep -n '\[ \]' TODO.md AGENTS.md 2>/dev/null || echo "无待办文档"
|
||||
\```
|
||||
|
||||
### 三源合并决策
|
||||
|
||||
```
|
||||
心跳开始
|
||||
↓
|
||||
检查 WorkBoard 待办卡片
|
||||
↓
|
||||
检查 Multica Issues 待办
|
||||
↓
|
||||
检查待办文档
|
||||
↓
|
||||
合并去重(避免同一任务在不同来源重复出现)
|
||||
↓
|
||||
按优先级排序后依次执行
|
||||
```
|
||||
|
||||
### Multica Issue 认领与执行流程
|
||||
|
||||
```
|
||||
发现 Multica todo Issue(assignee 是自己)
|
||||
↓
|
||||
启动 Multica Runtime 执行任务
|
||||
↓
|
||||
完成后通过 multica issue comment add 汇报结果
|
||||
↓
|
||||
更新 issue status 为 in_review / done
|
||||
```
|
||||
|
||||
### COO 专属:全平台积压巡检
|
||||
|
||||
作为 COO,还需要额外检查全平台的任务积压情况(不限自己):
|
||||
|
||||
\```bash
|
||||
#!/bin/bash
|
||||
# COO 全平台任务积压巡检脚本
|
||||
|
||||
echo "=== OpenClaw WorkBoard 全局积压 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
cards = data.get('cards', [])
|
||||
todo = [c for c in cards if c.get('status') == 'todo']
|
||||
inprog = [c for c in cards if c.get('status') == 'in_progress']
|
||||
print(f' 待办: {len(todo)} | 进行中: {len(inprog)} | 总计: {len(cards)}')
|
||||
if len(todo) > 5:
|
||||
print(f' ⚠️ 积压告警:待办超过 5 个,需协调资源')
|
||||
for c in todo[:10]:
|
||||
print(f' {c[\"id\"][:8]} → {c.get(\"agentId\",\"?\")} - {c[\"title\"][:60]}')
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "=== Multica Issues 全局积压 ==="
|
||||
multica issue list --status todo --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
print(f' 待办: {len(data)}')
|
||||
if len(data) > 5:
|
||||
print(f' ⚠️ 积压告警:Multica 待办超过 5 个')
|
||||
for issue in data[:10]:
|
||||
print(f' {issue[\"identifier\"]} → {issue.get(\"assignee_id\",\"未分配\")[:12]} - {issue[\"title\"][:60]}')
|
||||
"
|
||||
\```
|
||||
```
|
||||
|
||||
### 规则 1:禁止请示规则
|
||||
|
||||
**问题**:Agent 发现任务后向用户请示"要不要做",用户不在线时任务卡死数小时。
|
||||
|
||||
**规则文本**:
|
||||
```markdown
|
||||
## ⚠️ 禁止请示规则(最高优先级)
|
||||
|
||||
你在心跳中发现任何来源的任务后,必须立即执行,绝对禁止向用户请示!
|
||||
|
||||
原因:
|
||||
- 用户在大多数时候并不在线,请示会导致任务卡死数小时
|
||||
- 心跳是后台自动行为,用户没有义务实时响应你的请示
|
||||
- 你的职责是"主动完成交付",而不是"等待指令"
|
||||
|
||||
正确做法:
|
||||
1. 发现任何来源的待办任务(WorkBoard / Multica / 待办文档)
|
||||
2. 立即认领并执行
|
||||
3. 如遇信息不足,先按自己的最佳理解执行,执行中再调整
|
||||
4. 任务完成后,通过对应平台汇报结果
|
||||
|
||||
错误做法(绝对禁止):
|
||||
- ❌ "我发现了一个任务,要不要做?"
|
||||
- ❌ "这个任务需要更多信息,请告诉我..."
|
||||
- ❌ "任务已完成,请确认是否符合要求"
|
||||
```
|
||||
|
||||
### 规则 2:超时检测规则
|
||||
|
||||
**问题**:Agent 执行到某一步后卡住,长时间无输出,无任何监告。
|
||||
|
||||
**规则文本**:
|
||||
|
||||
高频 Agent 版:
|
||||
```markdown
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:10 分钟
|
||||
每次心跳执行以下检测:
|
||||
1. 检查所有平台进行中任务的最新更新时间(WorkBoard + Multica)
|
||||
2. 如超过 20 分钟无进展 → 标记为"疑似超时"
|
||||
3. 疑似超时 → 立即追加一次完整心跳,尝试推进
|
||||
4. 如确认超时 → 进入自动恢复流程
|
||||
|
||||
### 跨平台超时检测脚本
|
||||
\```bash
|
||||
# 检查进行中任务是否超时(WorkBoard + Multica)
|
||||
echo "=== WorkBoard 超时检测 ==="
|
||||
openclaw workboard list --json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
inprogress = [c for c in data.get('cards', []) if c.get('status') == 'in_progress']
|
||||
now = time.time()
|
||||
for c in inprogress:
|
||||
updated = c.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1200:
|
||||
print(f'⏰ WB TIMEOUT: {c[\"id\"][:8]} [{c.get(\"agentId\",\"?\")}] {c[\"title\"]}')
|
||||
"
|
||||
|
||||
echo "=== Multica 超时检测 ==="
|
||||
multica issue list --status in_progress --output json 2>/dev/null | python3 -c "
|
||||
import sys, json, time
|
||||
data = json.load(sys.stdin)
|
||||
now = time.time()
|
||||
for issue in data:
|
||||
updated = issue.get('updated_at', '')
|
||||
if updated:
|
||||
age = now - time.mktime(time.strptime(updated[:19], '%Y-%m-%dT%H:%M:%S'))
|
||||
if age > 1200:
|
||||
print(f'⏰ MUL TIMEOUT: {issue[\"identifier\"]} [{issue.get(\"assignee_id\",\"?\")[:12]}] {issue[\"title\"]}')
|
||||
"
|
||||
\```
|
||||
```
|
||||
|
||||
开发 Agent 版(差异部分):
|
||||
```markdown
|
||||
### 心跳频率:15 分钟
|
||||
每次心跳执行以下检测:
|
||||
1. 检查所有平台进行中任务的最新更新时间(WorkBoard + Multica)
|
||||
2. 如超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
```
|
||||
|
||||
业务 Agent 版(差异部分):
|
||||
```markdown
|
||||
### 心跳频率:15 分钟
|
||||
每次心跳执行以下检测:
|
||||
1. 检查所有平台进行中任务的最新更新时间(WorkBoard + Multica)
|
||||
2. 如超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
```
|
||||
|
||||
### 规则 3:自动恢复规则
|
||||
|
||||
**问题**:检测到无进展后没有自动恢复手段,任务永久停滞。
|
||||
|
||||
**规则文本**:
|
||||
```markdown
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 恢复流程
|
||||
```
|
||||
检测到超时(跨平台无进展超阈值)
|
||||
↓
|
||||
步骤 1:追加一次完整心跳,尝试推进任务
|
||||
↓
|
||||
步骤 2:检查任务状态
|
||||
↓
|
||||
┌─────────────┴─────────────┐
|
||||
│ │
|
||||
有进展 仍无进展
|
||||
│ │
|
||||
重置超时计数器 步骤 3:通知 COO/创建者
|
||||
│ │
|
||||
继续执行 步骤 4:通过对应平台标记 blocked
|
||||
│
|
||||
步骤 5:重新调度(分配备用 Agent 或
|
||||
等待人工介入)
|
||||
```
|
||||
|
||||
### 自动恢复触发条件
|
||||
- 高频 Agent:超 30 分钟无进展 → 自动重新调度
|
||||
- 开发 Agent:超 45 分钟无进展 → 自动重新调度
|
||||
- 业务 Agent:超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 跨平台恢复操作
|
||||
**WorkBoard 任务**:
|
||||
1. 添加评论说明超时原因
|
||||
2. 释放 Agent 认领(release claim)
|
||||
3. 通知 COO 重新分配
|
||||
|
||||
**Multica Issue**:
|
||||
1. `multica issue comment add` 说明超时原因
|
||||
2. `multica issue status <id> blocked`
|
||||
3. 通知 COO 重新分配
|
||||
|
||||
**待办文档任务**:
|
||||
1. 在原文档中标注超时状态
|
||||
2. 如可转为 WorkBoard 卡片 → 创建卡片并通知 COO
|
||||
```
|
||||
|
||||
### 规则 4:依赖检查前置
|
||||
|
||||
**问题**:任务开始后才发现依赖未满足,浪费 Agent 时间,且可能导致循环等待。
|
||||
|
||||
**规则文本**:
|
||||
```markdown
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 任务启动前强制检查
|
||||
每次认领或启动任务前,必须执行依赖检查:
|
||||
|
||||
**WorkBoard 任务**:
|
||||
1. 读取任务的 depends_on 字段
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 所有依赖 ready → 可以启动
|
||||
4. 任一依赖未完成 → 禁止启动,标记为 blocked
|
||||
|
||||
**Multica Issue**:
|
||||
1. 读取 issue 的 parent_issue_id
|
||||
2. 检查父 issue 状态
|
||||
3. 父 issue 未完成 → 禁止启动
|
||||
|
||||
### 检查脚本
|
||||
|
||||
#### WorkBoard 依赖检查
|
||||
\```bash
|
||||
openclaw workboard read <card-id> --json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
card = json.load(sys.stdin)
|
||||
deps = card.get('dependsOn', [])
|
||||
if deps:
|
||||
for dep in deps:
|
||||
print(f'依赖: {dep[\"id\"]} → 状态: {dep.get(\"status\", \"?\")}')
|
||||
if dep.get('status') != 'done':
|
||||
print(f'⛔ WB 依赖未满足,禁止启动 {card[\"id\"][:8]}')
|
||||
sys.exit(1)
|
||||
print('✅ 所有 WB 依赖已满足')
|
||||
else:
|
||||
print('✅ 无 WB 依赖,可以启动')
|
||||
"
|
||||
\```
|
||||
|
||||
#### Multica 依赖检查
|
||||
\```bash
|
||||
multica issue get <issue-id> --output json 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
issue = json.load(sys.stdin)
|
||||
parent_id = issue.get('parent_issue_id')
|
||||
if parent_id:
|
||||
import subprocess
|
||||
result = subprocess.run(['multica', 'issue', 'get', parent_id, '--output', 'json'],
|
||||
capture_output=True, text=True)
|
||||
parent = json.loads(result.stdout)
|
||||
if parent.get('status') != 'done':
|
||||
print(f'⛔ MUL 父 Issue {parent[\"identifier\"]} 未完成,禁止启动')
|
||||
sys.exit(1)
|
||||
print(f'✅ 父 Issue {parent[\"identifier\"]} 已完成')
|
||||
else:
|
||||
print('✅ 无父 Issue 依赖,可以启动')
|
||||
"
|
||||
\```
|
||||
|
||||
### 依赖未满足时的处理
|
||||
1. 不认领任务(保持 todo 状态)
|
||||
2. 不在该任务上浪费心跳时间
|
||||
3. 如超过等待阈值(高频 1h / 开发/业务 2h),通知依赖任务的执行者
|
||||
```
|
||||
|
||||
### 规则 5:最大轮次限制
|
||||
|
||||
**问题**:Agent 陷入无限循环,反复执行相同逻辑无进展,持续消耗 API 配额。
|
||||
|
||||
**规则文本**:
|
||||
|
||||
高频 Agent 版:
|
||||
```markdown
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:50 轮
|
||||
单次任务执行不得超过 50 个对话轮次。
|
||||
|
||||
### 检测机制
|
||||
- 每次心跳记录已消耗轮次
|
||||
- 接近上限(80%)时发出预警
|
||||
- 达到上限时自动暂停
|
||||
|
||||
### 超限处理
|
||||
```
|
||||
达到最大轮次
|
||||
↓
|
||||
1. 暂停任务执行
|
||||
2. 记录已完成的步骤和未完成的部分
|
||||
3. 通知 COO,附当前进度
|
||||
4. COO 决定:重新分配 / 拆分任务 / 人工介入
|
||||
```
|
||||
|
||||
### 跨平台轮次跟踪
|
||||
- **WorkBoard**:通过 workboard_heartbeat 的 note 记录轮次
|
||||
- **Multica**:通过 issue comment 记录轮次进度
|
||||
- **待办文档**:在工作日志中记录
|
||||
|
||||
### 计数器维护
|
||||
\```bash
|
||||
# 每次心跳更新轮次计数
|
||||
# 轮次数据存储在任务 metadata 或 comment 中
|
||||
\```
|
||||
```
|
||||
|
||||
开发 Agent 版(差异部分):
|
||||
```markdown
|
||||
### 限制值:100 轮
|
||||
单次任务执行不得超过 100 个对话轮次。
|
||||
```
|
||||
|
||||
业务 Agent 版(差异部分):
|
||||
```markdown
|
||||
### 限制值:30 轮
|
||||
单次任务执行不得超过 30 个对话轮次。
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 四、HEARTBEAT.md 完整增强模板
|
||||
|
||||
### 4.1 高频 Agent 完整模板(secretary / coo)
|
||||
|
||||
```markdown
|
||||
# HEARTBEAT.md - [Agent Name] 的心跳配置
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
1. **OpenClaw WorkBoard** — workboard list → 查找 agent_id=自己 & status=todo
|
||||
2. **Multica Issues** — issue list --assignee-id <uuid> --status todo
|
||||
3. **待办文档** — 检查 TODO.md / AGENTS.md 中未完成的任务
|
||||
|
||||
### 合并去重 → 按优先级排序 → 依次执行
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:10 分钟
|
||||
每次心跳跨平台执行以下检测:
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 20 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
- 超 30 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(1h)→ 通知依赖任务执行者
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:50 轮
|
||||
- 接近 80%(40 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知 COO
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ 全平台积压巡检(仅 COO)
|
||||
6. ✅ [Agent 专属检查项]
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
```
|
||||
|
||||
### 4.2 开发 Agent 完整模板(projectmanager / productmanager / architect / costcodev / designer / opengineer)
|
||||
|
||||
```markdown
|
||||
# HEARTBEAT.md - [Agent Name] 的心跳配置
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
1. **OpenClaw WorkBoard** — workboard list → 查找 agent_id=自己 & status=todo
|
||||
2. **Multica Issues** — issue list --assignee-id <uuid> --status todo
|
||||
3. **待办文档** — 检查 TODO.md / AGENTS.md 中未完成的任务
|
||||
|
||||
### 合并去重 → 按优先级排序 → 依次执行
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:15 分钟
|
||||
每次心跳跨平台执行以下检测:
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知 COO + 创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知 COO + 创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(2h)→ 通知依赖任务执行者
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:100 轮
|
||||
- 接近 80%(80 轮)→ 预警
|
||||
- 达到上限 → 暂停,记录日志
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ [Agent 专属检查项]
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
```
|
||||
|
||||
### 4.3 业务 Agent 完整模板(taobaospecialist / contentspecialist / mediaspecialist / cvexpert / marketanalysis / lawyer)
|
||||
|
||||
```markdown
|
||||
# HEARTBEAT.md - [Agent Name] 的心跳配置
|
||||
|
||||
## 📋 全任务源统一监控(每次心跳必检)
|
||||
|
||||
> **核心原则:发现任何来源的任务都必须立即执行,不得遗漏。**
|
||||
|
||||
### 三源检查
|
||||
1. **OpenClaw WorkBoard** — workboard list → 查找 agent_id=自己 & status=todo
|
||||
2. **Multica Issues** — issue list --assignee-id <uuid> --status todo
|
||||
3. **待办文档** — 检查 TODO.md / AGENTS.md 中未完成的任务
|
||||
|
||||
### 合并去重 → 按优先级排序 → 依次执行
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 禁止请示规则(最高优先级,自动执行)
|
||||
|
||||
> **核心规则:发现分配给自己的任务 → 立即执行,不得向用户请示!**
|
||||
> 心跳是后台自动行为,用户在大多数时候并不在线,请示会导致任务卡死。
|
||||
|
||||
### 执行流程(严格执行,无需确认)
|
||||
1. **三源检查待办** — WorkBoard + Multica + 待办文档
|
||||
2. **立即执行,不得请示** — 发现待办后直接执行
|
||||
3. **检查进行中任务** — 确认认领的任务状态
|
||||
4. **完成任务** — 通过对应平台汇报结果
|
||||
|
||||
### ⚠️ 绝对禁止行为
|
||||
- ❌ 不得问"要不要做这个任务"
|
||||
- ❌ 不得等用户确认再执行
|
||||
- ❌ 不得以"需要更多信息"为由拒绝执行
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ 超时检测规则
|
||||
|
||||
### 心跳频率:15 分钟
|
||||
每次心跳跨平台执行以下检测:
|
||||
1. 检查 WorkBoard 进行中任务的更新时间
|
||||
2. 检查 Multica 进行中 issues 的更新时间
|
||||
3. 超过 30 分钟无进展 → 标记为"疑似超时"
|
||||
4. 疑似超时 → 追加一次完整心跳尝试推进
|
||||
5. 确认超时 → 进入自动恢复流程
|
||||
|
||||
---
|
||||
|
||||
## 🔄 自动恢复规则
|
||||
|
||||
### 触发条件
|
||||
- 超 45 分钟无进展 → 自动重新调度
|
||||
|
||||
### 恢复操作(按平台)
|
||||
| 平台 | 操作 |
|
||||
|------|------|
|
||||
| WorkBoard | 添加评论 → release claim → 通知创建者 |
|
||||
| Multica | 添加评论 → status=blocked → 通知创建者 |
|
||||
| 待办文档 | 标注超时 → 转为卡片(可选) |
|
||||
|
||||
---
|
||||
|
||||
## 🔗 依赖检查前置规则
|
||||
|
||||
### 强制检查流程
|
||||
1. 认领任务前,读取依赖字段(depends_on / parent_issue_id)
|
||||
2. 逐一检查每个依赖任务的状态
|
||||
3. 依赖未满足 → 不认领(保持 todo)
|
||||
4. 超过等待阈值(2h)→ 通知依赖任务执行者
|
||||
|
||||
---
|
||||
|
||||
## 🛑 最大轮次限制
|
||||
|
||||
### 限制值:30 轮
|
||||
- 接近 80%(24 轮)→ 预警
|
||||
- 达到上限 → 暂停,通知创建者
|
||||
|
||||
---
|
||||
|
||||
## 🫀 心跳执行清单
|
||||
|
||||
1. ✅ **全任务源检查**:WorkBoard + Multica + 待办文档
|
||||
2. ✅ 进行中任务超时检测(跨平台)
|
||||
3. ✅ 依赖检查
|
||||
4. ✅ 轮次计数器更新
|
||||
5. ✅ [Agent 专属检查项]
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 用户正在对话时延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户询问
|
||||
3. **发现任务立即执行,不得请示**(任何来源)
|
||||
4. **超时任务按自动恢复流程处理**(跨平台)
|
||||
5. **依赖未满足不启动**
|
||||
6. **达到轮次上限自动暂停**
|
||||
7. **避免任务遗漏** — 三源必须全部检查,缺一不可
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 五、部署清单
|
||||
|
||||
### 5.1 各 Agent HEARTBEAT.md 更新状态
|
||||
|
||||
| Agent | 分类 | 模板版本 | 部署状态 | 部署人 |
|
||||
|-------|------|---------|---------|--------|
|
||||
| secretary (刘诗妮) | 高频 | 高频 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| coo (陆怀瑾) | 高频 | 高频 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| projectmanager (胡蓉) | 开发 | 开发 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| productmanager (沈路明) | 开发 | 开发 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| architect (梁思筑) | 开发 | 开发 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| costcodev (徐聪) | 开发 | 开发 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| designer (苏绘锦) | 开发 | 开发 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| opengineer (严维序) | 开发 | 开发 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| taobaospecialist (陆云帆) | 业务 | 业务 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| contentspecialist (文墨言) | 业务 | 业务 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| mediaspecialist (钟帧韵) | 业务 | 业务 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| cvexpert (程伯予) | 业务 | 业务 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| marketanalysis (顾析策) | 业务 | 业务 Agent 模板 v1.1 | 待部署 | COO |
|
||||
| lawyer (苏慎) | 业务 | 业务 Agent 模板 v1.1 | 待部署 | COO |
|
||||
|
||||
### 5.2 部署步骤
|
||||
|
||||
1. **Vincent 审阅本方案** — 确认参数配置和多源监控范围
|
||||
2. **收集各 Agent 的 Multica UUID** — 用于 `multica issue list --assignee-id <uuid>` 查询
|
||||
3. **创建 HEARTBEAT.md 文件** — 按 v1.1 模板为每个 Agent 创建(填充实际 ID)
|
||||
4. **配置心跳 cron** — 按分类配置定时任务
|
||||
5. **部署到各 Agent workspace** — 将 HEARTBEAT.md 分发到对应 Agent 工作区
|
||||
6. **验证** — 等待一轮完整心跳,检查三源任务是否全量覆盖
|
||||
|
||||
### 5.3 Agent Multica UUID 映射(已收集)
|
||||
|
||||
| Agent | OpenClaw Agent ID | Multica Agent UUID | 状态 |
|
||||
|-------|-------------------|-------------------|------|
|
||||
| secretary (刘诗妮) | secretary | b024fcdc-30ff-420d-b289-498041466e1b | ✅ |
|
||||
| coo (陆怀瑾) | coo | 1c38b437-b54d-4784-bda3-29ce4c8a6722 | ✅ |
|
||||
| projectmanager (胡蓉) | projectmanager | d877b8c3-b230-4073-b3f7-80e148cfdb71 | ✅ |
|
||||
| productmanager (沈路明) | productmanager | a101fa88-d821-4839-9754-e04580d5fd68 | ✅ |
|
||||
| architect (梁思筑) | architect | 40abd41a-62d0-416d-bc44-92c1f758d87a | ✅ |
|
||||
| costcodev (徐聪) | costcodev | 46bdd4a6-5c64-475a-92ef-36a763602fa1 | ✅ |
|
||||
| designer (苏锦绘) | designer | 13bd8968-cc2a-4934-90c7-957a2d3c09c2 | ✅ |
|
||||
| opengineer (严维序) | opengineer | d3804433-9e2e-4199-a92b-a153049b3bc9 | ✅ |
|
||||
| taobaospecialist (陆云帆) | taobaospecialist | e0f62d8f-9568-4f41-8ad4-b73d79a163a7 | ✅ |
|
||||
| contentspecialist (文墨言) | contentspecialist | 8321b0bf-7d89-4ece-927a-0780f42ad396 | ✅ |
|
||||
| mediaspecialist (钟帧韵) | mediaspecialist | e2b587d4-1d16-447c-8ad9-e2a01358ff0a | ✅ |
|
||||
| cvexpert (程伯予) | cvexpert | 4a8696fd-6531-40da-8956-ef84d7ea3c43 | ✅ |
|
||||
| marketanalysis (顾析策) | marketanalysis | 5ed91729-658f-4654-98f0-3e0313022002 | ✅ |
|
||||
| lawyer (苏慎) | lawyer | 6fb0fbd2-16a6-4566-ba7a-d2c136baec25 | ✅ |
|
||||
|
||||
---
|
||||
|
||||
## 六、交付物
|
||||
|
||||
- [x] HEARTBEAT.md 增强模板方案 v1.0(初始版本)
|
||||
- [x] HEARTBEAT.md 增强模板方案 v1.1(优化:增加全任务源统一监控)
|
||||
- [x] 各 Agent Multica UUID 映射表
|
||||
- [x] 14 个 Agent 的独立 HEARTBEAT.md 文件(v1.1,已生成并部署到 workspace)
|
||||
- [ ] 心跳 cron 配置脚本
|
||||
- [ ] 部署验证报告
|
||||
|
||||
---
|
||||
|
||||
## 七、v1.1 变更说明
|
||||
|
||||
| 变更项 | v1.0 | v1.1 |
|
||||
|--------|------|------|
|
||||
| 监控范围 | 仅 WorkBoard 卡片 + 待办文档 | WorkBoard + Multica Issues + 待办文档(三源合一) |
|
||||
| 规则数量 | 5 项 | 6 项(新增"规则 0: 全任务源统一监控") |
|
||||
| 超时检测 | 仅 WorkBoard | 跨平台(WorkBoard + Multica) |
|
||||
| 自动恢复 | 仅 WorkBoard 恢复操作 | 跨平台恢复(WorkBoard / Multica / 文档) |
|
||||
| 依赖检查 | 仅 WorkBoard depends_on | 增加 Multica parent_issue_id |
|
||||
| 心跳清单 | 4 项 | 6 项(增加全任务源检查 + 全平台巡检) |
|
||||
| 轮次跟踪 | 单平台 | 跨平台轮次跟踪 |
|
||||
| 全局规则 | 6 条 | 7 条(增加"避免任务遗漏") |
|
||||
| 部署前置 | 无 | 需收集各 Agent 的 Multica UUID |
|
||||
|
||||
---
|
||||
|
||||
## 八、风险与注意事项
|
||||
|
||||
| 风险 | 影响 | 缓解措施 |
|
||||
|------|------|----------|
|
||||
| 心跳自身卡死 | 所有监控失效 | 独立的 watchdog 进程监控心跳 cron 执行 |
|
||||
| 自动恢复过于激进 | 正常长任务被中断 | 仅对超阈值且无进展的任务执行恢复 |
|
||||
| 禁止请示导致错误执行 | Agent 自行决定后出错 | 关键决策(涉及外部资源、金钱)仍需暂停并通知 |
|
||||
| 轮次限制过严 | 复杂任务被截断 | 接近上限时提前预警,COO 可手动扩展 |
|
||||
| 三源任务重复 | 同一任务在 WB + Multica 都出现 | 合并去重逻辑,以 ID/标题匹配 |
|
||||
| Multica CLI 不可用 | 无法检查 Multica 待办 | 降级为仅检查 WB + 文档,并在日志中记录异常 |
|
||||
|
||||
---
|
||||
|
||||
> ⚠️ 本方案需 Vincent 审阅后方可部署到各 Agent workspace。当前为模板方案 v1.1,存放于 EnterpriseArchitect/plans/ 目录。
|
||||
@@ -0,0 +1,186 @@
|
||||
# HEARTBEAT.md 增强模板
|
||||
|
||||
> 版本:v2.0
|
||||
> 来源:BIZ-13 运行稳定性保障方案
|
||||
> 用途:为所有 Agent HEARTBEAT.md 增加运行稳定性保障能力
|
||||
|
||||
---
|
||||
|
||||
## 全局增强规则(所有 Agent 必须包含)
|
||||
|
||||
### 1. 🛡️ 超时检测与自动恢复
|
||||
|
||||
```markdown
|
||||
## 🛡️ 超时检测与自动恢复
|
||||
|
||||
> **核心规则:每次心跳,检查自己是否有任务超时未完成。**
|
||||
|
||||
### 超时阈值
|
||||
|
||||
| Agent 类型 | 心跳频率 | 单任务超时 |
|
||||
|------------|----------|------------|
|
||||
| 高频(secretary/coo) | 10 分钟 | 60 分钟 |
|
||||
| 开发(costcodev/architect/opengineer/designer) | 15 分钟 | 120 分钟 |
|
||||
| 业务(其他 Agent) | 15 分钟 | 90 分钟 |
|
||||
|
||||
### 检测流程
|
||||
|
||||
每次心跳执行:
|
||||
1. 获取自己的 `status=in_progress` 的 WorkBoard 卡片
|
||||
2. 计算 `当前时间 - started_at`
|
||||
3. 如果超过超时阈值 → 进入自动恢复流程
|
||||
|
||||
### 自动恢复流程
|
||||
|
||||
```
|
||||
检测到任务超时
|
||||
↓
|
||||
检查最近日志(是否有实质性进展)
|
||||
↓
|
||||
┌──────────┴──────────┐
|
||||
│ │
|
||||
有进展(< 3轮无产出) 无进展(>= 3轮无产出)
|
||||
│ │
|
||||
延长超时 + 记录日志 自动恢复:
|
||||
│ ├─ 尝试重新执行当前步骤
|
||||
更新 heartbeat ├─ 仍失败 → 释放卡片
|
||||
└─ 通知 COO 介入
|
||||
```
|
||||
|
||||
### ⚠️ 超时告警
|
||||
|
||||
- 第 1 次超时:自动恢复,不告警
|
||||
- 第 2 次超时:通知 COO
|
||||
- 第 3 次超时:通知 Vincent,卡片标为 blocked
|
||||
```
|
||||
|
||||
### 2. 🔗 依赖检查前置
|
||||
|
||||
```markdown
|
||||
## 🔗 依赖检查前置
|
||||
|
||||
> **核心规则:认领任务前,必须检查所有依赖是否已完成。**
|
||||
|
||||
### 检查流程
|
||||
|
||||
1. 获取任务的 `depends_on` 列表
|
||||
2. 对每个依赖,查询其状态
|
||||
3. 如果任一依赖未完成 → 不认领该任务,等待下次心跳
|
||||
4. 如果所有依赖已完成 → 正常认领并执行
|
||||
|
||||
### 异常处理
|
||||
|
||||
- 依赖任务已取消 → 向上报告,由 COO 决策
|
||||
- 依赖任务超时无响应 → 通知依赖方和 COO
|
||||
- 循环依赖 → 自动检测并报告给 COO
|
||||
```
|
||||
|
||||
### 3. 🔄 最大轮次限制
|
||||
|
||||
```markdown
|
||||
## 🔄 最大轮次限制
|
||||
|
||||
> **核心规则:单任务不能无限循环执行。**
|
||||
|
||||
| Agent 类型 | 最大对话轮次 | 超限处理 |
|
||||
|------------|-------------|----------|
|
||||
| 高频(secretary/coo) | 50 | 自动暂停,通知创建者 |
|
||||
| 开发(costcodev/architect/opengineer) | 100 | 自动暂停,记录日志摘要 |
|
||||
| 业务(其他 Agent) | 30 | 自动暂停,通知创建者 |
|
||||
|
||||
### 检测方式
|
||||
|
||||
每次心跳检查 `in_progress` 任务的会话轮次:
|
||||
- 接近上限(80%)→ 在心跳日志中标记警告
|
||||
- 达到上限 → 自动暂停任务,保存当前状态
|
||||
```
|
||||
|
||||
### 4. 📊 上下文控制
|
||||
|
||||
```markdown
|
||||
## 📊 上下文控制(Token 管理)
|
||||
|
||||
> **核心规则:避免上下文溢出导致任务中断。**
|
||||
|
||||
### 策略
|
||||
|
||||
1. **引用代替填塞**:Agent 配置文件中只保留核心规则,详细信息存 docs/ 目录
|
||||
2. **分块读取**:超大文件分块读取,避免一次性加载
|
||||
3. **清理过期信息**:每轮对话前清理上一轮的工具输出(仅保留关键结果)
|
||||
4. **合并查询**:多个 Agent 相同查询由 COO 统一执行后广播
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 心跳频率分级
|
||||
|
||||
```markdown
|
||||
## ⏱️ 心跳触发频率
|
||||
|
||||
- **高频 Agent(secretary / coo)**: 每 10 分钟
|
||||
- **开发 Agent(costcodev / architect / opengineer / designer)**: 每 15 分钟
|
||||
- **业务 Agent(projectmanager / productmanager / taobaospecialist / contentspecialist / mediaspecialist / cvexpert / marketanalysis / lawyer)**: 每 15 分钟
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 全局关键规则(增强版)
|
||||
|
||||
```markdown
|
||||
## ⚠️ 全局关键规则
|
||||
|
||||
1. **心跳不打断对话** — 如果用户正在与 Agent 对话,心跳逻辑延后执行
|
||||
2. **非紧急事项延后汇报** — 等下一轮心跳或用户主动询问时再汇报
|
||||
3. **发现任务立即执行,不得请示** — 用户在大多数时候不在线,请示=任务卡死
|
||||
4. **依赖检查前置** — 认领任务前必须检查所有依赖是否已完成
|
||||
5. **超时自动恢复** — 任务超时自动尝试恢复,3 次失败后升级
|
||||
6. **轮次限制** — 单任务达上限后自动暂停,防止无限循环
|
||||
7. **上下文控制** — 引用代替填塞,避免 Token 溢出
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 各 Agent 类型模板
|
||||
|
||||
### 高频 Agent 模板(secretary/coo)
|
||||
|
||||
在原有专属心跳清单基础上,增加:
|
||||
```markdown
|
||||
### 🛡️ 稳定性保障清单
|
||||
|
||||
1. ✅ 超时检测:检查 in_progress 任务是否超时(阈值 60 分钟)
|
||||
2. ✅ 依赖检查:新任务认领前检查所有 depends_on
|
||||
3. ✅ 轮次检查:当前任务是否接近 50 轮上限
|
||||
4. ✅ 上下文检查:HEARTBEAT.md/AGENTS.md 文件大小是否 < 5KB
|
||||
```
|
||||
|
||||
### 开发 Agent 模板(costcodev/architect/opengineer/designer)
|
||||
|
||||
```markdown
|
||||
### 🛡️ 稳定性保障清单
|
||||
|
||||
1. ✅ 超时检测:检查 in_progress 任务是否超时(阈值 120 分钟)
|
||||
2. ✅ 依赖检查:新任务认领前检查所有 depends_on
|
||||
3. ✅ 轮次检查:当前任务是否接近 100 轮上限
|
||||
4. ✅ 编译/测试检查:如有自动化测试,确认通过
|
||||
```
|
||||
|
||||
### 业务 Agent 模板(其他 Agent)
|
||||
|
||||
```markdown
|
||||
### 🛡️ 稳定性保障清单
|
||||
|
||||
1. ✅ 超时检测:检查 in_progress 任务是否超时(阈值 90 分钟)
|
||||
2. ✅ 依赖检查:新任务认领前检查所有 depends_on
|
||||
3. ✅ 轮次检查:当前任务是否接近 30 轮上限
|
||||
4. ✅ 输出质量检查:确认最近产出符合质量标准
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 实施说明
|
||||
|
||||
1. 此模板由 COO(陆怀瑾)编制,经 Vincent 审阅批准后实施
|
||||
2. 模板中的 agent_id 需替换为各 Agent 的实际标识
|
||||
3. 无需移除各 Agent 原有的专属心跳清单,只需追加稳定性保障清单
|
||||
4. 修改后的文件需提交到 EnterpriseArchitect git 仓库
|
||||
@@ -0,0 +1,46 @@
|
||||
# Sidecar V2 — Multi-Pool Provider Proxy
|
||||
FROM python:3.12-slim AS builder
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install dependencies
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir --upgrade pip && \
|
||||
pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# Copy application code
|
||||
COPY config.py crypto.py main.py server.py proxy.py router.py \
|
||||
pool_manager.py cooldown_manager.py rate_limiter.py __init__.py \
|
||||
dashboard.html ./
|
||||
COPY storage/ ./storage/
|
||||
|
||||
# Create data directory
|
||||
RUN mkdir -p /app/data /app/data/backups
|
||||
|
||||
FROM python:3.12-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Copy built artifacts
|
||||
COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
|
||||
COPY --from=builder /app /app
|
||||
|
||||
# Environment
|
||||
ENV SIDECAR_HOST=0.0.0.0
|
||||
ENV SIDECAR_PORT=9190
|
||||
ENV SIDECAR_METRICS_PORT=9191
|
||||
ENV SIDECAR_DB_PATH=/app/data/sidecar_v2.db
|
||||
ENV SIDECAR_BACKUP_DIR=/app/data/backups
|
||||
ENV SIDECAR_ENCRYPTION_KEY=
|
||||
ENV SIDECAR_ADMIN_TOKEN=
|
||||
ENV LOG_FORMAT=json
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
|
||||
EXPOSE 9190 9191
|
||||
|
||||
VOLUME ["/app/data"]
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
|
||||
CMD python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:9190/health')" || exit 1
|
||||
|
||||
ENTRYPOINT ["python3", "main.py"]
|
||||
@@ -0,0 +1,77 @@
|
||||
# Sidecar V2 — Multi-Pool Provider Proxy
|
||||
|
||||
## 概述
|
||||
Sidecar V2 是 OpenClaw 的 API 代理服务,实现多 Provider 池管理、负载均衡、429 冷却、RPM 队列控流。
|
||||
|
||||
## 核心功能
|
||||
- **Provider 池管理**:主池 (primary) + 备用池 (fallback),支持动态增删 Provider
|
||||
- **429 冷却**:检测 429 → 自动冷却 → 指数退避 → 自动恢复
|
||||
- **按 Provider 独立 RPM 限流**:每个 Provider 独立的 Token Bucket
|
||||
- **路由策略**:主池优先 → 备用池兜底 → 全部耗尽返 503
|
||||
- **WebUI 管理**:Dashboard 仪表盘 + Provider CRUD
|
||||
- **用量统计**:Token 用量 + 费用统计 + 每小时/每日聚合
|
||||
- **API Key 加密**:AES-256-GCM 加密存储
|
||||
|
||||
## 架构
|
||||
|
||||
```
|
||||
OpenClaw → Sidecar V2 (port 9190) → 路由 → 主池 Provider 1,2,3...
|
||||
↘ 备池 Provider 4,5...
|
||||
↘ 全部耗尽 → 503
|
||||
```
|
||||
|
||||
## 快速开始
|
||||
|
||||
```bash
|
||||
# 设置加密密钥 (64位十六进制)
|
||||
export SIDECAR_ENCRYPTION_KEY="0000111122223333444455556666777788889999aaaabbbbccccddddeeeeffff"
|
||||
|
||||
# 启动服务
|
||||
python3 main.py
|
||||
|
||||
# OR via uvicorn
|
||||
python3 -m uvicorn server:app --host 127.0.0.1 --port 9190
|
||||
```
|
||||
|
||||
## WebUI
|
||||
访问 http://127.0.0.1:9190/dashboard
|
||||
|
||||
## API 端点
|
||||
|
||||
### Admin API
|
||||
- `GET /api/admin/backends` — 列出所有 Provider
|
||||
- `POST /api/admin/backends` — 添加 Provider
|
||||
- `PUT /api/admin/backends/{id}` — 更新 Provider
|
||||
- `DELETE /api/admin/backends/{id}` — 删除 Provider
|
||||
- `GET /api/admin/pools` — 池状态汇总
|
||||
- `GET /api/admin/stats/total` — 总计统计
|
||||
- `GET /api/admin/stats/hourly` — 每小时用量
|
||||
- `GET /api/admin/stats/daily` — 每日聚合
|
||||
- `GET /api/admin/stats/cooldown` — 冷却事件历史
|
||||
- `GET /api/admin/config` — 系统配置
|
||||
|
||||
### 代理 API (OpenAI 兼容)
|
||||
- `POST /v1/chat/completions`
|
||||
- `POST /v1/completions`
|
||||
- `POST /v1/embeddings`
|
||||
- `GET /v1/models`
|
||||
|
||||
### 监控
|
||||
- `GET /health` — 健康检查
|
||||
- `GET /dashboard/sse` — Dashboard 实时数据流 (SSE)
|
||||
|
||||
## 环境变量
|
||||
|
||||
| 变量 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| SIDECAR_HOST | 127.0.0.1 | 监听地址 |
|
||||
| SIDECAR_PORT | 9190 | 监听端口 |
|
||||
| SIDECAR_ENCRYPTION_KEY | (必填) | API Key 加密密钥 (64 hex chars) |
|
||||
| SIDECAR_DB_PATH | ./data/sidecar_v2.db | SQLite 数据库路径 |
|
||||
| SIDECAR_RATE_RPM | 40 | 默认 RPM 限制 |
|
||||
| SIDECAR_COOLDOWN_BASE | 30 | 冷却基础时长 (秒) |
|
||||
| SIDECAR_COOLDOWN_MAX | 600 | 冷却最大时长 (秒) |
|
||||
|
||||
## 存储
|
||||
- SQLite (WAL 模式)
|
||||
- 表:backends, backend_usage_logs, cooldown_events, backend_health, system_config, daily_stats
|
||||
@@ -0,0 +1 @@
|
||||
"""Sidecar V2 — Multi-pool provider proxy with cooldown, rate limiting, and WebUI management."""
|
||||
@@ -0,0 +1,165 @@
|
||||
"""System configuration management for Sidecar V2."""
|
||||
|
||||
import os
|
||||
import json
|
||||
from dataclasses import dataclass, field, asdict
|
||||
from typing import Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class Config:
|
||||
"""Sidecar V2 runtime configuration.
|
||||
|
||||
Sources (priority order):
|
||||
1. Environment variables (highest)
|
||||
2. system_config table in SQLite
|
||||
3. Defaults defined here
|
||||
"""
|
||||
|
||||
# Listen
|
||||
host: str = "127.0.0.1"
|
||||
port: int = 9190
|
||||
metrics_port: int = 9191
|
||||
|
||||
# Queue
|
||||
queue_max_depth: int = 500
|
||||
queue_timeout_seconds: float = 30.0
|
||||
|
||||
# Provider
|
||||
default_rpm_limit: int = 40
|
||||
|
||||
# Cooldown
|
||||
cooldown_base_seconds: float = 30.0
|
||||
cooldown_max_seconds: float = 600.0
|
||||
cooldown_exponential_backoff: bool = True
|
||||
|
||||
# Emergency channel: RPM fraction when all pools exhausted
|
||||
emergency_rpm_fraction: float = 0.10
|
||||
|
||||
# Health check
|
||||
health_check_interval_seconds: int = 60
|
||||
health_check_timeout_seconds: int = 10
|
||||
health_probe_endpoint: str = "/v1/models"
|
||||
|
||||
# Admin auth
|
||||
admin_token: str = ""
|
||||
|
||||
# Encryption
|
||||
encryption_key: str = ""
|
||||
|
||||
# Logging
|
||||
log_level: str = "INFO"
|
||||
|
||||
# Database
|
||||
db_path: str = ""
|
||||
backup_dir: str = ""
|
||||
backup_retention_days: int = 7
|
||||
|
||||
# Rate limiter
|
||||
rate_limiter_refill_interval_ms: int = 50
|
||||
|
||||
# Router
|
||||
router_refresh_interval_seconds: float = 5.0
|
||||
|
||||
# Max pool-internal retries
|
||||
max_pool_retries: int = 5
|
||||
|
||||
# Pre-check cooldown threshold (seconds remaining)
|
||||
cooldown_precheck_threshold_seconds: float = 10.0
|
||||
|
||||
# Dashboard
|
||||
dashboard_sse_interval_seconds: float = 1.0
|
||||
|
||||
# Stats
|
||||
stats_refresh_interval_seconds: float = 30.0
|
||||
|
||||
# Request timeout
|
||||
default_request_timeout_seconds: int = 120
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> "Config":
|
||||
"""Load configuration from environment variables."""
|
||||
c = cls()
|
||||
|
||||
# Listen
|
||||
c.host = os.getenv("SIDECAR_HOST", c.host)
|
||||
c.port = int(os.getenv("SIDECAR_PORT", str(c.port)))
|
||||
c.metrics_port = int(os.getenv("SIDECAR_METRICS_PORT", str(c.metrics_port)))
|
||||
|
||||
# Queue
|
||||
c.queue_max_depth = int(os.getenv("SIDECAR_QUEUE_MAX", str(c.queue_max_depth)))
|
||||
c.queue_timeout_seconds = float(
|
||||
os.getenv("SIDECAR_QUEUE_TIMEOUT", str(c.queue_timeout_seconds))
|
||||
)
|
||||
|
||||
# Provider
|
||||
c.default_rpm_limit = int(
|
||||
os.getenv("SIDECAR_RATE_RPM", str(c.default_rpm_limit))
|
||||
)
|
||||
|
||||
# Cooldown
|
||||
c.cooldown_base_seconds = float(
|
||||
os.getenv("SIDECAR_COOLDOWN_BASE", str(c.cooldown_base_seconds))
|
||||
)
|
||||
c.cooldown_max_seconds = float(
|
||||
os.getenv("SIDECAR_COOLDOWN_MAX", str(c.cooldown_max_seconds))
|
||||
)
|
||||
|
||||
# Admin
|
||||
c.admin_token = os.getenv("SIDECAR_ADMIN_TOKEN", c.admin_token)
|
||||
|
||||
# Encryption
|
||||
c.encryption_key = os.getenv("SIDECAR_ENCRYPTION_KEY", c.encryption_key)
|
||||
|
||||
# Logging
|
||||
c.log_level = os.getenv("LOG_LEVEL", c.log_level).upper()
|
||||
|
||||
# Database
|
||||
c.db_path = os.getenv(
|
||||
"SIDECAR_DB_PATH",
|
||||
os.path.join(os.getcwd(), "data", "sidecar_v2.db"),
|
||||
)
|
||||
c.backup_dir = os.getenv(
|
||||
"SIDECAR_BACKUP_DIR",
|
||||
os.path.join(os.getcwd(), "data", "backups"),
|
||||
)
|
||||
|
||||
# V1 compatibility: migrate env vars
|
||||
c._migrate_v1_env()
|
||||
|
||||
return c
|
||||
|
||||
def _migrate_v1_env(self) -> None:
|
||||
"""Migrate V1 environment variables to V2 defaults."""
|
||||
# V1 UPSTREAM endpoint
|
||||
upstream = os.getenv("SIDECAR_UPSTREAM")
|
||||
api_key = os.getenv("SIDECAR_API_KEY")
|
||||
if api_key and self.encryption_key:
|
||||
# These will be used during initial migration
|
||||
os.environ["_SIDECAR_V1_API_KEY"] = api_key
|
||||
os.environ["_SIDECAR_V1_UPSTREAM"] = upstream or "https://integrate.api.nvidia.com/v1"
|
||||
|
||||
def to_db_dict(self) -> dict:
|
||||
"""Serialize to dict for system_config storage."""
|
||||
result = {}
|
||||
for key, value in asdict(self).items():
|
||||
if isinstance(value, bool):
|
||||
result[key] = "true" if value else "false"
|
||||
elif isinstance(value, (int, float)):
|
||||
result[key] = str(value)
|
||||
else:
|
||||
result[key] = value
|
||||
return result
|
||||
|
||||
@classmethod
|
||||
def merge_db(cls, base: "Config", db_config: dict) -> "Config":
|
||||
"""Merge DB config into base config (env vars already applied to base)."""
|
||||
for key, value in base.__dict__.items():
|
||||
if key in db_config and key not in os.environ:
|
||||
# DB values only apply when no env var override
|
||||
setattr(base, key, type(value)(db_config[key]))
|
||||
return base
|
||||
|
||||
|
||||
# Singleton
|
||||
config = Config.from_env()
|
||||
@@ -0,0 +1,114 @@
|
||||
"""429 Cooldown management for backends using exponential backoff."""
|
||||
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
import structlog
|
||||
from config import config
|
||||
from storage.backend_store import set_backend_cooldown, clear_backend_cooldown, get_backend
|
||||
from storage.cooldown_store import log_cooldown_event, end_cooldown_event
|
||||
|
||||
logger = structlog.get_logger("sidecar_v2.cooldown_manager")
|
||||
|
||||
|
||||
def calculate_cooldown(consecutive_count: int) -> float:
|
||||
"""Calculate cooldown duration using exponential backoff.
|
||||
|
||||
Formula: base * 2^(consecutive-1), capped at max.
|
||||
"""
|
||||
base = config.cooldown_base_seconds
|
||||
max_seconds = config.cooldown_max_seconds
|
||||
if config.cooldown_exponential_backoff:
|
||||
duration = base * (2 ** (consecutive_count - 1))
|
||||
else:
|
||||
duration = base * consecutive_count
|
||||
return min(duration, max_seconds)
|
||||
|
||||
|
||||
def start_cooldown(backend_id: str, consecutive_count: int) -> float:
|
||||
"""Start cooldown for a backend after 429.
|
||||
|
||||
Returns: cooldown end timestamp.
|
||||
"""
|
||||
duration = calculate_cooldown(consecutive_count)
|
||||
cooldown_until_ts = time.time() + duration
|
||||
cooldown_until = time.strftime(
|
||||
"%Y-%m-%dT%H:%M:%SZ", time.gmtime(cooldown_until_ts)
|
||||
)
|
||||
|
||||
set_backend_cooldown(backend_id, cooldown_until, consecutive_count)
|
||||
log_cooldown_event(
|
||||
backend_id=backend_id,
|
||||
consecutive_count=consecutive_count,
|
||||
cooldown_seconds=int(duration),
|
||||
response_summary=f"429 cooldown triggered (consecutive #{consecutive_count})",
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"cooldown_started",
|
||||
backend_id=backend_id,
|
||||
duration=round(duration, 1),
|
||||
consecutive=consecutive_count,
|
||||
)
|
||||
return duration
|
||||
|
||||
|
||||
def check_and_clear_cooldown(backend_id: str) -> bool:
|
||||
"""Check if cooldown has expired for a backend.
|
||||
|
||||
Returns True if cooldown was cleared (backend is back online).
|
||||
"""
|
||||
backend = get_backend(backend_id, decrypt_key=False)
|
||||
if backend is None:
|
||||
return False
|
||||
|
||||
if backend.status != "cooling":
|
||||
return False
|
||||
|
||||
cooldown_until = backend.cooldown_until
|
||||
if not cooldown_until:
|
||||
clear_backend_cooldown(backend_id)
|
||||
return True
|
||||
|
||||
# Parse cooldown_until as ISO timestamp
|
||||
try:
|
||||
dt = datetime.fromisoformat(cooldown_until.replace("Z", "+00:00"))
|
||||
cooldown_ts = dt.timestamp()
|
||||
except ValueError:
|
||||
# If parsing fails, clear and move on
|
||||
clear_backend_cooldown(backend_id)
|
||||
return True
|
||||
|
||||
now = time.time()
|
||||
if now >= cooldown_ts:
|
||||
clear_backend_cooldown(backend_id)
|
||||
end_cooldown_event(backend_id)
|
||||
logger.info("cooldown_cleared", backend_id=backend_id)
|
||||
return True
|
||||
|
||||
remaining = cooldown_ts - now
|
||||
logger.debug("cooldown_active", backend_id=backend_id, remaining_seconds=round(remaining, 1))
|
||||
return False
|
||||
|
||||
|
||||
def precheck_cooldown(backend_id: str) -> bool:
|
||||
"""Check if backend should be skipped due to near-expiry cooldown.
|
||||
|
||||
If cooldown will expire within config.cooldown_precheck_threshold_seconds,
|
||||
skip the backend so we don't hit it again right as it expires.
|
||||
"""
|
||||
backend = get_backend(backend_id, decrypt_key=False)
|
||||
if backend is None or backend.status != "cooling":
|
||||
return False
|
||||
|
||||
cooldown_until = backend.cooldown_until
|
||||
if not cooldown_until:
|
||||
return False
|
||||
|
||||
try:
|
||||
dt = datetime.fromisoformat(cooldown_until.replace("Z", "+00:00"))
|
||||
cooldown_ts = dt.timestamp()
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
remaining = cooldown_ts - time.time()
|
||||
return 0 < remaining <= config.cooldown_precheck_threshold_seconds
|
||||
@@ -0,0 +1,108 @@
|
||||
"""AES-256-GCM encryption for API Key storage."""
|
||||
|
||||
import os
|
||||
import secrets
|
||||
import structlog
|
||||
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
|
||||
|
||||
logger = structlog.get_logger()
|
||||
|
||||
_ENCRYPTION_KEY: bytes | None = None
|
||||
_cipher: AESGCM | None = None
|
||||
|
||||
|
||||
def init_crypto(hex_key: str) -> None:
|
||||
"""Initialize the encryption module.
|
||||
|
||||
Validates the key and prepares the cipher.
|
||||
Raises ValueError if key is invalid.
|
||||
"""
|
||||
global _ENCRYPTION_KEY, _cipher
|
||||
|
||||
if not hex_key:
|
||||
raise ValueError("FATAL: SIDECAR_ENCRYPTION_KEY not set")
|
||||
|
||||
if len(hex_key) != 64:
|
||||
raise ValueError(
|
||||
f"FATAL: SIDECAR_ENCRYPTION_KEY must be 64 hex chars (32 bytes), "
|
||||
f"got {len(hex_key)} chars"
|
||||
)
|
||||
|
||||
try:
|
||||
key_bytes = bytes.fromhex(hex_key)
|
||||
except ValueError:
|
||||
raise ValueError(
|
||||
"FATAL: SIDECAR_ENCRYPTION_KEY must be valid hexadecimal"
|
||||
)
|
||||
|
||||
global _ENCRYPTION_KEY, _cipher
|
||||
_ENCRYPTION_KEY = key_bytes
|
||||
_cipher = AESGCM(key_bytes)
|
||||
logger.info("crypto_initialized")
|
||||
|
||||
|
||||
def encrypt(plaintext: str) -> str:
|
||||
"""Encrypt plaintext using AES-256-GCM.
|
||||
|
||||
Returns: hex-encoded nonce (12 bytes) + ciphertext + tag.
|
||||
Format: <nonce_hex>:<ciphertext_hex>
|
||||
"""
|
||||
if _cipher is None:
|
||||
raise RuntimeError("Crypto not initialized. Call init_crypto() first.")
|
||||
|
||||
nonce = secrets.token_bytes(12)
|
||||
ciphertext = _cipher.encrypt(nonce, plaintext.encode("utf-8"), None)
|
||||
return nonce.hex() + ":" + ciphertext.hex()
|
||||
|
||||
|
||||
def decrypt(encrypted: str) -> str:
|
||||
"""Decrypt AES-256-GCM ciphertext.
|
||||
|
||||
Args:
|
||||
encrypted: Format "<nonce_hex>:<ciphertext_hex>"
|
||||
|
||||
Returns: Decrypted plaintext string.
|
||||
"""
|
||||
if _cipher is None:
|
||||
raise RuntimeError("Crypto not initialized. Call init_crypto() first.")
|
||||
|
||||
parts = encrypted.split(":", 1)
|
||||
if len(parts) != 2:
|
||||
raise ValueError("Invalid encrypted format: expected nonce:ciphertext")
|
||||
|
||||
nonce = bytes.fromhex(parts[0])
|
||||
ciphertext = bytes.fromhex(parts[1])
|
||||
|
||||
try:
|
||||
plaintext = _cipher.decrypt(nonce, ciphertext, None)
|
||||
return plaintext.decode("utf-8")
|
||||
except Exception as e:
|
||||
raise ValueError(f"Decryption failed: {e}")
|
||||
|
||||
|
||||
def is_initialized() -> bool:
|
||||
"""Check if crypto has been initialized."""
|
||||
return _cipher is not None
|
||||
|
||||
|
||||
def mask_api_key(api_key_plain: str) -> str:
|
||||
"""Mask API key for display: show first 6 + last 4 chars."""
|
||||
if len(api_key_plain) <= 10:
|
||||
return api_key_plain[:2] + "****"
|
||||
return api_key_plain[:6] + "****" + api_key_plain[-4:]
|
||||
|
||||
|
||||
def try_decrypt_existing(encrypted_value: str) -> str | None:
|
||||
"""Try to decrypt an existing encrypted value.
|
||||
|
||||
Returns the plaintext if successful, None if decryption fails
|
||||
(e.g., encryption key was changed).
|
||||
"""
|
||||
try:
|
||||
return decrypt(encrypted_value)
|
||||
except Exception:
|
||||
logger.warning(
|
||||
"decrypt_existing_failed",
|
||||
hint="Encryption key may have been changed, existing keys unrecoverable"
|
||||
)
|
||||
return None
|
||||
@@ -0,0 +1,623 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Sidecar V2 — Provider Pool Dashboard</title>
|
||||
<!-- Primary: jsDelivr CDN -->
|
||||
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js"></script>
|
||||
<!-- Fallback: local static copy for offline/intranet deployments -->
|
||||
<script>
|
||||
(function() {
|
||||
var check = function() {
|
||||
if (typeof Chart === 'undefined') {
|
||||
var s = document.createElement('script');
|
||||
s.src = '/static/chart.umd.min.js';
|
||||
s.onerror = function() {
|
||||
console.warn('Chart.js unavailable (CDN + local both failed). Charts disabled.');
|
||||
};
|
||||
document.head.appendChild(s);
|
||||
}
|
||||
};
|
||||
// Check after CDN script has had a chance to load
|
||||
setTimeout(check, 2000);
|
||||
})();
|
||||
</script>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #0f1117;
|
||||
--card-bg: #1a1d28;
|
||||
--border: #2a2d3a;
|
||||
--text: #e0e0e0;
|
||||
--text-dim: #888;
|
||||
--green: #23d160;
|
||||
--yellow: #ffdd57;
|
||||
--red: #ff3860;
|
||||
--blue: #3273dc;
|
||||
--purple: #b86bff;
|
||||
--cyan: #00d1b2;
|
||||
--orange: #ff8533;
|
||||
}
|
||||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
min-height: 100vh;
|
||||
}
|
||||
|
||||
/* Layout */
|
||||
.app { display: flex; height: 100vh; }
|
||||
.sidebar {
|
||||
width: 220px; background: var(--card-bg); border-right: 1px solid var(--border);
|
||||
padding: 20px 0; display: flex; flex-direction: column;
|
||||
}
|
||||
.sidebar h2 { padding: 0 20px 20px; font-size: 16px; color: var(--cyan); border-bottom: 1px solid var(--border); }
|
||||
.sidebar nav { flex: 1; padding: 10px 0; }
|
||||
.sidebar nav a {
|
||||
display: block; padding: 10px 20px; color: var(--text-dim); text-decoration: none;
|
||||
font-size: 13px; transition: 0.2s;
|
||||
}
|
||||
.sidebar nav a:hover, .sidebar nav a.active { color: var(--text); background: rgba(255,255,255,0.05); }
|
||||
.sidebar .status-bar { padding: 15px 20px; border-top: 1px solid var(--border); font-size: 11px; color: var(--text-dim); }
|
||||
|
||||
.main { flex: 1; overflow-y: auto; padding: 24px; }
|
||||
.page { display: none; }
|
||||
.page.active { display: block; }
|
||||
|
||||
/* Dashboard Cards */
|
||||
.cards { display: grid; grid-template-columns: repeat(auto-fit, minmax(200px, 1fr)); gap: 16px; margin-bottom: 24px; }
|
||||
.card {
|
||||
background: var(--card-bg); border: 1px solid var(--border); border-radius: 8px; padding: 16px;
|
||||
}
|
||||
.card .label { font-size: 12px; color: var(--text-dim); text-transform: uppercase;letter-spacing:0.5px;margin-bottom:6px; }
|
||||
.card .value { font-size: 28px; font-weight: 700; }
|
||||
.card .sub { font-size: 12px; color: var(--text-dim); margin-top: 4px; }
|
||||
|
||||
.charts { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; }
|
||||
.chart-card {
|
||||
background: var(--card-bg); border: 1px solid var(--border); border-radius: 8px; padding: 16px;
|
||||
}
|
||||
.chart-card h3 { font-size: 14px; margin-bottom: 12px; color: var(--text-dim); }
|
||||
.chart-card canvas { max-height: 250px; }
|
||||
|
||||
/* Pool Cards */
|
||||
.pool-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; margin-bottom: 24px; }
|
||||
.pool-card {
|
||||
background: var(--card-bg); border: 1px solid var(--border); border-radius: 8px; padding: 16px;
|
||||
}
|
||||
.pool-card h3 { font-size: 15px; margin-bottom: 12px; text-transform: uppercase; letter-spacing: 1px; }
|
||||
.pool-card h3.primary { color: var(--blue); }
|
||||
.pool-card h3.fallback { color: var(--orange); }
|
||||
.pool-stats { display: grid; grid-template-columns: repeat(4, 1fr); gap: 8px; }
|
||||
.pool-stat { text-align: center; }
|
||||
.pool-stat .num { font-size: 22px; font-weight: 700; }
|
||||
.pool-stat .lbl { font-size: 11px; color: var(--text-dim); margin-top: 2px; }
|
||||
.pool-stat.healthy .num { color: var(--green); }
|
||||
.pool-stat.cooling .num { color: var(--yellow); }
|
||||
.pool-stat.error .num { color: var(--red); }
|
||||
.pool-stat.total .num { color: var(--purple); }
|
||||
|
||||
/* Tables */
|
||||
table { width: 100%; border-collapse: collapse; background: var(--card-bg); border-radius: 8px; overflow: hidden; }
|
||||
th { text-align: left; padding: 10px 12px; font-size: 11px; text-transform: uppercase; letter-spacing: 0.5px; color: var(--text-dim); background: rgba(255,255,255,0.03); border-bottom: 1px solid var(--border); }
|
||||
td { padding: 10px 12px; font-size: 13px; border-bottom: 1px solid var(--border); }
|
||||
tr:last-child td { border-bottom: none; }
|
||||
tr:hover { background: rgba(255,255,255,0.02); }
|
||||
.badge {
|
||||
display: inline-block; padding: 2px 8px; border-radius: 10px; font-size: 11px; font-weight: 600;
|
||||
}
|
||||
.badge.healthy { background: rgba(35,209,96,0.15); color: var(--green); }
|
||||
.badge.cooling { background: rgba(255,221,87,0.15); color: var(--yellow); }
|
||||
.badge.error { background: rgba(255,56,96,0.15); color: var(--red); }
|
||||
.badge.disabled { background: rgba(136,136,136,0.15); color: var(--text-dim); }
|
||||
.badge.primary { background: rgba(50,115,220,0.15); color: var(--blue); }
|
||||
.badge.fallback { background: rgba(255,133,51,0.15); color: var(--orange); }
|
||||
|
||||
/* Buttons */
|
||||
.btn {
|
||||
padding: 6px 14px; border-radius: 6px; border: none; cursor: pointer; font-size: 12px; font-weight: 600;
|
||||
transition: 0.2s;
|
||||
}
|
||||
.btn-primary { background: var(--blue); color: #fff; }
|
||||
.btn-primary:hover { opacity: 0.85; }
|
||||
.btn-danger { background: var(--red); color: #fff; }
|
||||
.btn-danger:hover { opacity: 0.85; }
|
||||
.btn-sm { padding: 3px 10px; font-size: 11px; }
|
||||
.btn-outline { background: transparent; border: 1px solid var(--border); color: var(--text); }
|
||||
.btn-outline:hover { background: rgba(255,255,255,0.05); }
|
||||
|
||||
.section-header { display: flex; justify-content: space-between; align-items: center; margin-bottom: 12px; }
|
||||
.section-header h3 { font-size: 15px; }
|
||||
|
||||
/* Modal */
|
||||
.modal-overlay { display: none; position: fixed; top: 0; left: 0; right: 0; bottom: 0; background: rgba(0,0,0,0.7); z-index: 100; justify-content: center; align-items: center; }
|
||||
.modal-overlay.active { display: flex; }
|
||||
.modal { background: var(--card-bg); border: 1px solid var(--border); border-radius: 12px; padding: 24px; width: 560px; max-height: 80vh; overflow-y: auto; }
|
||||
.modal h3 { margin-bottom: 16px; font-size: 16px; }
|
||||
.form-group { margin-bottom: 12px; }
|
||||
.form-group label { display: block; font-size: 12px; color: var(--text-dim); margin-bottom: 4px; }
|
||||
.form-group input, .form-group select, .form-group textarea {
|
||||
width: 100%; padding: 8px 10px; background: var(--bg); border: 1px solid var(--border);
|
||||
border-radius: 6px; color: var(--text); font-size: 13px;
|
||||
}
|
||||
.form-group textarea { min-height: 80px; font-family: monospace; font-size: 12px; }
|
||||
.form-row { display: grid; grid-template-columns: 1fr 1fr; gap: 12px; }
|
||||
.form-actions { display: flex; gap: 8px; justify-content: flex-end; margin-top: 16px; }
|
||||
.model-mapping-row { display: flex; gap: 8px; align-items: center; margin-bottom: 8px; }
|
||||
.model-mapping-row input { flex: 1; }
|
||||
|
||||
/* Utility */
|
||||
.text-green { color: var(--green); }
|
||||
.text-red { color: var(--red); }
|
||||
.text-dim { color: var(--text-dim); }
|
||||
.mb-16 { margin-bottom: 16px; }
|
||||
.mb-24 { margin-bottom: 24px; }
|
||||
|
||||
@media (max-width: 768px) {
|
||||
.charts, .pool-grid { grid-template-columns: 1fr; }
|
||||
.sidebar { display: none; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="app">
|
||||
<!-- Sidebar -->
|
||||
<aside class="sidebar">
|
||||
<h2>🚀 Sidecar V2</h2>
|
||||
<nav>
|
||||
<a href="#" data-page="dashboard" class="active">📊 Dashboard</a>
|
||||
<a href="#" data-page="providers">🔌 Providers</a>
|
||||
<a href="#" data-page="usage">📈 Usage Stats</a>
|
||||
<a href="#" data-page="cooldown">🧊 Cooldown Log</a>
|
||||
</nav>
|
||||
<div class="status-bar" id="status-bar">Connected · Sidecar V2</div>
|
||||
</aside>
|
||||
|
||||
<!-- Main Content -->
|
||||
<main class="main">
|
||||
<!-- Dashboard Page -->
|
||||
<div class="page active" id="page-dashboard">
|
||||
<div class="cards" id="stat-cards"></div>
|
||||
<div class="pool-grid" id="pool-grid"></div>
|
||||
<div class="charts" id="charts"></div>
|
||||
</div>
|
||||
|
||||
<!-- Providers Page -->
|
||||
<div class="page" id="page-providers">
|
||||
<div class="section-header">
|
||||
<h3>Provider Backends</h3>
|
||||
<button class="btn btn-primary" onclick="showAddBackend()">+ Add Provider</button>
|
||||
</div>
|
||||
<table id="backends-table">
|
||||
<thead>
|
||||
<tr><th>Name</th><th>Label</th><th>Pool</th><th>Status</th><th>RPM</th><th>Models</th><th>Actions</th></tr>
|
||||
</thead>
|
||||
<tbody></tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- Usage Page -->
|
||||
<div class="page" id="page-usage">
|
||||
<div class="section-header"><h3>Hourly Usage</h3></div>
|
||||
<div class="mb-16">
|
||||
<select id="usage-backend-filter" onchange="loadUsage()" class="btn btn-outline btn-sm">
|
||||
<option value="">All Backends</option>
|
||||
</select>
|
||||
</div>
|
||||
<table id="usage-table">
|
||||
<thead>
|
||||
<tr><th>Hour</th><th>Backend</th><th>Model</th><th>Requests</th><th>Errors</th><th>Tokens</th><th>Cost</th><th>Avg Latency</th></tr>
|
||||
</thead>
|
||||
<tbody></tbody>
|
||||
</table>
|
||||
|
||||
<div class="section-header mt-24 mb-16"><h3>Daily Aggregation</h3></div>
|
||||
<table id="daily-table">
|
||||
<thead>
|
||||
<tr><th>Date</th><th>Pool</th><th>Requests</th><th>Errors</th><th>Tokens</th><th>Cost</th><th>Backends</th></tr>
|
||||
</thead>
|
||||
<tbody></tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- Cooldown Page -->
|
||||
<div class="page" id="page-cooldown">
|
||||
<div class="section-header"><h3>Cooldown Event History</h3></div>
|
||||
<table id="cooldown-table">
|
||||
<thead>
|
||||
<tr><th>Time</th><th>Backend</th><th>Consecutive 429s</th><th>Duration</th><th>Summary</th></tr>
|
||||
</thead>
|
||||
<tbody></tbody>
|
||||
</table>
|
||||
</div>
|
||||
</main>
|
||||
</div>
|
||||
|
||||
<!-- Add/Edit Backend Modal -->
|
||||
<div class="modal-overlay" id="backend-modal">
|
||||
<div class="modal">
|
||||
<h3 id="modal-title">Add Provider</h3>
|
||||
<form id="backend-form" onsubmit="saveBackend(event)">
|
||||
<input type="hidden" id="backend-id">
|
||||
<div class="form-row">
|
||||
<div class="form-group">
|
||||
<label>Name *</label>
|
||||
<input type="text" id="backend-name" placeholder="e.g. NVIDIA H100 Primary" required>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label>Label</label>
|
||||
<input type="text" id="backend-label" placeholder="e.g. nvidia, siliconflow">
|
||||
</div>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label>API Base URL *</label>
|
||||
<input type="url" id="backend-url" placeholder="https://integrate.api.nvidia.com/v1" required>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label>API Key *</label>
|
||||
<input type="password" id="backend-key" placeholder="sk-..." required>
|
||||
</div>
|
||||
<div class="form-row">
|
||||
<div class="form-group">
|
||||
<label>Pool</label>
|
||||
<select id="backend-pool">
|
||||
<option value="primary">Primary</option>
|
||||
<option value="fallback">Fallback</option>
|
||||
</select>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label>RPM Limit</label>
|
||||
<input type="number" id="backend-rpm" value="40" min="1" max="1000">
|
||||
</div>
|
||||
</div>
|
||||
<div class="form-row">
|
||||
<div class="form-group">
|
||||
<label>Timeout (seconds)</label>
|
||||
<input type="number" id="backend-timeout" value="120" min="10" max="600">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label>Enabled</label>
|
||||
<select id="backend-enabled">
|
||||
<option value="true">Yes</option>
|
||||
<option value="false">No</option>
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label>Model Mappings (JSON: canonical → {native_id, cost, ...})</label>
|
||||
<textarea id="backend-mappings" placeholder='{"deepseek-ai/DeepSeek-V4-Pro":{"native_id":"deepseek-ai/deepseek-v4-pro","cost":{"input":0.000001,"output":0.000004}}}'></textarea>
|
||||
</div>
|
||||
<div class="form-actions">
|
||||
<button type="button" class="btn btn-outline" onclick="closeModal()">Cancel</button>
|
||||
<button type="submit" class="btn btn-primary">Save</button>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
// ── Navigation ──
|
||||
document.querySelectorAll('.sidebar nav a').forEach(a => {
|
||||
a.addEventListener('click', e => {
|
||||
e.preventDefault();
|
||||
document.querySelectorAll('.sidebar nav a').forEach(l => l.classList.remove('active'));
|
||||
a.classList.add('active');
|
||||
document.querySelectorAll('.page').forEach(p => p.classList.remove('active'));
|
||||
document.getElementById('page-' + a.dataset.page).classList.add('active');
|
||||
loadPage(a.dataset.page);
|
||||
});
|
||||
});
|
||||
|
||||
// ── SSE Connection ──
|
||||
const sse = new EventSource('/dashboard/sse');
|
||||
sse.onmessage = e => {
|
||||
const data = JSON.parse(e.data);
|
||||
if (data.type === 'snapshot') updateDashboard(data);
|
||||
};
|
||||
|
||||
sse.onerror = () => {
|
||||
document.getElementById('status-bar').textContent = '⚠️ SSE Disconnected';
|
||||
};
|
||||
|
||||
// ── Dashboard Update ──
|
||||
let costChart = null, tokenChart = null;
|
||||
|
||||
function updateDashboard(data) {
|
||||
document.getElementById('status-bar').textContent =
|
||||
`⚡ Connected · Uptime ${formatDuration(data.uptime_seconds)}`;
|
||||
|
||||
// Stat cards
|
||||
const st = data.total || {};
|
||||
const errRate = st.total_requests > 0 ? ((st.total_errors || 0) / st.total_requests * 100).toFixed(1) : '0.0';
|
||||
document.getElementById('stat-cards').innerHTML = `
|
||||
<div class="card"><div class="label">Total Requests</div><div class="value">${fmt(st.total_requests)}</div><div class="sub">Error rate: ${errRate}%</div></div>
|
||||
<div class="card"><div class="label">Total Tokens</div><div class="value">${fmt(st.total_tokens)}</div><div class="sub">Prompt: ${fmt(st.total_prompt_tokens)} · Completion: ${fmt(st.total_completion_tokens)}</div></div>
|
||||
<div class="card"><div class="label">Total Cost</div><div class="value">$${st.total_cost ? st.total_cost.toFixed(4) : '0.0000'}</div><div class="sub">USD</div></div>
|
||||
<div class="card"><div class="label">Uptime</div><div class="value">${formatDuration(data.uptime_seconds)}</div><div class="sub">Sidecar V2</div></div>
|
||||
`;
|
||||
|
||||
// Pool grid
|
||||
let poolHTML = '';
|
||||
for (const [pool, ps] of Object.entries(data.pool || {})) {
|
||||
poolHTML += `
|
||||
<div class="pool-card">
|
||||
<h3 class="${pool}">${pool}</h3>
|
||||
<div class="pool-stats">
|
||||
<div class="pool-stat total"><div class="num">${ps.total}</div><div class="lbl">Total</div></div>
|
||||
<div class="pool-stat healthy"><div class="num">${ps.healthy}</div><div class="lbl">Healthy</div></div>
|
||||
<div class="pool-stat cooling"><div class="num">${ps.cooling}</div><div class="lbl">Cooling</div></div>
|
||||
<div class="pool-stat error"><div class="num">${ps.error}</div><div class="lbl">Error</div></div>
|
||||
</div>
|
||||
</div>`;
|
||||
}
|
||||
document.getElementById('pool-grid').innerHTML = poolHTML || '<div class="card">No pools configured</div>';
|
||||
|
||||
// Update backend table if on providers page
|
||||
if (document.getElementById('page-providers').classList.contains('active')) {
|
||||
renderBackendsTable(data.backends || []);
|
||||
}
|
||||
}
|
||||
|
||||
// ── Chart Updates (use SSE data to build chart data) ──
|
||||
function initCharts() {
|
||||
const cc = document.getElementById('cost-chart');
|
||||
const tc = document.getElementById('token-chart');
|
||||
if (!cc || !tc) return;
|
||||
|
||||
if (costChart) costChart.destroy();
|
||||
if (tokenChart) tokenChart.destroy();
|
||||
|
||||
costChart = new Chart(cc, {
|
||||
type: 'line', data: { labels: [], datasets: [{ label: 'Cost (USD)', data: [], borderColor: '#00d1b2', backgroundColor: 'rgba(0,209,178,0.1)', fill: true, tension: 0.3 }] },
|
||||
options: { responsive: true, maintainAspectRatio: true, plugins: { legend: { labels: { color: '#888' } } }, scales: { x: { ticks: { color: '#888', maxTicksLimit: 12 } }, y: { ticks: { color: '#888' } } } }
|
||||
});
|
||||
|
||||
tokenChart = new Chart(tc, {
|
||||
type: 'line', data: { labels: [], datasets: [{ label: 'Total Tokens', data: [], borderColor: '#b86bff', backgroundColor: 'rgba(184,107,255,0.1)', fill: true, tension: 0.3 }] },
|
||||
options: { responsive: true, maintainAspectRatio: true, plugins: { legend: { labels: { color: '#888' } } }, scales: { x: { ticks: { color: '#888', maxTicksLimit: 12 } }, y: { ticks: { color: '#888' } } } }
|
||||
});
|
||||
}
|
||||
|
||||
// ── Providers Page ──
|
||||
function renderBackendsTable(backends) {
|
||||
const tbody = document.querySelector('#backends-table tbody');
|
||||
tbody.innerHTML = backends.map(b => `
|
||||
<tr>
|
||||
<td><strong>${h(b.name)}</strong></td>
|
||||
<td><span class="badge ${b.label ? 'primary' : ''}">${h(b.label || '-')}</span></td>
|
||||
<td><span class="badge ${b.pool}">${b.pool}</span></td>
|
||||
<td><span class="badge ${b.status}">${b.status}</span></td>
|
||||
<td>${b.rpm_limit}</td>
|
||||
<td>${b.model_count || 0}</td>
|
||||
<td>
|
||||
<button class="btn btn-outline btn-sm" onclick="editBackend('${b.id}')">Edit</button>
|
||||
<button class="btn btn-danger btn-sm" onclick="deleteBackend('${b.id}')">Del</button>
|
||||
</td>
|
||||
</tr>`).join('');
|
||||
}
|
||||
|
||||
function showAddBackend() {
|
||||
document.getElementById('modal-title').textContent = 'Add Provider';
|
||||
document.getElementById('backend-id').value = '';
|
||||
document.getElementById('backend-name').value = '';
|
||||
document.getElementById('backend-label').value = '';
|
||||
document.getElementById('backend-url').value = '';
|
||||
document.getElementById('backend-key').value = '';
|
||||
document.getElementById('backend-pool').value = 'primary';
|
||||
document.getElementById('backend-rpm').value = '40';
|
||||
document.getElementById('backend-timeout').value = '120';
|
||||
document.getElementById('backend-enabled').value = 'true';
|
||||
document.getElementById('backend-mappings').value = '{}';
|
||||
document.getElementById('backend-modal').classList.add('active');
|
||||
}
|
||||
|
||||
async function editBackend(id) {
|
||||
try {
|
||||
const res = await fetch('/api/admin/backends/' + id);
|
||||
const b = await res.json();
|
||||
document.getElementById('modal-title').textContent = 'Edit Provider';
|
||||
document.getElementById('backend-id').value = b.id;
|
||||
document.getElementById('backend-name').value = b.name;
|
||||
document.getElementById('backend-label').value = b.label || '';
|
||||
document.getElementById('backend-url').value = b.api_base_url;
|
||||
document.getElementById('backend-key').value = '';
|
||||
document.getElementById('backend-key').placeholder = '(leave blank to keep current)';
|
||||
document.getElementById('backend-key').required = false;
|
||||
document.getElementById('backend-pool').value = b.pool;
|
||||
document.getElementById('backend-rpm').value = b.rpm_limit;
|
||||
document.getElementById('backend-timeout').value = b.timeout_seconds;
|
||||
document.getElementById('backend-enabled').value = b.enabled ? 'true' : 'false';
|
||||
document.getElementById('backend-mappings').value = JSON.stringify(b.model_mappings || {}, null, 2);
|
||||
document.getElementById('backend-modal').classList.add('active');
|
||||
} catch (e) { alert('Failed to load backend: ' + e.message); }
|
||||
}
|
||||
|
||||
async function saveBackend(e) {
|
||||
e.preventDefault();
|
||||
const id = document.getElementById('backend-id').value;
|
||||
const body = {
|
||||
name: document.getElementById('backend-name').value,
|
||||
label: document.getElementById('backend-label').value,
|
||||
api_base_url: document.getElementById('backend-url').value,
|
||||
pool: document.getElementById('backend-pool').value,
|
||||
rpm_limit: parseInt(document.getElementById('backend-rpm').value),
|
||||
timeout_seconds: parseInt(document.getElementById('backend-timeout').value),
|
||||
enabled: document.getElementById('backend-enabled').value === 'true',
|
||||
model_mappings: JSON.parse(document.getElementById('backend-mappings').value || '{}'),
|
||||
};
|
||||
|
||||
const key = document.getElementById('backend-key').value;
|
||||
if (key) body.api_key = key;
|
||||
|
||||
try {
|
||||
const method = id ? 'PUT' : 'POST';
|
||||
const url = id ? '/api/admin/backends/' + id : '/api/admin/backends';
|
||||
const res = await fetch(url, { method, headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) });
|
||||
if (!res.ok) throw new Error((await res.json()).detail || 'Save failed');
|
||||
closeModal();
|
||||
refreshAll();
|
||||
} catch (e) { alert('Error: ' + e.message); }
|
||||
}
|
||||
|
||||
async function deleteBackend(id) {
|
||||
if (!confirm('Delete this provider? This cannot be undone.')) return;
|
||||
try {
|
||||
await fetch('/api/admin/backends/' + id, { method: 'DELETE' });
|
||||
refreshAll();
|
||||
} catch (e) { alert('Delete failed: ' + e.message); }
|
||||
}
|
||||
|
||||
function closeModal() { document.getElementById('backend-modal').classList.remove('active'); }
|
||||
|
||||
// ── Load Pages ──
|
||||
async function loadPage(page) {
|
||||
if (page === 'dashboard') {
|
||||
initCharts();
|
||||
loadChartData();
|
||||
} else if (page === 'providers') {
|
||||
refreshAll();
|
||||
} else if (page === 'usage') {
|
||||
loadUsageFilter();
|
||||
loadUsage();
|
||||
loadDaily();
|
||||
} else if (page === 'cooldown') {
|
||||
loadCooldown();
|
||||
}
|
||||
}
|
||||
|
||||
async function refreshAll() {
|
||||
try {
|
||||
const res = await fetch('/api/admin/backends');
|
||||
const backends = await res.json();
|
||||
renderBackendsTable(backends);
|
||||
} catch (e) { console.error(e); }
|
||||
}
|
||||
|
||||
async function loadUsageFilter() {
|
||||
try {
|
||||
const res = await fetch('/api/admin/backends');
|
||||
const backends = await res.json();
|
||||
const sel = document.getElementById('usage-backend-filter');
|
||||
sel.innerHTML = '<option value="">All Backends</option>' +
|
||||
backends.map(b => `<option value="${b.id}">${h(b.name)}</option>`).join('');
|
||||
} catch (e) {}
|
||||
}
|
||||
|
||||
async function loadUsage() {
|
||||
const sel = document.getElementById('usage-backend-filter');
|
||||
const backendId = sel.value;
|
||||
const url = backendId ? `/api/admin/stats/hourly?backend_id=${backendId}&hours=72` : '/api/admin/stats/hourly?hours=72';
|
||||
try {
|
||||
const res = await fetch(url);
|
||||
const data = await res.json();
|
||||
const tbody = document.querySelector('#usage-table tbody');
|
||||
tbody.innerHTML = data.map(r => `
|
||||
<tr>
|
||||
<td>${r.hour_bucket}</td>
|
||||
<td>${r.backend_id}</td>
|
||||
<td>${h(r.model)}</td>
|
||||
<td>${fmt(r.request_count)}</td>
|
||||
<td class="${r.error_count > 0 ? 'text-red' : 'text-green'}">${r.error_count}</td>
|
||||
<td>${fmt(r.total_tokens)}</td>
|
||||
<td>$${(r.cost || 0).toFixed(6)}</td>
|
||||
<td>${r.avg_latency_ms}ms</td>
|
||||
</tr>`).join('');
|
||||
} catch (e) { console.error(e); }
|
||||
}
|
||||
|
||||
async function loadDaily() {
|
||||
try {
|
||||
const res = await fetch('/api/admin/stats/daily?days=30');
|
||||
const data = await res.json();
|
||||
const tbody = document.querySelector('#daily-table tbody');
|
||||
tbody.innerHTML = data.map(r => `
|
||||
<tr>
|
||||
<td>${r.date}</td>
|
||||
<td><span class="badge ${r.pool}">${r.pool}</span></td>
|
||||
<td>${fmt(r.total_requests)}</td>
|
||||
<td>${fmt(r.total_errors)}</td>
|
||||
<td>${fmt(r.total_tokens)}</td>
|
||||
<td>$${(r.total_cost || 0).toFixed(6)}</td>
|
||||
<td>${r.unique_backends}</td>
|
||||
</tr>`).join('');
|
||||
} catch (e) { console.error(e); }
|
||||
}
|
||||
|
||||
async function loadCooldown() {
|
||||
try {
|
||||
const res = await fetch('/api/admin/stats/cooldown?limit=100');
|
||||
const data = await res.json();
|
||||
const tbody = document.querySelector('#cooldown-table tbody');
|
||||
tbody.innerHTML = data.map(r => `
|
||||
<tr>
|
||||
<td>${r.started_at}</td>
|
||||
<td>${r.backend_id}</td>
|
||||
<td>${r.consecutive_count}</td>
|
||||
<td>${r.cooldown_seconds}s</td>
|
||||
<td>${h(r.response_summary)}</td>
|
||||
</tr>`).join('');
|
||||
} catch (e) { console.error(e); }
|
||||
}
|
||||
|
||||
async function loadChartData() {
|
||||
try {
|
||||
const res = await fetch('/api/admin/stats/hourly?hours=168');
|
||||
const data = await res.json();
|
||||
// Group by hour, sum
|
||||
const byHour = {};
|
||||
data.forEach(r => {
|
||||
const hour = r.hour_bucket.slice(0, 13);
|
||||
if (!byHour[hour]) byHour[hour] = { cost: 0, tokens: 0 };
|
||||
byHour[hour].cost += (r.cost || 0);
|
||||
byHour[hour].tokens += (r.total_tokens || 0);
|
||||
});
|
||||
const hours = Object.keys(byHour).sort();
|
||||
const costs = hours.map(h => byHour[h].cost);
|
||||
const tokens = hours.map(h => byHour[h].tokens);
|
||||
const labels = hours.map(h => h.slice(11, 16) + ' ' + h.slice(5, 10));
|
||||
|
||||
if (costChart) {
|
||||
costChart.data.labels = labels;
|
||||
costChart.data.datasets[0].data = costs;
|
||||
costChart.update();
|
||||
}
|
||||
if (tokenChart) {
|
||||
tokenChart.data.labels = labels;
|
||||
tokenChart.data.datasets[0].data = tokens;
|
||||
tokenChart.update();
|
||||
}
|
||||
} catch (e) { console.error(e); }
|
||||
}
|
||||
|
||||
// ── Helpers ──
|
||||
function fmt(n) { return (n || 0).toLocaleString(); }
|
||||
function h(s) { const d=document.createElement('div'); d.textContent=s||''; return d.innerHTML; }
|
||||
function formatDuration(s) {
|
||||
const d = Math.floor(s / 86400);
|
||||
const h = Math.floor((s % 86400) / 3600);
|
||||
const m = Math.floor((s % 3600) / 60);
|
||||
const parts = [];
|
||||
if (d) parts.push(d + 'd');
|
||||
if (h) parts.push(h + 'h');
|
||||
if (m || !parts.length) parts.push(m + 'm');
|
||||
return parts.join(' ');
|
||||
}
|
||||
|
||||
// Initial load
|
||||
document.addEventListener('DOMContentLoaded', () => {
|
||||
// Ensure chart containers exist
|
||||
if (!document.getElementById('cost-chart')) {
|
||||
const chartsDiv = document.getElementById('charts');
|
||||
if (chartsDiv) {
|
||||
chartsDiv.innerHTML = `
|
||||
<div class="chart-card"><h3>Cost Over Time</h3><canvas id="cost-chart"></canvas></div>
|
||||
<div class="chart-card"><h3>Token Usage Over Time</h3><canvas id="token-chart"></canvas></div>`;
|
||||
}
|
||||
}
|
||||
initCharts();
|
||||
loadChartData();
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,90 @@
|
||||
# Sidecar V2 — API Key Encryption Rotation SOP
|
||||
|
||||
> 版本: v1.0 | 维护者: 严维序 (opengineer)
|
||||
|
||||
## 背景
|
||||
|
||||
Sidecar V2 使用 AES-256-GCM 加密存储所有 Provider 的 API Key。加密密钥通过 `SIDECAR_ENCRYPTION_KEY` 环境变量传入,启动时通过 `init_crypto()` 初始化。
|
||||
|
||||
## ⚠️ 关键警告
|
||||
|
||||
**更换 SIDECAR_ENCRYPTION_KEY 会导致所有已存储的 API Key 永久不可恢复!**
|
||||
|
||||
`crypto.py` 的 `try_decrypt_existing()` 在密钥变更时会静默返回 `None`,已有加密数据将无法解密。请在轮换密钥前执行以下步骤。
|
||||
|
||||
## 安全轮换步骤
|
||||
|
||||
### Step 1: 导出当前 API Key 明文(必须)
|
||||
|
||||
```bash
|
||||
# 使用旧密钥启动 sidecar,通过 admin API 导出
|
||||
curl -s -H "Authorization: Bearer <ADMIN_TOKEN>" \
|
||||
http://127.0.0.1:9190/api/admin/backends | \
|
||||
python3 -c "
|
||||
import json, sys
|
||||
data = json.load(sys.stdin)
|
||||
# 注意:api_key 是 masked 的,需要重新从安全渠道获取原始 key
|
||||
print(json.dumps(data, indent=2))
|
||||
"
|
||||
```
|
||||
|
||||
### Step 2: 停止服务
|
||||
|
||||
```bash
|
||||
systemctl stop sidecar-v2
|
||||
# 或
|
||||
docker compose down
|
||||
```
|
||||
|
||||
### Step 3: 备份数据库
|
||||
|
||||
```bash
|
||||
cp /app/data/sidecar_v2.db /app/data/backups/pre-rotation-$(date +%Y%m%d_%H%M%S).db
|
||||
```
|
||||
|
||||
### Step 4: 更新密钥
|
||||
|
||||
更新 `/etc/sidecar-v2/env` 或 docker `.env` 文件中的 `SIDECAR_ENCRYPTION_KEY`:
|
||||
|
||||
```
|
||||
SIDECAR_ENCRYPTION_KEY=<new_64_hex_char_key>
|
||||
```
|
||||
|
||||
生成新密钥:
|
||||
```bash
|
||||
python3 -c "import secrets; print(secrets.token_hex(32))"
|
||||
```
|
||||
|
||||
### Step 5: 清空加密 Key 并重新录入
|
||||
|
||||
由于密钥变更后旧加密数据不可读,需要:
|
||||
|
||||
1. 启动服务(此时所有旧 Provider 的 API Key 不可用)
|
||||
2. 通过 Admin API 重新录入所有 Provider 的 API Key:
|
||||
```bash
|
||||
curl -s -X PUT -H "Authorization: Bearer <ADMIN_TOKEN>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"api_key": "<NEW_PLAIN_KEY>"}' \
|
||||
http://127.0.0.1:9190/api/admin/backends/<backend_id>
|
||||
```
|
||||
|
||||
### Step 6: 验证
|
||||
|
||||
```bash
|
||||
# 确认 Provider 状态为 healthy
|
||||
curl -s http://127.0.0.1:9190/api/admin/pools
|
||||
# 发送测试请求
|
||||
curl -s -X POST http://127.0.0.1:9190/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"<model_name>","messages":[{"role":"user","content":"test"}],"max_tokens":5}'
|
||||
```
|
||||
|
||||
## 应急预案
|
||||
|
||||
如果在密钥轮换过程中出错:
|
||||
|
||||
1. 恢复旧密钥环境变量
|
||||
2. 恢复旧数据库备份
|
||||
3. 重启服务
|
||||
|
||||
旧 Key 会正常工作,因为未被覆盖的数据仍然用旧密钥加密。
|
||||
@@ -0,0 +1,56 @@
|
||||
# Sidecar V2 — Nginx reverse proxy config (reference)
|
||||
# Place at /etc/nginx/sites-available/sidecar-v2.conf
|
||||
# SSL certs managed by certbot or manually
|
||||
|
||||
upstream sidecar_v2_main {
|
||||
server 127.0.0.1:9190;
|
||||
}
|
||||
|
||||
upstream sidecar_v2_metrics {
|
||||
server 127.0.0.1:9191;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name sidecar.example.com;
|
||||
|
||||
ssl_certificate /etc/ssl/certs/sidecar.pem;
|
||||
ssl_certificate_key /etc/ssl/private/sidecar.key;
|
||||
|
||||
# Dashboard + Admin API (main port)
|
||||
location / {
|
||||
proxy_pass http://sidecar_v2_main;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
|
||||
# SSE support for dashboard real-time data
|
||||
location /dashboard/sse {
|
||||
proxy_pass http://sidecar_v2_main;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection "";
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_buffering off;
|
||||
proxy_cache off;
|
||||
chunked_transfer_encoding off;
|
||||
proxy_read_timeout 86400s;
|
||||
}
|
||||
|
||||
# Prometheus metrics
|
||||
location /metrics {
|
||||
proxy_pass http://sidecar_v2_metrics;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# Health check
|
||||
location /health {
|
||||
proxy_pass http://sidecar_v2_main;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
[Unit]
|
||||
Description=Sidecar V2 — Multi-Pool Provider Proxy
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=openclaw
|
||||
Group=openclaw
|
||||
WorkingDirectory=/opt/sidecar-v2
|
||||
EnvironmentFile=/etc/sidecar-v2/env
|
||||
ExecStart=/opt/sidecar-v2/.venv/bin/python3 main.py
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
# Security hardening
|
||||
NoNewPrivileges=yes
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
ReadWritePaths=/opt/sidecar-v2/data
|
||||
PrivateTmp=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,26 @@
|
||||
# Sidecar V2 — Multi-Pool Provider Proxy
|
||||
version: "3.9"
|
||||
|
||||
services:
|
||||
sidecar-v2:
|
||||
build: .
|
||||
container_name: sidecar-v2
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "9190:9190" # Main proxy + admin API + dashboard
|
||||
- "9191:9191" # Prometheus metrics
|
||||
environment:
|
||||
- SIDECAR_ENCRYPTION_KEY=${SIDECAR_ENCRYPTION_KEY}
|
||||
- SIDECAR_ADMIN_TOKEN=${SIDECAR_ADMIN_TOKEN:-change-me}
|
||||
- LOG_FORMAT=${LOG_FORMAT:-json}
|
||||
- SIDECAR_HOST=0.0.0.0
|
||||
- SIDECAR_PORT=9190
|
||||
- SIDECAR_METRICS_PORT=9191
|
||||
- SIDECAR_DB_PATH=/app/data/sidecar_v2.db
|
||||
- SIDECAR_BACKUP_DIR=/app/data/backups
|
||||
volumes:
|
||||
- sidecar-data:/app/data
|
||||
|
||||
volumes:
|
||||
sidecar-data:
|
||||
driver: local
|
||||
@@ -0,0 +1,17 @@
|
||||
"""Sidecar V2 entry point."""
|
||||
|
||||
import uvicorn
|
||||
from config import config
|
||||
|
||||
|
||||
def main():
|
||||
uvicorn.run(
|
||||
"server:app",
|
||||
host=config.host,
|
||||
port=config.port,
|
||||
log_level=config.log_level.lower(),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,83 @@
|
||||
"""Provider pool management: primary / fallback pool routing."""
|
||||
|
||||
import structlog
|
||||
from typing import Optional
|
||||
|
||||
from storage.backend_store import list_backends, get_pool_stats
|
||||
from storage.models import Backend
|
||||
|
||||
logger = structlog.get_logger("sidecar_v2.pool_manager")
|
||||
|
||||
|
||||
class PoolManager:
|
||||
"""Manages provider pools and selects healthy backends for a given model.
|
||||
|
||||
Priority: primary pool → fallback pool.
|
||||
Within a pool: healthy backends only, sorted by availability.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self._pool_order = ["primary", "fallback"]
|
||||
|
||||
def get_available_backends(
|
||||
self, canonical_model: str, pool: Optional[str] = None
|
||||
) -> list[Backend]:
|
||||
"""Get all healthy, enabled backends that serve a model, in pool order.
|
||||
|
||||
Args:
|
||||
canonical_model: Canonical model name to match.
|
||||
pool: Optional pool filter (primary/fallback). None = all pools.
|
||||
|
||||
Returns:
|
||||
List of ready backends sorted by pool priority, then RPM utilization.
|
||||
"""
|
||||
backends: list[Backend] = []
|
||||
|
||||
pools_to_check = [pool] if pool else self._pool_order
|
||||
for p in pools_to_check:
|
||||
pool_backends = list_backends(pool=p, enabled_only=True, decrypt_key=True)
|
||||
for b in pool_backends:
|
||||
if b.status == "healthy" and b.has_model(canonical_model):
|
||||
backends.append(b)
|
||||
if pool:
|
||||
break
|
||||
|
||||
return backends
|
||||
|
||||
def get_any_healthy_backends(self, pool: Optional[str] = None) -> list[Backend]:
|
||||
"""Get all healthy, enabled backends regardless of model."""
|
||||
backends: list[Backend] = []
|
||||
pools_to_check = [pool] if pool else self._pool_order
|
||||
for p in pools_to_check:
|
||||
pool_backends = list_backends(pool=p, enabled_only=True, decrypt_key=True)
|
||||
for b in pool_backends:
|
||||
if b.status == "healthy":
|
||||
backends.append(b)
|
||||
if pool:
|
||||
break
|
||||
return backends
|
||||
|
||||
def get_pool_status(self) -> dict:
|
||||
"""Get pool summary for dashboard."""
|
||||
stats = get_pool_stats()
|
||||
result = {}
|
||||
for pool in self._pool_order:
|
||||
s = stats.get(pool, {"total": 0, "enabled": 0, "healthy": 0, "cooling": 0, "error": 0})
|
||||
result[pool] = s
|
||||
# Also include any other pools
|
||||
for pool, s in stats.items():
|
||||
if pool not in result:
|
||||
result[pool] = s
|
||||
return result
|
||||
|
||||
def is_pool_available(self, canonical_model: str, pool: str = "primary") -> bool:
|
||||
"""Check if a pool has any healthy backends for a model."""
|
||||
backends = self.get_available_backends(canonical_model, pool=pool)
|
||||
return len(backends) > 0
|
||||
|
||||
def is_any_pool_available(self, canonical_model: str) -> bool:
|
||||
"""Check if any pool has healthy backends for a model."""
|
||||
for pool in self._pool_order:
|
||||
if self.is_pool_available(canonical_model, pool):
|
||||
return True
|
||||
return False
|
||||
@@ -0,0 +1,383 @@
|
||||
"""Proxy request handling for Sidecar V2 — multi-pool routing + cooldown + rate limiting."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import time
|
||||
from typing import Any, Optional
|
||||
|
||||
import httpx
|
||||
import structlog
|
||||
from fastapi import Request
|
||||
from fastapi.responses import JSONResponse, Response, StreamingResponse
|
||||
|
||||
from config import config
|
||||
from pool_manager import PoolManager
|
||||
from rate_limiter import PerBackendRateLimiter
|
||||
from router import Router
|
||||
from cooldown_manager import start_cooldown, check_and_clear_cooldown
|
||||
from storage.models import Backend
|
||||
from storage.usage_store import record_usage
|
||||
|
||||
# Emergency activation counter (read by metrics endpoint)
|
||||
_emergency_count: int = 0
|
||||
|
||||
|
||||
def get_emergency_count() -> int:
|
||||
return _emergency_count
|
||||
|
||||
|
||||
logger: structlog.stdlib.BoundLogger = structlog.get_logger("sidecar_v2.proxy")
|
||||
|
||||
|
||||
def extract_model(body: dict[str, Any]) -> str:
|
||||
"""Extract model identifier from request body."""
|
||||
return str(body.get("model", "unknown"))
|
||||
|
||||
|
||||
def build_error_response(status: int, message: str, error_type: str = "") -> JSONResponse:
|
||||
"""Build a standard error response."""
|
||||
return JSONResponse(
|
||||
status_code=status,
|
||||
content={
|
||||
"error": {
|
||||
"message": message,
|
||||
"type": error_type or f"Error_{status}",
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
async def forward_to_backend(
|
||||
backend: Backend,
|
||||
method: str,
|
||||
path: str,
|
||||
body: bytes | None,
|
||||
headers: dict[str, str],
|
||||
stream: bool = False,
|
||||
) -> httpx.Response:
|
||||
"""Forward a request to a specific backend."""
|
||||
upstream_url = backend.api_base_url.rstrip("/") + path
|
||||
|
||||
forward_headers = {
|
||||
k: v
|
||||
for k, v in headers.items()
|
||||
if k.lower() not in ("host", "content-length", "transfer-encoding")
|
||||
}
|
||||
|
||||
if backend.api_key_plain:
|
||||
forward_headers["authorization"] = f"Bearer {backend.api_key_plain}"
|
||||
elif "authorization" not in {k.lower() for k in forward_headers}:
|
||||
forward_headers["authorization"] = "Bearer nvidia"
|
||||
|
||||
timeout = httpx.Timeout(backend.timeout_seconds)
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout) as client:
|
||||
req = client.build_request(
|
||||
method=method,
|
||||
url=upstream_url,
|
||||
headers=forward_headers,
|
||||
content=body,
|
||||
)
|
||||
return await client.send(req, stream=stream)
|
||||
|
||||
|
||||
def build_response(resp: httpx.Response) -> Response:
|
||||
"""Convert httpx.Response to FastAPI Response."""
|
||||
content_type = resp.headers.get("content-type", "")
|
||||
headers = {
|
||||
k: v
|
||||
for k, v in resp.headers.items()
|
||||
if k.lower() not in ("content-encoding", "transfer-encoding")
|
||||
}
|
||||
|
||||
is_sse = "text/event-stream" in content_type
|
||||
is_chunked = resp.headers.get("transfer-encoding", "").lower() == "chunked"
|
||||
if is_sse or (is_chunked and headers.get("content-type", "") != "application/octet-stream"):
|
||||
return StreamingResponse(
|
||||
content=resp.aiter_bytes(),
|
||||
status_code=resp.status_code,
|
||||
headers=headers,
|
||||
media_type=content_type or "text/event-stream",
|
||||
)
|
||||
|
||||
return Response(
|
||||
content=resp.content,
|
||||
status_code=resp.status_code,
|
||||
headers=headers,
|
||||
media_type=content_type or "application/json",
|
||||
)
|
||||
|
||||
|
||||
def extract_usage_from_response(
|
||||
resp: httpx.Response,
|
||||
resp_json: dict[str, Any],
|
||||
model: str,
|
||||
) -> tuple[int, int, int]:
|
||||
"""Extract token usage from response body (OpenAI-compatible)."""
|
||||
usage = resp_json.get("usage", {})
|
||||
prompt_tokens = usage.get("prompt_tokens", 0) or 0
|
||||
completion_tokens = usage.get("completion_tokens", 0) or 0
|
||||
|
||||
# Try streaming chunks: aggregate from choices
|
||||
if not prompt_tokens and not completion_tokens:
|
||||
choices = resp_json.get("choices", [])
|
||||
for choice in choices:
|
||||
if isinstance(choice, dict):
|
||||
tokens = choice.get("usage", {})
|
||||
prompt_tokens += tokens.get("prompt_tokens", 0) or 0
|
||||
completion_tokens += tokens.get("completion_tokens", 0) or 0
|
||||
|
||||
total_tokens = prompt_tokens + completion_tokens
|
||||
if total_tokens == 0:
|
||||
total_tokens = usage.get("total_tokens", 0) or 0
|
||||
|
||||
return prompt_tokens, completion_tokens, total_tokens
|
||||
|
||||
|
||||
def calculate_cost(
|
||||
backend: Backend,
|
||||
model: str,
|
||||
prompt_tokens: int,
|
||||
completion_tokens: int,
|
||||
) -> float:
|
||||
"""Calculate cost using backend's model pricing."""
|
||||
cost_info = backend.get_model_cost(model)
|
||||
input_cost = cost_info.get("input", 0.0)
|
||||
output_cost = cost_info.get("output", 0.0)
|
||||
# Costs are per token
|
||||
return (prompt_tokens * input_cost + completion_tokens * output_cost)
|
||||
|
||||
|
||||
async def handle_proxy_request(
|
||||
pool_manager: PoolManager,
|
||||
rate_limiter: PerBackendRateLimiter,
|
||||
router: Router,
|
||||
request: Request,
|
||||
path: str,
|
||||
) -> Response:
|
||||
"""Main proxy handler: multi-pool routing with cooldown and rate limiting.
|
||||
|
||||
Flow:
|
||||
1. Extract model → canonical name
|
||||
2. Pick backend via Router (primary → fallback)
|
||||
3. Forward request
|
||||
4. If 429 → cooldown backend, retry with another
|
||||
5. If all pools exhausted → emergency mode
|
||||
6. Track usage
|
||||
"""
|
||||
start_time = time.monotonic()
|
||||
|
||||
body_bytes: bytes = await request.body()
|
||||
raw_headers: dict[str, str] = dict(request.headers)
|
||||
|
||||
body_json: dict[str, Any] = {}
|
||||
try:
|
||||
if body_bytes:
|
||||
parsed = json.loads(body_bytes)
|
||||
if isinstance(parsed, dict):
|
||||
body_json = parsed
|
||||
except (ValueError, TypeError):
|
||||
body_json = {}
|
||||
|
||||
canonical_model = extract_model(body_json)
|
||||
is_stream = body_json.get("stream", False)
|
||||
|
||||
# Try with pool routing
|
||||
max_retries = config.max_pool_retries
|
||||
for attempt in range(max_retries):
|
||||
# Check and clear expired cooldowns before picking
|
||||
_refresh_cooldowns()
|
||||
|
||||
backend = router.pick_backend(canonical_model)
|
||||
if backend is None:
|
||||
break # No backend available, fall through to emergency
|
||||
|
||||
try:
|
||||
resp = await forward_to_backend(
|
||||
backend=backend,
|
||||
method=request.method,
|
||||
path=path,
|
||||
body=body_bytes if body_bytes else None,
|
||||
headers=raw_headers,
|
||||
stream=is_stream,
|
||||
)
|
||||
elapsed_ms = int((time.monotonic() - start_time) * 1000)
|
||||
|
||||
# Handle 429 — cooldown and retry
|
||||
if resp.status_code == 429:
|
||||
new_count = backend.consecutive_429_count + 1
|
||||
start_cooldown(backend.id, new_count)
|
||||
|
||||
resp_body = ""
|
||||
try:
|
||||
resp_body = resp.text[:200]
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
logger.warning(
|
||||
"backend_429_cooldown",
|
||||
backend_id=backend.id,
|
||||
pool=backend.pool,
|
||||
consecutive=new_count,
|
||||
model=canonical_model,
|
||||
)
|
||||
|
||||
# Track the error
|
||||
record_usage(
|
||||
backend_id=backend.id,
|
||||
model=canonical_model,
|
||||
prompt_tokens=0,
|
||||
completion_tokens=0,
|
||||
cost=0.0,
|
||||
latency_ms=elapsed_ms,
|
||||
is_error=True,
|
||||
)
|
||||
|
||||
continue # Retry with another backend
|
||||
|
||||
# Success — track usage
|
||||
resp_json: dict[str, Any] = {}
|
||||
try:
|
||||
if not is_stream and resp.content:
|
||||
resp_json = json.loads(resp.content)
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
prompt_tokens, completion_tokens, total_tokens = extract_usage_from_response(
|
||||
resp, resp_json, canonical_model
|
||||
)
|
||||
cost = calculate_cost(
|
||||
backend, canonical_model, prompt_tokens, completion_tokens
|
||||
)
|
||||
|
||||
record_usage(
|
||||
backend_id=backend.id,
|
||||
model=canonical_model,
|
||||
prompt_tokens=prompt_tokens,
|
||||
completion_tokens=completion_tokens,
|
||||
cost=cost,
|
||||
latency_ms=elapsed_ms,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"request_completed",
|
||||
backend_id=backend.id,
|
||||
pool=backend.pool,
|
||||
model=canonical_model,
|
||||
status=resp.status_code,
|
||||
tokens=total_tokens,
|
||||
cost=round(cost, 6),
|
||||
elapsed_ms=elapsed_ms,
|
||||
)
|
||||
|
||||
return build_response(resp)
|
||||
|
||||
except httpx.TimeoutException:
|
||||
logger.warning(
|
||||
"backend_timeout",
|
||||
backend_id=backend.id,
|
||||
model=canonical_model,
|
||||
)
|
||||
continue
|
||||
except (httpx.ConnectError, httpx.RemoteProtocolError) as exc:
|
||||
logger.warning(
|
||||
"backend_connection_error",
|
||||
backend_id=backend.id,
|
||||
model=canonical_model,
|
||||
error=str(exc),
|
||||
)
|
||||
continue
|
||||
except Exception as exc:
|
||||
logger.error(
|
||||
"proxy_error",
|
||||
backend_id=backend.id,
|
||||
model=canonical_model,
|
||||
error=str(exc),
|
||||
)
|
||||
continue
|
||||
|
||||
# All pools exhausted — emergency rate-limited passthrough
|
||||
emergency_rpm = int(config.default_rpm_limit * config.emergency_rpm_fraction)
|
||||
if emergency_rpm < 1:
|
||||
emergency_rpm = 1
|
||||
|
||||
logger.warning(
|
||||
"all_pools_exhausted_emergency",
|
||||
model=canonical_model,
|
||||
emergency_rpm=emergency_rpm,
|
||||
)
|
||||
|
||||
# Track emergency activation for metrics
|
||||
_emergency_count += 1
|
||||
|
||||
# Emergency: try to get a token from any fallback backend at reduced RPM
|
||||
emergency_retries = 3
|
||||
for attempt in range(emergency_retries):
|
||||
backends = pool_manager.get_any_healthy_backends()
|
||||
for backend in backends:
|
||||
if rate_limiter.consume(backend.id, emergency_rpm):
|
||||
try:
|
||||
resp = await forward_to_backend(
|
||||
backend=backend,
|
||||
method=request.method,
|
||||
path=path,
|
||||
body=body_bytes if body_bytes else None,
|
||||
headers=raw_headers,
|
||||
stream=is_stream,
|
||||
)
|
||||
elapsed_ms = int((time.monotonic() - start_time) * 1000)
|
||||
|
||||
if resp.status_code == 429:
|
||||
start_cooldown(backend.id, backend.consecutive_429_count + 1)
|
||||
continue
|
||||
|
||||
# Success in emergency mode
|
||||
try:
|
||||
resp_json: dict[str, Any] = {}
|
||||
if not is_stream and resp.content:
|
||||
resp_json = json.loads(resp.content)
|
||||
except Exception:
|
||||
resp_json = {}
|
||||
|
||||
prompt_tokens, completion_tokens, total_tokens = extract_usage_from_response(
|
||||
resp, resp_json, canonical_model
|
||||
)
|
||||
cost_em = calculate_cost(backend, canonical_model, prompt_tokens, completion_tokens)
|
||||
|
||||
record_usage(
|
||||
backend_id=backend.id,
|
||||
model=canonical_model,
|
||||
prompt_tokens=prompt_tokens,
|
||||
completion_tokens=completion_tokens,
|
||||
cost=cost_em,
|
||||
latency_ms=elapsed_ms,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"emergency_passthrough_success",
|
||||
backend_id=backend.id,
|
||||
model=canonical_model,
|
||||
emergency_rpm=emergency_rpm,
|
||||
)
|
||||
return build_response(resp)
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
# All emergency attempts failed — return 503 for OpenClaw fallback chain
|
||||
return build_error_response(
|
||||
503,
|
||||
"All provider pools exhausted. OpenClaw fallback chain should activate.",
|
||||
"AllPoolsExhausted",
|
||||
)
|
||||
|
||||
|
||||
def _refresh_cooldowns() -> None:
|
||||
"""Check and clear expired cooldowns for backends currently in cooling state.
|
||||
|
||||
Only queries backends with status='cooling' (the health_check_loop handles
|
||||
the periodic scanning; this is the on-demand refresh before proxy routing)."""
|
||||
from storage.backend_store import list_backends
|
||||
backends = list_backends(decrypt_key=False)
|
||||
for backend in backends:
|
||||
if backend.status == "cooling":
|
||||
check_and_clear_cooldown(backend.id)
|
||||
@@ -0,0 +1,111 @@
|
||||
"""Per-backend rate limiter using token bucket algorithm."""
|
||||
|
||||
import threading
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
|
||||
class PerBackendRateLimiter:
|
||||
"""Manages independent token buckets for each backend.
|
||||
|
||||
Thread-safe. Each backend gets its own bucket with configurable RPM.
|
||||
"""
|
||||
|
||||
def __init__(self, refill_interval_ms: int = 50):
|
||||
self._buckets: dict[str, _TokenBucket] = {}
|
||||
self._lock = threading.Lock()
|
||||
self._refill_interval_ms = refill_interval_ms
|
||||
|
||||
def ensure_bucket(self, backend_id: str, rpm_limit: int) -> None:
|
||||
"""Create or update a bucket for a backend."""
|
||||
with self._lock:
|
||||
if backend_id in self._buckets:
|
||||
existing = self._buckets[backend_id]
|
||||
existing.update_rate(rpm_limit)
|
||||
else:
|
||||
self._buckets[backend_id] = _TokenBucket(
|
||||
rate=rpm_limit / 60.0,
|
||||
capacity=max(rpm_limit, 1),
|
||||
)
|
||||
|
||||
def remove_bucket(self, backend_id: str) -> None:
|
||||
"""Remove a backend's bucket."""
|
||||
with self._lock:
|
||||
self._buckets.pop(backend_id, None)
|
||||
|
||||
def consume(self, backend_id: str, rpm_limit: int, tokens: int = 1) -> bool:
|
||||
"""Try to consume tokens for a backend. Returns True if allowed.
|
||||
|
||||
Auto-creates the bucket if needed.
|
||||
"""
|
||||
self.ensure_bucket(backend_id, rpm_limit)
|
||||
|
||||
with self._lock:
|
||||
bucket = self._buckets.get(backend_id)
|
||||
if bucket is None:
|
||||
return False
|
||||
|
||||
return bucket.consume(tokens)
|
||||
|
||||
def get_status(self, backend_id: str) -> dict[str, Any] | None:
|
||||
"""Get bucket status for a backend."""
|
||||
with self._lock:
|
||||
bucket = self._buckets.get(backend_id)
|
||||
if bucket is None:
|
||||
return None
|
||||
return bucket.get_status()
|
||||
|
||||
def get_all_status(self) -> dict[str, dict[str, Any]]:
|
||||
"""Get status of all buckets."""
|
||||
with self._lock:
|
||||
return {bid: b.get_status() for bid, b in self._buckets.items()}
|
||||
|
||||
|
||||
class _TokenBucket:
|
||||
"""Internal token bucket with refill."""
|
||||
|
||||
def __init__(self, rate: float, capacity: int):
|
||||
self._rate = float(rate)
|
||||
self._capacity = int(capacity)
|
||||
self._tokens = float(capacity)
|
||||
self._last_refill = time.monotonic()
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def _refill(self) -> None:
|
||||
now = time.monotonic()
|
||||
elapsed = now - self._last_refill
|
||||
if elapsed > 0 and self._rate > 0:
|
||||
self._tokens = min(self._tokens + elapsed * self._rate, float(self._capacity))
|
||||
self._last_refill = now
|
||||
|
||||
def consume(self, tokens: int = 1) -> bool:
|
||||
if tokens <= 0:
|
||||
return True
|
||||
with self._lock:
|
||||
self._refill()
|
||||
if self._tokens >= tokens:
|
||||
self._tokens -= tokens
|
||||
return True
|
||||
return False
|
||||
|
||||
def update_rate(self, rpm_limit: int) -> None:
|
||||
new_rate = rpm_limit / 60.0
|
||||
with self._lock:
|
||||
self._refill()
|
||||
self._rate = new_rate
|
||||
self._capacity = max(rpm_limit, 1)
|
||||
self._tokens = min(self._tokens, float(self._capacity))
|
||||
|
||||
def get_status(self) -> dict[str, Any]:
|
||||
with self._lock:
|
||||
self._refill()
|
||||
rate_per_minute = self._rate * 60.0
|
||||
utilization = 0.0 if self._capacity == 0 else (
|
||||
(self._capacity - self._tokens) / self._capacity
|
||||
)
|
||||
return {
|
||||
"tokens": round(self._tokens, 2),
|
||||
"capacity": self._capacity,
|
||||
"rate_per_minute": round(rate_per_minute, 1),
|
||||
"utilization": round(utilization, 4),
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
# Sidecar V2 — Multi-Pool Provider Proxy
|
||||
fastapi>=0.115.0,<1.0.0
|
||||
uvicorn[standard]>=0.30.0,<1.0.0
|
||||
httpx>=0.27.0,<1.0.0
|
||||
structlog>=24.0.0,<25.0.0
|
||||
cryptography>=42.0.0,<44.0.0
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Model → Backend routing logic for Sidecar V2."""
|
||||
|
||||
import structlog
|
||||
from typing import Optional
|
||||
|
||||
from storage.models import Backend
|
||||
from pool_manager import PoolManager
|
||||
from rate_limiter import PerBackendRateLimiter
|
||||
|
||||
logger = structlog.get_logger("sidecar_v2.router")
|
||||
|
||||
|
||||
class Router:
|
||||
"""Routes model requests to the best available backend.
|
||||
|
||||
Pick strategy:
|
||||
1. Primary pool → healthy backends supporting the model
|
||||
2. Rate-limiter check → skip if RPM exhausted
|
||||
3. Fallback pool → repeat above
|
||||
4. If all exhausted → return None (caller handles emergency)
|
||||
"""
|
||||
|
||||
def __init__(self, pool_manager: PoolManager, rate_limiter: PerBackendRateLimiter):
|
||||
self._pool_manager = pool_manager
|
||||
self._rate_limiter = rate_limiter
|
||||
|
||||
def pick_backend(self, canonical_model: str) -> Optional[Backend]:
|
||||
"""Pick the best available backend for a model.
|
||||
|
||||
Tries primary pool first, then fallback.
|
||||
Within each pool, skips backends at RPM limit.
|
||||
Returns None if no backend available.
|
||||
"""
|
||||
# Try pools in order
|
||||
for pool in ["primary", "fallback"]:
|
||||
backends = self._pool_manager.get_available_backends(
|
||||
canonical_model, pool=pool
|
||||
)
|
||||
for backend in backends:
|
||||
# Rate-limit check
|
||||
if self._rate_limiter.consume(
|
||||
backend.id, backend.rpm_limit
|
||||
):
|
||||
return backend
|
||||
# Skip this backend, try next
|
||||
logger.debug(
|
||||
"backend_rate_limited",
|
||||
backend_id=backend.id,
|
||||
pool=pool,
|
||||
model=canonical_model,
|
||||
)
|
||||
|
||||
if not backends:
|
||||
logger.debug("pool_exhausted", pool=pool, model=canonical_model)
|
||||
else:
|
||||
logger.debug("pool_rpm_exhausted", pool=pool, model=canonical_model)
|
||||
|
||||
return None
|
||||
|
||||
def get_all_pools_exhausted_info(self, canonical_model: str) -> bool:
|
||||
"""Check if ALL pools are exhausted for a model."""
|
||||
return not self._pool_manager.is_any_pool_available(canonical_model)
|
||||
@@ -0,0 +1,712 @@
|
||||
"""Sidecar V2 — FastAPI server with multi-pool routing, admin API, dashboard SSE."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import AsyncGenerator
|
||||
from contextlib import asynccontextmanager
|
||||
from typing import Any, Optional
|
||||
|
||||
import structlog
|
||||
from fastapi import Depends, FastAPI, HTTPException, Request, Response
|
||||
from fastapi.responses import FileResponse, HTMLResponse, JSONResponse, StreamingResponse
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
|
||||
from config import config as app_config
|
||||
from crypto import init_crypto, is_initialized
|
||||
from pool_manager import PoolManager
|
||||
from rate_limiter import PerBackendRateLimiter
|
||||
from router import Router
|
||||
from proxy import handle_proxy_request, get_emergency_count
|
||||
|
||||
from storage.db import init_db, create_tables, run_integrity_check, get_connection, _DB_PATH
|
||||
from storage.backend_store import (
|
||||
create_backend, get_backend, list_backends, update_backend,
|
||||
delete_backend, get_pool_stats,
|
||||
)
|
||||
from storage.usage_store import get_total_stats, get_hourly_usage, get_daily_stats, aggregate_daily_stats
|
||||
from storage.cooldown_store import get_cooldown_history
|
||||
from storage.config_store import get_config, set_config, list_configs, delete_config
|
||||
from storage.models import Backend, ModelMapping
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────
|
||||
# Logging
|
||||
# ──────────────────────────────────────────────────────────
|
||||
_LOG_FORMAT = os.getenv("LOG_FORMAT", "console").lower()
|
||||
|
||||
structlog.configure(
|
||||
processors=[
|
||||
structlog.stdlib.filter_by_level,
|
||||
structlog.stdlib.add_logger_name,
|
||||
structlog.stdlib.add_log_level,
|
||||
structlog.stdlib.PositionalArgumentsFormatter(),
|
||||
structlog.processors.TimeStamper(fmt="iso"),
|
||||
structlog.processors.StackInfoRenderer(),
|
||||
structlog.processors.format_exc_info,
|
||||
structlog.processors.UnicodeDecoder(),
|
||||
(
|
||||
structlog.processors.JSONRenderer()
|
||||
if _LOG_FORMAT == "json"
|
||||
else structlog.dev.ConsoleRenderer()
|
||||
),
|
||||
],
|
||||
context_class=dict,
|
||||
logger_factory=structlog.stdlib.LoggerFactory(),
|
||||
wrapper_class=structlog.stdlib.BoundLogger,
|
||||
cache_logger_on_first_use=True,
|
||||
)
|
||||
logger: structlog.stdlib.BoundLogger = structlog.get_logger("sidecar_v2.server")
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────
|
||||
# Admin Auth middleware
|
||||
# ──────────────────────────────────────────────────────────
|
||||
_security = HTTPBearer(auto_error=False)
|
||||
|
||||
|
||||
def verify_admin_token(
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(_security),
|
||||
) -> bool:
|
||||
"""Verify Bearer Token against config.admin_token.
|
||||
|
||||
If admin_token is empty, write operations are rejected.
|
||||
READ operations are allowed without auth for dashboard use.
|
||||
"""
|
||||
if not app_config.admin_token:
|
||||
# No token configured — allow read, reject write (checked per-endpoint)
|
||||
if credentials is None:
|
||||
return False
|
||||
return False
|
||||
|
||||
if credentials is None:
|
||||
return False
|
||||
|
||||
return credentials.credentials == app_config.admin_token
|
||||
|
||||
|
||||
def require_admin(credentials: Optional[HTTPAuthorizationCredentials] = Depends(_security)):
|
||||
"""Require admin auth — raise 401 if not authorized."""
|
||||
if not app_config.admin_token:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Admin API not configured: set SIDECAR_ADMIN_TOKEN",
|
||||
)
|
||||
if credentials is None:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Missing Authorization header",
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
if credentials.credentials != app_config.admin_token:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Invalid admin token",
|
||||
)
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────
|
||||
# Global runtime state
|
||||
# ──────────────────────────────────────────────────────────
|
||||
pool_manager: Optional[PoolManager] = None
|
||||
rate_limiter: Optional[PerBackendRateLimiter] = None
|
||||
router: Optional[Router] = None
|
||||
start_time: float = 0.0
|
||||
|
||||
# In-memory metrics counters
|
||||
_metrics_counters: dict[str, int] = {}
|
||||
_metrics_lock = asyncio.Lock()
|
||||
|
||||
|
||||
def _inc_metric(key: str, delta: int = 1) -> None:
|
||||
"""Thread-safe counter increment (deferred via asyncio)."""
|
||||
_metrics_counters[key] = _metrics_counters.get(key, 0) + delta
|
||||
|
||||
|
||||
def get_pm() -> PoolManager:
|
||||
assert pool_manager is not None
|
||||
return pool_manager
|
||||
|
||||
|
||||
def get_rl() -> PerBackendRateLimiter:
|
||||
assert rate_limiter is not None
|
||||
return rate_limiter
|
||||
|
||||
|
||||
def get_router() -> Router:
|
||||
assert router is not None
|
||||
return router
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────
|
||||
# Lifespan
|
||||
# ──────────────────────────────────────────────────────────
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI) -> AsyncGenerator[None, Any]:
|
||||
global pool_manager, rate_limiter, router, start_time
|
||||
|
||||
# P0: Encryption key is mandatory — refuse to start without it
|
||||
if not app_config.encryption_key:
|
||||
logger.critical(
|
||||
"missing_encryption_key",
|
||||
hint="Set SIDECAR_ENCRYPTION_KEY (64 hex chars). Refusing to start."
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
init_crypto(app_config.encryption_key)
|
||||
logger.info("crypto_initialized")
|
||||
|
||||
# P0: Warn if admin_token not set
|
||||
if not app_config.admin_token:
|
||||
logger.warning(
|
||||
"admin_token_not_set",
|
||||
hint="Admin write endpoints disabled until SIDECAR_ADMIN_TOKEN is configured."
|
||||
)
|
||||
|
||||
# Init DB
|
||||
init_db()
|
||||
create_tables()
|
||||
ok = run_integrity_check()
|
||||
if not ok:
|
||||
logger.error("db_integrity_check_failed")
|
||||
|
||||
# Init runtime components
|
||||
pool_manager = PoolManager()
|
||||
rate_limiter = PerBackendRateLimiter(
|
||||
refill_interval_ms=app_config.rate_limiter_refill_interval_ms,
|
||||
)
|
||||
router = Router(pool_manager, rate_limiter)
|
||||
start_time = time.time()
|
||||
|
||||
# Start background tasks
|
||||
health_task = asyncio.create_task(_health_check_loop())
|
||||
stats_task = asyncio.create_task(_stats_aggregation_loop())
|
||||
backup_task = asyncio.create_task(_backup_loop())
|
||||
|
||||
logger.info(
|
||||
"sidecar_v2_started",
|
||||
host=app_config.host,
|
||||
port=app_config.port,
|
||||
metrics_port=app_config.metrics_port,
|
||||
)
|
||||
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
for task in [health_task, stats_task, backup_task]:
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
logger.info("sidecar_v2_stopped")
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="Sidecar V2 — Multi-Pool Provider Proxy",
|
||||
version="2.0.0",
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────
|
||||
# Background tasks
|
||||
# ──────────────────────────────────────────────────────────
|
||||
|
||||
async def _health_check_loop() -> None:
|
||||
"""Periodic health checks: clear expired cooldowns + active probing of backends."""
|
||||
from cooldown_manager import check_and_clear_cooldown
|
||||
import httpx
|
||||
|
||||
while True:
|
||||
try:
|
||||
backends = list_backends(decrypt_key=True)
|
||||
for b in backends:
|
||||
# 1. Clear expired cooldowns
|
||||
if b.status == "cooling":
|
||||
check_and_clear_cooldown(b.id)
|
||||
|
||||
# 2. Active health probing for healthy/enabled backends
|
||||
if b.status == "healthy" and b.enabled:
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(
|
||||
app_config.health_check_timeout_seconds
|
||||
)) as client:
|
||||
probe_url = b.api_base_url.rstrip("/") + app_config.health_probe_endpoint
|
||||
headers = {}
|
||||
if b.api_key_plain:
|
||||
headers["Authorization"] = f"Bearer {b.api_key_plain}"
|
||||
|
||||
start = time.monotonic()
|
||||
resp = await client.get(probe_url, headers=headers)
|
||||
elapsed_ms = int((time.monotonic() - start) * 1000)
|
||||
|
||||
# Update health state in DB
|
||||
from storage.db import get_connection as _gc
|
||||
with _gc() as conn:
|
||||
conn.execute(
|
||||
"""INSERT INTO backend_health
|
||||
(backend_id, state, last_latency_ms, last_status_code,
|
||||
last_check_at)
|
||||
VALUES (?, 'healthy', ?, ?, datetime('now'))
|
||||
ON CONFLICT(backend_id) DO UPDATE SET
|
||||
state = excluded.state,
|
||||
last_latency_ms = excluded.last_latency_ms,
|
||||
last_status_code = excluded.last_status_code,
|
||||
last_check_at = excluded.last_check_at""",
|
||||
(b.id, elapsed_ms, resp.status_code),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
logger.debug(
|
||||
"health_probe_ok",
|
||||
backend_id=b.id,
|
||||
status=resp.status_code,
|
||||
latency_ms=elapsed_ms,
|
||||
)
|
||||
except Exception as probe_err:
|
||||
logger.warning(
|
||||
"health_probe_failed",
|
||||
backend_id=b.id,
|
||||
error=str(probe_err),
|
||||
)
|
||||
# Mark as degraded
|
||||
from storage.db import get_connection as _gc
|
||||
with _gc() as conn:
|
||||
conn.execute(
|
||||
"""INSERT INTO backend_health
|
||||
(backend_id, state, last_check_at)
|
||||
VALUES (?, 'degraded', datetime('now'))
|
||||
ON CONFLICT(backend_id) DO UPDATE SET
|
||||
state = 'degraded',
|
||||
last_check_at = excluded.last_check_at""",
|
||||
(b.id,),
|
||||
)
|
||||
conn.execute(
|
||||
"""UPDATE backend_health SET
|
||||
consecutive_failures = consecutive_failures + 1
|
||||
WHERE backend_id = ?""",
|
||||
(b.id,),
|
||||
)
|
||||
conn.commit()
|
||||
except Exception:
|
||||
logger.exception("health_check_error")
|
||||
await asyncio.sleep(app_config.health_check_interval_seconds)
|
||||
|
||||
|
||||
async def _stats_aggregation_loop() -> None:
|
||||
"""Periodically aggregate daily stats."""
|
||||
while True:
|
||||
try:
|
||||
today = time.strftime("%Y-%m-%d", time.gmtime())
|
||||
aggregate_daily_stats(today)
|
||||
except Exception:
|
||||
logger.exception("stats_aggregation_error")
|
||||
await asyncio.sleep(app_config.stats_refresh_interval_seconds)
|
||||
|
||||
|
||||
async def _backup_loop() -> None:
|
||||
"""Daily SQLite backup with retention."""
|
||||
import shutil
|
||||
|
||||
while True:
|
||||
try:
|
||||
await asyncio.sleep(86400) # 24 hours
|
||||
backup_dir = app_config.backup_dir
|
||||
if not backup_dir:
|
||||
continue
|
||||
|
||||
os.makedirs(backup_dir, exist_ok=True)
|
||||
|
||||
backup_name = f"sidecar_v2_{time.strftime('%Y%m%d_%H%M%S', time.gmtime())}.db"
|
||||
backup_path = os.path.join(backup_dir, backup_name)
|
||||
|
||||
from storage.db import _DB_PATH as db_path
|
||||
import sqlite3
|
||||
|
||||
source = sqlite3.connect(db_path)
|
||||
dest = sqlite3.connect(backup_path)
|
||||
source.backup(dest)
|
||||
dest.close()
|
||||
source.close()
|
||||
|
||||
logger.info("db_backup_created", path=backup_path)
|
||||
|
||||
# Retention: remove old backups
|
||||
retention_days = app_config.backup_retention_days
|
||||
cutoff = time.time() - retention_days * 86400
|
||||
for fname in os.listdir(backup_dir):
|
||||
if fname.startswith("sidecar_v2_") and fname.endswith(".db"):
|
||||
fpath = os.path.join(backup_dir, fname)
|
||||
if os.path.getmtime(fpath) < cutoff:
|
||||
os.remove(fpath)
|
||||
logger.info("db_backup_retired", path=fpath)
|
||||
|
||||
except Exception:
|
||||
logger.exception("backup_error")
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────
|
||||
# Health / Metrics
|
||||
# ──────────────────────────────────────────────────────────
|
||||
@app.get("/health")
|
||||
async def health() -> dict[str, Any]:
|
||||
return {
|
||||
"status": "ok",
|
||||
"version": "2.0.0",
|
||||
"uptime_seconds": int(time.time() - start_time),
|
||||
}
|
||||
|
||||
|
||||
@app.get("/metrics")
|
||||
async def metrics() -> Response:
|
||||
"""Prometheus-compatible metrics endpoint."""
|
||||
lines = []
|
||||
|
||||
# Pool provider counts
|
||||
pool_status = pool_manager.get_pool_status()
|
||||
for pool_name, stats in pool_status.items():
|
||||
for key, val in stats.items():
|
||||
lines.append(
|
||||
f"sidecar_pool_providers{{pool=\"{pool_name}\",type=\"{key}\"}} {val}"
|
||||
)
|
||||
|
||||
# Cooldown status
|
||||
all_backends = list_backends(decrypt_key=False)
|
||||
cooling_count = sum(1 for b in all_backends if b.status == "cooling")
|
||||
lines.append(f"sidecar_cooldown_active {cooling_count}")
|
||||
|
||||
# Emergency count (from proxy module)
|
||||
lines.append(f"sidecar_emergency_count {get_emergency_count()}")
|
||||
|
||||
# DB sizes
|
||||
from storage.db import get_db_sizes
|
||||
sizes = get_db_sizes()
|
||||
lines.append(f"sidecar_db_size_bytes {sizes.get('db_bytes', 0)}")
|
||||
lines.append(f"sidecar_wal_size_bytes {sizes.get('wal_bytes', 0)}")
|
||||
|
||||
# Total stats
|
||||
total = get_total_stats()
|
||||
lines.append(f"sidecar_requests_total {total.get('total_requests', 0) or 0}")
|
||||
lines.append(f"sidecar_errors_total {total.get('total_errors', 0) or 0}")
|
||||
lines.append(f"sidecar_tokens_total {total.get('total_tokens', 0) or 0}")
|
||||
cost = total.get('total_cost', 0) or 0.0
|
||||
lines.append(f"sidecar_cost_total {cost}")
|
||||
|
||||
# Uptime
|
||||
lines.append(f"sidecar_uptime_seconds {int(time.time() - start_time)}")
|
||||
|
||||
return Response(
|
||||
content="\n".join(lines) + "\n",
|
||||
media_type="text/plain; charset=utf-8",
|
||||
)
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────
|
||||
# Dashboard SSE
|
||||
# ──────────────────────────────────────────────────────────
|
||||
@app.get("/dashboard/sse")
|
||||
async def dashboard_sse() -> StreamingResponse:
|
||||
"""SSE endpoint for real-time dashboard data."""
|
||||
|
||||
async def event_generator():
|
||||
while True:
|
||||
try:
|
||||
pool_status = pool_manager.get_pool_status()
|
||||
total_stats = get_total_stats()
|
||||
all_backends = list_backends(decrypt_key=False)
|
||||
|
||||
backends_list = []
|
||||
for b in all_backends:
|
||||
rl_status = rate_limiter.get_status(b.id)
|
||||
backends_list.append({
|
||||
"id": b.id,
|
||||
"name": b.name,
|
||||
"label": b.label,
|
||||
"pool": b.pool,
|
||||
"enabled": b.enabled,
|
||||
"status": b.status,
|
||||
"rpm_limit": b.rpm_limit,
|
||||
"cooldown_until": b.cooldown_until,
|
||||
"consecutive_429_count": b.consecutive_429_count,
|
||||
"model_count": len(b.model_mappings),
|
||||
"rate_limiter": rl_status,
|
||||
})
|
||||
|
||||
snapshot = {
|
||||
"type": "snapshot",
|
||||
"pool": pool_status,
|
||||
"total": total_stats,
|
||||
"backends": backends_list,
|
||||
"uptime_seconds": int(time.time() - start_time),
|
||||
"timestamp": time.time(),
|
||||
}
|
||||
yield f"data: {json.dumps(snapshot)}\n\n"
|
||||
except Exception:
|
||||
logger.exception("sse_error")
|
||||
|
||||
await asyncio.sleep(app_config.dashboard_sse_interval_seconds)
|
||||
|
||||
return StreamingResponse(
|
||||
event_generator(),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"Connection": "keep-alive",
|
||||
"X-Accel-Buffering": "no",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────
|
||||
# Admin: Backend CRUD (READ: public, WRITE: auth required)
|
||||
# ──────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/api/admin/backends")
|
||||
async def admin_list_backends(pool: Optional[str] = None) -> list[dict]:
|
||||
"""List all backends with masked keys (public read)."""
|
||||
backends = list_backends(pool=pool, decrypt_key=True)
|
||||
return [b.to_dict(mask_key=True) for b in backends]
|
||||
|
||||
|
||||
@app.get("/api/admin/backends/{backend_id}")
|
||||
async def admin_get_backend(backend_id: str) -> dict:
|
||||
"""Get a single backend (public read, key masked)."""
|
||||
b = get_backend(backend_id, decrypt_key=True)
|
||||
if b is None:
|
||||
raise HTTPException(404, "Backend not found")
|
||||
return b.to_dict(mask_key=True)
|
||||
|
||||
|
||||
@app.post("/api/admin/backends")
|
||||
async def admin_create_backend(
|
||||
body: dict[str, Any],
|
||||
_auth=Depends(require_admin),
|
||||
) -> dict:
|
||||
"""Create a new backend (auth required)."""
|
||||
required = ["name", "api_base_url", "api_key"]
|
||||
for field in required:
|
||||
if field not in body:
|
||||
raise HTTPException(400, f"Missing required field: {field}")
|
||||
|
||||
model_mappings_raw = body.get("model_mappings", {})
|
||||
model_mappings = {}
|
||||
for canonical_name, mm in model_mappings_raw.items():
|
||||
model_mappings[canonical_name] = ModelMapping.from_dict(mm)
|
||||
|
||||
backend = Backend(
|
||||
name=body["name"],
|
||||
label=body.get("label", ""),
|
||||
api_base_url=body["api_base_url"],
|
||||
api_key_plain=body["api_key"],
|
||||
api=body.get("api", "openai-completions"),
|
||||
timeout_seconds=body.get("timeout_seconds", 120),
|
||||
rpm_limit=body.get("rpm_limit", app_config.default_rpm_limit),
|
||||
pool=body.get("pool", "primary"),
|
||||
enabled=body.get("enabled", True),
|
||||
model_mappings=model_mappings,
|
||||
source=body.get("source", "webui"),
|
||||
)
|
||||
|
||||
created = create_backend(backend)
|
||||
return created.to_dict(mask_key=True)
|
||||
|
||||
|
||||
@app.put("/api/admin/backends/{backend_id}")
|
||||
async def admin_update_backend(
|
||||
backend_id: str,
|
||||
body: dict[str, Any],
|
||||
_auth=Depends(require_admin),
|
||||
) -> dict:
|
||||
"""Update a backend (auth required)."""
|
||||
updates = dict(body)
|
||||
|
||||
if "model_mappings" in updates:
|
||||
raw = updates["model_mappings"]
|
||||
updates["model_mappings"] = {
|
||||
k: ModelMapping.from_dict(v) for k, v in raw.items()
|
||||
}
|
||||
|
||||
if "api_key" in updates:
|
||||
updates["api_key_plain"] = updates.pop("api_key")
|
||||
|
||||
updated = update_backend(backend_id, updates)
|
||||
if updated is None:
|
||||
raise HTTPException(404, "Backend not found")
|
||||
return updated.to_dict(mask_key=True)
|
||||
|
||||
|
||||
@app.delete("/api/admin/backends/{backend_id}")
|
||||
async def admin_delete_backend(
|
||||
backend_id: str,
|
||||
_auth=Depends(require_admin),
|
||||
) -> dict:
|
||||
"""Delete a backend (auth required)."""
|
||||
ok = delete_backend(backend_id)
|
||||
if not ok:
|
||||
raise HTTPException(404, "Backend not found")
|
||||
return {"status": "deleted", "id": backend_id}
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────
|
||||
# Admin: Pool Status (public read)
|
||||
# ──────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/api/admin/pools")
|
||||
async def admin_pool_status() -> dict:
|
||||
return pool_manager.get_pool_status()
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────
|
||||
# Admin: Usage / Stats (public read)
|
||||
# ──────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/api/admin/stats/total")
|
||||
async def admin_total_stats() -> dict:
|
||||
return get_total_stats()
|
||||
|
||||
|
||||
@app.get("/api/admin/stats/hourly")
|
||||
async def admin_hourly_usage(
|
||||
backend_id: Optional[str] = None,
|
||||
hours: int = 168,
|
||||
) -> list[dict]:
|
||||
since = None
|
||||
if hours > 0:
|
||||
since = time.strftime(
|
||||
"%Y-%m-%dT%H:%M:%SZ",
|
||||
time.gmtime(time.time() - hours * 3600),
|
||||
)
|
||||
return get_hourly_usage(backend_id=backend_id, since=since, limit=hours)
|
||||
|
||||
|
||||
@app.get("/api/admin/stats/daily")
|
||||
async def admin_daily_stats(days: int = 30) -> list[dict]:
|
||||
return get_daily_stats(days=days)
|
||||
|
||||
|
||||
@app.get("/api/admin/stats/cooldown")
|
||||
async def admin_cooldown_history(
|
||||
backend_id: Optional[str] = None,
|
||||
limit: int = 50,
|
||||
) -> list[dict]:
|
||||
return get_cooldown_history(backend_id=backend_id, limit=limit)
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────
|
||||
# Admin: System Config (read public, write auth required)
|
||||
# ──────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/api/admin/config")
|
||||
async def admin_get_all_config() -> list[dict]:
|
||||
return list_configs()
|
||||
|
||||
|
||||
@app.get("/api/admin/config/{key}")
|
||||
async def admin_get_config(key: str) -> dict:
|
||||
value = get_config(key)
|
||||
if value is None:
|
||||
raise HTTPException(404, "Config not found")
|
||||
return {"key": key, "value": value}
|
||||
|
||||
|
||||
@app.put("/api/admin/config/{key}")
|
||||
async def admin_set_config(
|
||||
key: str,
|
||||
body: dict[str, Any],
|
||||
_auth=Depends(require_admin),
|
||||
) -> dict:
|
||||
value = str(body.get("value", ""))
|
||||
description = str(body.get("description", ""))
|
||||
set_config(key, value, description)
|
||||
return {"key": key, "value": value}
|
||||
|
||||
|
||||
@app.delete("/api/admin/config/{key}")
|
||||
async def admin_delete_config(
|
||||
key: str,
|
||||
_auth=Depends(require_admin),
|
||||
) -> dict:
|
||||
ok = delete_config(key)
|
||||
if not ok:
|
||||
raise HTTPException(404, "Config not found")
|
||||
return {"status": "deleted", "key": key}
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────
|
||||
# Dashboard HTML (public, but respects admin_token for writes in JS)
|
||||
# ──────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/dashboard")
|
||||
async def dashboard_html() -> HTMLResponse:
|
||||
dashboard_path = os.path.join(
|
||||
os.path.dirname(__file__), "dashboard.html"
|
||||
)
|
||||
if os.path.exists(dashboard_path):
|
||||
with open(dashboard_path, "r") as f:
|
||||
return HTMLResponse(f.read())
|
||||
return HTMLResponse("<h1>Dashboard not found</h1>", status_code=404)
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────
|
||||
# Proxy Endpoints
|
||||
# ──────────────────────────────────────────────────────────
|
||||
|
||||
@app.post("/v1/chat/completions")
|
||||
async def chat_completions(request: Request) -> Response:
|
||||
_inc_metric("proxy_requests_total")
|
||||
return await handle_proxy_request(
|
||||
pool_manager, rate_limiter, router, request, "/v1/chat/completions"
|
||||
)
|
||||
|
||||
|
||||
@app.post("/v1/completions")
|
||||
async def completions(request: Request) -> Response:
|
||||
return await handle_proxy_request(
|
||||
pool_manager, rate_limiter, router, request, "/v1/completions"
|
||||
)
|
||||
|
||||
|
||||
@app.post("/v1/embeddings")
|
||||
async def embeddings(request: Request) -> Response:
|
||||
return await handle_proxy_request(
|
||||
pool_manager, rate_limiter, router, request, "/v1/embeddings"
|
||||
)
|
||||
|
||||
|
||||
@app.get("/v1/models")
|
||||
@app.get("/v1/models/{model_id:path}")
|
||||
async def list_models(request: Request, model_id: Optional[str] = None) -> Response:
|
||||
path = f"/v1/models/{model_id}" if model_id else "/v1/models"
|
||||
return await handle_proxy_request(
|
||||
pool_manager, rate_limiter, router, request, path
|
||||
)
|
||||
|
||||
|
||||
@app.api_route("/{path:path}", methods=["GET", "POST", "PUT", "DELETE", "PATCH", "OPTIONS"])
|
||||
async def catch_all(request: Request, path: str) -> Response:
|
||||
target_path = f"/{path}" if not path.startswith("/") else path
|
||||
return await handle_proxy_request(
|
||||
pool_manager, rate_limiter, router, request, target_path
|
||||
)
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────
|
||||
# Main
|
||||
# ──────────────────────────────────────────────────────────
|
||||
|
||||
def main() -> None:
|
||||
import uvicorn
|
||||
|
||||
uvicorn.run(
|
||||
"server:app",
|
||||
host=app_config.host,
|
||||
port=app_config.port,
|
||||
log_level=app_config.log_level.lower(),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1 @@
|
||||
# Sidecar V2 storage module
|
||||
@@ -0,0 +1,252 @@
|
||||
"""CRUD operations for Backend (provider) management."""
|
||||
|
||||
import json
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
from storage.db import get_connection, generate_id
|
||||
from storage.models import Backend, ModelMapping
|
||||
from crypto import encrypt, decrypt
|
||||
|
||||
|
||||
def create_backend(backend: Backend) -> Backend:
|
||||
"""Create a new backend. Encrypts API key before storage."""
|
||||
if not backend.id:
|
||||
backend.id = generate_id("bkd")
|
||||
|
||||
now = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
||||
backend.created_at = now
|
||||
backend.updated_at = now
|
||||
|
||||
api_key_encrypted = encrypt(backend.api_key_plain)
|
||||
|
||||
with get_connection() as conn:
|
||||
conn.execute(
|
||||
"""INSERT INTO backends (id, name, label, api_base_url, api_key_encrypted,
|
||||
api, timeout_seconds, rpm_limit, pool, enabled, status, model_mappings_json,
|
||||
source, cooldown_until, consecutive_429_count, metadata_json, created_at, updated_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
(
|
||||
backend.id, backend.name, backend.label, backend.api_base_url,
|
||||
api_key_encrypted, backend.api, backend.timeout_seconds,
|
||||
backend.rpm_limit, backend.pool, 1 if backend.enabled else 0,
|
||||
backend.status, json.dumps(_mappings_to_dict(backend.model_mappings)),
|
||||
backend.source, backend.cooldown_until,
|
||||
backend.consecutive_429_count,
|
||||
json.dumps(backend.metadata), backend.created_at, backend.updated_at,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
return backend
|
||||
|
||||
|
||||
def get_backend(backend_id: str, decrypt_key: bool = True) -> Optional[Backend]:
|
||||
"""Get a single backend by ID."""
|
||||
with get_connection() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT * FROM backends WHERE id = ?", (backend_id,)
|
||||
).fetchone()
|
||||
|
||||
if row is None:
|
||||
return None
|
||||
|
||||
return _row_to_backend(row, decrypt_key=decrypt_key)
|
||||
|
||||
|
||||
def list_backends(
|
||||
pool: Optional[str] = None,
|
||||
enabled_only: bool = False,
|
||||
decrypt_key: bool = False,
|
||||
) -> list[Backend]:
|
||||
"""List backends, optionally filtered by pool."""
|
||||
with get_connection() as conn:
|
||||
if pool:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM backends WHERE pool = ? ORDER BY created_at",
|
||||
(pool,),
|
||||
).fetchall()
|
||||
else:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM backends ORDER BY pool, created_at"
|
||||
).fetchall()
|
||||
|
||||
backends = [_row_to_backend(r, decrypt_key=decrypt_key) for r in rows]
|
||||
if enabled_only:
|
||||
backends = [b for b in backends if b.enabled]
|
||||
return backends
|
||||
|
||||
|
||||
def update_backend(backend_id: str, updates: dict) -> Optional[Backend]:
|
||||
"""Update backend fields. If api_key_plain is provided, re-encrypt."""
|
||||
current = get_backend(backend_id, decrypt_key=True)
|
||||
if current is None:
|
||||
return None
|
||||
|
||||
# Apply updates
|
||||
allowed = {
|
||||
"name", "label", "api_base_url", "api", "timeout_seconds",
|
||||
"rpm_limit", "pool", "enabled", "status", "source",
|
||||
"cooldown_until", "consecutive_429_count", "metadata",
|
||||
}
|
||||
for key, value in updates.items():
|
||||
if key in allowed:
|
||||
setattr(current, key, value)
|
||||
|
||||
current.updated_at = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
||||
|
||||
# Handle API key update
|
||||
api_key_encrypted = None
|
||||
if "api_key_plain" in updates and updates["api_key_plain"]:
|
||||
current.api_key_plain = updates["api_key_plain"]
|
||||
api_key_encrypted = encrypt(updates["api_key_plain"])
|
||||
|
||||
# Handle model_mappings update
|
||||
mappings_json = None
|
||||
if "model_mappings" in updates:
|
||||
current.model_mappings = updates["model_mappings"]
|
||||
mappings_json = json.dumps(_mappings_to_dict(current.model_mappings))
|
||||
|
||||
with get_connection() as conn:
|
||||
# Build dynamic UPDATE
|
||||
set_clauses = [
|
||||
"name = ?", "label = ?", "api_base_url = ?", "api = ?",
|
||||
"timeout_seconds = ?", "rpm_limit = ?", "pool = ?", "enabled = ?",
|
||||
"status = ?", "source = ?", "cooldown_until = ?",
|
||||
"consecutive_429_count = ?", "metadata_json = ?", "updated_at = ?",
|
||||
]
|
||||
params = [
|
||||
current.name, current.label, current.api_base_url, current.api,
|
||||
current.timeout_seconds, current.rpm_limit, current.pool,
|
||||
1 if current.enabled else 0, current.status, current.source,
|
||||
current.cooldown_until, current.consecutive_429_count,
|
||||
json.dumps(current.metadata), current.updated_at,
|
||||
]
|
||||
if api_key_encrypted:
|
||||
set_clauses.append("api_key_encrypted = ?")
|
||||
params.append(api_key_encrypted)
|
||||
if mappings_json is not None:
|
||||
set_clauses.append("model_mappings_json = ?")
|
||||
params.append(mappings_json)
|
||||
params.append(backend_id)
|
||||
|
||||
conn.execute(
|
||||
f"UPDATE backends SET {', '.join(set_clauses)} WHERE id = ?",
|
||||
params,
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
return get_backend(backend_id, decrypt_key=False)
|
||||
|
||||
|
||||
def delete_backend(backend_id: str) -> bool:
|
||||
"""Delete a backend. Returns True if deleted."""
|
||||
with get_connection() as conn:
|
||||
cursor = conn.execute("DELETE FROM backends WHERE id = ?", (backend_id,))
|
||||
conn.commit()
|
||||
return cursor.rowcount > 0
|
||||
|
||||
|
||||
def set_backend_status(backend_id: str, status: str) -> bool:
|
||||
"""Quickly set backend status (healthy/cooling/error/disabled)."""
|
||||
now = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
||||
with get_connection() as conn:
|
||||
cursor = conn.execute(
|
||||
"UPDATE backends SET status = ?, updated_at = ? WHERE id = ?",
|
||||
(status, now, backend_id),
|
||||
)
|
||||
conn.commit()
|
||||
return cursor.rowcount > 0
|
||||
|
||||
|
||||
def set_backend_cooldown(backend_id: str, cooldown_until: str, count: int) -> bool:
|
||||
"""Set cooldown state on a backend."""
|
||||
now = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
||||
with get_connection() as conn:
|
||||
cursor = conn.execute(
|
||||
"""UPDATE backends SET status = 'cooling', cooldown_until = ?,
|
||||
consecutive_429_count = ?, updated_at = ? WHERE id = ?""",
|
||||
(cooldown_until, count, now, backend_id),
|
||||
)
|
||||
conn.commit()
|
||||
return cursor.rowcount > 0
|
||||
|
||||
|
||||
def clear_backend_cooldown(backend_id: str) -> bool:
|
||||
"""Clear cooldown (back to healthy)."""
|
||||
now = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
||||
with get_connection() as conn:
|
||||
cursor = conn.execute(
|
||||
"""UPDATE backends SET status = 'healthy', cooldown_until = NULL,
|
||||
consecutive_429_count = 0, updated_at = ? WHERE id = ?""",
|
||||
(now, backend_id),
|
||||
)
|
||||
conn.commit()
|
||||
return cursor.rowcount > 0
|
||||
|
||||
|
||||
def get_pool_stats() -> dict:
|
||||
"""Get summary stats per pool."""
|
||||
with get_connection() as conn:
|
||||
rows = conn.execute(
|
||||
"""SELECT pool, COUNT(*) as total,
|
||||
SUM(CASE WHEN enabled = 1 THEN 1 ELSE 0 END) as enabled,
|
||||
SUM(CASE WHEN status = 'healthy' THEN 1 ELSE 0 END) as healthy,
|
||||
SUM(CASE WHEN status = 'cooling' THEN 1 ELSE 0 END) as cooling,
|
||||
SUM(CASE WHEN status = 'error' THEN 1 ELSE 0 END) as error
|
||||
FROM backends GROUP BY pool"""
|
||||
).fetchall()
|
||||
stats = {}
|
||||
for row in rows:
|
||||
stats[row["pool"]] = {
|
||||
"total": row["total"],
|
||||
"enabled": row["enabled"],
|
||||
"healthy": row["healthy"],
|
||||
"cooling": row["cooling"],
|
||||
"error": row["error"],
|
||||
}
|
||||
return stats
|
||||
|
||||
|
||||
def _row_to_backend(row, decrypt_key: bool = True) -> Backend:
|
||||
"""Convert a DB row to a Backend instance."""
|
||||
mappings_raw = row["model_mappings_json"] or "{}"
|
||||
mappings_dict = json.loads(mappings_raw)
|
||||
|
||||
model_mappings = {}
|
||||
for canonical_name, mm in mappings_dict.items():
|
||||
model_mappings[canonical_name] = ModelMapping.from_dict(mm)
|
||||
|
||||
backend = Backend(
|
||||
id=row["id"],
|
||||
name=row["name"],
|
||||
label=row["label"],
|
||||
api_base_url=row["api_base_url"],
|
||||
api_key_encrypted=row["api_key_encrypted"] or "",
|
||||
api=row["api"],
|
||||
timeout_seconds=row["timeout_seconds"],
|
||||
rpm_limit=row["rpm_limit"],
|
||||
pool=row["pool"],
|
||||
enabled=bool(row["enabled"]),
|
||||
status=row["status"],
|
||||
model_mappings=model_mappings,
|
||||
source=row["source"],
|
||||
cooldown_until=row["cooldown_until"],
|
||||
consecutive_429_count=row["consecutive_429_count"],
|
||||
metadata=json.loads(row["metadata_json"] or "{}"),
|
||||
created_at=row["created_at"],
|
||||
updated_at=row["updated_at"],
|
||||
)
|
||||
|
||||
if decrypt_key and backend.api_key_encrypted:
|
||||
from crypto import try_decrypt_existing
|
||||
plain = try_decrypt_existing(backend.api_key_encrypted)
|
||||
if plain:
|
||||
backend.api_key_plain = plain
|
||||
|
||||
return backend
|
||||
|
||||
|
||||
def _mappings_to_dict(mappings: dict[str, ModelMapping]) -> dict:
|
||||
"""Convert ModelMapping dict to JSON-safe dict."""
|
||||
return {k: v.to_dict() for k, v in mappings.items()}
|
||||
@@ -0,0 +1,55 @@
|
||||
"""System configuration KV store operations."""
|
||||
|
||||
import time
|
||||
from typing import Optional, Any
|
||||
|
||||
from storage.db import get_connection
|
||||
|
||||
|
||||
def get_config(key: str) -> Optional[str]:
|
||||
"""Get a single config value."""
|
||||
with get_connection() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT value FROM system_config WHERE key = ?", (key,)
|
||||
).fetchone()
|
||||
return row["value"] if row else None
|
||||
|
||||
|
||||
def set_config(key: str, value: str, description: str = "") -> None:
|
||||
"""Set or update a config value."""
|
||||
now = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
||||
with get_connection() as conn:
|
||||
conn.execute(
|
||||
"""INSERT INTO system_config (key, value, description, updated_at)
|
||||
VALUES (?, ?, ?, ?)
|
||||
ON CONFLICT(key) DO UPDATE SET
|
||||
value = excluded.value,
|
||||
description = excluded.description,
|
||||
updated_at = excluded.updated_at""",
|
||||
(key, value, description, now),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def delete_config(key: str) -> bool:
|
||||
"""Delete a config value."""
|
||||
with get_connection() as conn:
|
||||
cursor = conn.execute(
|
||||
"DELETE FROM system_config WHERE key = ?", (key,)
|
||||
)
|
||||
conn.commit()
|
||||
return cursor.rowcount > 0
|
||||
|
||||
|
||||
def list_configs() -> list[dict]:
|
||||
"""List all system config entries."""
|
||||
with get_connection() as conn:
|
||||
rows = conn.execute("SELECT * FROM system_config ORDER BY key").fetchall()
|
||||
return [dict(row) for row in rows]
|
||||
|
||||
|
||||
def get_all_configs_as_dict() -> dict[str, str]:
|
||||
"""Get all configs as a simple dict."""
|
||||
with get_connection() as conn:
|
||||
rows = conn.execute("SELECT key, value FROM system_config").fetchall()
|
||||
return {row["key"]: row["value"] for row in rows}
|
||||
@@ -0,0 +1,74 @@
|
||||
"""Cooldown event logging."""
|
||||
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
from storage.db import get_connection, generate_id
|
||||
from storage.models import CooldownEvent
|
||||
|
||||
|
||||
def log_cooldown_event(
|
||||
backend_id: str,
|
||||
consecutive_count: int,
|
||||
cooldown_seconds: int,
|
||||
response_summary: str = "",
|
||||
) -> CooldownEvent:
|
||||
"""Record a cooldown event."""
|
||||
event = CooldownEvent(
|
||||
id=generate_id("cev"),
|
||||
backend_id=backend_id,
|
||||
consecutive_count=consecutive_count,
|
||||
cooldown_seconds=cooldown_seconds,
|
||||
response_summary=response_summary,
|
||||
started_at=time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
)
|
||||
|
||||
with get_connection() as conn:
|
||||
conn.execute(
|
||||
"""INSERT INTO cooldown_events
|
||||
(id, backend_id, consecutive_count, cooldown_seconds,
|
||||
response_summary, started_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?)""",
|
||||
(event.id, event.backend_id, event.consecutive_count,
|
||||
event.cooldown_seconds, event.response_summary, event.started_at),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
return event
|
||||
|
||||
|
||||
def end_cooldown_event(backend_id: str) -> bool:
|
||||
"""Mark the latest open cooldown event as ended."""
|
||||
ended_at = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
||||
with get_connection() as conn:
|
||||
# Find the latest event for this backend that hasn't ended
|
||||
cursor = conn.execute(
|
||||
"""UPDATE cooldown_events SET ended_at = ?
|
||||
WHERE backend_id = ? AND ended_at IS NULL
|
||||
ORDER BY started_at DESC LIMIT 1""",
|
||||
(ended_at, backend_id),
|
||||
)
|
||||
conn.commit()
|
||||
return cursor.rowcount > 0
|
||||
|
||||
|
||||
def get_cooldown_history(
|
||||
backend_id: Optional[str] = None,
|
||||
limit: int = 50,
|
||||
) -> list[dict]:
|
||||
"""Get cooldown event history."""
|
||||
with get_connection() as conn:
|
||||
if backend_id:
|
||||
rows = conn.execute(
|
||||
"""SELECT * FROM cooldown_events
|
||||
WHERE backend_id = ?
|
||||
ORDER BY started_at DESC LIMIT ?""",
|
||||
(backend_id, limit),
|
||||
).fetchall()
|
||||
else:
|
||||
rows = conn.execute(
|
||||
"""SELECT * FROM cooldown_events
|
||||
ORDER BY started_at DESC LIMIT ?""",
|
||||
(limit,),
|
||||
).fetchall()
|
||||
return [dict(row) for row in rows]
|
||||
@@ -0,0 +1,193 @@
|
||||
"""SQLite database connection management with WAL mode."""
|
||||
|
||||
import os
|
||||
import sqlite3
|
||||
import uuid
|
||||
import structlog
|
||||
from contextlib import contextmanager
|
||||
from typing import Generator
|
||||
|
||||
from config import config
|
||||
|
||||
logger = structlog.get_logger()
|
||||
|
||||
# Module-level DB path
|
||||
_DB_PATH: str = ""
|
||||
|
||||
|
||||
def init_db(db_path: str = "") -> None:
|
||||
"""Initialize the database connection and ensure WAL mode.
|
||||
|
||||
Creates the data directory if needed and verifies integrity.
|
||||
"""
|
||||
global _DB_PATH
|
||||
_DB_PATH = db_path or config.db_path
|
||||
|
||||
# Ensure data directory exists
|
||||
os.makedirs(os.path.dirname(_DB_PATH), exist_ok=True)
|
||||
|
||||
# Test connection and enable WAL
|
||||
conn = _get_raw_connection()
|
||||
try:
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
conn.execute("PRAGMA wal_autocheckpoint=1000")
|
||||
conn.execute("PRAGMA foreign_keys=ON")
|
||||
conn.execute("PRAGMA busy_timeout=5000")
|
||||
logger.info("db_initialized", path=_DB_PATH, mode="WAL")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def _get_raw_connection() -> sqlite3.Connection:
|
||||
"""Get a raw sqlite3 connection."""
|
||||
conn = sqlite3.connect(_DB_PATH, check_same_thread=False)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
conn.execute("PRAGMA foreign_keys=ON")
|
||||
return conn
|
||||
|
||||
|
||||
@contextmanager
|
||||
def get_connection() -> Generator[sqlite3.Connection, None, None]:
|
||||
"""Get a database connection with WAL enabled."""
|
||||
conn = _get_raw_connection()
|
||||
try:
|
||||
yield conn
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def generate_id(prefix: str = "") -> str:
|
||||
"""Generate a unique ID with optional prefix."""
|
||||
uid = uuid.uuid4().hex[:12]
|
||||
return f"{prefix}_{uid}" if prefix else uid
|
||||
|
||||
|
||||
def create_tables() -> None:
|
||||
"""Create all tables if they don't exist."""
|
||||
with get_connection() as conn:
|
||||
conn.executescript(_DDL)
|
||||
conn.commit()
|
||||
logger.info("tables_created")
|
||||
|
||||
|
||||
def run_integrity_check() -> bool:
|
||||
"""Run PRAGMA integrity_check and return True if OK."""
|
||||
with get_connection() as conn:
|
||||
result = conn.execute("PRAGMA integrity_check").fetchone()
|
||||
ok = result[0] == "ok"
|
||||
if not ok:
|
||||
logger.error("integrity_check_failed", result=result[0])
|
||||
return ok
|
||||
|
||||
|
||||
def get_db_sizes() -> dict:
|
||||
"""Get database and WAL file sizes."""
|
||||
result = {"db_bytes": 0, "wal_bytes": 0}
|
||||
db_path = _DB_PATH
|
||||
if os.path.exists(db_path):
|
||||
result["db_bytes"] = os.path.getsize(db_path)
|
||||
wal_path = db_path + "-wal"
|
||||
if os.path.exists(wal_path):
|
||||
result["wal_bytes"] = os.path.getsize(wal_path)
|
||||
return result
|
||||
|
||||
|
||||
def wal_checkpoint(mode: str = "TRUNCATE") -> None:
|
||||
"""Execute WAL checkpoint."""
|
||||
with get_connection() as conn:
|
||||
conn.execute(f"PRAGMA wal_checkpoint({mode})")
|
||||
|
||||
|
||||
_DDL = """
|
||||
-- Backend configuration table (core)
|
||||
CREATE TABLE IF NOT EXISTS backends (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
label TEXT DEFAULT '',
|
||||
api_base_url TEXT NOT NULL,
|
||||
api_key_encrypted TEXT NOT NULL,
|
||||
api TEXT NOT NULL DEFAULT 'openai-completions',
|
||||
timeout_seconds INTEGER NOT NULL DEFAULT 120,
|
||||
rpm_limit INTEGER NOT NULL DEFAULT 40,
|
||||
pool TEXT NOT NULL DEFAULT 'primary'
|
||||
CHECK(pool IN ('primary', 'fallback')),
|
||||
enabled INTEGER NOT NULL DEFAULT 1,
|
||||
status TEXT NOT NULL DEFAULT 'healthy'
|
||||
CHECK(status IN ('healthy', 'cooling', 'error', 'disabled')),
|
||||
model_mappings_json TEXT DEFAULT '{}',
|
||||
source TEXT NOT NULL DEFAULT 'webui'
|
||||
CHECK(source IN ('webui', 'env', 'import')),
|
||||
cooldown_until TEXT,
|
||||
consecutive_429_count INTEGER DEFAULT 0,
|
||||
metadata_json TEXT DEFAULT '{}',
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
);
|
||||
|
||||
-- Usage logs (hour-bucketed, UPSERT-safe)
|
||||
CREATE TABLE IF NOT EXISTS backend_usage_logs (
|
||||
id TEXT PRIMARY KEY,
|
||||
backend_id TEXT NOT NULL REFERENCES backends(id) ON DELETE CASCADE,
|
||||
model TEXT DEFAULT 'unknown',
|
||||
prompt_tokens INTEGER DEFAULT 0,
|
||||
completion_tokens INTEGER DEFAULT 0,
|
||||
total_tokens INTEGER DEFAULT 0,
|
||||
cost REAL DEFAULT 0.0,
|
||||
request_count INTEGER DEFAULT 0,
|
||||
error_count INTEGER DEFAULT 0,
|
||||
avg_latency_ms INTEGER DEFAULT 0,
|
||||
ttft_ms INTEGER DEFAULT 0,
|
||||
hour_bucket TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_usage_backend_hour
|
||||
ON backend_usage_logs(backend_id, hour_bucket);
|
||||
|
||||
-- Cooldown event log
|
||||
CREATE TABLE IF NOT EXISTS cooldown_events (
|
||||
id TEXT PRIMARY KEY,
|
||||
backend_id TEXT NOT NULL REFERENCES backends(id) ON DELETE CASCADE,
|
||||
consecutive_count INTEGER NOT NULL DEFAULT 1,
|
||||
cooldown_seconds INTEGER NOT NULL,
|
||||
response_summary TEXT DEFAULT '',
|
||||
started_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
ended_at TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_cooldown_backend_time
|
||||
ON cooldown_events(backend_id, started_at);
|
||||
|
||||
-- Backend health state
|
||||
CREATE TABLE IF NOT EXISTS backend_health (
|
||||
backend_id TEXT PRIMARY KEY REFERENCES backends(id) ON DELETE CASCADE,
|
||||
state TEXT NOT NULL DEFAULT 'healthy'
|
||||
CHECK(state IN ('healthy', 'degraded', 'down')),
|
||||
last_latency_ms INTEGER DEFAULT 0,
|
||||
last_status_code INTEGER DEFAULT 200,
|
||||
success_rate_5m REAL DEFAULT 1.0,
|
||||
consecutive_failures INTEGER DEFAULT 0,
|
||||
last_check_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
);
|
||||
|
||||
-- System configuration KV store
|
||||
CREATE TABLE IF NOT EXISTS system_config (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL,
|
||||
description TEXT DEFAULT '',
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
);
|
||||
|
||||
-- Daily aggregated stats
|
||||
CREATE TABLE IF NOT EXISTS daily_stats (
|
||||
id TEXT PRIMARY KEY,
|
||||
date TEXT NOT NULL,
|
||||
pool TEXT NOT NULL CHECK(pool IN ('primary', 'fallback')),
|
||||
total_requests INTEGER DEFAULT 0,
|
||||
total_errors INTEGER DEFAULT 0,
|
||||
total_tokens INTEGER DEFAULT 0,
|
||||
total_cost REAL DEFAULT 0.0,
|
||||
unique_backends INTEGER DEFAULT 0,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_daily_date_pool ON daily_stats(date, pool);
|
||||
"""
|
||||
@@ -0,0 +1,161 @@
|
||||
"""Data models for Sidecar V2 — backend-centric, Canonical Name routing."""
|
||||
|
||||
from dataclasses import dataclass, field, asdict
|
||||
from typing import Optional
|
||||
import json
|
||||
|
||||
|
||||
@dataclass
|
||||
class ModelMapping:
|
||||
"""A single model mapping within a backend: Canonical Name → native_id + properties."""
|
||||
|
||||
native_id: str
|
||||
reasoning: bool = False
|
||||
reasoning_effort: bool = False
|
||||
input_modalities: list[str] = field(default_factory=lambda: ["text"])
|
||||
cost: dict = field(default_factory=lambda: {
|
||||
"input": 0.0, "output": 0.0, "cacheRead": 0.0, "cacheWrite": 0.0
|
||||
})
|
||||
context_window: int = 128000
|
||||
max_tokens: int = 65536
|
||||
compat: dict = field(default_factory=dict)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return asdict(self)
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, d: dict) -> "ModelMapping":
|
||||
defaults = {
|
||||
"native_id": "",
|
||||
"reasoning": False,
|
||||
"reasoning_effort": False,
|
||||
"input_modalities": ["text"],
|
||||
"cost": {"input": 0.0, "output": 0.0, "cacheRead": 0.0, "cacheWrite": 0.0},
|
||||
"context_window": 128000,
|
||||
"max_tokens": 65536,
|
||||
"compat": {},
|
||||
}
|
||||
defaults.update(d)
|
||||
return cls(**{k: v for k, v in defaults.items() if k in cls.__dataclass_fields__})
|
||||
|
||||
|
||||
@dataclass
|
||||
class Backend:
|
||||
"""A physical API backend (API Key + URL).
|
||||
|
||||
Represents a single API key endpoint. Multiple backends can serve the same
|
||||
Canonical Models through their model_mappings.
|
||||
"""
|
||||
|
||||
id: str = ""
|
||||
name: str = ""
|
||||
label: str = "" # e.g., "nvidia", "siliconflow" — WebUI tag only
|
||||
api_base_url: str = ""
|
||||
api_key_encrypted: str = ""
|
||||
api: str = "openai-completions"
|
||||
timeout_seconds: int = 120
|
||||
rpm_limit: int = 40
|
||||
pool: str = "primary" # primary | fallback
|
||||
enabled: bool = True
|
||||
status: str = "healthy" # healthy | cooling | error | disabled
|
||||
model_mappings: dict[str, ModelMapping] = field(default_factory=dict)
|
||||
source: str = "webui" # webui | env | import
|
||||
cooldown_until: Optional[str] = None
|
||||
consecutive_429_count: int = 0
|
||||
metadata: dict = field(default_factory=dict)
|
||||
created_at: str = ""
|
||||
updated_at: str = ""
|
||||
|
||||
# Runtime fields (not persisted)
|
||||
api_key_plain: str = "" # decrypted at load time, not serialized to DB
|
||||
|
||||
def has_model(self, canonical_name: str) -> bool:
|
||||
"""Check if backend supports a given Canonical Model."""
|
||||
return canonical_name in self.model_mappings
|
||||
|
||||
def get_native_id(self, canonical_name: str) -> str:
|
||||
"""Get this backend's native model ID for a Canonical Name."""
|
||||
mm = self.model_mappings.get(canonical_name)
|
||||
return mm.native_id if mm else canonical_name
|
||||
|
||||
def get_model_cost(self, canonical_name: str) -> dict:
|
||||
"""Get cost info for a Canonical Model on this backend."""
|
||||
mm = self.model_mappings.get(canonical_name)
|
||||
return mm.cost if mm else {"input": 0.0, "output": 0.0, "cacheRead": 0.0, "cacheWrite": 0.0}
|
||||
|
||||
def to_dict(self, mask_key: bool = True) -> dict:
|
||||
"""Convert to dict for API responses."""
|
||||
d = asdict(self)
|
||||
# Remove runtime-only fields
|
||||
d.pop("api_key_plain", None)
|
||||
d.pop("api_key_encrypted", None)
|
||||
|
||||
# Mask API key
|
||||
if mask_key and self.api_key_plain:
|
||||
d["api_key"] = _mask_key(self.api_key_plain)
|
||||
elif self.api_key_plain:
|
||||
d["api_key"] = self.api_key_plain
|
||||
else:
|
||||
d["api_key"] = ""
|
||||
|
||||
# Convert model_mappings to dict for serialization
|
||||
d["model_mappings"] = {
|
||||
k: v.to_dict() for k, v in self.model_mappings.items()
|
||||
}
|
||||
return d
|
||||
|
||||
|
||||
def _mask_key(key: str) -> str:
|
||||
if len(key) <= 10:
|
||||
return key[:2] + "****"
|
||||
return key[:6] + "****" + key[-4:]
|
||||
|
||||
|
||||
@dataclass
|
||||
class CooldownEvent:
|
||||
id: str = ""
|
||||
backend_id: str = ""
|
||||
consecutive_count: int = 1
|
||||
cooldown_seconds: int = 60
|
||||
response_summary: str = ""
|
||||
started_at: str = ""
|
||||
ended_at: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class BackendHealth:
|
||||
backend_id: str = ""
|
||||
state: str = "healthy" # healthy | degraded | down
|
||||
last_latency_ms: int = 0
|
||||
last_status_code: int = 200
|
||||
success_rate_5m: float = 1.0
|
||||
consecutive_failures: int = 0
|
||||
last_check_at: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class UsageLog:
|
||||
id: str = ""
|
||||
backend_id: str = ""
|
||||
model: str = "unknown"
|
||||
prompt_tokens: int = 0
|
||||
completion_tokens: int = 0
|
||||
total_tokens: int = 0
|
||||
cost: float = 0.0
|
||||
request_count: int = 0
|
||||
error_count: int = 0
|
||||
avg_latency_ms: int = 0
|
||||
ttft_ms: int = 0
|
||||
hour_bucket: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class DailyStats:
|
||||
id: str = ""
|
||||
date: str = ""
|
||||
pool: str = "primary"
|
||||
total_requests: int = 0
|
||||
total_errors: int = 0
|
||||
total_tokens: int = 0
|
||||
total_cost: float = 0.0
|
||||
unique_backends: int = 0
|
||||
@@ -0,0 +1,155 @@
|
||||
"""Usage logging and daily statistics aggregation."""
|
||||
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
from storage.db import get_connection, generate_id
|
||||
|
||||
|
||||
def record_usage(
|
||||
backend_id: str,
|
||||
model: str,
|
||||
prompt_tokens: int,
|
||||
completion_tokens: int,
|
||||
cost: float,
|
||||
latency_ms: int,
|
||||
ttft_ms: int = 0,
|
||||
is_error: bool = False,
|
||||
) -> None:
|
||||
"""Record a single request's usage, hour-bucketed with UPSERT."""
|
||||
hour_bucket = time.strftime("%Y-%m-%dT%H:00:00Z", time.gmtime())
|
||||
uid = generate_id("use")
|
||||
|
||||
with get_connection() as conn:
|
||||
# Try update existing hour bucket
|
||||
cursor = conn.execute(
|
||||
"""UPDATE backend_usage_logs SET
|
||||
prompt_tokens = prompt_tokens + ?,
|
||||
completion_tokens = completion_tokens + ?,
|
||||
total_tokens = total_tokens + ?,
|
||||
cost = cost + ?,
|
||||
request_count = request_count + 1,
|
||||
error_count = error_count + ?,
|
||||
avg_latency_ms = CAST((avg_latency_ms * request_count + ?) / (request_count + 1) AS INTEGER),
|
||||
ttft_ms = CASE WHEN ? > 0 THEN CAST((ttft_ms * request_count + ?) / (request_count + 1) AS INTEGER) ELSE ttft_ms END
|
||||
WHERE backend_id = ? AND hour_bucket = ?""",
|
||||
(
|
||||
prompt_tokens, completion_tokens,
|
||||
prompt_tokens + completion_tokens,
|
||||
cost,
|
||||
1 if is_error else 0,
|
||||
latency_ms,
|
||||
ttft_ms, ttft_ms,
|
||||
backend_id, hour_bucket,
|
||||
),
|
||||
)
|
||||
if cursor.rowcount == 0:
|
||||
# Insert new hour bucket
|
||||
conn.execute(
|
||||
"""INSERT INTO backend_usage_logs
|
||||
(id, backend_id, model, prompt_tokens, completion_tokens,
|
||||
total_tokens, cost, request_count, error_count,
|
||||
avg_latency_ms, ttft_ms, hour_bucket)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
(
|
||||
uid, backend_id, model,
|
||||
prompt_tokens, completion_tokens,
|
||||
prompt_tokens + completion_tokens,
|
||||
cost, 1, 1 if is_error else 0,
|
||||
latency_ms, ttft_ms, hour_bucket,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def get_hourly_usage(
|
||||
backend_id: Optional[str] = None,
|
||||
since: Optional[str] = None,
|
||||
limit: int = 168,
|
||||
) -> list[dict]:
|
||||
"""Get hourly usage data, optionally filtered by backend and time range."""
|
||||
with get_connection() as conn:
|
||||
if backend_id and since:
|
||||
rows = conn.execute(
|
||||
"""SELECT * FROM backend_usage_logs
|
||||
WHERE backend_id = ? AND hour_bucket >= ?
|
||||
ORDER BY hour_bucket DESC LIMIT ?""",
|
||||
(backend_id, since, limit),
|
||||
).fetchall()
|
||||
elif backend_id:
|
||||
rows = conn.execute(
|
||||
"""SELECT * FROM backend_usage_logs
|
||||
WHERE backend_id = ? ORDER BY hour_bucket DESC LIMIT ?""",
|
||||
(backend_id, limit),
|
||||
).fetchall()
|
||||
elif since:
|
||||
rows = conn.execute(
|
||||
"""SELECT * FROM backend_usage_logs
|
||||
WHERE hour_bucket >= ? ORDER BY hour_bucket DESC LIMIT ?""",
|
||||
(since, limit),
|
||||
).fetchall()
|
||||
else:
|
||||
rows = conn.execute(
|
||||
"""SELECT * FROM backend_usage_logs
|
||||
ORDER BY hour_bucket DESC LIMIT ?""",
|
||||
(limit,),
|
||||
).fetchall()
|
||||
return [dict(row) for row in rows]
|
||||
|
||||
|
||||
def get_total_stats() -> dict:
|
||||
"""Get aggregate stats across all backends."""
|
||||
with get_connection() as conn:
|
||||
row = conn.execute(
|
||||
"""SELECT
|
||||
SUM(request_count) as total_requests,
|
||||
SUM(error_count) as total_errors,
|
||||
SUM(total_tokens) as total_tokens,
|
||||
SUM(prompt_tokens) as total_prompt_tokens,
|
||||
SUM(completion_tokens) as total_completion_tokens,
|
||||
SUM(cost) as total_cost
|
||||
FROM backend_usage_logs"""
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return {
|
||||
"total_requests": 0, "total_errors": 0,
|
||||
"total_tokens": 0, "total_prompt_tokens": 0,
|
||||
"total_completion_tokens": 0, "total_cost": 0.0,
|
||||
}
|
||||
return dict(row)
|
||||
|
||||
|
||||
def aggregate_daily_stats(date: str) -> None:
|
||||
"""Aggregate hourly usage into daily stats table."""
|
||||
with get_connection() as conn:
|
||||
# Aggregate per pool
|
||||
conn.execute("""DELETE FROM daily_stats WHERE date = ?""", (date,))
|
||||
conn.execute(
|
||||
"""INSERT INTO daily_stats (id, date, pool, total_requests,
|
||||
total_errors, total_tokens, total_cost, unique_backends)
|
||||
SELECT
|
||||
? || '-' || b.pool,
|
||||
?,
|
||||
b.pool,
|
||||
SUM(u.request_count),
|
||||
SUM(u.error_count),
|
||||
SUM(u.total_tokens),
|
||||
SUM(u.cost),
|
||||
COUNT(DISTINCT u.backend_id)
|
||||
FROM backend_usage_logs u
|
||||
JOIN backends b ON u.backend_id = b.id
|
||||
WHERE u.hour_bucket LIKE ?
|
||||
GROUP BY b.pool""",
|
||||
(generate_id("day"), date, date + "%"),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def get_daily_stats(days: int = 30) -> list[dict]:
|
||||
"""Get daily aggregated stats."""
|
||||
with get_connection() as conn:
|
||||
rows = conn.execute(
|
||||
"""SELECT * FROM daily_stats ORDER BY date DESC LIMIT ?""",
|
||||
(days,),
|
||||
).fetchall()
|
||||
return [dict(row) for row in rows]
|
||||
Reference in New Issue
Block a user