feat: Sidecar V2 — multi-pool provider proxy with 429 cooldown
- proxy.py: Fix route path duplication (v1/v1 → v1) when upstream base URL already includes /v1 prefix - proxy.py: Fix _emergency_count global variable for metrics tracking - server.py: Add logging.basicConfig(level=logging.INFO) for structlog INFO-level log visibility - Full multi-pool routing: primary → fallback → emergency passthrough - Per-backend rate limiting with RPM-based token bucket - 429 cooldown mechanism with automatic recovery - Dashboard with SSE real-time monitoring - Admin API for backend/pool/config management - SQLite-backed persistence with encrypted API key storage - Docker compose deployment Deployed by opengineer 严维序 as BIZ-50 Step 4
This commit is contained in:
@@ -0,0 +1,90 @@
|
||||
# Sidecar V2 — API Key Encryption Rotation SOP
|
||||
|
||||
> 版本: v1.0 | 维护者: 严维序 (opengineer)
|
||||
|
||||
## 背景
|
||||
|
||||
Sidecar V2 使用 AES-256-GCM 加密存储所有 Provider 的 API Key。加密密钥通过 `SIDECAR_ENCRYPTION_KEY` 环境变量传入，启动时通过 `init_crypto()` 初始化。
|
||||
|
||||
## ⚠️ 关键警告
|
||||
|
||||
**更换 SIDECAR_ENCRYPTION_KEY 会导致所有已存储的 API Key 永久不可恢复！**
|
||||
|
||||
`crypto.py` 的 `try_decrypt_existing()` 在密钥变更时会静默返回 `None`，已有加密数据将无法解密。请在轮换密钥前执行以下步骤。
|
||||
|
||||
## 安全轮换步骤
|
||||
|
||||
### Step 1: 导出当前 API Key 明文（必须）
|
||||
|
||||
```bash
|
||||
# 使用旧密钥启动 sidecar，通过 admin API 导出
|
||||
curl -s -H "Authorization: Bearer <ADMIN_TOKEN>" \
|
||||
http://127.0.0.1:9190/api/admin/backends | \
|
||||
python3 -c "
|
||||
import json, sys
|
||||
data = json.load(sys.stdin)
|
||||
# 注意：api_key 是 masked 的，需要重新从安全渠道获取原始 key
|
||||
print(json.dumps(data, indent=2))
|
||||
"
|
||||
```
|
||||
|
||||
### Step 2: 停止服务
|
||||
|
||||
```bash
|
||||
systemctl stop sidecar-v2
|
||||
# 或
|
||||
docker compose down
|
||||
```
|
||||
|
||||
### Step 3: 备份数据库
|
||||
|
||||
```bash
|
||||
cp /app/data/sidecar_v2.db /app/data/backups/pre-rotation-$(date +%Y%m%d_%H%M%S).db
|
||||
```
|
||||
|
||||
### Step 4: 更新密钥
|
||||
|
||||
更新 `/etc/sidecar-v2/env` 或 docker `.env` 文件中的 `SIDECAR_ENCRYPTION_KEY`：
|
||||
|
||||
```
|
||||
SIDECAR_ENCRYPTION_KEY=<new_64_hex_char_key>
|
||||
```
|
||||
|
||||
生成新密钥：
|
||||
```bash
|
||||
python3 -c "import secrets; print(secrets.token_hex(32))"
|
||||
```
|
||||
|
||||
### Step 5: 清空加密 Key 并重新录入
|
||||
|
||||
由于密钥变更后旧加密数据不可读，需要：
|
||||
|
||||
1. 启动服务（此时所有旧 Provider 的 API Key 不可用）
|
||||
2. 通过 Admin API 重新录入所有 Provider 的 API Key：
|
||||
```bash
|
||||
curl -s -X PUT -H "Authorization: Bearer <ADMIN_TOKEN>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"api_key": "<NEW_PLAIN_KEY>"}' \
|
||||
http://127.0.0.1:9190/api/admin/backends/<backend_id>
|
||||
```
|
||||
|
||||
### Step 6: 验证
|
||||
|
||||
```bash
|
||||
# 确认 Provider 状态为 healthy
|
||||
curl -s http://127.0.0.1:9190/api/admin/pools
|
||||
# 发送测试请求
|
||||
curl -s -X POST http://127.0.0.1:9190/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"<model_name>","messages":[{"role":"user","content":"test"}],"max_tokens":5}'
|
||||
```
|
||||
|
||||
## 应急预案
|
||||
|
||||
如果在密钥轮换过程中出错：
|
||||
|
||||
1. 恢复旧密钥环境变量
|
||||
2. 恢复旧数据库备份
|
||||
3. 重启服务
|
||||
|
||||
旧 Key 会正常工作，因为未被覆盖的数据仍然用旧密钥加密。
|
||||
@@ -0,0 +1,56 @@
|
||||
# Sidecar V2 — Nginx reverse proxy config (reference)
|
||||
# Place at /etc/nginx/sites-available/sidecar-v2.conf
|
||||
# SSL certs managed by certbot or manually
|
||||
|
||||
upstream sidecar_v2_main {
|
||||
server 127.0.0.1:9190;
|
||||
}
|
||||
|
||||
upstream sidecar_v2_metrics {
|
||||
server 127.0.0.1:9191;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name sidecar.example.com;
|
||||
|
||||
ssl_certificate /etc/ssl/certs/sidecar.pem;
|
||||
ssl_certificate_key /etc/ssl/private/sidecar.key;
|
||||
|
||||
# Dashboard + Admin API (main port)
|
||||
location / {
|
||||
proxy_pass http://sidecar_v2_main;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
|
||||
# SSE support for dashboard real-time data
|
||||
location /dashboard/sse {
|
||||
proxy_pass http://sidecar_v2_main;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection "";
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_buffering off;
|
||||
proxy_cache off;
|
||||
chunked_transfer_encoding off;
|
||||
proxy_read_timeout 86400s;
|
||||
}
|
||||
|
||||
# Prometheus metrics
|
||||
location /metrics {
|
||||
proxy_pass http://sidecar_v2_metrics;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# Health check
|
||||
location /health {
|
||||
proxy_pass http://sidecar_v2_main;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
[Unit]
|
||||
Description=Sidecar V2 — Multi-Pool Provider Proxy
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=openclaw
|
||||
Group=openclaw
|
||||
WorkingDirectory=/opt/sidecar-v2
|
||||
EnvironmentFile=/etc/sidecar-v2/env
|
||||
ExecStart=/opt/sidecar-v2/.venv/bin/python3 main.py
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
# Security hardening
|
||||
NoNewPrivileges=yes
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
ReadWritePaths=/opt/sidecar-v2/data
|
||||
PrivateTmp=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Reference in New Issue
Block a user