Muse Spark 1.3·续2

Muse Spark 1.3·续2

Here's a reversible plan — old queue stays source-of-truth until parity is proven. All switches behind flags.

**1. Monitor - lag monitoring:**
Add publish rate, `queue_depth_old`, `log_end_offset - consumer_offset` per partition/fleet, e2e latency, DLQ counts. Rollback trigger: metrics missing/divergent in audit -> stop.

**2. Dual write - old primary:**
Producers sync-write old, best-effort async-write new log with same `job_id`. Compare counts. Rollback trigger: new-write error >0.1% or p99 +10% -> flag off, ignore new log.

**3. Shadow consumers:**
Deploy 3 new fleets reading new log, execute but suppress side-effects, diff vs old. Rollback trigger: mismatch or crash -> disable shadow, fix.

**4. Deduplicate:**
Add `job_id` idempotency table, TTL > retention. New workers dedup; reconcile old vs new processed sets hourly. Rollback trigger: duplicate side-effect or reconcile gap -> stay on old.

**5. Cutover - new primary:**
Flip to new log primary, old as fallback, dual-read + dedup. Move one fleet at a time. Rollback trigger: lag >30s, error-budget burn, or DLQ spike -> flag back to old instantly.

**6. Drain:**
Stop writes to old, drain workers, dark-launch 48h. Rollback trigger: old-queue reappearance -> re-enable.

**Invariant before deleting old queue:** `queue_depth_old=0` for full retention window AND producer refs=0 AND new-log lag ~0 AND 100% reconciled `job_ids` with zero pending dedup keys.

{
  "id": "resp_6ac7ad2d29eeea2059184ed9",
  "object": "response",
  "created_at": 1791470893,
  "model": "muse-spark-1.3",
  "output": [
    {
      "type": "reasoning",
      "id": "rs_6ac7ad2d29eeea2059184ed9:rs_9a03ef30-1d43-4ddb-92ff-faea1c123a3f",
      "summary": [],
      "status": "completed"
    },
    {
      "type": "message",
      "id": "msg_807b479b-0d11-45d6-9fb5-2e304375c590",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "Here's a reversible plan — old queue stays source-of-truth until parity is proven. All switches behind flags.\n\n**1. Monitor - lag monitoring:**\nAdd publish rate, `queue_depth_old`, `log_end_offset - consumer_offset` per partition/fleet, e2e latency, DLQ counts. Rollback trigger: metrics missing/divergent in audit -> stop.\n\n**2. Dual write - old primary:**\nProducers sync-write old, best-effort async-write new log with same `job_id`. Compare counts. Rollback trigger: new-write error >0.1% or p99 +10% -> flag off, ignore new log.\n\n**3. Shadow consumers:**\nDeploy 3 new fleets reading new log, execute but suppress side-effects, diff vs old. Rollback trigger: mismatch or crash -> disable shadow, fix.\n\n**4. Deduplicate:**\nAdd `job_id` idempotency table, TTL > retention. New workers dedup; reconcile old vs new processed sets hourly. Rollback trigger: duplicate side-effect or reconcile gap -> stay on old.\n\n**5. Cutover - new primary:**\nFlip to new log primary, old as fallback, dual-read + dedup. Move one fleet at a time. Rollback trigger: lag >30s, error-budget burn, or DLQ spike -> flag back to old instantly.\n\n**6. Drain:**\nStop writes to old, drain workers, dark-launch 48h. Rollback trigger: old-queue reappearance -> re-enable.\n\n**Invariant before deleting old queue:** `queue_depth_old=0` for full retention window AND producer refs=0 AND new-log lag ~0 AND 100% reconciled `job_ids` with zero pending dedup keys.",
          "annotations": [],
          "logprobs": []
        }
      ],
      "status": "completed"
    }
  ],
  "status": "completed",
  "usage": {
    "input_tokens": 126,
    "output_tokens": 2390,
    "total_tokens": 2516,
    "input_tokens_details": {
      "cached_tokens": 0
    },
    "output_tokens_details": {
      "reasoning_tokens": 2024
    }
  },
  "top_logprobs": 0,
  "temperature": 1,
  "top_p": 1,
  "service_tier": "auto",
  "reasoning": {
    "effort": "high"
  },
  "background": false,
  "tools": [],
  "tool_choice": "auto",
  "truncation": "disabled",
  "completed_at": 1791470893,
  "error": null,
  "incomplete_details": null,
  "instructions": null,
  "parallel_tool_calls": true,
  "store": true,
  "max_output_tokens": 8192
}

</details>

<summary>Incident Timeline Analysis — Analyze an outage timeline, identify the likely contributing factors, and propose follow-up actions.</summary>

const response = await env.AI.run(
  'meta/muse-spark-1.3',
  {
    messages: [
      {
        role: 'user',
        content:
          'Analyze this incident timeline:\n\n09:00: deploy increases worker concurrency from 20 to 80.\n09:07: queue depth begins rising.\n09:12: database connection utilization reaches 95%.\n09:16: retries increase and queue depth doubles.\n09:25: rollback starts.\n09:31: queue depth returns to normal.\n\nWrite a blameless root-cause summary, distinguish direct cause from contributing factors, and propose four follow-up actions with measurable success criteria.',
      },
    ],
    max_completion_tokens: 4096,
  },
)
console.log(response)

curl https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/ai/run \
  --header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
  "model": "meta/muse-spark-1.3",
  "input": {
    "messages": [
      {
        "role": "user",
        "content": "Analyze this incident timeline:\n\n09:00: deploy increases worker concurrency from 20 to 80.\n09:07: queue depth begins rising.\n09:12: database connection utilization reaches 95%.\n09:16: retries increase and queue depth doubles.\n09:25: rollback starts.\n09:31: queue depth returns to normal.\n\nWrite a blameless root-cause summary, distinguish direct cause from contributing factors, and propose four follow-up actions with measurable success criteria."
      }
    ],
    "max_completion_tokens": 4096
  }
}'

### Blameless Root-Cause Summary

**TL;DR:** A 4x increase in worker concurrency at 09:00 saturated a downstream bottleneck - the database connection pool - which reduced effective throughput, causing queue depth to grow. Retries then amplified the load and created a feedback loop. Rollback removed the overload and service recovered, which strongly implicates the concurrency change as the trigger.

Increasing workers from 20 to 80 would normally *decrease* queue depth. That the queue *rose* 7 minutes after the deploy is a key signal: workers were not processing faster, they were getting slower/blocked waiting on a shared resource.

What we can infer from the timeline alone:
09:00 -> 09:07 deploy precedes queue growth
09:12 -> DB at 95% shows downstream saturation
09:16 -> retries + doubling queue shows amplification
09:25 -> 09:31 rollback -> recovery confirms causal link

This analysis is based on metrics correlation. Trace-level confirmation - worker wait time, DB query latency/timeouts, connection acquire timeouts - would finalize it.

### Direct Cause vs. Contributing Factors

**Direct Cause:**
Worker demand exceeded database serving capacity. 80 concurrent workers contended for a finite pool of DB connections. Once utilization approached saturation (~95%), connection wait time, query latency, and timeouts increased. Throughput fell below arrival rate, so queue depth rose even though there were more workers.

**Contributing Factors - the system allowed this to happen:**

1. **Unsafe config change:** The system permitted a 4x concurrency jump with no validation against downstream DB capacity.
2. **No backpressure / load-shedding:** Workers kept accepting/polling work even when the DB could not keep up, instead of throttling.
3. **Retry amplification:** Retries starting at 09:16 added load to an already saturated DB, doubling queue depth - a classic retry storm without sufficient backoff.
4. **Slow detection and mitigation:** First signal at 09:07, but rollback did not start until 09:25 - 18 minutes of growing impact. There was no early joint alert on queue + DB, no canary, and no auto-rollback.

No individual is at fault. The process lacked guardrails, safe rollout, and resilient defaults.

### 4 Follow-Up Actions

**1. Enforce concurrency-to-DB capacity guardrail**
Cap `worker_concurrency * max_connections_per_worker` to a safe fraction of provisioned DB connections.
*Owner: Platform / DBA*
*Success criteria:* Within 2 weeks, CI blocks any deploy where projected usage >70% of DB max connections. Achieve 100% coverage of worker pools and 0 violations over the next 30 days. Verified by a load test at 80 workers staying <70% DB utilization.

**2. Make retries safe under saturation**
Add bounded retries with exponential backoff + jitter, timeouts, and a circuit-breaker on DB connection timeouts.
*Owner: Backend*
*Success criteria:* In a staging reproduction of 95% DB utilization, retry amplification is <1.2x baseline traffic vs. ~2x seen on 09:16, and p95 job latency does not degrade >20%. Merged and load-tested within 3 weeks.

**3. Require progressive rollout for scaling changes**
Any concurrency increase >20% must go via canary: 5% -> 25% -> 100% with automatic halt/rollback.
*Owner: SRE / Release*
*Success criteria:* Within 1 month, all worker scaling uses canary. Canary auto-halts if queue depth growth >20% over 5-min baseline OR DB utilization >80% for 3 min. Demonstrated in one game day with time-to-halt <10 min.

**4. Add joint alerting + mitigation runbook**
Create a paging alert that correlates leading and lagging signals, not just one metric.
*Owner: On-call / Observability*
*Success criteria:* Within 2 weeks, new alert: `queue depth increasing for 5 min AND DB utilization >80%` pages within 5 min. Runbook for "worker saturation" documented and drill-tested. In next drill: Time-to-detect <5 min, Time-to-start-mitigation <10 min.

{
  "id": "resp_6ac7ad5f3c5042d4703149a8",
  "object": "response",
  "created_at": 1791470943,
  "model": "muse-spark-1.3",
  "output": [
    {
      "type": "reasoning",
      "id": "rs_6ac7ad5f3c5042d4703149a8:rs_c616723d-3712-4ee3-96ad-72d2a63569fa",
      "summary": [],
      "status": "completed"
    },
    {
      "type": "message",
      "id": "msg_36a2fc5c-fce5-43d7-808a-6603c27b4779",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "### Blameless Root-Cause Summary\n\n**TL;DR:** A 4x increase in worker concurrency at 09:00 saturated a downstream bottleneck - the database connection pool - which reduced effective throughput, causing queue depth to grow. Retries then amplified the load and created a feedback loop. Rollback removed the overload and service recovered, which strongly implicates the concurrency change as the trigger.\n\nIncreasing workers from 20 to 80 would normally *decrease* queue depth. That the queue *rose* 7 minutes after the deploy is a key signal: workers were not processing faster, they were getting slower/blocked waiting on a shared resource.\n\nWhat we can infer from the timeline alone:\n09:00 -> 09:07 deploy precedes queue growth\n09:12 -> DB at 95% shows downstream saturation\n09:16 -> retries + doubling queue shows amplification\n09:25 -> 09:31 rollback -> recovery confirms causal link\n\nThis analysis is based on metrics correlation. Trace-level confirmation - worker wait time, DB query latency/timeouts, connection acquire timeouts - would finalize it.\n\n### Direct Cause vs. Contributing Factors\n\n**Direct Cause:**\nWorker demand exceeded database serving capacity. 80 concurrent workers contended for a finite pool of DB connections. Once utilization approached saturation (~95%), connection wait time, query latency, and timeouts increased. Throughput fell below arrival rate, so queue depth rose even though there were more workers.\n\n**Contributing Factors - the system allowed this to happen:**\n\n1. **Unsafe config change:** The system permitted a 4x concurrency jump with no validation against downstream DB capacity.\n2. **No backpressure / load-shedding:** Workers kept accepting/polling work even when the DB could not keep up, instead of throttling.\n3. **Retry amplification:** Retries starting at 09:16 added load to an already saturated DB, doubling queue depth - a classic retry storm without sufficient backoff.\n4. **Slow detection and mitigation:** First signal at 09:07, but rollback did not start until 09:25 - 18 minutes of growing impact. There was no early joint alert on queue + DB, no canary, and no auto-rollback.\n\nNo individual is at fault. The process lacked guardrails, safe rollout, and resilient defaults.\n\n### 4 Follow-Up Actions\n\n**1. Enforce concurrency-to-DB capacity guardrail**\nCap `worker_concurrency * max_connections_per_worker` to a safe fraction of provisioned DB connections.\n*Owner: Platform / DBA*\n*Success criteria:* Within 2 weeks, CI blocks any deploy where projected usage >70% of DB max connections. Achieve 100% coverage of worker pools and 0 violations over the next 30 days. Verified by a load test at 80 workers staying <70% DB utilization.\n\n**2. Make retries safe under saturation**\nAdd bounded retries with exponential backoff + jitter, timeouts, and a circuit-breaker on DB connection timeouts.\n*Owner: Backend*\n*Success criteria:* In a staging reproduction of 95% DB utilization, retry amplification is <1.2x baseline traffic vs. ~2x seen on 09:16, and p95 job latency does not degrade >20%. Merged and load-tested within 3 weeks.\n\n**3. Require progressive rollout for scaling changes**\nAny concurrency increase >20% must go via canary: 5% -> 25% -> 100% with automatic halt/rollback.\n*Owner: SRE / Release*\n*Success criteria:* Within 1 month, all worker scaling uses canary. Canary auto-halts if queue depth growth >20% over 5-min baseline OR DB utilization >80% for 3 min. Demonstrated in one game day with time-to-halt <10 min.\n\n**4. Add joint alerting + mitigation runbook**\nCreate a paging alert that correlates leading and lagging signals, not just one metric.\n*Owner: On-call / Observability*\n*Success criteria:* Within 2 weeks, new alert: `queue depth increasing for 5 min AND DB utilization >80%` pages within 5 min. Runbook for \"worker saturation\" documented and drill-tested. In next drill: Time-to-detect <5 min, Time-to-start-mitigation <10 min.",
          "annotations": [],
          "logprobs": []
        }
      ],
      "status": "completed"
    }
  ],
  "status": "completed",
  "usage": {
    "input_tokens": 102,
    "output_tokens": 2497,
    "total_tokens": 2599,
    "input_tokens_details": {
      "cached_tokens": 0
    },
    "output_tokens_details": {
      "reasoning_tokens": 1600
    }
  },
  "top_logprobs": 0,
  "temperature": 1,
  "top_p": 1,
  "service_tier": "auto",
  "reasoning": {
    "effort": "high"
  },
  "background": false,
  "tools": [],
  "tool_choice": "auto",
  "truncation": "disabled",
  "completed_at": 1791470943,
  "error": null,
  "incomplete_details": null,
  "instructions": null,
  "parallel_tool_calls": true,
  "store": true,
  "max_output_tokens": 4096
}