Incident 482
Ongoing · 12 minError rate is 4.8% on GET /v1/search
It started 12 minutes ago, after v1.4.0 went live. About one request in twenty fails.
Error rate4.8%
p951.9 s
Failed requests2 104
Live versionv1.4.0
Error rate, last hour over 2%
Endpoints
| Endpoint | Latency, 1 h | p95 | Errors |
|---|---|---|---|
GET /v1/searchFailing | 1.9 s | 4.8% | |
GET /v1/items | 290 ms | 0.4% | |
POST /v1/items | 340 ms | 0.3% |
What happened
- Error rate over 5% on GET /v1/search14:20
Paged ada, on call this week.
- Latency doubled on GET /v1/search14:12
- v1.4.0 is live14:05
Production · 7 commits from main
- Tests passed14:03
214 tests in 38 s
Runbook
Do not debug in production. Go back to the last healthy version first (R), then find the cause in peace.
R roll back · A acknowledge