HLD-225·Search & Indexing
Medium · 1,000 ptsCentralized Log Ingestion & Full-Text Search Pipeline
Design a Datadog Logs / ELK-style observability pipeline ingesting structured JSON logs at high write throughput while serving incident search queries.
Max p99
≤ 85 ms
Availability
≥ 99.4%
Budget
≤ $8,650
Functional Requirements
- FR-1POST /v1/logs/ingest — Buffers high-throughput service log batches keyed by tenant_service.
- FR-2GET /v1/logs/search?service={tenant_service} — Queries recent indexed logs for incident triage.
- FR-3PATCH & DELETE /v1/logs/ingest/{tenant_service} — Updates, revokes, or deletes tenant_service entries with bounded TTL or explicit cache/index invalidation.
- FR-4Must survive indexing storage gray latency spikes during production outage log storms.
Workload Profile
Steady Fleet Logs14,000 Query RPS / 16,000 Log Batch Write RPS
Incident Log Storm36,000 Query RPS / 42,000 Log Write RPS
Active Working Set28 GB hot 15-minute inverted log segments
Traffic DistributionWrite-heavy burst from crashing tenant_service clusters
Consistency Model
Near-real-time log search visibility (< 5s segment refresh); zero dropped accepted log batches.
Relevant practice for
Solutions Locked
Sign in and either achieve AC or complete 3 official submissions on this question to inspect community diagrams and writeups.
Loading solutions...
Submission History (0)Select to replay
No submissions yet
Click Submit in the top bar to evaluate against all 7 judge cases and save each run here.
Click an item in Inventory above, or press B to add a component, double-click to name, C to connect.
Design Write-up
Write under ### Architecture for Diagram to take that up directly
Compulsory section · if not written yet, Diagram reads from the start of your write-up.
Architecture: required·FRs mentioned: 0/4·0 boxes · 0 edges
0 words