HLDArena
HLD-225·Search & Indexing
Medium · 1,000 pts

Centralized Log Ingestion & Full-Text Search Pipeline

Design a Datadog Logs / ELK-style observability pipeline ingesting structured JSON logs at high write throughput while serving incident search queries.

Max p99
≤ 85 ms
Availability
≥ 99.4%
Budget
≤ $8,650

Functional Requirements

  • FR-1POST /v1/logs/ingest — Buffers high-throughput service log batches keyed by tenant_service.
  • FR-2GET /v1/logs/search?service={tenant_service} — Queries recent indexed logs for incident triage.
  • FR-3PATCH & DELETE /v1/logs/ingest/{tenant_service} — Updates, revokes, or deletes tenant_service entries with bounded TTL or explicit cache/index invalidation.
  • FR-4Must survive indexing storage gray latency spikes during production outage log storms.

Workload Profile

Steady Fleet Logs14,000 Query RPS / 16,000 Log Batch Write RPS
Incident Log Storm36,000 Query RPS / 42,000 Log Write RPS
Active Working Set28 GB hot 15-minute inverted log segments
Traffic DistributionWrite-heavy burst from crashing tenant_service clusters

Consistency Model

Near-real-time log search visibility (< 5s segment refresh); zero dropped accepted log batches.

Relevant practice for

Click an item in Inventory above, or press B to add a component, double-click to name, C to connect.

Design Write-up

Write under ### Architecture for Diagram to take that up directly

Compulsory section · if not written yet, Diagram reads from the start of your write-up.
Architecture: required·FRs mentioned: 0/4·0 boxes · 0 edges
0 words
Hidden Cases #4–#7 run automatically on Submit
Case 1: Smoke & API ContractInput Workload
Read RPS
1,400 rps
Write RPS
400 rps
Key Distribution
Uniform (Low Skew)
Chaos Fault
None (Healthy Cluster)
Expected Pass CriteriaASSERTIONS
• Tail Latency: p99 ≤ 170 ms
• Availability: ≥ 99.4%
Valid end-to-end Client → Compute → Storage topology