Distributed job scheduler
Schedule one-off delayed tasks and recurring cron jobs across a fleet of workers with at-least-once execution, lease heartbeats, fencing tokens so zombie workers cannot corrupt downstream state, and isolation when every tenant schedules a job for 00:00 UTC.
Component labels and connections
- Clients (client) · calls: Load balancer
- Load balancer (loadBalancer) · calls: Scheduler API 1, Scheduler API 2
- Scheduler API 1 (service) · calls: Submission idempotency, Execution status cache, Jobs & executions DB, Ready execution queue
- Scheduler API 2 (service) · calls: Submission idempotency, Execution status cache, Jobs & executions DB, Ready execution queue
- Submission idempotency (dedupStore) · calls: none
- Execution status cache (cache) · calls: none
- Jobs & executions DB (database) · calls: none
- Ready execution queue (queue) · calls: none
- Job worker fleet (worker) · calls: Execution lease & fencing store, Jobs & executions DB
- Execution lease & fencing store (dedupStore) · calls: none
Requirements
Functional
- Register one-time delayed jobs (`runAt`) and recurring cron jobs (`0 * * * *`), with payload, retry policy, and timeout.
- Dispatch due jobs to worker nodes within ~1 second of their scheduled time, track execution status, and retry failed runs with exponential backoff up to `maxAttempts` before moving to a dead-letter queue.
- Allow tenants to cancel a scheduled job or inspect execution history.
Non-functional
- No lost jobs once acknowledged: every due job executes at least once even if schedulers or workers crash.
- Prevent duplicate side effects on worker lease expiry via fencing tokens (`leaseEpoch`) and execution idempotency keys.
- Absorb the top-of-the-hour ('00:00 UTC') thundering herd without starving real-time high-priority jobs.
Worked estimates
Registered jobs
≈ 25 GB of job metadata — easily indexed by `(shardId, nextRunAt)`
50M active cron & delayed definitions × ~500 B
Executions
≈ 5,800 executions/s average
500M executions/day ÷ 86,400
Top-of-hour spike
≈ 50k–100k jobs due in the same second (~15× average) — requires pre-bucketing and jitter
20% of daily cron jobs fire at 00:00 UTC (or on the hour)
Worker fleet
≈ 2,900 concurrent worker slots on average (Little's law: L = λW)
5,800 jobs/s × 500 ms mean execution duration
Heartbeat traffic
≈ 300 lease extensions/s — negligible
3,000 running jobs ÷ 10 s heartbeat interval
API
POST /v1/jobs
Request: { schedule, payload, retryPolicy, timeoutSec } + Idempotency-Key
Response: 201 { jobId, nextRunAt }
GET /v1/jobs/{jobId}/executions
Request: —
Response: 200 { executions[{ executionId, status, leaseEpoch, startedAt, finishedAt }] }
POST /v1/executions/{executionId}/heartbeat
Request: { workerId, leaseEpoch }
Response: 200 { extendUntil } · 409 lease lost / cancelled
Worker aborts immediately if it receives 409.
Data model
- jobs — store: jobdb
- executions — store: jobdb
jobs stores the definition and `nextRunAt`, partitioned across `N` virtual buckets (e.g., 64 shards) with a composite index `(bucketId, status, nextRunAt)` so each scheduler partition scans only its own due slice.
executions records each deterministic run `(jobId, scheduledTime)` with a monotonically increasing `leaseEpoch` (fencing token) incremented every time a worker claims or steals the lease.
When a recurring job is claimed at `t`, the scheduler atomically inserts the execution for `t` and advances `jobs.nextRunAt` to the next cron tick.
High-level design
Trace permitted calls in the architecture above, then open the reference canvas to inspect routing.
Deep dives
Two-tier architecture: durable time-bucket store + ready queue
Message brokers (Kafka, SQS) are great at dispatching ready work, but most do not natively sort millions of arbitrary future timestamps days in advance. Store future jobs in a sharded database indexed by `(bucketId, status, nextRunAt)`. Each scheduler node owns a subset of `bucketId`s (assigned via consensus/leases), queries `nextRunAt <= now + 5s`, atomically transitions those rows to `ENQUEUED` (incrementing `leaseEpoch`), and publishes them to the ready queue where workers pull them immediately.
Leases, heartbeats, and fencing tokens against zombie workers
At-least-once delivery is mandatory because a worker can die mid-job. When a worker claims an execution, it receives a lease (visibility timeout, e.g., 60 s) and a monotonically increasing integer `leaseEpoch = 1` (the fencing token). Long jobs heartbeat every 15 s to extend the lease. If Worker A pauses in a long GC or network partition, its lease expires and Worker B claims the execution with `leaseEpoch = 2`. When Worker A wakes up and tries to write its result or call a downstream storage service, the store rejects `leaseEpoch = 1 < 2`, preventing the zombie worker from overwriting Worker B's state.
Taming the 00:00 UTC thundering herd and noisy tenants
Human-written crons overwhelmingly specify `0 * * * *` or `0 0 * * *`. Mitigate this in three ways: (1) support fuzzy cron expressions (`H * * * *` in Jenkins) that hash the `jobId` to a deterministic offset minute/second across the window; (2) separate queues by priority tier (latency-sensitive vs batch); and (3) enforce per-tenant concurrency quotas or weighted fair queuing so one tenant submitting 100,000 jobs cannot starve everyone else.
Failure scenarios
Worker fleet crashes mid-execution
Scenario and expected result
From t = 1.3 s to 1.8 s the worker fleet dies while holding in-flight executions. Unacknowledged jobs exceed the 400 ms visibility timeout and are redelivered.
Expected: Job submissions and status queries remain 100% available. When workers recover, redelivered executions are deduplicated via `executionId` and fencing state, completing cleanly without duplicate effects.
00:00 UTC cron spike: traffic triples
Scenario and expected result
At the top of the hour (t = 1.2 s to 2.0 s), scheduled job volume triples.
Expected: The ready queue absorbs the burst and feeds the worker fleet at its max concurrency without dropping submissions.
The jobs database leader fails over
Scenario and expected result
At t = 1.5 s the primary jobs database crashes; replica promotion completes in ~0.5 s.
Expected: Cached status reads keep succeeding during the election window; semi-sync replication and leader fencing ensure no committed job schedule is lost or split-brained.
Alternative designs
Reference
- Inspect the architecture above and run the same scenario.
No execution idempotency or fencing token on worker retry
- When a worker crashes or pauses longer than the visibility timeout, the queue redelivers the execution to a second worker. If the execution handler applies side effects (such as appending to an execution ledger or charging an account) without deduplicating by `(executionId, leaseEpoch)`, redeliveries cause duplicate side effects. Run the runner-crash scenario against this alternative to see `duplicateEffect` violations.
Reference
- Inspect the architecture above and run the same scenario.
Unindexed table polling (`SELECT * FROM jobs WHERE next_run_at <= NOW()`)
- Having workers poll a single unsharded SQL table without a partitioning scheme or ready queue causes lock contention (`FOR UPDATE SKIP LOCKED` helps up to a few thousand rows/s) or full-table scans. In the simulator, replacing the cached status and indexed lookup with unindexed table queries pushes p99 latency above 14 ms.
Reference
- Inspect the architecture above and run the same scenario.
In-memory hierarchical timing wheel vs database range scan
- A hashed timing wheel (Netty / Kafka style) gives O(1) timer insertion and tick advancement in RAM, ideal for sub-second delays at 100k+ timers/s, but loses state on crash unless backed by a write-ahead log. A hybrid loads the next 60 seconds of due jobs from the sharded DB into an in-memory timing wheel.
Clarifying questions
- What is the acceptable dispatch delay between a job's scheduled time and when a worker starts running it (e.g., <1 s vs <30 s)?
- How long can a single job execution run (milliseconds vs hours), and do workers need heartbeats?
- Are jobs strictly one-off, recurring cron, or DAG workflows with dependencies?
- Is the system multi-tenant, where one tenant might dump 1M jobs at once?
Common mistakes
- Running `SELECT * FROM jobs WHERE next_run_at <= NOW()` across an unpartitioned table from hundreds of workers.
- Claiming 'exactly-once execution' without explaining that worker crashes require at-least-once dispatch plus idempotent/fenced side effects.
- Setting a fixed short visibility timeout with no heartbeat mechanism for long-running jobs.
- Ignoring the 00:00 UTC cron spike where 20% of the day's jobs fire in the same second.
- Retrying a deterministic poison-pill job forever instead of capping `maxAttempts` and routing to a dead-letter queue.
Self-assessment rubric
requirements / levels 1-4
- 1/4 Run jobs at a time.
- 2/4 Adds cron, retries, and status.
- 3/4 Defines dispatch lag SLO, at-least-once + idempotency, and dead-letter handling.
- 4/4 Addresses lease heartbeats, cancellation, and multi-tenant fairness at 00:00 UTC.
estimates / levels 1-4
- 1/4 None.
- 2/4 Executions/s only.
- 3/4 Applies Little's law (`L = λW`) to size worker concurrency and estimates midnight spike.
- 4/4 Sizes the due-index scan rate per shard and heartbeat QPS.
api / levels 1-4
- 1/4 Submit job.
- 2/4 Submit, cancel, get status.
- 3/4 Idempotent submission, deterministic `(jobId, scheduledAt)` execution IDs, and worker heartbeat API.
- 4/4 Includes `leaseEpoch` fencing token in worker heartbeat and completion calls.
data model / levels 1-4
- 1/4 Jobs table.
- 2/4 Jobs + executions tables.
- 3/4 Composite `(bucketId, status, nextRunAt)` index + `executions` state machine with `leaseEpoch`.
- 4/4 Explains atomic claim + `nextRunAt` advancement for recurring cron jobs.
high level / levels 1-4
- 1/4 Cron daemon on one box.
- 2/4 DB + workers.
- 3/4 Sharded time-bucket store, partitioned scheduler pollers, ready queue, and worker fleet.
- 4/4 Adds DLQ, priority queues, and lease reaper for expired in-flight runs.
deep dive / levels 1-4
- 1/4 None.
- 2/4 Mentions `SKIP LOCKED` or Kafka.
- 3/4 Explains zombie workers and how `leaseEpoch` fencing tokens stop stale writes.
- 4/4 Details 00:00 UTC jittering (`H`), timing wheels, and per-tenant fair queuing.
wrap-up / levels 1-4
- 1/4 Stops.
- 2/4 Mentions monitoring.
- 3/4 Highlights schedule lag, poison-pill jobs, and zombie workers as top risks.
- 4/4 Defines SLOs on dispatch delay (`startedAt - scheduledAt`) and DLQ alerting.