Case study: Distributed job scheduler

Why can't a distributed job scheduler guarantee 'pure at-most-once AND never-lost' execution without cooperation from the downstream resource?

Your scheduler dispatches 4,000 jobs/s and each job takes an average of 3 seconds to run on a worker. By Little's law ($L = \lambda W$), what is the minimum number of concurrent worker execution slots required just to keep up?

A worker running a 10-minute report job sends its 15-second heartbeat and receives `409 Conflict (stale leaseEpoch)`. What should the worker do immediately?

Why should a cron job's `executionId` be derived deterministically from `(jobId, scheduledTimestamp)` rather than generated as a random UUID when polled?

Worker A claims a job with `leaseEpoch = 4`, freezes for two minutes, and wakes up after Worker B has already re-claimed the job with `leaseEpoch = 5`. What prevents Worker A from overwriting Worker B's result?

A job payload triggers an unhandled NullPointerException in the worker on every attempt, crashing the worker process before it can mark the job failed. What stops this single 'poison-pill' job from crashing the entire worker fleet in a loop?

All workers in Availability Zone A die simultaneously. How long does it take before the jobs they were running are picked up by surviving workers in Zone B?

Which SLI directly measures whether the scheduler itself is keeping up with due jobs?