You’ll definitely want to read this if you’d like to know how Slack handles their job queue.

Robustly handling billions of tasks in milliseconds using Kafka and Redis.

On our busiest days, the system processes over 1.4 billion jobs at a peak rate of 33,000 per second. Job execution times range from a few milliseconds to (in some cases) several minutes.

Also, check out The Changelog #278 — we talked about blockchains and databases at OSCON, including a conversation with Tague Griffith on Redis.