Updated stats about LinkedIn's experience with Kafka:
- Writes: >10B messages/day (>2TB compressed data)
- Reads: >50B messages/day (>1PB compressed data)
- Typical failover time after a broker failure: <10ms
Slides 14, 18-20 talk about its replication model for eventual consistency, interesting as it intentionally makes tradeoffs to take advantage of intra-datacenter latency being an order of magnitude(ish) better than that between DCs connected by the open internet. In exchange for some extra chatter, they tolerate 2f failures among 2f+1 replicas. Clever, and clearly it works for them. (See slides 21-22 for unhelpful diagrams, 27-31 for interesting performance numbers, excepting slide 28's totally inexplicable durability column using highly scientific measures like "some data loss" vs "a few data loss". What.)
Pretty neat stuff, and it's great to see a built-in solution for cross-DC replication.
--
David Schoonover