Designing a system to identify the top k heavy hitters in a data stream is a common yet challenging system design interview problem. Real-world examples include tracking the most played songs on a music platform or the most viewed videos on a streaming service. I’m taking on this problem next to sharpen my skills in building real-time, conflict-resilient systems that operate reliably at scale. Let’s dive in and see how it all comes together.
Functional Requirements
Let’s define the core functionality our Top K Songs system should support:
- Top K Query: Allow users to fetch the top k most played songs
- Custom Time Ranges: Support queries like "Top K all time", “Top K this hour”, “Top K today”, or “Top K this month”
- Dynamic K: Allow configurable values of k, typically in the range of 100 to 1000 songs
- Real-Time Ingestion: Process millions of song plays per day in near real-time
Non-Functional Requirements
The system should be scalable, fast, and fault-tolerant:
- Scale: 10B songs play per day
- Storage: 100M songs in total
- Low Latency Fetch: Serve top K queries in under 100ms
- Inclusion Delay: A new play should reflect in the Top K within 1 minute
Back of the envelope estimations
- DAU = 500M and each listens 20 songs = 10B songs
- Number of click per second (QPS) = 10B clicks/day / (100K seconds/day) = 100K QPS
- Total songs = 50K/day, store for 10 years = 200M
- Storage estimated = 200M * (8 bytes/ID + 8 bytes/count) = 3.2GB = ~ 4GB
API Design
We expose a simple REST API for fetching the top K songs for a given time window:
GET /top-k-songs?k=1000&time=1m (1 month)Query parameters:
k: Number of top songs to retrieve (e.g., 1000)time: Time window (e.g.,1h,1d,1m)
High-Level Design
Stream Processing with Flink
Apache Flink aggregates song play counts over sliding or tumbling time windows. It maintains a Min Heap to track the top K songs efficiently. Aggregated results are then pushed to Redis.
- Input: Song play event stream from Kafka
- Windowing: Supports 1 hour, 1 day, and 1 month windows
- Output: Top K songs per time window
Data Partitioning Strategy
Kafka partitions song play events by songId, ensuring scalability. To manage load from popular songs, those can be sharded into multiple partitions.
Result Caching and Retrieval
Aggregated results are stored in:
- Redis: For fast, in-memory access to recent top K data
- OLAP DB: For historical analytics and backup persistence
TopKSongs
- songId: string
- timestamp: ISO8601
- playCount: integer
Deep Dive
Kafka captures and stores the incoming song play data, ensuring durability and fault tolerance (Partitioned by song ID, when partitions for popular songs to manage load effectively)
I hope this gives you a clear breakdown of how to design a scalable Top K Songs system like Spotify or YouTube. See you next time!