Case Study: Rumble’s Real-Time Leaderboards Empower Users to Lead Healthier Lifestyles

Many of us have become more conscious about how much activity we’re getting in a day--and it shows. Purchases for smartwatches that track calories and activities have dramatically increased since 2014. These smartwatches have helped people train for races, track different types of workouts, and be mindful of how much movement they are getting in a day. However, people tracking daily activity levels for casual or semi-competitive reasons have never received the same fanfare as those who track to compete--no medals, post-race swag, or high-fives. That’s changing.

Rumble, an Israeli company, is building applications to encourage and inspire people to maintain healthy daily habits by converting the user’s steps to reward coins. From there, users can make purchases to unique products or services at hundreds of shops and websites like cafes and stores.

Encountering Performance Challenges with User Growth

Rumble originally used PostgreSQL to handle data comprising users’ step counts. There are three different tables that track the user’s steps: daily, weekly, and monthly. A new row is added every day to the daily table, weekly to the weekly table, and monthly to the monthly table. They originally computed the weekly and monthly steps from the daily collection. However, this became very compute intensive due to the large number of queries. To offset the compute, they preaggregated daily steps into weekly and monthly data, resulting in the three tables.

Rumble displays the leaderboards in real-time to users and also engages with them when new companies and coupons are available by sending them notifications. Since they have high engagement with their users, maintaining the platform performance is vital. As user growth started to increase, PostgreSQL performance began declining. The evenings are usually their peak times, with a high number of concurrent queries, and this is where the application responsiveness declined. At around 20+ requests per second, PostgreSQL becomes unable to maintain the latency required to serve the leaderboards. Eventually, it runs out of CPU and memory.

Rumble users are goal-oriented. Being able to instantaneously see their steps and purchase coupons from companies because of their healthy habits encourages them to maintain their active lifestyles. Rumble needs to deliver real-time, data-driven applications to meet those needs. Their SQL queries to power leaderboards involve JOINs, ORDER BY, DESC, LIMIT, and WHERE. In addition to handling complex queries, they need a database that can easily scale as their number of users grows: effortlessly handle high concurrency, maintain low-latency queries, and require low ops. If they stayed with PostgreSQL, they would continuously have to scale vertically as their user base grows, which is untenable for them. Rumble decided to evaluate other technical solutions to see if those requirements can be met.

Evaluating Other Analytics Solutions

Imply Cloud

There were other solutions Rumble considered before deciding to go with Rockset. They initially evaluated Imply Cloud to run OLAP queries in real-time with high traffic. Imply Cloud is a managed Druid service on Amazon Web Services. However, there were some obstacles:

• Difficult to get started: Rumble had a challenging time getting started with Druid because there was no self-service flow.

• The need to build a data team: To run, maintain, and scale Druid required expertise. Rumble would need to build a data team to do this.

• Druid doesn’t have full support for JOINs: Rumble would need to denormalize the data in order to do JOINs in a performant manner.

Yaron Levi, the lead architect of Rumble, examined Druid as a possible solution. However, he decided against it:

"But their [Druid] solution didn't work for us for two reasons. It's expensive. It has a steep learning curve and requires certain expertise both in designing and preparing Druid for your workload."

Snowflake

Rumble also initially looked at Snowflake to handle the real-time data for clickthroughs on pages and coupons, so they can provide that report to their retailers. Snowflake is a fully managed data warehouse that also has a data ingestion tool called Snowpipe. Snowpipe loads data in micro-batches, making it available to users within minutes. However, Snowpipe was not a feasible solution for Rumble due to cost and latency:

• Continuous ingest involves always-on compute: Rumble would have to constantly turn on compute to ingest to Snowflake, which makes it very expensive for continuous live ingest.

• Snowpipe cannot deliver the real-time data they need: It can take 5 to 10 minutes for data to be available. To power real-time analytics, Rumble needed a low-latency option.

These solutions had a number of drawbacks for Rumble that centered around ops, cost, and latency. They continued their search and came across Rockset.

Using Rockset for Real-Time Analytics

Rockset was able to meet Rumble’s real-time analytical needs where the alternatives didn’t. Within 30 minutes of creating an account, Rumble was able to power their leaderboards in real-time using the Write API to write data into Rockset. In the days to follow, Rumble was committed to integrating Rockset into their product. The diagram below shows how Rockset fits within their architecture:

Rumble's Architecture Diagram: In step 1, data flows into Node.js. In step 2, Rumble simultaneously writes data to PostgreSQL and Rockset. From there, Rumble updates the leaderboards in real-time in step 3.

Real-time applications require a database to merge data from multiple sources and perform JOINs, aggregations, and searches. In many cases where JOINs or aggregations are minimally supported, developers have to use other technologies or write extensive code. This adds operational burden. Rockset supports ANSI SQL with JOINs, aggregations, ordering and grouping on any field in your documents.

This is a simplified example of Rumble’s leaderboard query. In this query, we are gathering the steps that a particular user did from September 9th to September 13th. We’re grouping and ordering by the day. Here, Rumble needs to JOIN 2 collections in order to get the daily steps:

Embedded content: https://gist.github.com/nfarah86/52754379f36add4526960082f19f6ea3

In order to return this query within milliseconds, Rockset uses its Converged Index™. The Converged Index™ indexes each field through an inverted index, row index, and column index. Having three different indexes allows for queries to be executed in the most efficient manner. For example, Rockset uses the columnar index for low-selectivity aggregations queries and an inverted index for highly selective queries. If we analyze this query, we would notice different indexes are used in order for the results to return in milliseconds:

• On line 11, the inverted index will be used to find all document ids where userId = 1.

• One line 7 and 8, the inverted index will also be used to find document ids where the day is between the specific bounds.

• On line 2, the row index is used to lookup the (d.steps).

• On line 9 and 10, the inverted index is used for the user collection to get all the document ids where subSegmentId = 1914 and appType = 3 and intersect them.

• Finally, the join will take place to combine the two collections.

Rumble Wellness chose Rockset over the alternatives because ops, scale, latency, and developer velocity were critical to their business success:

"Rockset is pure magic. We chose Rockset over Druid, because it requires no planning whatsoever in terms of indexes or scaling. In one hour, we were up and running, serving complex OLAP queries for our live leaderboards and dashboards at very high queries per second. As we grow in traffic, we can just 'turn a knob' and Rockset scales with us," said Yaron Levi, Chief Architect at Rumble Wellness.

Rumble started on Rockset with around 400,000 users. Since then, they have more than tripled their user base by having two incredible partnerships with Clalit Health Services and Histadrut-General Federation of Labor in Israel. As they continue to grow and expand, even beyond Israel, Rumble will rely on Rockset to seamlessly scale with them while maintaining the high performance their applications require.