One Size Can't Fit All: The Case for a Real-Time Data Warehouse

In the last decade, the rise of the cloud data warehouse, led by platforms like Snowflake, BigQuery, and Redshift, has helped to modernize data warehousing by providing scalability, convenience, and most importantly flexibility and openness to a very important class of data workloads. Once this data was available in the cloud, it was then tempting to blur the boundaries of the use cases beyond data warehousing, presenting cloud data warehouses as a "one-size fits all" approach for all use cases from user-facing analytics to server side-transformations, dashboarding, observability, machine learning, and so on. This led to recurrent performance challenges, a degraded user experience, and significant runaway costs, calling for a need to reevaluate the data architecture. In this talk, we will introduce the concept of a real-time data warehouse by describing how it addresses these challenges, explaining where it sits in the data architecture, and showing why it is evolving to be a key part of any modern data stack.