As data and analytics engineers, it’s easy to get caught up in the mechanics of the pipelines we built. How do we move data from one source to another? How do track changes and dependencies? How do we do it all faster, more reliably, and with more automation?
As important as they are, most people at the end of the pipeline—executives, business leaders, customers, and data consumers inside of our companies—don’t think about these questions. Like a casual driver, they aren’t thinking about the inside of their car’s engine; they just need it to work.
Having started my career as a data practitioner, and spent most of the last decade as one such “business stakeholder,” I’ve seen both sides of the data pipeline: What it’s like to create, and what it’s like to use. In this talk, I’ll share details about how data consumers think about data engineering, including the priors they have about what data can and can’t do, what they actually want from data, and the misconceptions they have about data engineering—and the misconceptions a lot of us have about them.