As data and analytics engineers, it’s easy to get caught up in the mechanics of the pipelines we built. How do we move data from one source to another? How do track changes and dependencies? How do we do it all faster, more reliably, and with more automation?
To efficiently apply the necessary changes to a pipeline requires running it parallel to production to test the effect of a change. Most data engineers would agree that the best way to do this is far from a solved problem.