stage(squidington)

Automating Airbyte Connection Management @ Scale

Calendar Icon - Evently Webflow Template
08 Dec
 
Clock Icon - Evently Webflow Template
10:50 am
 - 
11:00 am PST

About the Session

At Bluesky, we manage a large scale of Airbyte connections, replicating from and to various Snowflake and BigQuery clusters.

Airbyte provides great user experience for managing the connections via its UI, however, its API to manage data sources, destinations, and connections are still in its early stages. Moreover, with the new trend of “infrastructure as code”, the configurations of Airbyte connections need to be stored as code to meet our infrastructure goal.

We tried multiple approaches with pros and cons, and were able to come up with a scalable solution that works well with our external YAML configured connection list file.

In this presentation, we will share the whole story. We look forward to discussions and collaborations.

Add to Calendar
Want to Join the movement?

Register today for move(data)!