Empowering Business Teams in Small Companies Using Open-Source Tools

This session provides a step-by-step guide to setting up a data infrastructure for an e-commerce company using open-source tools on AWS EC2 instances. The tools covered include:

- Airbyte for data ingestion
- dbt for data transformation
- Airflow for orchestration
- Postgres as the data warehouse
- Metabase for data visualization

I'll walk through the process of selecting and implementing these tools to empower 10 business teams with data-driven decision-making. The session will cover:

1. Understanding Business Needs – Engaging with teams to identify pain points and data requirements.
2. Tool Selection Criteria – Key factors considered when choosing tools based on business needs and constraints.
3. Project Outcomes – The impact of this setup on efficiency and decision-making.

Lastly, this session aims to enlighten Data engineers to avoid overcomplicating infrastructure with tools designed for large enterprises. Small companies also need data-driven decision-making, but with solutions that are simpler, cost-effective, and tailored to their scale.