Astronomer is a data engineering platform that collects, processes and unifies an organization’s data so that they can get straight to analytics and data science. Initially, Astronomer built their pipeline on top of AWS-managed services to get moving quickly, but ran into some limitations down the road.

In this talk, Astronomer outlines their transition to open source software like Mesos, Docker, Kafka, Airflow, Spark and more. They will review the challenges they faced along the way, and the solutions that most excite them for the future.