I just released my first OSS library! Introducing Aqueducts, a framework to build ETL pipelines using rust
I just released my first OSS library! Introducing Aqueducts, a framework to build ETL pipelines using rust
Framework to build data pipelines declaratively. Contribute to vigimite/aqueducts development by creating an account on GitHub.
This is my first try at anything open source so any feedback is welcome :)
You're viewing a single thread.
Very similar to something I’ve been working on but this seems better (I’m still a rust programming newb). I am essentially making a DBT clone but in rust so it doesn’t take 6 hours to do a DBT run and focuses more on Rust managing and transforming the data.
Oh no worries I am quite new to rust myself but am lucky to be able to use it at work and already got some experience with datafusion and delta-rs :). Accessing postgresql with this is not supported yet but am trying to figure out using OpenDAL for that which should hopefully make it quite easy to implement
What was your programming experience before rust? I’m coming from Python so at lot of times I feel like I’m learning from scratch but in a good way. I’m glad rust is teaching me how to break bad habits.
I have a couple of years of experience writing functional scala as a backend web dev and switched to doing data engineering 2 years ago. Before that some C/C++ (this is where my rust interest came from).
I definitely understand the feeling of learning from scratch, I had the same experience learning functional programming but having learnt that made learning rust much easier