Apache-Airflow is the current plateform to orchestrate our ETL(Extract Transform and Load).
Basically it let's us add automation to our different workflow, primarly for machine learning needs.
It could be usefull for other repetitive tasks.
Official documentation can be found at Airflow Docs
NOTE: Your environment should be running on linux (dev or prod) e.g. Ubuntu.
Please note that you must install docker and docker-compose to be able to use airflow
Docker and docker-compose installation tutorials:
Currently we are running Airflow under a VM with the following configuration :
NOTE:Docker and docker-compose must be installed on the VM for more information see (see note in Environment Setup) :