Airflow Deployed Infrastructure

[averyniceday]

Airflow server has the ability to spin up a dedicated machine for a job. Machine can either be a preconfigured EC2 machine (from snapshot), worker node, and or specified EC2 machine with params. Airflow should manage the start/stop of the instance.

The main benefit is to have on-demand infrastructure and avoid conflicting jobs while using a shared machine (e.g. pipelines3 node).