Follow your inner conductor: A comparison of file versus abstract dependency-based orchestration
TLDR; GNU Make and Apache Airflow are two DAG orchestration tools. Make gets the job done and is, without a doubt, simpler than Airflow. However, conformance to a purely file dependency-based orchestration (Make) could require refactoring that Airflow would not. It’s worth learning how to use Airflow (or another abstract orchestration tool), even if you have no immediate need for most of its features. Every quarter or so, I deliberately slow down to learn and, if I find it worthwhile, incorporate a new way of working....