Hi Team, As part of our ongoing efforts to streamline our development workflows and align with best practices, we will be transitioning away from using Airflow run_dev https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_guide#Development_instance instances. Going forward, all development work shall be done using Airflow development environments (devenvs). Documented on https://gitlab.wikimedia.org/repos/data-engineering/airflow-devenv and https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_... This move will help standardize our processes, improve local testing, and ensure better compatibility with our production setup. This also helps overcome the main ` ./run_dev_instance.sh` limitation which is the fact that there's only one process to do the job of the worker and the scheduler’s job. Please begin migrating your work to the new Airflow devenv setup. Documentation and setup instructions are available on https://gitlab.wikimedia.org/repos/data-engineering/airflow-devenv and https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_... and the DPE SRE team is available to help with the transition to airflow-devenv over on #data-platform-sre. If you have any questions or run into issues during the transition, don’t hesitate to reach out. Best, Steve Munene. DPE SRE.
analytics-announce@lists.wikimedia.org