brouberol and I worked through and improved the pytest CI job from 8-9
minutes to under 2 minutes. On my work laptop, make test now finishes in
under 30 seconds.
Action required: rebase any in-progress patches that add DagProperties
DagProperties no longer infers the DAG file from the call stack. You must
pass it explicitly:
props = DagProperties(
_dag_file=__file__,
...
)
Every DAG in main has already been updated.
If your branch adds a DagProperties without either _dag_file or
_variable_name, DAG parsing now fails with an AssertionError.
This can be easy to miss: your branch's CI may pass, but the merge result
can fail because CI tests your branch, not the rebased merge. Rebase onto
main before merging.
Action required: rebuild your dev image once
pytest-xdist is now a dependency. Because the development image is rebuilt
only when Dockerfile.dev changes, you'll need to rebuild it manually once:
make test.image.rm && make test
Otherwise, pytest will fail with:
unrecognized arguments: -n
Other test changes
make test now runs with 4 workers by default.
Use make test jobs=1 for a small subset of tests.