brouberol and I worked through and improved the pytest CI job from 8-9 minutes to under 2 minutes. On my work laptop, make test now finishes in under 30 seconds.

Action required: rebase any in-progress patches that add DagProperties
DagProperties no longer infers the DAG file from the call stack. You must pass it explicitly:
props = DagProperties(
    _dag_file=__file__,
    ...
)
Every DAG in main has already been updated.
If your branch adds a DagProperties without either _dag_file or _variable_name, DAG parsing now fails with an AssertionError.
This can be easy to miss: your branch's CI may pass, but the merge result can fail because CI tests your branch, not the rebased merge. Rebase onto main before merging.

Action required: rebuild your dev image once
pytest-xdist is now a dependency. Because the development image is rebuilt only when Dockerfile.dev changes, you'll need to rebuild it manually once:
make test.image.rm && make test
Otherwise, pytest will fail with:
unrecognized arguments: -n
Other test changes
make test now runs with 4 workers by default.
Use make test jobs=1 for a small subset of tests.

--
Analytics-announce mailing list -- analytics-announce@lists.wikimedia.org
To unsubscribe send an email to analytics-announce-leave@lists.wikimedia.org