[QA] [Wikitech-l] Migrating away from nose (was 2017-11-01 Scrum of Scrums meeting notes)

Thu Nov 2 11:49:37 UTC 2017

I've migrated Cumin from 'nose' to 'pytest' a while ago, here's a
summary of my experience.

First of all we need to distinguish the two main parts of Python
tests: writing the tests and collecting and running them.

In the classical setup, 'unittest' would be used for writing the tests
and 'nose' for collecting and running them, often wrapped in a call to
'coverage'.
To use 'pytest' just as a test collector and runner, basically means
replacing 'nose' with 'pytest' and 'coverage' with the 'pytest-cov'
module and adding 'pytest-runner' to the 'setup_requires' in setup.py.
There are some caveats though, depending how many nose-specific
features are used and if any unsupported idiom is used, see also [1]
and [2].

Given the status of 'nose', I've looked at 'nose2', but it didn't seem
mature enough and actively developed. When I looked at it few months
ago that was no commit in the last 6-months and the latest release is
from mid-2016. I've seen that there has been some recent activity
though.
I then chose 'pytest' as I got the impression that the Python
community is slowly moving towards it. In the process I've also got
some quick and friendly help on their issue tracker.

I've done this migration in Cumin primarily because I wanted to
overcome some limitations in the previous setup, namely:
- more compact output of passing tests, listing dots per file and not
naming them all
- more verbose output on failing tests with the stacktrace, the code
involved and an easy way to collect separately stderr and stdout
- coverage plugin works with multiprocess testing (very useful for
integration tests) while nose+coverage did not work (known limitation)

Once this step is done, the other, optional but somehow natural step,
is to use 'pytest' also for writing the tests, replacing 'unittest'.
This could be partly automated by a tool like [3], although in my
experience it doesn't cover all the cases and needs some manual
cleanup, but surely helps.
This allows to take full advantage of all the pytest features, as some
are not compatible with unittest-based tests.

I've done this migration for the integration tests of Cumin to take
advantage of 'pytest' parametrization feature [4] that was very useful
in my case for integration tests that were automatically generated.

Once that was done, I've decided to migrate everything to 'pytest' because:
- coherence, to not have integration tests written with 'pytest' and
unit test written with 'unittest'
- to be able in the future to use advanced features of pytest in the
unittests, because they will most likely not work in unittest.TestCase
subclasses (and indeed it happened)
- style-wise 'unittest' doesn't follow PEP8

The main style-related drawback of using 'pytest' for unit tests is
that all the 'self.assertSomeThing(foo)' must be replaced by standard
asserts like 'assert foo == something', and the assert lines don't go
nicely multiline because is not possible to use parentheses. But this
can be easily avoided just assigning the expected and test result
values to variables just before doing the assert.

If you have specific questions while migrating your project, feel free
to contact me [5] or add me as code reviewer on Gerrit (user: Volans).

Riccardo

[1] https://docs.pytest.org/en/latest/nose.html
[2] https://docs.pytest.org/en/latest/unittest.html
[3] https://github.com/pytest-dev/unittest2pytest
[4] https://docs.pytest.org/en/latest/parametrize.html
[5] https://meta.wikimedia.org/wiki/User:RCoccioli_(WMF)

On Wed, Nov 1, 2017 at 7:34 PM, Greg Grossmeier <greg at wikimedia.org> wrote:
>
> (adding in qa@ list)
>
> <quote name="Kunal Mehta" date="2017-11-01" time="11:15:42 -0700">
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA512
> >
> > Hi,
> >
> > On 11/01/2017 10:53 AM, Grace Gellerman wrote:
> > > === Scoring Platform === ** Converted our tests from nosetests to
> > > pytest (Anyone else making the transition?) *** nosetests is
> > > under-maintained *** python community is moving towards pytest ***
> > > converting our repos was pretty straightforward
> >
> > I didn't realize nose was in such a state :( They put a notice[1] on
> > their website, but I always skipped it.
> >
> > pytest[2] claims that it's compatible with nose out of the box - was
> > that the experience you had as well?
> >
> > Off the top of my head, Pywikibot, CI, and other miscellaneous
> > projects all use nose and should probably be migrated over to pytest.
> >
> > [1] https://nose.readthedocs.io/en/latest/#note-to-users
> > [2] https://docs.pytest.org/en/latest/
> >
> > - -- Legoktm
> > -----BEGIN PGP SIGNATURE-----
> >
> > iQJLBAEBCgA1FiEE+h6fmkHn9DUCyl1jUvyOe+23/KIFAln6D0gXHGxlZ29rdG1A
> > bWVtYmVyLmZzZi5vcmcACgkQUvyOe+23/KKbMw/+Nh/tGRagrvZgJqHBZakvvAYE
> > IHCUFPTUhfQ4cQ++oUqDpblAxR/FNO6BN2UQIrIh9gUylHCf/hwZ3BToC5GGPFY5
> > D9Sgi/ce9v5dp3IIvNZq/NZgb0DkityVkxWg8tiZJPwhJ6hTKw9LjaPl++ee9Po/
> > 940WYy7UQ7L/xwgy7rNJrViW0j4VD4mM7+PVvxPuv6rovOpa/H0VMbSlEuoXQYFG
> > P2DPJE1f8B9tn/xun6O6tveWdD7cdna2DVCXhFF+Tldnpa+XtOerE3P+YBA8Gp/+
> > sRJZuDjbCljM0GCzYYVR00klp8xDFQJXn+ELWaE1E3pAitZQdjb1A+xPmxagk1Up
> > ysqSc9nndVm6c59lkzdMcqz1myWyzLVho/RBmRVAAm37Wv7F7tvxHufuwWe8QO/I
> > sUJsRXkH7/TdsSzs8TU9YTZECE5QntTOM7gGgP4jPQgiwZrHob9si1jDUaQMuFoC
> > i8U9dylhrh280VAJj9LRaoRaSb48Ln6sZrCMdpCW8gt4UiRsUcSYCXbD45AgUS9Z
> > 3oPLMEpt+ESqDxJrBl9rgxyFZF19KhpeE3FPCvlAMZQShLZCAlRdDozczV4nFSHC
> > zhRmuRITmnDomf8prk7NwFSM0q601kTOgVqVZiBn9AY3Axvsa1TS1HM582y4rJ8U
> > O+eQbaSCyiI6HTE0lcc=
> > =4lTI
> > -----END PGP SIGNATURE-----
> >
> > _______________________________________________
> > Wikitech-l mailing list
> > Wikitech-l at lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> --
> | Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
> | Release Team Manager            A18D 1138 8E47 FAC8 1C7D |
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l