On Mon, May 15, 2023 at 10:23 AM Roy Smith <roy(a)panix.com> wrote:
The past few days, I've worked on T336630 and T336624, both of which describe
failures I was seeing which other people weren't. In both cases, it turned out to be
caused by something in my user-config.py. Many of these tests also rely on external
resources (i.e. live testing against testwiki or metawiki). I come from a shop where our
test environment was hermetic. That eliminated this kind of flakiness due to
environmental differences. It's a great way to work, but I recognize that's not
what we've got here, and not easy to get to that point.
I want to point out that there should remain a difference between unit
tests and integration tests. For integration tests, I believe it's a
great idea to actually spawn a mediawiki instance and check how a page
has been affected by various pywikibot API calls. However, that's a
massive overkill for unit tests. Here's how I would write such tests
for pywikibot API: mock the internal API that calls mediawiki API, and
assert on requests to have certain parameters and provide hardcoded
responses to these requests. This way, if a unit test fails, you know
it's Pywikibot's code that is to blame and not something like a
MediaWiki change or the environment setup.
Though, having tests depend on user-supplied user-config.py is just
... broken. Unless the goal is to be able to test the user-supplied
user-config.py, that is.
YiFei Zhu
> Given all that, any suggestions on how I should set up a dev environment on my local
machine to minimize this kind of problem in the future?
>
> Also, once I figured out what was going on, I closed T336630 as "declined".
Was that the right state for "I'm not going to do any more work on this because
it turned out to be a problem with the test environment". Should I have used
"invalid"? Something else?
> _______________________________________________
> pywikibot mailing list -- pywikibot(a)lists.wikimedia.org
> Public archives at
https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/m…
> To unsubscribe send an email to pywikibot-leave(a)lists.wikimedia.org