The past few days, I've worked on T336630 and T336624, both of which describe failures I was seeing which other people weren't. In both cases, it turned out to be caused by something in my user-config.py. Many of these tests also rely on external resources (i.e. live testing against testwiki or metawiki). I come from a shop where our test environment was hermetic. That eliminated this kind of flakiness due to environmental differences. It's a great way to work, but I recognize that's not what we've got here, and not easy to get to that point.
Given all that, any suggestions on how I should set up a dev environment on my local machine to minimize this kind of problem in the future?
Also, once I figured out what was going on, I closed T336630 as "declined". Was that the right state for "I'm not going to do any more work on this because it turned out to be a problem with the test environment". Should I have used "invalid"? Something else?
On Mon, May 15, 2023 at 10:23 AM Roy Smith roy@panix.com wrote:
The past few days, I've worked on T336630 and T336624, both of which describe failures I was seeing which other people weren't. In both cases, it turned out to be caused by something in my user-config.py. Many of these tests also rely on external resources (i.e. live testing against testwiki or metawiki). I come from a shop where our test environment was hermetic. That eliminated this kind of flakiness due to environmental differences. It's a great way to work, but I recognize that's not what we've got here, and not easy to get to that point.
I want to point out that there should remain a difference between unit tests and integration tests. For integration tests, I believe it's a great idea to actually spawn a mediawiki instance and check how a page has been affected by various pywikibot API calls. However, that's a massive overkill for unit tests. Here's how I would write such tests for pywikibot API: mock the internal API that calls mediawiki API, and assert on requests to have certain parameters and provide hardcoded responses to these requests. This way, if a unit test fails, you know it's Pywikibot's code that is to blame and not something like a MediaWiki change or the environment setup.
Though, having tests depend on user-supplied user-config.py is just ... broken. Unless the goal is to be able to test the user-supplied user-config.py, that is.
YiFei Zhu
Given all that, any suggestions on how I should set up a dev environment on my local machine to minimize this kind of problem in the future?
Also, once I figured out what was going on, I closed T336630 as "declined". Was that the right state for "I'm not going to do any more work on this because it turned out to be a problem with the test environment". Should I have used "invalid"? Something else? _______________________________________________ pywikibot mailing list -- pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/me... To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org
On May 15, 2023, at 3:11 PM, YiFei Zhu zhuyifei1999@gmail.com wrote:
having tests depend on user-supplied user-config.py is just ... broken.
I hear you!
So, still trying to find my way around the test suite, I see appveyor.yml. I don't know anything about Appveyor, but it looks like this is building a known user-config.py before kicking off a test run. Is there some reason this couldn't be part of the global test setup for all tests? I tried tracing back how the test suite works, but to be honest, I stopped reading once I found MetaTestCaseClass.
On Wed, May 17, 2023 at 10:49 AM Roy Smith roy@panix.com wrote:
On May 15, 2023, at 3:11 PM, YiFei Zhu zhuyifei1999@gmail.com wrote:
having tests depend on user-supplied user-config.py is just ... broken.
I hear you!
So, still trying to find my way around the test suite, I see appveyor.yml. I don't know anything about Appveyor, but it looks like this is building a known user-config.py before kicking off a test run. Is there some reason this couldn't be part of the global test setup for all tests?
My educated guess: it contains login credentials to live projects. Not sure how they can be shared. But I don't think live testing against production wikis in unit tests makes sense anyways.
I tried tracing back how the test suite works, but to be honest, I stopped reading once I found MetaTestCaseClass.
I find the github workflow files easier to follow. .github/workflows/pywikibot-ci.yml for example has a step called "Generate user files" that creates the user-config.py.
YiFei Zhu
pywikibot mailing list -- pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/me... To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org