It appears there is no clear consensus in the developer community how to resolve the problem of the parserTests known-to-fail tests. Consequently, the most useful thing I can do is provide the patches I have developed and explain how they can be used to implement one of a number of options. These are neither the only options nor is one of them necessarily the best. They simply correspond to suggestions made on the wikitech-1 email list (the second option is additional). There is, of course, the possibility of developing an option not specified here and implementing it.
[Mark Clements added another option this morning - 7/27/09 - that requires additional development. If that is what people want, I (or others) can do the necessary implementation. However, I thought it best to get out what I have done so far in case it meets consensus needs.]
I have attached the patches to bug 19957 (I couldn't find an appropriate existing bug for this issue, so I created one. Also, I attached a tar file with 7 patches. When I looked at the bug after submitting it, it wasn't clear the tar file made it. Any advice?). These patches can be used to implement the options specified below. Details of the patches are found in text I have added to the bug report.
+ Install none of these patches and do nothing else:
+ Pros: parserTests behavior remains as currently defined.
+ Cons: parserTests behavior remains as currently defined.
+ Install none of these patches and define successful output as that which the Parser currently returns (this would require changing parserTests.txt using a unprovided patch):
+ Pros: parserTests is brought into compliance with the test set.
+ Cons: It is generally bad practice to define success without understanding the logic behind it.
+ Install none of these patches, file bugs against the 14 failing tests and then fix the bugs:
+ Pros: parserTests is brought into compliance with the test set.
+ Cons: It has been suggested that the parserTests logic is sufficiently complex that it may be difficult to modify it to pass the known to fail tests.
+ File bugs against the 14 failing tests and patch parserTests.txt to comment out the known to fail tests:
+ Pros: Testing the Parser using parserTests does not return confusing test results. Developers can concentrate on those tests that fail due to the addition of new functionality or intefering bug fixes.
+ Cons: Since the known to fail tests no longer report their presence, it may be easy for the community to forget they are still a problem.
+ Patch parserTests.txt so the known to fail tests are disabled. Optionally file bugs against these tests:
+ Pros: Testing the Parser using parserTests does not return confusing test results. Developers can concentrate on those tests that fail due to the addition of new functionality or intefering bug fixes.
+ Cons: There are other disabled tests in parserTests.txt. These could become confused with the known to fail tests.
+ Patch parserTests|.inc|.php to provide a --run-disabled option:
+ Pros: This would allow disabled known to fail tests to run without editing parserTests.txt
+ Cons: Specifciation of this option runs all disabled tests, not just those known to fail. Test results using this option might confuse developers.
+ Patch parserTests|.inc|.txt to provide a knowntofail option:
+ Pros: parserTests output and summary statistics indicate which tests succeeded, failed and returned known-to-fail status.
+ Cons: Creates a new test status for what should be a temporary problem. This is a questionable software architecture decision.
+ Patch parserTests.php to support a ktf-in-fail option:
+ Pros: those of the previous option. Adds the ability to accumulate ktf failure status in failure statistics.
+ Cons: those of the previous option. Raises the question of what is the proper statistics strategy when the known-to-fail option is missing (on runs with this option missing, known-to-fail tests accumulate against success statistics).
The provided patches do not address whether the 'testrun' and 'testitem' tables should be modifed to note the specification of statistics changing application options (i.e., either --ktf-in-fail or --run-disabled) or the use of the knowntofail option for a particular test. These tables are not modified by the patches.
So far no one has responded to Brion's comments about how parserTests should be modified. You are the customers. What do you want?
Should I just modify parserTests.txt to disable those tests that always have failed? Should I do that and add an option to run disabled tests?
Or should I retain the known-to-fail status? If so, should I retain the option that known-to-fail results accumulate as fails? If so, what should the option be called? Right now it is --ktf-to-fail. There is one proposal to change it to --with-known-to-fail. Any others? Which do people prefer?
Should we note anything new in the testrun and testitem tables? Specifically, should we add a column to testrun that records whether --ktf-to-fail (or whatever we call it) or --run-disabled was set (obviously the table would be modified only to note one of these depending on which implementation option people choose). If we go with the known-to-fail status, should we add a column to testitem that indicates that a test returned a known-to-fail status? Should we leave these tables alone?
I do not have enough experience to determine which of these are the best options. You do, so you need to choose. [Of course, you can remain silent, which I will interpret to mean you really don't care]
It seems that reply doesn't work. So I'll send a new message.
Since the static HTML Wikipedia is not updating (please update), and XML
updates like everyday, the logical choice is to go with XML. Is there any
way to convert XML to HTML, like the static HTML version? I need it in HTML,
and I don't want a one year old version of Wikipedia, with all the useless
information on user talk, discussions, etc.
I don't have mad computer skills like most of you. I need a simple way
(preferably a GUI) to convert XML to HTML. Also, how does the converted XML
look like compared to the real Wikipedia? I've use Bzreader to open it, and
it looks TERRIBLE, without any skin or format organization. Please tell me
the converted XML won't look like this, and looks like the Wikipedia
website.
If the static HTML Wikipedia does update at some time, what are your
preferred method of deleting the user talk, discussion, etc pages? I tried
using Vista's search function and delete all of them with the name "user",
etc. But Vista doesn't like deleting millions of files. Even deleting 1 file
takes minutes (probably due to the sheer number of folders). Is there like a
program that can delete more efficiently? Or a program that deletes while
searching (like finds a page, delete it, move on to search for the next
file).
Thank you SOOOOOO much.
I'm not sure what to do about this; it seems like a good idea but a
major security risk:
http://www.watchlistr.com/ is a site that creates aggregate watchlists
across multiple projects. See
http://en.wikipedia.org/w/index.php?title=Wikipedia:Bounty_board#Transwiki_…
The user who made it has very little editing history, and the site
aggregates watchlists across multiple projects, but requires inputting
your Wikimedia password into the watchlistr.com site. I have no
specific reason to think it's a scam, but if I was trying to phish
passwords I would do something like this.
-Sage Ross (User:Ragesoss)
2009/7/23 Aryeh Gregor <Simetrical+wikilist(a)gmail.com>:
> On Thu, Jul 23, 2009 at 1:02 AM, Ryan Lane<rlane32(a)gmail.com> wrote:
>> Check out how the Flickr API works. Users can give web and desktop
>> apps privileges (read/write/delete).
>>
>> It isn't really that bizarre of a concept.
>
> Read/write/delete access to what? The only cases where read access
> would be relevant would be what, watchlist and preferences, pretty
> much? I don't think we'd want this for editing, or admin-only stuff
> like viewing deleted pages.
Eh? I do. Else why bother even having a write API? Why bother even
having the login aspect to the API?
I can imagine someone building an alternative edit interface for a
subset of Wikipedia content, say a WikiProject. Then the interface can
strip away all the general crud and just provide information relevant
to that topic area.
The OAuth provider typically has a page on the service (say en.wp)
that lists all the third party apps you have granted authorisation to.
This authorisation can be time-limited in itself, but if an app starts
misbehaving (say, doing edits you didn't tell it to do), you can
revoke its authorisation from the service directly (rather than having
to change your password to stop it).
Brianna
--
They've just been waiting in a mountain for the right moment:
http://modernthings.org/
parserTests is a developer tool to stay informed about potential problems in a release. Consequently, its customer base is the developer community. I am working on the assumption that user requirements for changes to it should come from this community (hence the use of this list for discussions about them).
Originally, I changed parserTests by adding a flipresult option that changed a success to a failure and a failure to a success. This was universally panned and a requirement for a know-to-fail status indicated (see emails from Meijssen 7/21 12:19am and 2:56am and Gregor 7/21 4:09am and 9:51am). So, I added this status. In addition, Stephen Bain suggested a command line option that controls whether known-to-fail test results are accumulated in the failure or success statistics. So, I added --ktf-to-fail (there is controversy whether this is the best name. I am open to change it to whatever people want or to get rid of it if that is the consensus).
Your suggestions are to use the disabled option to turn off those tests known to fail and to add a --run-disabled command line switch that when set runs the disabled tests. I can change parserTests to do this is if that is what people want, but I see at least one issue. You may want to disable specific tests for reasons other than they are known to fail (e.g., a test exercises functionality undergoing modification and currently not working). So, you loose the option of separating the known to fail cases from others.
The interactions with --compare and --record are as follows. If we keep the status of known-to-fail, then should its statistics accumulate with success, failure or neither. The motivation of the --ktf-to-fail switch is to specify that known-to-fail tests accumulate with failure statistics. If it is missing, do you want the records to indicate that all of a sudden you have 14 more successes? If that is acceptable, then the issue is resolved. Or should known-to-fail results accumulate against neither? In either case, what information do you want to record and compare against in the testrun and testitem tables? Do you want to add a column to testrun that indicates whether the --ktf-to-fail or --run-disabled flags were set? Do you want to add a column to testitem that records known-to-fail status? Do you want to leave these tables as they are?
In any case, as I stated previously, I can do pretty much whatever the community thinks is right. I just need a concrete set of user requirements.
--- On Thu, 7/23/09, Brion Vibber <brion(a)wikimedia.org> wrote:
> From: Brion Vibber <brion(a)wikimedia.org>
> Subject: Re: [Wikitech-l] What to do about --compare and --record. Second request for comments
> To: wikitech-l(a)lists.wikimedia.org
> Date: Thursday, July 23, 2009, 4:09 PM
> On 07/23/2009 11:00 AM, dan nessett
> wrote:
> >
> > So far no one has responded to my question about how
> --ktf-to-fail should interact with --compare and --record.
> Also, at least one commenter has suggested a different name
> for --ktf-to-fail. Before I open a bug and attach the
> patches, I would like to resolve these issues. Since Brion
> suggested this task, would he comment?
>
> Offhand I'm not sure I see a need for a switch
> specifically.
>
> Couple thoughts offhand:
>
> * There appears to already be a "disabled" option which can
> be added to
> test cases. Since this already exists, it doesn't need to
> be developed
> and could simply be added to the tests we know don't
> currently work.
>
> * If there's a desire to run those tests anyway, I'd
> probably call the
> option --run-disabled. This should be easy to add.
>
> * Not sure there's any need for specific handling w/
> compare and record;
> we can just record whatever we run.
>
>
> If on the other hand we want to run and record these tests,
> but not
> whinge at the user about them, then we'd want another
> option on them.
> Probably just having another completion state for the
> output would do it
> (grouping known-to-fail tests separately from others that
> fail). I'm not
> sure how important that is, though.
>
> -- brion
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l(a)lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
So far no one has responded to my question about how --ktf-to-fail should interact with --compare and --record. Also, at least one commenter has suggested a different name for --ktf-to-fail. Before I open a bug and attach the patches, I would like to resolve these issues. Since Brion suggested this task, would he comment?
Hi!
What's up with the translation extension? The messages of the new
Vector skin are translated into hungarian weeks ago on translatewiki,
but since no update, and no translation extension to transfer these
strings :[ I don't want to create all these pages on Hungarian
Wikipedia, it's unnecessary on the long term. Having the translatewiki
is good, but so slow transfer from it is very frustrating.
And what is the status of including JQuery? Life would be much much
easier with it...
Farewell,
Glanthor Reviol
2009/7/22 Pavlo Shevelo <pavlo.shevelo(a)gmail.com>:
> There should not be any real problem to link wikimedia.org.uk directly
> to Wikimedia UK chapter wiki (wherever it's hosted).
It depends on how the WMF has everything set up. They have a
complicated setup for hosting multiple wikis, it may well be
hard-coded that they all use the WMF domains. I'm cross-posting this
to wikitech-l, hopefully someone there can clarify the situation. Can
a wiki hosted on the WMF servers use a non-WMF domain?