Open GWT to all Commons users

List overview All Threads
Download

newer

older

GWToolset beta: url from xml not...

Re: [Glamtools] Internship at...

Emmanuel Engelhart

27 Apr 2014 27 Apr '14

2:39 p.m.

Hi, Currently on Commons, the access to the GWT is limited to a small group of "happy fews" (who have the GWT permission). Becoming a part of this group seems to be difficult, even for prominent players of our community: https://commons.wikimedia.org/wiki/Commons:Bureaucrats%27_noticeboard#GWToo… To my opinion, we have nothing to loose (and maybe a lot to win) by opening the GWT to all Commons users. This is the way we use to work and what makes us successful. I don't see why we should proceed differently here. AFAIK, the only risk of this move would be to be flooded by inadequate files. That's why we should maybe limit the number of parallel GWT downloads to 1 or limit the overall number of uploads to 10 per XML file for non-GWT permitted users. An other solution would be to adapt current admin tools to allow them to efficiently deal with this new kind of challenges. Do you see any other risk? Regards Emmanuel -- Volunteer Technology, GLAM, Trainings Zurich +41 797 670 398

Show replies by date

Micha L. Rieser

27 Apr 27 Apr

3 p.m.

Hello Emmanuel, i think it is a good idea to open the GWT to all Commons users. But I think a limit to ten files is not a good idea. Ten files can be easily uploaded with the current wizard. If we want to limit it, it should be 100 files. This should work for active photographs. And we can limit it to users who have already uploaded several files to avoid that a new sock puppet can misuse the tool. I would prefer to have no limitation. If somebody misuses the tools than there are administrators who can block that user. Maybe we need a easy rollback function for administrators to rollback such an bad upload of several unwanted files and the possibility to delete that pictures permanently. Best regards, Micha

...

Am 27.04.2014 um 14:39 schrieb Emmanuel Engelhart <emmanuel.engelhart(a)wikimedia.ch>ch>: Hi, Currently on Commons, the access to the GWT is limited to a small group of "happy fews" (who have the GWT permission). Becoming a part of this group seems to be difficult, even for prominent players of our community: https://commons.wikimedia.org/wiki/Commons:Bureaucrats%27_noticeboard#GWToo… To my opinion, we have nothing to loose (and maybe a lot to win) by opening the GWT to all Commons users. This is the way we use to work and what makes us successful. I don't see why we should proceed differently here. AFAIK, the only risk of this move would be to be flooded by inadequate files. That's why we should maybe limit the number of parallel GWT downloads to 1 or limit the overall number of uploads to 10 per XML file for non-GWT permitted users. An other solution would be to adapt current admin tools to allow them to efficiently deal with this new kind of challenges. Do you see any other risk? Regards Emmanuel -- Volunteer Technology, GLAM, Trainings Zurich +41 797 670 398

Steinsplitter Wiki

28 Apr 28 Apr

4:21 p.m.

No. This is a very bad idea. Noraltemits is ""NOT"" possible for all user do to seurety reasons. It is wasting of traffic and resources. Pleas read the shellpolicy. Regards,

...

From: micha(a)rieser.ch Date: Sun, 27 Apr 2014 15:00:43 +0200 To: emmanuel.engelhart(a)wikimedia.ch CC: glamtools(a)lists.wikimedia.org Subject: Re: [Glamtools] Open GWT to all Commons users Hello Emmanuel, i think it is a good idea to open the GWT to all Commons users. But I think a limit to ten files is not a good idea. Ten files can be easily uploaded with the current wizard. If we want to limit it, it should be 100 files. This should work for active photographs. And we can limit it to users who have already uploaded several files to avoid that a new sock puppet can misuse the tool. I would prefer to have no limitation. If somebody misuses the tools than there are administrators who can block that user. Maybe we need a easy rollback function for administrators to rollback such an bad upload of several unwanted files and the possibility to delete that pictures permanently. Best regards, Micha

_______________________________________________ Glamtools mailing list Glamtools(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/glamtools

Fæ

27 Apr 27 Apr

3:15 p.m.

On 27/04/2014, Emmanuel Engelhart <emmanuel.engelhart(a)wikimedia.ch> wrote: ...

...

Do you see any other risk?

Yes. * If a user has 10 files to upload they should use the well designed standard wizard. The GWT is an amazingly cumbersome way to upload small numbers of files. * The GWT has a number of bugs and restricted features that would be very odd to explain to a user who was not part of a major GLAM upload project or had not had experience with batch uploads. For example xml validity testing is a complex issue and will never be covered by on-wiki step by step documentation. * We have a small user group, if this were doubled by random interested users without skills to bring to the party, or involvement in large batch upload projects, it is highly likely that most of us more engaged folks would drift off into yet another clique. In general, if anyone wanted to play with the tool, I would let them use the beta cluster to knock themselves out. I would encourage this approach as after playing around they might later come up with useful and interesting batch upload projects. At the moment we are have had very few requests to play with the tool on beta. With regard to barriers, the bureaucrats are being a bit harsh, but we have to remember that the point of GWT is that it is designed to be an powerful tool with the ability to rapidly upload 10,000 or 1000,000 images, each of which may be up to 1GB. We want it to stay that powerful, not be re-designed to be small-upload friendly when there are several other tools and the upload wizard, to do precisely that and do it well. Fae -- faewik(a)gmail.com https://commons.wikimedia.org/wiki/User:Fae

Steinsplitter Wiki

6:22 p.m.

On 27/04/2014, Emmanuel Engelhart <emmanuel.engelhart(a)wikimedia.ch> wrote: ...

...

Do you see any other risk?

Yes. The userrightmanegment for GWT was changed for a very good reason to crat management. Huge potential for abuse. GWT is powerful like botflag (noratelimits, uploadbyurl, etc). Regards, > Date: Sun, 27 Apr 2014 14:15:10 +0100 > From: faewik(a)gmail.com > To: glamtools(a)lists.wikimedia.org > Subject: Re: [Glamtools] Open GWT to all Commons users > > On 27/04/2014, Emmanuel Engelhart <emmanuel.engelhart(a)wikimedia.ch> wrote: > ... >

...

Do you see any other risk?

> > Yes. > > * If a user has 10 files to upload they should use the well designed > standard wizard. The GWT is an amazingly cumbersome way to upload > small numbers of files. > > * The GWT has a number of bugs and restricted features that would be > very odd to explain to a user who was not part of a major GLAM upload > project or had not had experience with batch uploads. For example xml > validity testing is a complex issue and will never be covered by > on-wiki step by step documentation. > > * We have a small user group, if this were doubled by random > interested users without skills to bring to the party, or involvement > in large batch upload projects, it is highly likely that most of us > more engaged folks would drift off into yet another clique. > > In general, if anyone wanted to play with the tool, I would let them > use the beta cluster to knock themselves out. I would encourage this > approach as after playing around they might later come up with useful > and interesting batch upload projects. At the moment we are have had > very few requests to play with the tool on beta. > > With regard to barriers, the bureaucrats are being a bit harsh, but we > have to remember that the point of GWT is that it is designed to be an > powerful tool with the ability to rapidly upload 10,000 or 1000,000 > images, each of which may be up to 1GB. We want it to stay that > powerful, not be re-designed to be small-upload friendly when there > are several other tools and the upload wizard, to do precisely that > and do it well. > > Fae > -- > faewik(a)gmail.com https://commons.wikimedia.org/wiki/User:Fae > > _______________________________________________ > Glamtools mailing list > Glamtools(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/glamtools

Emmanuel Engelhart

6:28 p.m.

On 27.04.2014 18:22, Steinsplitter Wiki wrote:

...

Yes. The userrightmanegment for GWT was changed for a very good reason to crat management. Huge potential for abuse. GWT is powerful like botflag (noratelimits, uploadbyurl, etc).

May you please give examples? Emmanuel -- Volunteer Technology, GLAM, Trainings Zurich +41 797 670 398

Steinsplitter Wiki

6:32 p.m.

Pleas read Fae's reply. :) You can flood Server with thousands of upload... etc. ect. Regards, steinsplitter(a)wikipedia.de

...

Date: Sun, 27 Apr 2014 18:28:04 +0200 From: emmanuel.engelhart(a)wikimedia.ch To: glamtools(a)lists.wikimedia.org Subject: Re: [Glamtools] Open GWT to all Commons users On 27.04.2014 18:22, Steinsplitter Wiki wrote:

Yes. The userrightmanegment for GWT was changed for a very good reason to crat management. Huge potential for abuse. GWT is powerful like botflag (noratelimits, uploadbyurl, etc).

May you please give examples? Emmanuel -- Volunteer Technology, GLAM, Trainings Zurich +41 797 670 398 _______________________________________________ Glamtools mailing list Glamtools(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/glamtools

Emmanuel Engelhart

6:34 p.m.

On 27.04.2014 18:32, Steinsplitter Wiki wrote:

...

Pleas read Fae's reply. :) You can flood Server with thousands of upload... etc. ect.

Please read my first email :) == AFAIK, the only risk of this move would be to be flooded by inadequate files. That's why we should maybe limit the number of parallel GWT downloads to 1 or limit the overall number of uploads to 10 per XML file for non-GWT permitted users. An other solution would be to adapt current admin tools to allow them to efficiently deal with this new kind of challenges. == -- Volunteer Technology, GLAM, Trainings Zurich +41 797 670 398

Steinsplitter Wiki

6:39 p.m.

We have changed from admin to cratmanegment for a good reason (GWT is powerful tool like botflag). 10 per XML sounds like nonsense. The user can use UploadWirzard... This tool has a specific SCOPE and should not be open for all users. Pleas note: Such a change is not possible without community consensus and huge changes in the GWT code. Regards,

...

Date: Sun, 27 Apr 2014 18:34:25 +0200 From: emmanuel.engelhart(a)wikimedia.ch To: glamtools(a)lists.wikimedia.org Subject: Re: [Glamtools] Open GWT to all Commons users On 27.04.2014 18:32, Steinsplitter Wiki wrote:

Pleas read Fae's reply. :) You can flood Server with thousands of upload... etc. ect.

Emmanuel Engelhart

7:19 p.m.

On 27.04.2014 18:39, Steinsplitter Wiki wrote:

...

We have changed from admin to cratmanegment for a good reason (GWT is powerful tool like botflag). 10 per XML sounds like nonsense. The user can use UploadWirzard...

Here are a few reasons why it makes sense and why the uploadWizard is not adapted in some cases: 0 - You don't have enough bandwidth on your personal computer 1 - You don't have the data on your computer 2 - You already have the metadata in a structured way 3 - Before making massive uploads, people want to try it *with the real tool* (GLAMs too). But, you make a point, maybe 10 is too low or we should remove this kind of limit and just rely on the upload speed. Anyway, this was just a starter proposition. Emmanuel -- Volunteer Technology, GLAM, Trainings Zurich +41 797 670 398

Steinsplitter Wiki

7:33 p.m.

...

0 - You don't have enough bandwidth on your personal computer

UploadWirzard supports uploadbyurl, but it is only enabled for licenser and admins because upload by url is not 100% stable. GWToolset has bugs too.

...

3 - Before making massive uploads, people want to try it *with the real tool* (GLAMs too).

We have the betacluster (http://commons.wikimedia.beta.wmflabs.org/) to test. It is a real clone. Per request (mailinglist/talkpage/mail) i can grant the gwtoolset right to trusted user without bureaucracy on the betacluster. Regards, > Date: Sun, 27 Apr 2014 19:19:50 +0200 > From: emmanuel.engelhart(a)wikimedia.ch > To: glamtools(a)lists.wikimedia.org > Subject: Re: [Glamtools] Open GWT to all Commons users > > On 27.04.2014 18:39, Steinsplitter Wiki wrote: > > We have changed from admin to cratmanegment for a good reason (GWT is > > powerful tool like botflag). > > > > 10 per XML sounds like nonsense. The user can use UploadWirzard... > > Here are a few reasons why it makes sense and why the uploadWizard is > not adapted in some cases:

...

0 - You don't have enough bandwidth on your personal computer

> 1 - You don't have the data on your computer > 2 - You already have the metadata in a structured way

...

3 - Before making massive uploads, people want to try it *with the real tool* (GLAMs too).

> > But, you make a point, maybe 10 is too low or we should remove this kind > of limit and just rely on the upload speed. Anyway, this was just a > starter proposition. > > Emmanuel > > -- > Volunteer > Technology, GLAM, Trainings > Zurich > +41 797 670 398 > > _______________________________________________ > Glamtools mailing list > Glamtools(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/glamtools

dan entous

28 Apr 28 Apr

10:29 a.m.

open to all ----------- my main concern in regards to opening the tool to everyone is whether or not bad up loaders could be dealt with quickly enough. too many uploads at once ------------------------ there are already “throttles” in place that limit the amount of files that can be uploaded at once and per background job run and the total number of GWToolset uploads that can exist in the job queue at once: • http://www.mediawiki.org/wiki/Extension:GWToolset/Technical_Design#Throttle… • http://git.wikimedia.org/blob/mediawiki%2Fextensions%2FGWToolset.git/306957… thus far i don’t think we’ve run into issues with those limit. big upload issue ---------------- as far as i currently understand the issue, the problem was not that GWToolset uploaded too many images; the problem was that the tool that generates thumbnails tried to create too many thumbnails at once. if that’s true, then it seems to me that the tool that takes care of that should have it’s own throttle and job queue so that it doesn’t try to create too many thumbnails at once. with kind regards, dan On Apr 27, 2014, at 19:33 , Steinsplitter Wiki <steinsplitter(a)wikipedia.de> wrote:

...

0 - You don't have enough bandwidth on your personal computer

UploadWirzard supports uploadbyurl, but it is only enabled for licenser and admins because upload by url is not 100% stable. GWToolset has bugs too.

3 - Before making massive uploads, people want to try it *with the real tool* (GLAMs too).

0 - You don't have enough bandwidth on your personal computer

> 1 - You don't have the data on your computer > 2 - You already have the metadata in a structured way

3 - Before making massive uploads, people want to try it *with the real tool* (GLAMs too).

Steinsplitter Wiki

4:26 p.m.

...

open to all -----------

@dan: So eine Änderung wird nicht duch's codereview kommen (community-consensus-needed@shellpolicy). Bevor ihr an den Rechteconfigs aufschraubt bitte RFC. > From: d_entous(a)yahoo.com > Date: Mon, 28 Apr 2014 10:29:31 +0200 > To: glamtools(a)lists.wikimedia.org > Subject: Re: [Glamtools] Open GWT to all Commons users >

...

open to all -----------

> my main concern in regards to opening the tool to everyone is whether or not bad up loaders could be dealt with quickly enough. > > > too many uploads at once > ------------------------ > there are already “throttles” in place that limit the amount of files that can be uploaded at once and per background job run and the total number of GWToolset uploads that can exist in the job queue at once: > > • http://www.mediawiki.org/wiki/Extension:GWToolset/Technical_Design#Throttle… > • http://git.wikimedia.org/blob/mediawiki%2Fextensions%2FGWToolset.git/306957… > > thus far i don’t think we’ve run into issues with those limit. > > > big upload issue > ---------------- > as far as i currently understand the issue, the problem was not that GWToolset uploaded too many images; the problem was that the tool that generates thumbnails tried to create too many thumbnails at once. if that’s true, then it seems to me that the tool that takes care of that should have it’s own throttle and job queue so that it doesn’t try to create too many thumbnails at once. > > > with kind regards, > dan > > > On Apr 27, 2014, at 19:33 , Steinsplitter Wiki <steinsplitter(a)wikipedia.de> wrote: > > > > 0 - You don't have enough bandwidth on your personal computer > > UploadWirzard supports uploadbyurl, but it is only enabled for licenser and admins because upload by url is not 100% stable. > > GWToolset has bugs too. > > > > > 3 - Before making massive uploads, people want to try it *with the real > > > tool* (GLAMs too). > > We have the betacluster (http://commons.wikimedia.beta.wmflabs.org/) to test. It is a real clone. > > Per request (mailinglist/talkpage/mail) i can grant the gwtoolset right to trusted user without bureaucracy on the betacluster. > > > > Regards, > > > > > Date: Sun, 27 Apr 2014 19:19:50 +0200 > > > From: emmanuel.engelhart(a)wikimedia.ch > > > To: glamtools(a)lists.wikimedia.org > > > Subject: Re: [Glamtools] Open GWT to all Commons users > > > > > > On 27.04.2014 18:39, Steinsplitter Wiki wrote: > > > > We have changed from admin to cratmanegment for a good reason (GWT is > > > > powerful tool like botflag). > > > > > > > > 10 per XML sounds like nonsense. The user can use UploadWirzard... > > > > > > Here are a few reasons why it makes sense and why the uploadWizard is > > > not adapted in some cases: > > > 0 - You don't have enough bandwidth on your personal computer > > > 1 - You don't have the data on your computer > > > 2 - You already have the metadata in a structured way > > > 3 - Before making massive uploads, people want to try it *with the real > > > tool* (GLAMs too). > > > > > > But, you make a point, maybe 10 is too low or we should remove this kind > > > of limit and just rely on the upload speed. Anyway, this was just a > > > starter proposition. > > > > > > Emmanuel > > > > > > -- > > > Volunteer > > > Technology, GLAM, Trainings > > > Zurich > > > +41 797 670 398 > > > > > > _______________________________________________ > > > Glamtools mailing list > > > Glamtools(a)lists.wikimedia.org > > > https://lists.wikimedia.org/mailman/listinfo/glamtools > > _______________________________________________ > > Glamtools mailing list > > Glamtools(a)lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/glamtools > > > _______________________________________________ > Glamtools mailing list > Glamtools(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/glamtools

Emmanuel Engelhart

27 Apr 27 Apr

6:26 p.m.

On 27.04.2014 15:15, Fæ wrote:

...

On 27/04/2014, Emmanuel Engelhart <emmanuel.engelhart(a)wikimedia.ch> wrote:

Do you see any other risk?

Yes. * If a user has 10 files to upload they should use the well designed standard wizard. The GWT is an amazingly cumbersome way to upload small numbers of files.

This is not a risk. Let people choose the tool they prefer.

...

* The GWT has a number of bugs and restricted features that would be very odd to explain to a user who was not part of a major GLAM upload project or had not had experience with batch uploads. For example xml validity testing is a complex issue and will never be covered by on-wiki step by step documentation.

All our tools have bugs (think to the uploadWizard) and many of them are not easy at all to use. As far as I know, this was never an argument to limit artificially their usage. It's possible that we still need to wait a little more to fix the most critical bugs before doing this, but then we should IMO setup now the list of show-stoppers. Anyway, if we want to increase the GWT bug fixing speed, we should increase the GWT audience. It's a basic rule of FOSS development.

...

* We have a small user group, if this were doubled by random interested users without skills to bring to the party, or involvement in large batch upload projects, it is highly likely that most of us more engaged folks would drift off into yet another clique.

I believe that people are not idiots. Most of them will keep using the uploadWizard because it's what they need and a few others will learn what they have to learn to use the GWT, they are not less skilled or more stupid than GLAM's employees. Emmanuel -- Volunteer Technology, GLAM, Trainings Zurich +41 797 670 398

3650

days inactive

3651

days old

glamtools@lists.wikimedia.org

Manage subscription

13 comments

6 participants

tags (0)

participants (6)

dan entous
Emmanuel Engelhart
Fæ
Micha L. Rieser
Steinsplitter Wiki
Steinsplitter Wiki