Help test chunked uploading

List overview All Threads
Download

newer

older

Desktop upload tool

Fwd: [Wikitech-l] UploadWizard...

Erik Moeller

4 May 2012 4 May '12

5:16 a.m.

http://commons.wikimedia.org/wiki/Special:Preferences#mw-prefsection-uploads There's a new user preference "chunked uploads for files over 1 MB" for Upload Wizard. The intent of this feature is to reliably transfer large files, mitigating connection reliability issues. This will enable us to raise the file size limit for all users. Please help test this feature and report issues (performance, file corruption, errors, etc.). The carrot is that with this preference enabled, you can upload files <500 MB today (as opposed to <100 MB) :-) The feature currently only works with Upload Wizard. More information: http://commons.wikimedia.org/wiki/Commons:Chunked_uploads Please report bugs on the talk page or against UploadWizard in Bugzilla: https://bugzilla.wikimedia.org/enter_bug.cgi?product=MediaWiki%20extensions… -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

Show replies by date

Raimond Spekking

5 May 5 May

10:46 a.m.

Am 04.05.2012 05:16, schrieb Erik Moeller:

...

Thanks for the info, Erik. Does anyone know, if the pywikipediabot framework supports chunked uploading now or in the near future? Raimond.

Maarten Dammers

1 p.m.

Hi Raimond, Op 5-5-2012 10:46, Raimond Spekking schreef:

...

Am 04.05.2012 05:16, schrieb Erik Moeller:

http://commons.wikimedia.org/wiki/Special:Preferences#mw-prefsection-uploads There's a new user preference "chunked uploads for files over 1 MB" for Upload Wizard. The intent of this feature is to reliably transfer large files, mitigating connection reliability issues. This will enable us to raise the file size limit for all users. Please help test this feature and report issues (performance, file corruption, errors, etc.). The carrot is that with this preference enabled, you can upload files<500 MB today (as opposed to<100 MB) :-) The feature currently only works with Upload Wizard. More information: http://commons.wikimedia.org/wiki/Commons:Chunked_uploads Please report bugs on the talk page or against UploadWizard in Bugzilla: https://bugzilla.wikimedia.org/enter_bug.cgi?product=MediaWiki%20extensions…

Thanks for the info, Erik. Does anyone know, if the pywikipediabot framework supports chunked uploading now or in the near future?

I was about to forward this email to the pywikipedia-l list to suggest someone to implement this. Based on http://www.mediawiki.org/wiki/API:Upload#Chunked_uploading this shouldn't be too hard. Maarten

Federico Leva (Nemo)

7 May 7 May

11:54 a.m.

Very nice news! I hope we'll soon be able to reach the sweetness of the archive.org uploader, which allowed me to upload files up to 7-8 GB smoothly and without errors (on Chromium). Erik Moeller, 04/05/2012 05:16:

...

Please help test this feature and report issues (performance, file corruption, errors, etc.). The carrot is that with this preference enabled, you can upload files<500 MB today (as opposed to<100 MB) :-)

This is indeed a sweet carrot; I've tried to upload the new WMIT WikiGuide video (190 MB) but it failed with internal_api_error_UploadStashFileNotFoundException (took too long?). Now I'll try again, but I had also the following (non blocking) problems which make it unusable (or very user-unfriendly) at least on my platform: * upload is terribly slow, averaged at 25-50 KB/s and never went above 100 (but I didn't stare at it for hours); * especially when you add the file and start upload, it's very resource-intensive and made my browser stall for a short while: 400 MB of RAM used for several minutes, CPU usage hopping, a whole core waiting for disk (?). I suppose you're already working on these problems, but if they're platform-specific (I used Firefox on Ubuntu) and you need more details I can file some bugs. Nemo

Prolineserver

12:24 p.m.

Hi, On Mon, May 7, 2012 at 11:54 AM, Federico Leva (Nemo) <nemowiki(a)gmail.com> wrote:

...

This is indeed a sweet carrot; I've tried to upload the new WMIT WikiGuide video (190 MB) but it failed with internal_api_error_UploadStashFileNotFoundException (took too long?).

I got the same error, but: The file was uploaded correctly to Commons.

...

* especially when you add the file and start upload, it's very resource-intensive and made my browser stall for a short while: 400 MB of RAM used for several minutes, CPU usage hopping, a whole core waiting for disk (?).

Chromium on Ubuntu is just crashing, Firefox complains about a slow script, both on Windows and Ubuntu, but it still works. Regards, Holger

Federico Leva (Nemo)

12:33 p.m.

Prolineserver, 07/05/2012 12:24:

...

Chromium on Ubuntu is just crashing, Firefox complains about a slow script, both on Windows and Ubuntu, but it still works.

On the contrary, with Chromium 18.0.1025.142 (Build 129054 Linux) I had a much better experience. It still failed though, filed as https://bugzilla.wikimedia.org/show_bug.cgi?id=36587 Nemo

Erik Moeller

7:13 p.m.

On Mon, May 7, 2012 at 2:54 AM, Federico Leva (Nemo) <nemowiki(a)gmail.com> wrote:

...

Yeah, UW was trying to extract JPEG metadata from video files. :-P This should be fixed once https://gerrit.wikimedia.org/r/#/c/6722/ is reviewed and deployed. The internal API error bug will probably be a bit harder to track down; hazarding a guess it might be a timing issue with the copy operation on very large files. Can you file a bug for the upload performance issue as well? That one is new to me. Thanks, Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

Federico Leva (Nemo)

9:06 p.m.

Erik Moeller, 07/05/2012 19:13:

...

On Mon, May 7, 2012 at 2:54 AM, Federico Leva (Nemo) wrote:

What about the automatic clearing of the stash? Wasn't that made more aggressive at some point?

...

Can you file a bug for the upload performance issue as well? That one is new to me.

Done: https://bugzilla.wikimedia.org/show_bug.cgi?id=36599 Nemo

とある白い猫

8 May 8 May

10:08 p.m.

Limit still is 100MB. I am willing to test but I can't with the 100MB restriction. -- とある白い猫 (To Aru Shiroi Neko) On Mon, May 7, 2012 at 9:06 PM, Federico Leva (Nemo) <nemowiki(a)gmail.com>wrote;wrote:

...

Erik Moeller, 07/05/2012 19:13:

On Mon, May 7, 2012 at 2:54 AM, Federico Leva (Nemo) wrote: * especially when you add the file and start upload, it's very

resource-intensive and made my browser stall for a short while: 400 MB of RAM used for several minutes, CPU usage hopping, a whole core waiting for disk (?).

Yeah, UW was trying to extract JPEG metadata from video files. :-P This should be fixed once https://gerrit.wikimedia.org/**r/#/c/6722/<https://gerrit.wikimedia.org/… reviewed and deployed. The internal API error bug will probably be a bit harder to track down; hazarding a guess it might be a timing issue with the copy operation on very large files.

What about the automatic clearing of the stash? Wasn't that made more aggressive at some point? Can you file a bug for the upload performance issue as well? That one

is new to me.

Done: https://bugzilla.wikimedia.**org/show_bug.cgi?id=36599<https://bugzilla.… Nemo ______________________________**_________________ Commons-l mailing list Commons-l(a)lists.wikimedia.org https://lists.wikimedia.org/**mailman/listinfo/commons-l<https://lists.w…

Erik Moeller

11:51 p.m.

On Tue, May 8, 2012 at 1:08 PM, とある白い猫 <to.aru.shiroi.neko(a)gmail.com> wrote:

...

Limit still is 100MB. I am willing to test but I can't with the 100MB restriction.

とある白い猫

9 May 9 May

11:18 p.m.

Firefox, IE and Chorme on Windows 7 - IE upload just goes on forever. It attempted to upload for over a day and nothing happened. - Chrome just dies. - Firefox does nothing when I pick to upload any file I am using Special:UploadWizard to upload a 1.4gb file. -- とある白い猫 (To Aru Shiroi Neko) On Tue, May 8, 2012 at 11:51 PM, Erik Moeller <erik(a)wikimedia.org> wrote:

...

On Tue, May 8, 2012 at 1:08 PM, とある白い猫 <to.aru.shiroi.neko(a)gmail.com> wrote:

Limit still is 100MB. I am willing to test but I can't with the 100MB restriction.

It's not -- if you get that message, it's likely that your browser doesn't support chunked uploading (or you don't have the user preference enabled). Which browser were you testing with? -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate _______________________________________________ Commons-l mailing list Commons-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/commons-l

Federico Leva (Nemo)

11:56 p.m.

とある白い猫, 09/05/2012 23:18:

...

I am using Special:UploadWizard to upload a 1.4gb file.

Erik Moeller, 04/05/2012 05:16:

...

The carrot is that with this preference enabled, you can upload files<500 MB today (as opposed to<100 MB)

Nemo

とある白い猫

11 May 11 May

8:22 p.m.

I get the same problem with a 173 mb file. -- とある白い猫 (To Aru Shiroi Neko) On Wed, May 9, 2012 at 11:56 PM, Federico Leva (Nemo) <nemowiki(a)gmail.com>wrote;wrote:

...

とある白い猫, 09/05/2012 23:18: I am using Special:UploadWizard to upload a 1.4gb file.

Erik Moeller, 04/05/2012 05:16:

The carrot is that with this preference enabled, you can upload files<500 MB today (as opposed to<100 MB)

Nemo ______________________________**_________________ Commons-l mailing list Commons-l(a)lists.wikimedia.org https://lists.wikimedia.org/**mailman/listinfo/commons-l<https://lists.w…

Erik Moeller

10:34 p.m.

On Fri, May 11, 2012 at 11:22 AM, とある白い猫 <to.aru.shiroi.neko(a)gmail.com> wrote:

...

I get the same problem with a 173 mb file.

Until this change is merged and deployed: https://gerrit.wikimedia.org/r/#/c/6722/ You'll experience browser crashes/slowness with very large files. However, _if_ (and only if) you're able to manage to start a chunked upload of a large file in a recent version of Firefox and Chrome, please tell me if you're still seeing errors on the first step. We just deployed a timeout fix that may have resolved the error message sometimes returned on the first step. -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

とある白い猫

11:16 p.m.

I have a no go on IE (attempts to upload forever) and Chrome (crashes the first second). I managed to upload a file on Firefox: http://commons.wikimedia.org/wiki/File:Iridescent_Glory_of_Nearby_Helix_Neb… Could someone review if the upload is corrupted or not? -- とある白い猫 (To Aru Shiroi Neko) On Fri, May 11, 2012 at 10:34 PM, Erik Moeller <erik(a)wikimedia.org> wrote:

...

On Fri, May 11, 2012 at 11:22 AM, とある白い猫 <to.aru.shiroi.neko(a)gmail.com> wrote:

I get the same problem with a 173 mb file.

Erik Moeller

25 May 25 May

1:25 a.m.

On Fri, May 11, 2012 at 1:34 PM, Erik Moeller <erik(a)wikimedia.org> wrote:

...

On Fri, May 11, 2012 at 11:22 AM, とある白い猫 <to.aru.shiroi.neko(a)gmail.com> wrote:

I get the same problem with a 173 mb file.

Until this change is merged and deployed: https://gerrit.wikimedia.org/r/#/c/6722/

It is now. Please try chunked uploading again -- it should work now, at least in recent versions of Firefox and Chrome, for files up to 500MB (some steps may take a long time to complete, but you shouldn't get errors). -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

4353

days inactive

4373

days old

commons-l@lists.wikimedia.org

Manage subscription

15 comments

6 participants

tags (0)

participants (6)

Erik Moeller
Federico Leva (Nemo)
Maarten Dammers
Prolineserver
Raimond Spekking
とある白い猫