Bot upload of files over 100MB - Wikitech-l

Techman224

25 Jun 25 Jun

11:47 p.m.

Right now the wiki is setup to only allow up to 100 MB files. The only way I see to upload a file greater than that is to file a bugzilla request and a system administrator with shell access can manually upload it using a script. Techmn224 On 2013-06-25, at 4:43 PM, Daniel Mietchen <daniel.mietchen(a)googlemail.com> wrote:

...

Hi, my bot[1] occasionally stumbles upon files that are above 100MB and thus does not upload them[2]. What do I have to do to get it set up for handling these files too? Thanks for any pointers, Daniel [1] http://commons.wikimedia.org/wiki/User:Open_Access_Media_Importer_Bot [2] https://github.com/erlehmann/open-access-media-importer/issues/22#issuecomm… _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Reply

Techman224

12:18 a.m.

https://commons.wikimedia.org/w/index.php?title=Special:Upload shows 100 MB. I turned on chunked uploads and it still shows 100 MB. Techman224 On 2013-06-25, at 5:13 PM, Greg Grossmeier <greg(a)wikimedia.org> wrote:

...

Right now the wiki is setup to only allow up to 100 MB files. The only way I see to upload a file greater than that is to file a bugzilla request and a system administrator with shell access can manually upload it using a script.

I thought it was set to 500mb? https://git.wikimedia.org/blob/operations%2Fmediawiki-config/40f5cf38a00edc… Greg -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D | _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Reply

Techman224

12:25 a.m.

Oh yea, I forgot it was for the Upload Wizard. It doesn't say the maximum file size allowed. Techman224 On 2013-06-25, at 5:22 PM, Techman224 <techman224(a)techman224.ca> wrote:

...

There's a comment for that variable, "Only affects URL uploads; web uploads are enforced by PHP." Techman224 On 2013-06-25, at 5:18 PM, Techman224 <techman224(a)techman224.ca> wrote:

https://commons.wikimedia.org/w/index.php?title=Special:Upload shows 100 MB. I turned on chunked uploads and it still shows 100 MB. Techman224 On 2013-06-25, at 5:13 PM, Greg Grossmeier <greg(a)wikimedia.org> wrote:

Right now the wiki is setup to only allow up to 100 MB files. The only way I see to upload a file greater than that is to file a bugzilla request and a system administrator with shell access can manually upload it using a script.

I thought it was set to 500mb? https://git.wikimedia.org/blob/operations%2Fmediawiki-config/40f5cf38a00edc… Greg -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D | _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Reply

Jeremy Baron

25 Jun 25 Jun

11:49 p.m.

On Jun 25, 2013 5:44 PM, "Daniel Mietchen" <daniel.mietchen(a)googlemail.com> wrote:

...

my bot[1] occasionally stumbles upon files that are above 100MB and thus does not upload them[2]. What do I have to do to get it set up for handling these files too?

this looks like the relevant section: https://www.mediawiki.org/wiki/API:Upload#Chunked_uploading I don't know the current settings; you might need to enable chunked uploads in the MediaWiki prefs for the user you're uploading as. -Jeremy

Reply

Tilman Bayer

11:58 p.m.

With the https://commons.wikimedia.org/wiki/Commons:Chunked_uploads enabled, users can currently upload files up to 500MB, when using a supported browser (e.g. a current Firefox or Chrome). I think Daniel's question was about how to make this method work for a bot too, i.e. via the API rather than in a browser. On Tue, Jun 25, 2013 at 2:52 PM, OQ <overlordq(a)gmail.com> wrote:

...

I would assume given the previous reply that regardless of upload method, chunked or otherwise, there is still the hard limit of how big the resultant file can be. On Tue, Jun 25, 2013 at 5:49 PM, Jeremy Baron <jeremy(a)tuxmachine.com> wrote:

On Jun 25, 2013 5:44 PM, "Daniel Mietchen" <daniel.mietchen(a)googlemail.com

wrote:

my bot[1] occasionally stumbles upon files that are above 100MB and thus does not upload them[2]. What do I have to do to get it set up for handling these files too?

this looks like the relevant section: https://www.mediawiki.org/wiki/API:Upload#Chunked_uploading I don't know the current settings; you might need to enable chunked uploads in the MediaWiki prefs for the user you're uploading as. -Jeremy _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

-- Tilman Bayer Senior Operations Analyst (Movement Communications) Wikimedia Foundation IRC (Freenode): HaeB

Reply

Daniel Mietchen

2:23 a.m.

Thanks to all the replies so far. Looks like chunked upload is the way to go. Cheers, Daniel On Wed, Jun 26, 2013 at 12:06 AM, Brian Wolff <bawolff(a)gmail.com> wrote:

...

When browsing through commons, I happened to stumble upon https://commons.wikimedia.org/wiki/User:Smallman12q/PyCJWiki which appears to be a python bot that uses chunked uploading, and thus work with files up to 500 mb big. Perhaps it would be helpful to you. --bawolff On 6/25/13, Tilman Bayer <tbayer(a)wikimedia.org> wrote:

With the https://commons.wikimedia.org/wiki/Commons:Chunked_uploads enabled, users can currently upload files up to 500MB, when using a supported browser (e.g. a current Firefox or Chrome). I think Daniel's question was about how to make this method work for a bot too, i.e. via the API rather than in a browser. On Tue, Jun 25, 2013 at 2:52 PM, OQ <overlordq(a)gmail.com> wrote:

I would assume given the previous reply that regardless of upload method, chunked or otherwise, there is still the hard limit of how big the resultant file can be. On Tue, Jun 25, 2013 at 5:49 PM, Jeremy Baron <jeremy(a)tuxmachine.com> wrote:

On Jun 25, 2013 5:44 PM, "Daniel Mietchen" <daniel.mietchen(a)googlemail.com > wrote: > my bot[1] occasionally stumbles upon files that are above 100MB and > thus does not upload them[2]. What do I have to do to get it set up > for handling these files too? this looks like the relevant section: https://www.mediawiki.org/wiki/API:Upload#Chunked_uploading I don't know the current settings; you might need to enable chunked uploads in the MediaWiki prefs for the user you're uploading as. -Jeremy _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

-- Tilman Bayer Senior Operations Analyst (Movement Communications) Wikimedia Foundation IRC (Freenode): HaeB _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Reply