[merged thread]

Hi Bawolff and Daniel,

This explanation makes sense, as I'm running the same set of unit tests on 3 different environments, each environment runs in parallel with one another. And I've just realized 20191116051316 is just a timestamp with accuracy up to seconds XD

So for now I think I will just append a random suffix to the file name for this unit-testing purpose. Thanks for your help!

Regards, Xinyan

-----Original Message [1/2] ----- From: Wikitech-l wikitech-l-bounces@lists.wikimedia.org On Behalf Of bawolff Sent: Tuesday, November 19, 2019 9:37 AM To: Wikimedia developers wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] The difference between fileexists-no-change and backend-fail-alreadyexists for action=upload

So fileexists-no-change happens when your uploading a new version of a file, MediaWiki calculates the sha1 sum, and notices that the img_sha1 field in the db for the current version of the file is the same as what was just calculated for the new file. Which probably (although not necessarily now that sha1 is broken) means the file you're uploading is the same as the one already there.

backend-fail-alreadyexists basically means that MediaWiki was about to write a file somewhere, but there already was a file at that place, and the overwrite/overwriteSame flags were not set, so there should not have already been a file in that place (The file already there, may or may not be the same as the new file). Normally I would assume that something more frontend-y would give an error before this error would be encountered. Perhaps it might be encountered during a race condition. This is mostly speculation as I'm not super familiar with the FileBackend code.

-- bawolff

-----Original Message [2/2] ----- From: Daniel Kinzler dkinzler@wikimedia.org Sent: Monday, November 18, 2019 8:56 PM To: Chen Xinyan cxuesongc@outlook.com Subject: Re: [Wikitech-l] The difference between fileexists-no-change and backend-fail-alreadyexists for action=upload

Hi Xinyan!

I may be wrong about this, but from my understanding backend-fail-alreadyexists is an unexpected internal error that indicates a conflict in the storage backend. My best guess is a problem with the granularity of the timestamp used to construct the filename - I could imagine that, if you manage to upload two versions of a file within the same second, the backend fails because the versions are distinguished only by timestamp (20191116051316 is 2019-11-16 05:13:16). An alternative explanation might be a race conidtion triggered when two processes are trying to upload a file at the same time - this should be prevented by applying the appropriate locking, but who knows.

This is all just an "informed guess", my understanding on how the file backend works is superficial.

HTH Daniel

On Sat, Nov 16, 2019 at 6:51 AM Chen Xinyan cxuesongc@outlook.com wrote:

Hi there,

Hope I've found the correct list for asking this question.

I was setting up a CI test case for my MediaWiki Client Library to run a bot that will upload a file to https://en.wikipedia.beta.wmflabs.org in chunked stash mode. For most of the time, when I perform the final upload with filekey parameter, I will receive a fileexists-no-change error from the MW API server. This is expected, as there is already a same file with the same title on the site. However, today I received backend-fail-alreadyexists. The error message looks like

backend-fail-alreadyexists: The file "mwstore://local-swift-eqiad/local-public/archive/9/95/20191116051316!Test_image.jpg" already exists.

Does this error usually occurs when user tries to upload the same file? What's the difference between fileexists-no-change and backend-fail-alreadyexists?

Thanks, Xinyan

---

Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l