Hi guys,
We are trying to assess how many images are stored locally on Wikimedia sites, rather than hosted on Commons. We also want to better understand what templates they are using for their metadata.
Does anyone know where we could find reliable statistics and links for these non-Commons files?
We would like to check how well they work with Media Viewer, as well as get a sense of scope.
If they use the same templates and data structure as Commons, they should display well in Media Viewer. But if they use different templates, these would need to be modified for their meta-data to appear in Media Viewer.
Last time we checked, there were over 800k image files hosted on English Wikipedia alone, so this could be a significant number, which may require some advance work to get them ready for Media Viewer.
Gergo is preparing a document to address the template issue, which he can share with us when it’s ready.
Thanks,
Fabrice
_______________________________
Fabrice Florin Product Manager Wikimedia Foundation
We are trying to assess how many images are stored locally on Wikimedia sites, rather than hosted on Commons. We also want to better understand what templates they are using for their metadata.
Does anyone know where we could find reliable statistics and links for these non-Commons files?
Does not ring a bell to me. :-(
Closer I recall is the opposite - projects that do not need to be checked for non-CommonsFiles https://commons.wikimedia.org/wiki/Commons:Turning_off_local_uploads
Hi,
English Wikinews has a number of "non-free" images that can be found locally at https://en.wikinews.org/wiki/Category:Non-free_media , https://en.wikinews.org/wiki/Category:CC-BY-NC-SA-2.5 and https://en.wikinews.org/wiki/Category:Non-commercial .
Sincerely, Laura Hale
On Wed, Apr 9, 2014 at 6:00 PM, Fabrice Florin fflorin@wikimedia.orgwrote:
Hi guys,
We are trying to assess how many images are stored locally on Wikimedia sites, rather than hosted on Commons. We also want to better understand what templates they are using for their metadata.
Does anyone know where we could find reliable statistics and links for these non-Commons files?
We would like to check how well they work with Media Viewer, as well as get a sense of scope.
If they use the same templates and data structure as Commons, they should display well in Media Viewer. But if they use different templates, these would need to be modified for their meta-data to appear in Media Viewer.
Last time we checked, there were over 800k image files hosted on English Wikipedia alone, so this could be a significant number, which may require some advance work to get them ready for Media Viewer.
Gergo is preparing a document to address the template issue, which he can share with us when it's ready.
Thanks,
Fabrice
Fabrice Florin Product Manager Wikimedia Foundation
http://en.wikipedia.org/wiki/User:Fabrice_Florin_(WMF)
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
On Wed, Apr 9, 2014 at 4:16 PM, Laura Hale laura@fanhistory.com wrote:
English Wikinews has a number of "non-free" images that can be found locally at https://en.wikinews.org/wiki/Category:Non-free_media , https://en.wikinews.org/wiki/Category:CC-BY-NC-SA-2.5 and https://en.wikinews.org/wiki/Category:Non-commercial .
The easiest way to get a count is to just use Special:Statisticshttps://en.wikinews.org/wiki/Special:Statistics. Don't know of any tool which would give these numbers cross-wiki, though.
On Apr 9, 2014 8:33 PM, "Gergo Tisza" gtisza@wikimedia.org wrote:
On Wed, Apr 9, 2014 at 4:16 PM, Laura Hale laura@fanhistory.com wrote:
English Wikinews has a number of "non-free" images that can be found
locally at https://en.wikinews.org/wiki/Category:Non-free_media , https://en.wikinews.org/wiki/Category:CC-BY-NC-SA-2.5 and https://en.wikinews.org/wiki/Category:Non-commercial .
The easiest way to get a count is to just use Special:Statistics. Don't
know of any tool which would give these numbers cross-wiki, though.
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
Shell script on toolserver querying all the dbs would probably be pretty easy.
As for metadata similar to commons - almost certainly not. Especially on smaller non english wikis.
-bawolff
Gergo Tisza, 10/04/2014 01:33:
Don't know of any tool which would give these numbers cross-wiki, though.
http://wikistats.wmflabs.org/ "images" column. For size follow links at https://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_XML_dumps#Media_tarballs.
Nemo
Dear Jean-Fred, Laura, Gergo, Brian and Nemo,
Thanks so much for your helpful answers to this question!
Nemo, your wikistats link is particularly useful, and suggests that about 15% of files on Wikimedia files may be stored outside of Commons, which is significant (3M out of 23M total).
I would be grateful for any URLs that would make it easy for us to test Media Viewer performance on the largest sites that have that practice of local hosting (e.g. English Wikipedia, who else?).
Laura, your URLs of local files on Wikinews is exactly what I’m looking for on other sites, thanks. A casual check suggests that these files open in Media Viewer, but their metadata doesn’t display, which could be due to Wikinews using a different template than Commons.
So my next question to the group is what you think can be done to encourage communities like Wikinews to consider adapting their templates to match the Commons more closely? Otherwise, their local images will not be as useful in Media Viewer.
This may not be all that big of a change, and we can document this more specifically, but could use some help from community members to figure out practical solutions for updating these local templates.
What do you think?
Thanks again for all your help with this project.
Fabrice
On Apr 9, 2014, at 11:54 PM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
Gergo Tisza, 10/04/2014 01:33:
Don't know of any tool which would give these numbers cross-wiki, though.
http://wikistats.wmflabs.org/ "images" column. For size follow links at https://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_XML_dumps#Media_tarballs.
Nemo
On Apr 9, 2014 8:33 PM, "Gergo Tisza" gtisza@wikimedia.org wrote:
On Wed, Apr 9, 2014 at 4:16 PM, Laura Hale laura@fanhistory.com wrote:
English Wikinews has a number of "non-free" images that can be found locally at https://en.wikinews.org/wiki/Category:Non-free_media , https://en.wikinews.org/wiki/Category:CC-BY-NC-SA-2.5 and https://en.wikinews.org/wiki/Category:Non-commercial .
The easiest way to get a count is to just use Special:Statistics. Don't know of any tool which would give these numbers cross-wiki, though.
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
Shell script on toolserver querying all the dbs would probably be pretty easy.
As for metadata similar to commons - almost certainly not. Especially on smaller non english wikis.
-bawolff
On Apr 9, 2014, at 4:33 PM, Gergo Tisza gtisza@wikimedia.org wrote:
On Wed, Apr 9, 2014 at 4:16 PM, Laura Hale laura@fanhistory.com wrote: English Wikinews has a number of "non-free" images that can be found locally at https://en.wikinews.org/wiki/Category:Non-free_media , https://en.wikinews.org/wiki/Category:CC-BY-NC-SA-2.5 and https://en.wikinews.org/wiki/Category:Non-commercial .
The easiest way to get a count is to just use Special:Statistics. Don't know of any tool which would give these numbers cross-wiki, though.
On Apr 9, 2014, at 4:16 PM, Laura Hale laura@fanhistory.com wrote:
Hi,
English Wikinews has a number of "non-free" images that can be found locally at https://en.wikinews.org/wiki/Category:Non-free_media , https://en.wikinews.org/wiki/Category:CC-BY-NC-SA-2.5 and https://en.wikinews.org/wiki/Category:Non-commercial .
Sincerely, Laura Hale
On Apr 9, 2014, at 4:13 PM, Jean-Frédéric jeanfrederic.wiki@gmail.com wrote:
We are trying to assess how many images are stored locally on Wikimedia sites, rather than hosted on Commons. We also want to better understand what templates they are using for their metadata.
Does anyone know where we could find reliable statistics and links for these non-Commons files?
Does not ring a bell to me. :-(
Closer I recall is the opposite − projects that do not need to be checked for non-CommonsFiles https://commons.wikimedia.org/wiki/Commons:Turning_off_local_uploads
-- Jean-Fred
On Wed, Apr 9, 2014 at 6:00 PM, Fabrice Florin fflorin@wikimedia.org wrote: Hi guys,
We are trying to assess how many images are stored locally on Wikimedia sites, rather than hosted on Commons. We also want to better understand what templates they are using for their metadata.
Does anyone know where we could find reliable statistics and links for these non-Commons files?
We would like to check how well they work with Media Viewer, as well as get a sense of scope.
If they use the same templates and data structure as Commons, they should display well in Media Viewer. But if they use different templates, these would need to be modified for their meta-data to appear in Media Viewer.
Last time we checked, there were over 800k image files hosted on English Wikipedia alone, so this could be a significant number, which may require some advance work to get them ready for Media Viewer.
Gergo is preparing a document to address the template issue, which he can share with us when it’s ready.
Thanks,
Fabrice
Fabrice Florin Product Manager Wikimedia Foundation
On Thu, Apr 10, 2014 at 4:55 PM, Fabrice Florin fflorin@wikimedia.orgwrote:
Dear Jean-Fred, Laura, Gergo, Brian and Nemo,
Thanks so much for your helpful answers to this question!
Nemo, your wikistats link http://wikistats.wmflabs.org/ is particularly useful, and suggests that about 15% of files on Wikimedia files may be stored outside of Commons, which is significant (3M out of 23M total).
I would be grateful for any URLs that would make it easy for us to test Media Viewer performance on the largest sites that have that practice of local hosting (e.g. English Wikipedia, who else?).
Laura, your URLs of local files on Wikinews is exactly what I'm looking for on other sites, thanks. A casual check suggests that these files open in Media Viewer, but their metadata doesn't display, which could be due to Wikinews using a different template than Commons.
So my next question to the group is what you think can be done to encourage communities like Wikinews to consider adapting their templates to match the Commons more closely? Otherwise, their local images will not be as useful in Media Viewer.
Some one would need to come and tweak the templates themselves (and then automate to fix that). Beyond that, not sure how they would conform as we need fair use rationales for some images, and we need to have CC-BY-NC licences for that. They aren't hosted on Commons deliberately, and making them more like Commons is something I suspect could cause problems globally because of that. Templates for images are not something we've spent much time on. :/ If some one wants to come in and tweak our more common ones that are already in use, that would be awesome.
Sincerely, Laura Hale
On 10 apr. 2014, at 23:55, Fabrice Florin fflorin@wikimedia.org wrote:
So my next question to the group is what you think can be done to encourage communities like Wikinews to consider adapting their templates to match the Commons more closely? Otherwise, their local images will not be as useful in Media Viewer.
This may not be all that big of a change, and we can document this more specifically, but could use some help from community members to figure out practical solutions for updating these local templates.
In many cases a good well prepared call for action is all that is needed. Explain, what needs to be changed and where. For that last part, we could identify all the categories on wiki's that we think identify license templates, then rank the templates that are 'most used' and give the wiki's trough village pump messaging a list so that they can fix it pre launch. I'm positive that would have a big impact. The call for action can also be integrated right in the UI post launch, where MMV can say "Unable to show all information for this image. Please report this image to the community".
What is left after that, is probably not worth investing time in, until the time that we have true structured metadata.
Also important, we need a good cross wiki meta tag for fair use. (and figure out what to do with those in MMV).
DJ
Thanks, DJ!
Great advice, as always. :)
We will send out a call to action, as you recommend. Gergo is preparing a list of recommended templates, which we will link to in our outreach.
This will be good preparation for the wider range of updates that will be needed once we implement structured data on Commons later in the year.
Your suggestion to include a call to action in the UI for this use case makes good sense as well. I have updated our error message spec to that end:
https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/299
Lastly, we will create a special FAQ for this as well on this help page:
https://www.mediawiki.org/wiki/Multimedia/Media_Viewer/Help
Not sure how to get a good cross-wiki meta tag for fair use. Any suggestions from community veterans on how to do that?
Fabrice
On Apr 12, 2014, at 5:26 AM, Derk-Jan Hartman d.j.hartman@gmail.com wrote:
On 10 apr. 2014, at 23:55, Fabrice Florin fflorin@wikimedia.org wrote:
So my next question to the group is what you think can be done to encourage communities like Wikinews to consider adapting their templates to match the Commons more closely? Otherwise, their local images will not be as useful in Media Viewer.
This may not be all that big of a change, and we can document this more specifically, but could use some help from community members to figure out practical solutions for updating these local templates.
In many cases a good well prepared call for action is all that is needed. Explain, what needs to be changed and where. For that last part, we could identify all the categories on wiki's that we think identify license templates, then rank the templates that are 'most used' and give the wiki's trough village pump messaging a list so that they can fix it pre launch. I'm positive that would have a big impact. The call for action can also be integrated right in the UI post launch, where MMV can say "Unable to show all information for this image. Please report this image to the community".
What is left after that, is probably not worth investing time in, until the time that we have true structured metadata.
Also important, we need a good cross wiki meta tag for fair use. (and figure out what to do with those in MMV).
DJ
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
_______________________________
Fabrice Florin Product Manager Wikimedia Foundation
On Sat, Apr 12, 2014 at 5:26 AM, Derk-Jan Hartman d.j.hartman@gmail.comwrote:
In many cases a good well prepared call for action is all that is needed. Explain, what needs to be changed and where.
There is a tutorial (although probably not a user-friendly one) at Multimedia/Media_Viewer/Template_compatibilityhttps://www.mediawiki.org/wiki/Multimedia/Media_Viewer/Template_compatibility .
The call for action can also be integrated right in the UI post launch, where MMV can say "Unable to show all information for this image. Please report this image to the community".
Not sure how we can tell that there is information that we can't display, unless we just show this for every image which does not have a compatible information template.
What is left after that, is probably not worth investing time in, until the time that we have true structured metadata.
Images that are uploaded to other places than Commons will not have structured metadata. (Not in the foreseeable future, at the very least.)
Also important, we need a good cross wiki meta tag for fair use. (and figure out what to do with those in MMV).
Non-free images usually have a license-like template (such as {{ Non-free_fair_use https://en.wikipedia.org/wiki/Template:Non-free_fair_use}}), that can be marked up as a license template so it's displayed by MediaViewer.
multimedia@lists.wikimedia.org