Dear GLAMWiki-verse,
[tl;dr Seeking feedback. Please read the GWT 2.0 grant application draft before I submit it officially: https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset ]
As you're aware, within *Europeana* I (and MANY others) have been working on building a major grant application to the Wikimedia Foundation to continue development of the GLAMwiki Toolset.
I think the last few months - since the toolset has been "public" - have shown that there is widespread interest in (and usage of) the tool. It's already become part of the "standard" things that we use when talking to potential partners. However, that there are lots of things that can still be improved in it.
As anyone who has ever written a large grant proposal knows, there is a LOT of negotiation, behind the scenes that goes in to getting the document supported by all the relevant stakeholders, and this is no exception. [Heck - I chaired the first formal meeting about this project during Wikimania Haifa!!]
So, it is with great pleasure that I can say that the GLAMwiki Toolset grant application is *almost *ready to be submitted officially.
This is where *you* come in... Many of you have already seen this grant in various states of construction or have been part of the consultation process about what gets prioritised and what doesn't. Now I would really appreciate if you could have another look at the [hopefully] finished application. This is the chance to identify any errors, missing elements, excellent examples I've forgotten, logical flaws, spelling mistakes...
*https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset*
- If you like it, excellent. Please put your name on the "endorse" list at the bottom. - If you've got a comment/critique/suggestion, excellent. Tell me (on the talkpage, email, morse-code, carrier-pigeon etc...)
Once any major points arising have been addressed, we'll then "flip the switch" and formally submit the grant application.
-Liam / Wittylama Europeana GLAMwiki liaison
p.s. In case you're wondering, we do have permission from the WMF grants team to submit this application at this time, even thought it is during the 'inspire grants' campaign https://meta.wikimedia.org/wiki/Grants:IdeaLab/Inspire_Grants_%E2%80%93_Gender_gap_campaign, because it was already so far down the track of negotiation with various stakeholders.
wittylama.com Peace, love & metadata
On Feb 12, 2015 1:11 PM, "Liam Wyatt" liamwyatt@gmail.com wrote:
Dear GLAMWiki-verse,
[tl;dr Seeking feedback. Please read the GWT 2.0 grant application
draft before I submit it officially: https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset ]
As you're aware, within Europeana I (and MANY others) have been working
on building a major grant application to the Wikimedia Foundation to continue development of the GLAMwiki Toolset.
I think the last few months - since the toolset has been "public" - have
shown that there is widespread interest in (and usage of) the tool. It's already become part of the "standard" things that we use when talking to potential partners. However, that there are lots of things that can still be improved in it.
As anyone who has ever written a large grant proposal knows, there is a
LOT of negotiation, behind the scenes that goes in to getting the document supported by all the relevant stakeholders, and this is no exception. [Heck - I chaired the first formal meeting about this project during Wikimania Haifa!!]
So, it is with great pleasure that I can say that the GLAMwiki Toolset
grant application is almost ready to be submitted officially.
This is where you come in... Many of you have already seen this grant in various states of
construction or have been part of the consultation process about what gets prioritised and what doesn't. Now I would really appreciate if you could have another look at the [hopefully] finished application. This is the chance to identify any errors, missing elements, excellent examples I've forgotten, logical flaws, spelling mistakes...
https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset
If you like it, excellent. Please put your name on the "endorse" list at
the bottom.
If you've got a comment/critique/suggestion, excellent. Tell me (on the
talkpage, email, morse-code, carrier-pigeon etc...)
Once any major points arising have been addressed, we'll then "flip the
switch" and formally submit the grant application.
-Liam / Wittylama Europeana GLAMwiki liaison
p.s. In case you're wondering, we do have permission from the WMF grants
team to submit this application at this time, even thought it is during the 'inspire grants' campaign, because it was already so far down the track of negotiation with various stakeholders.
wittylama.com Peace, love & metadata
Glamtools mailing list Glamtools@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/glamtools
First of all a nitpick:
" It is the only “integrated upload tool” for Wikimedia commons (aside from the WMF-developed Upload Wizard)"
Technically you also have the original special:upload, and also that mobile thing thats less popular.
---
One thing i would like to see in future gwt development is bringing it further in line with mediawiki coding conventions. There are several parts that do things differently than the normal mediawiki way, and also some parts that duplicate mediawiki code, but is subtly different (e.g. some of the sanitation code stands out in that regard). Ancedotally i believe this inconsistency with MW core code conventions is significantly reducing the amount of patches that interested unaffiliated developers would otherwise contribute. I also think that more closely following mw core conventions would make it easier for a wmf staff (or volunteer) who isnt previously familar with the code to debug critical bugs in emergency situations (e.g. things like the hhvm job runner bug).
---
On the subject of code review, i believe a major contributing factor to difficulties last time, was development outside of gerrit, and waiting until the end to do code review. I strongly urge that all development be done on wmf's version control (because people watch what happens on wmf version control, and even if nobody is providing code review, they may still point out things that could be significant problems later) . I know first hand that getting code review, especially for non-wmf projects can be extremely difficult. If at all possible, i strongly suggest code review be done as incremently as possible.
Good luck with the grant. I really hope it goes through, gwt is an important project.
--bawolff
Thank you Brian for these comments. Details like this - practical, specific, based-on-developer-experience suggestions, are GOLD and I'm really hoping that you'll be able to provide more of them along the way.
Replies inline:
On 13 February 2015 at 16:42, Brian Wolff bawolff@gmail.com wrote:
First of all a nitpick:
" It is the only “integrated upload tool” for Wikimedia commons (aside from the WMF-developed Upload Wizard)"
Technically you also have the original special:upload, and also that mobile thing thats less popular.
That phrase came from the fact that the Upload Wizard and the GWT were the only things listed in the "integrated" section of the page [[Commons:Upload Tools]]. However, I see that as of a couple of weeks ago someone's added a new item there too: https://commons.wikimedia.org/w/index.php?title=Commons:Upload_tools&dif...
I'll take your recommendation for how to better phrase this sentence to be both factually accurate, but also to try to emphasise that this tool is trying its best to be "of the system" rather than sitting outside of it.
One thing i would like to see in future gwt development is bringing it further in line with mediawiki coding conventions. There are several parts that do things differently than the normal mediawiki way, and also some parts that duplicate mediawiki code, but is subtly different (e.g. some of the sanitation code stands out in that regard). Ancedotally i believe this inconsistency with MW core code conventions is significantly reducing the amount of patches that interested unaffiliated developers would otherwise contribute. I also think that more closely following mw core conventions would make it easier for a wmf staff (or volunteer) who isnt previously familar with the code to debug critical bugs in emergency situations (e.g. things like the hhvm job runner bug).
Is it feasible for you to identify the specific parts that you're talking about and add them to either the bug is on phabricator, or simply to email them back to me and Dan? While noting that I'm not qualified enough to know, I would think that "ensure consistency with MW core code conventions" is an applicable thing to be added to very first item in this grant proposal ("Identify major bugs in current system and fix"). While these are not "bugs" per-se, if you can identify a hit-list of the places where the current system diverges from standard practice, that would be very helpful. One of the things I've already heard back from the WMF grants team is that they'd like to see some demonstration of volunteer developer interest in the GWT, and if addressing this point would help in that regard - that would be excellent.
On the subject of code review, i believe a major contributing factor to difficulties last time, was development outside of gerrit, and waiting until the end to do code review. I strongly urge that all development be done on wmf's version control (because people watch what happens on wmf version control, and even if nobody is providing code review, they may still point out things that could be significant problems later) . I know first hand that getting code review, especially for non-wmf projects can be extremely difficult. If at all possible, i strongly suggest code review be done as incremently as possible.
I'll take any suggestions for how point might be integrated into the grant application text, but suffice to say that I agree with you, and the plan is always to try and do things 'by the book' and to 'bring people along for the ride'. I think both we (europeana) and the WMF have lessons-learned from the code-review experience last time. I'll let Dan respond with any technical aspects to this point if he wishes?
Good luck with the grant. I really hope it goes through, gwt is an important project.
comments and endorsements posted to the bottom of the grant application are always welcome! ;-P
--bawolff
Glamtools mailing list Glamtools@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/glamtools
On Fri, Feb 13, 2015 at 8:50 AM, Liam Wyatt liamwyatt@gmail.com wrote:
On 13 February 2015 at 16:42, Brian Wolff bawolff@gmail.com wrote:
One thing i would like to see in future gwt development is bringing it further in line with mediawiki coding conventions. There are several parts that do things differently than the normal mediawiki way, and also some parts that duplicate mediawiki code, but is subtly different (e.g. some of the sanitation code stands out in that regard). Ancedotally i believe this inconsistency with MW core code conventions is significantly reducing the amount of patches that interested unaffiliated developers would otherwise contribute. I also think that more closely following mw core conventions would make it easier for a wmf staff (or volunteer) who isnt previously familar with the code to debug critical bugs in emergency situations (e.g. things like the hhvm job runner bug).
Is it feasible for you to identify the specific parts that you're talking about and add them to either the bug is on phabricator, or simply to email them back to me and Dan? While noting that I'm not qualified enough to know, I would think that "ensure consistency with MW core code conventions" is an applicable thing to be added to very first item in this grant proposal ("Identify major bugs in current system and fix"). While these are not "bugs" per-se, if you can identify a hit-list of the places where the current system diverges from standard practice, that would be very helpful.
Fixing bugs and paying down technical debt are different thing, and the second typically takes much effort than the first (but also has higher benefits in the long term). In my experience, paying down technical debt has a high chance of not happening unless it is explicitly included in the objectives, with dedicated resources.
My top 2 for areas where GWT could benefit from following standard software engineering best practices (admittedly much of MediaWiki core doesn't do these either, especially the older parts, but IMO they are good goals for new code that's intended to be high-quality):
1. Cleaner information flow, with components only passing as much information to each other as needed, passing it explicitly, and, where a large amount of information is needed, passing it as objects with a well-defined structure. GWT currently does most of its information passing via huge unstructured arrays, which has practically the same effect as using globals: methods pass each other the full application state as an array reference, and any method can modify it in any way, which means that changes in the system can have effects on remote and not obviously connected parts, in ways that are very hard to reason about just by reading the code. Looking at a method signature gives no real information about what data the method will use itself, what data it will forward and what data it can change. This is a major pain point in code review as it means that reviewing the classes whose code has been changed is often not enough to understand the effects of the change.
2. Good unit test coverage. Nothing reduces code reviewer anxiety like a solid test suite that is green for the change. This will also become a lot easier if #1is tackled - it is easy to test methods which only depend on their inputs and do that in an obvious way.
On the subject of code review, i believe a major contributing factor to
difficulties last time, was development outside of gerrit, and waiting until the end to do code review. I strongly urge that all development be done on wmf's version control (because people watch what happens on wmf version control, and even if nobody is providing code review, they may still point out things that could be significant problems later) . I know first hand that getting code review, especially for non-wmf projects can be extremely difficult. If at all possible, i strongly suggest code review be done as incremently as possible.
I'll take any suggestions for how point might be integrated into the grant application text, but suffice to say that I agree with you, and the plan is always to try and do things 'by the book' and to 'bring people along for the ride'. I think both we (europeana) and the WMF have lessons-learned from the code-review experience last time. I'll let Dan respond with any technical aspects to this point if he wishes?
I wonder if you can make this part of the grant application as non-monetary support, ie. request the WMF to reserve code review time. The development plan outlined in the grant application comes with a significant code review burden (the usual advice in software development methodology books is that code review time be no less than 50% of coding time) and the WMF is, at this time, doing poorly at providing timely reviews for non-WMF patches (or, for that matter, WMF patches which need to be reviewed by a different team), with average review time being multiple weeks. If code reviews take more than a few days, developers need to multitask between the task they are working on right now and a bunch of older tasks which need to be updated based on reviews, and efficiency goes down. While I agree with Brian that leaving code review to the end makes the problem worse, it is bad either way, and without explicit commitment from the WMF (or someone else with code review rights) you risk significant delays in the project if reviewers have their own high-priority tasks and can't find the time.
(This assumes that you need external code review. Two Europeana devs working on the project and doing code reviews for each other could be an alternative.)
On Mon, Feb 16, 2015 at 1:18 PM, Gergo Tisza gtisza@wikimedia.org wrote:
(This assumes that you need external code review. Two Europeana devs working on the project and doing code reviews for each other could be an alternative.)
Thanks for all the smart comments in the thread so far, and big thanks to everyone who's worked on this proposal. GWT is a pretty amazing reflection of the unique value Wikimedia can provide to the cultural and educational sector already, so I am really happy this is getting additional attention.
A couple of points:
1) I agree with Gergo's point quoted above, and within the context of the current proposal, would recommend budgeting for at least a 20 hour/week developer supporting Dan with code review and integration, ideally someone with prior MW experience. This cannot entirely eliminate integration effort on the WMF side of things, but will ensure that the people who have the greatest interest in seeing the project through to completion are set up for success in doing so. With or without this, let's really carefully negotiate what exactly everyone's commitments are so we avoid a repeat of phase 1.
2) I say "within the context of the current proposal", because I would ask you to give serious consideration to the following question: Does GLAMWikiToolset need to existing within MediaWiki? When it was first developed, we didn't have OAuth (so a tool outside MediaWiki couldn't perform user actions), and our APIs were less mature. Today we have many examples of external tools that are doing amazing things. Magnus' tools have made tens of millions of edits to Wikidata. The Wiki Edu Foundation has created wizard.wikiedu.org and dashboard.wikiedu.org for managing student assignments and courses.
I know we have GWT and so it seems natural to just fix bugs and improve it. But consider the long term development velocity. GWT is used by a very small subset of Wikimedia users, it's not "core site functionality" and does, as far as I can tell (I may be missing something), not benefit dramatically from deep integration. You pay a lot of cost for this integration without necessarily getting a lot of "bang for the buck".
I would wager that if you started over with a new external tool, applying all the lessons learned so far and spending extra effort on UX, you could pretty quickly catch up with current functionality and then would move at a faster velocity from there. Consider where we want to be in 2016, 2018, 2020 -- is the strategy of maintaining a deeply integrated MediaWiki extension for this really sustainable or desirable? I think it's at least worth seriously considering the alternatives.
Erik
This is a high impact suggestion. But I do follow your reasoning. Maybe somebody from the GWT can try to discuss this question with Magnus?
Best,
Maarten
Op 19 feb. 2015, om 09:14 heeft Erik Moeller erik@wikimedia.org het volgende geschreven:
I would wager that if you started over with a new external tool, applying all the lessons learned so far and spending extra effort on UX, you could pretty quickly catch up with current functionality and then would move at a faster velocity from there. Consider where we want to be in 2016, 2018, 2020 -- is the strategy of maintaining a deeply integrated MediaWiki extension for this really sustainable or desirable? I think it's at least worth seriously considering the alternatives.
On Feb 19, 2015, at 3:14 AM, Erik Moeller erik@wikimedia.org wrote:
- I say "within the context of the current proposal", because I would ask you to give serious consideration to the following question: Does GLAMWikiToolset need to existing within MediaWiki? When it was first developed, we didn't have OAuth (so a tool outside MediaWiki couldn't perform user actions), and our APIs were less mature. Today we have many examples of external tools that are doing amazing things. Magnus' tools have made tens of millions of edits to Wikidata. The Wiki Edu Foundation has created wizard.wikiedu.org and dashboard.wikiedu.org for managing student assignments and courses.
I know we have GWT and so it seems natural to just fix bugs and improve it. But consider the long term development velocity. GWT is used by a very small subset of Wikimedia users, it's not "core site functionality" and does, as far as I can tell (I may be missing something), not benefit dramatically from deep integration. You pay a lot of cost for this integration without necessarily getting a lot of "bang for the buck".
I would wager that if you started over with a new external tool, applying all the lessons learned so far and spending extra effort on UX, you could pretty quickly catch up with current functionality and then would move at a faster velocity from there. Consider where we want to be in 2016, 2018, 2020 -- is the strategy of maintaining a deeply integrated MediaWiki extension for this really sustainable or desirable? I think it's at least worth seriously considering the alternatives.
Erik
Erik Möller VP of Product & Strategy, Wikimedia Foundation _______________________________________________ Glamtools mailing list Glamtools@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/glamtools
I'll be blunt: I will be using the toolset to upload millions of files. Taking that into consideration: what kind of marginal cost are we looking at having an external tool interfacing with the API instead of something built directly into the software? These are media files, not byte-sized edits to Wikidata. Also, how is uploading files—even large numbers of them—not a core function of a media repository?
Good point James, thanks for raising this perspective. However building within the MW environment proved to be quite a challenge for the team in the first phase. So it’s also a question about what solution is most efficient.
Best,
Maarten
Op 19 feb. 2015, om 13:10 heeft James Hare james.hare@wikidc.org het volgende geschreven:
On Feb 19, 2015, at 3:14 AM, Erik Moeller <erik@wikimedia.org mailto:erik@wikimedia.org> wrote:
- I say "within the context of the current proposal", because I would ask you to give serious consideration to the following question: Does GLAMWikiToolset need to existing within MediaWiki? When it was first developed, we didn't have OAuth (so a tool outside MediaWiki couldn't perform user actions), and our APIs were less mature. Today we have many examples of external tools that are doing amazing things. Magnus' tools have made tens of millions of edits to Wikidata. The Wiki Edu Foundation has created wizard.wikiedu.org http://wizard.wikiedu.org/ and dashboard.wikiedu.org http://dashboard.wikiedu.org/for managing student assignments and courses.
I know we have GWT and so it seems natural to just fix bugs and improve it. But consider the long term development velocity. GWT is used by a very small subset of Wikimedia users, it's not "core site functionality" and does, as far as I can tell (I may be missing something), not benefit dramatically from deep integration. You pay a lot of cost for this integration without necessarily getting a lot of "bang for the buck".
I would wager that if you started over with a new external tool, applying all the lessons learned so far and spending extra effort on UX, you could pretty quickly catch up with current functionality and then would move at a faster velocity from there. Consider where we want to be in 2016, 2018, 2020 -- is the strategy of maintaining a deeply integrated MediaWiki extension for this really sustainable or desirable? I think it's at least worth seriously considering the alternatives.
Erik
Erik Möller VP of Product & Strategy, Wikimedia Foundation _______________________________________________ Glamtools mailing list Glamtools@lists.wikimedia.org mailto:Glamtools@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/glamtools https://lists.wikimedia.org/mailman/listinfo/glamtools
I'll be blunt: I will be using the toolset to upload millions of files. Taking that into consideration: what kind of marginal cost are we looking at having an external tool interfacing with the API instead of something built directly into the software? These are media files, not byte-sized edits to Wikidata. Also, how is uploading files—even large numbers of them—not a core function of a media repository? _______________________________________________ Glamtools mailing list Glamtools@lists.wikimedia.org mailto:Glamtools@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/glamtools https://lists.wikimedia.org/mailman/listinfo/glamtools
James Hare wrote:
I'll be blunt: I will be using the toolset to upload millions of files. Taking that into consideration: what kind of marginal cost are we looking at having an external tool interfacing with the API instead of something built directly into the software? These are media files, not byte-sized edits to Wikidata. Also, how is uploading files—even large numbers of them—not a core function of a media repository?
Hi.
I personally think we must give a lot more thought to the broad strategy being used here. For example, if you want to upload millions of files, why not put them on a hard drive and ship them? It will be dramatically faster and a lot less wasteful of bandwidth and time. In my opinion, we need to figure out what the actual scope of this tool is and then build around that, recognizing that the scope probably doesn't (or shouldn't) include the ability to import a nation's archives into Wikimedia Commons.
There's also a larger conversation that needs to happen about whether Wikimedia Commons is ready to accept such large media donations. Yes, I realize that people have been bulk-uploading to Commons for years now, but that doesn't mean that this is acceptable nor sustainable, it's just currently tolerated. There are both technical costs (disk space comes to mind) and social costs (flooding a wiki with content that can't be reviewed or curated in a timely manner) to understand and account for.
MZMcBride
On 19 February 2015 at 14:37, MZMcBride z@mzmcbride.com wrote:
I personally think we must give a lot more thought to the broad strategy being used here. For example, if you want to upload millions of files, why not put them on a hard drive and ship them? It will be dramatically faster and a lot less wasteful of bandwidth and time. In my opinion, we need to
Agree 100%, but there is no working procedure for doing this. Unfortunately when I attempted to get an important upload of 100,000 image on to Commons this way, timed to be just before last year's Wikimania in London, I failed dismally. The first disk was "lost" at the goods-in stage, the second disk was never acted on. After 3 months of going back and forth, I gave up and just uploaded the 300GB of files over my home broadband connection (which took around another 10 weeks, relied on my home-cooked upload scripts and made it darn hard to watch any on-line video streaming at home!)
The results are great[1] however by the end of the process, I felt like a schoolboy being patted on the head and given platitudes but otherwise ignored. One of our most important GLAM partners is no doubt carefully considering the risks of working in the future with the somewhat unreliable Wikimedia community.
P.S. The GWT does not do direct disk uploads, it is one of the things that needs a better solution and has been discussed several times, however my understanding is that it is not part of the current proposed project and most often we suggest uploading to an intermediary site or server without recommending any particular workflow.
Links: 1. http://blog.wikimedia.org/2015/01/20/wellcome-library-donation/
On Thu, Feb 19, 2015 at 4:10 AM, James Hare james.hare@wikidc.org wrote:
These are media files, not byte-sized edits to Wikidata.
Magnus' OAuth uploader has been used to upload >600K files to Commons, nothing to sneeze at (GWT's count is at 375K): https://tools.wmflabs.org/magnustools/oauth_uploader.php
User:Prolineserver's video conversion tool offers large file uploads and conversion with 7 day expiry: https://tools.wmflabs.org/videoconvert/
OAuth-created contributions can be tracked: https://commons.wikimedia.org/w/index.php?title=Special:RecentChanges&ta...
There's nothing that precludes external tools (web-based or not) from dealing with large sets of files and uploading them via the API. If we wanted such stashing and use Labs we'd need to clear the storage requirements with the Labs team, but it's certainly not an automatic blocker.
There are also examples of tools that are dealing with the kind of complex metadata issues that GWT is dealing with, such as the Remixing Archival Metadata Project: https://tools.wmflabs.org/ramp/
Consider also how easy it is for others to contribute to the GWT. GWT right now is basically Dan's baby -- and almost necessarily so, because the intersection of "know how to write a MediaWiki extension" and "can figure out the complex GLAM metadata problems GWT solves for" is pretty small. If you can reduce the level of deep MW experience required for development, you may have a better chance of the project becoming self-sustaining in the long run, with active participation by GLAMs in Europeana's network.
Erik
I say "within the context of the current proposal", because I would ask you to give serious consideration to the following question: Does GLAMWikiToolset need to existing within MediaWiki? When it was first developed, we didn't have OAuth (so a tool outside MediaWiki couldn't perform user actions), and our APIs were less mature. Today we have many examples of external tools that are doing amazing things.
I disagree that OAuth really represents a significant difference in the landscape. Building it as a bot/separate tool was always an option. I doubt having the uploader field being some user, vs "glam upload service" really makes a difference. From what I've seen, there's even been several instances with gwtoolset where the uploader is some random user not associated with the museum in question. There may still be arguments for an external tool, and I know that originally several people felt it should have been an external tool (well others disagreed), but I don't think there are any new arguments.
As for our apis. With the exception of oAuth, nothing has changed on that front that's relavent. The last big change to the upload API was 2012, and for upload by url (which such a tool would likely use, that hasn't changed much since 2010.
I'll be blunt: I will be using the toolset to upload millions of files. Taking that into >consideration: what kind of marginal cost are we looking at having an external tool >interfacing with the API instead of something built directly into the software? These are >media files, not byte-sized edits to Wikidata. Also, how is uploading files—even large >numbers of them—not a core function of a media repository?
Marginal cost isn't really different (If one uses the upload by url api, its doing almost essentially the same thing). If you want to get marginal cost down further, you'd have to do it like MZMcBride said, and send in a hard disk. That's quite a bit quicker, but has a higher cost in preparing the hard disk properly (Needs to have all the information in a very specific format). Also requires time on the part of wmf staff to plug the disk in somewhere (I imagine that that works for one off requests, but would be tiresome if it happened all the time). Most people who have gone this route have had success (Fae being the exception, which from what I've read of the situation, was really beyond WMF's control).
Consider also how easy it is for others to contribute to the GWT. GWT right now is basically Dan's baby -- and almost necessarily so, because the intersection of "know how to write a MediaWiki extension" and "can figure out the complex GLAM metadata problems GWT solves for" is pretty small. If you can reduce the level of deep MW experience required for development, you may have a better chance of the project becoming self-sustaining in the long run, with active participation by GLAMs in Europeana's network.
OTOH, typically external tools are also a single person's baby, and the exposure of being integrated into MW may help with attracting people.
However I agree, it would probably be a good idea for the reasons why gwtoolset wants to be directly integrated to be explicitly enumerated.
--bawolff
for me, i’m definitely not attached to the project codebase. my main concern is that a tool is in place that achieves the goals thus far achieved, those remaining open, and any future requirements that may arise.
there were reason’s why we didn’t develop the tool externally at first, and reasons why it was developed in the way it was, but looking at the bigger picture, i think erik’s suggestion for developing the tool externally is the most encouraging. i also agree that if the tool is built externally it should not be hosted, in a production state, in labs.
with kind regards, dan
On Thu, Feb 19, 2015 at 7:26 PM, Brian Wolff bawolff@gmail.com wrote:
I say "within the context of the current proposal", because I would ask you to give serious consideration to the following question: Does GLAMWikiToolset need to existing within MediaWiki? When it was first developed, we didn't have OAuth (so a tool outside MediaWiki couldn't perform user actions), and our APIs were less mature. Today we have many examples of external tools that are doing amazing things.
I disagree that OAuth really represents a significant difference in the landscape. Building it as a bot/separate tool was always an option. I doubt having the uploader field being some user, vs "glam upload service" really makes a difference. From what I've seen, there's even been several instances with gwtoolset where the uploader is some random user not associated with the museum in question. There may still be arguments for an external tool, and I know that originally several people felt it should have been an external tool (well others disagreed), but I don't think there are any new arguments.
As for our apis. With the exception of oAuth, nothing has changed on that front that's relavent. The last big change to the upload API was 2012, and for upload by url (which such a tool would likely use, that hasn't changed much since 2010.
I'll be blunt: I will be using the toolset to upload millions of files. Taking that into >consideration: what kind of marginal cost are we looking at having an external tool >interfacing with the API instead of something built directly into the software? These are >media files, not byte-sized edits to Wikidata. Also, how is uploading files—even large >numbers of them—not a core function of a media repository?
Marginal cost isn't really different (If one uses the upload by url api, its doing almost essentially the same thing). If you want to get marginal cost down further, you'd have to do it like MZMcBride said, and send in a hard disk. That's quite a bit quicker, but has a higher cost in preparing the hard disk properly (Needs to have all the information in a very specific format). Also requires time on the part of wmf staff to plug the disk in somewhere (I imagine that that works for one off requests, but would be tiresome if it happened all the time). Most people who have gone this route have had success (Fae being the exception, which from what I've read of the situation, was really beyond WMF's control).
Consider also how easy it is for others to contribute to the GWT. GWT right now is basically Dan's baby -- and almost necessarily so, because the intersection of "know how to write a MediaWiki extension" and "can figure out the complex GLAM metadata problems GWT solves for" is pretty small. If you can reduce the level of deep MW experience required for development, you may have a better chance of the project becoming self-sustaining in the long run, with active participation by GLAMs in Europeana's network.
OTOH, typically external tools are also a single person's baby, and the exposure of being integrated into MW may help with attracting people.
However I agree, it would probably be a good idea for the reasons why gwtoolset wants to be directly integrated to be explicitly enumerated.
--bawolff
Glamtools mailing list Glamtools@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/glamtools
Just a quick note on behalf of Europeana to acknowledge that this suggestion has been received, and that we're actively discussing its implications. We'll respond as soon as we can. Thanks,
-Liam
On 19 February 2015 at 09:14, Erik Moeller erik@wikimedia.org wrote:
On Mon, Feb 16, 2015 at 1:18 PM, Gergo Tisza gtisza@wikimedia.org wrote:
(This assumes that you need external code review. Two Europeana devs working on the project and doing code reviews for each other could be an alternative.)
Thanks for all the smart comments in the thread so far, and big thanks to everyone who's worked on this proposal. GWT is a pretty amazing reflection of the unique value Wikimedia can provide to the cultural and educational sector already, so I am really happy this is getting additional attention.
A couple of points:
- I agree with Gergo's point quoted above, and within the context of the
current proposal, would recommend budgeting for at least a 20 hour/week developer supporting Dan with code review and integration, ideally someone with prior MW experience. This cannot entirely eliminate integration effort on the WMF side of things, but will ensure that the people who have the greatest interest in seeing the project through to completion are set up for success in doing so. With or without this, let's really carefully negotiate what exactly everyone's commitments are so we avoid a repeat of phase 1.
- I say "within the context of the current proposal", because I would ask
you to give serious consideration to the following question: Does GLAMWikiToolset need to existing within MediaWiki? When it was first developed, we didn't have OAuth (so a tool outside MediaWiki couldn't perform user actions), and our APIs were less mature. Today we have many examples of external tools that are doing amazing things. Magnus' tools have made tens of millions of edits to Wikidata. The Wiki Edu Foundation has created wizard.wikiedu.org and dashboard.wikiedu.org for managing student assignments and courses.
I know we have GWT and so it seems natural to just fix bugs and improve it. But consider the long term development velocity. GWT is used by a very small subset of Wikimedia users, it's not "core site functionality" and does, as far as I can tell (I may be missing something), not benefit dramatically from deep integration. You pay a lot of cost for this integration without necessarily getting a lot of "bang for the buck".
I would wager that if you started over with a new external tool, applying all the lessons learned so far and spending extra effort on UX, you could pretty quickly catch up with current functionality and then would move at a faster velocity from there. Consider where we want to be in 2016, 2018, 2020 -- is the strategy of maintaining a deeply integrated MediaWiki extension for this really sustainable or desirable? I think it's at least worth seriously considering the alternatives.
Erik
Erik Möller VP of Product & Strategy, Wikimedia Foundation
Glamtools mailing list Glamtools@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/glamtools
Hi Erik,
Responding to your points.
Erik Moeller schreef op 19-2-2015 om 9:14:
- I agree with Gergo's point quoted above, and within the context of
the current proposal, would recommend budgeting for at least a 20 hour/week developer supporting Dan with code review and integration, ideally someone with prior MW experience. This cannot entirely eliminate integration effort on the WMF side of things, but will ensure that the people who have the greatest interest in seeing the project through to completion are set up for success in doing so. With or without this, let's really carefully negotiate what exactly everyone's commitments are so we avoid a repeat of phase 1.
Finding such a developer is a big challenge. Probably nearly impossible. Any good MediaWiki dev seems to already have a real job or is hired by the Wikimedia movement already.
- I say "within the context of the current proposal", because I would
ask you to give serious consideration to the following question: Does GLAMWikiToolset need to existing within MediaWiki? When it was first developed, we didn't have OAuth (so a tool outside MediaWiki couldn't perform user actions), and our APIs were less mature. Today we have many examples of external tools that are doing amazing things. Magnus' tools have made tens of millions of edits to Wikidata. The Wiki Edu Foundation has created wizard.wikiedu.org http://wizard.wikiedu.org and dashboard.wikiedu.org http://dashboard.wikiedu.org for managing student assignments and courses.
No, it doesn't need to exist with MediaWiki. We wanted to create a tool that ran on a production grade platform with production grade support. The only organization offering that is the Wikimedia Foundation. The only production grade platform operated by the WMF suitable for this application is based on MediaWiki. That's the reason why it was build as a MediaWiki extention. What other production grade platform could we use?
Maarten
Ps. I'll ridicule anyone who says Wikimedia Labs is a suitable given the production grade requirement.
On Thu, Feb 19, 2015 at 10:18 AM, Maarten Dammers maarten@mdammers.nl wrote:
No, it doesn't need to exist with MediaWiki. We wanted to create a tool that ran on a production grade platform with production grade support.
What level of downtime per month / per year do you consider acceptable for this type of tool?
Thanks,
Erik
in regards to codebase changes, as liam mentioned, a list of specific items to address would be very helpful. i’m still not 100% familiar with the mediawiki framework and the wiki way of developing, so any help that guides me or any developer through it is very much appreciated. gergo’s comments have broader reach and probably relate more to my own lack of knowledge regarding to standard software engineering best practices. his comments are clear enough that i could move forward, but i would want to be able to confirm with him my approach to making the changes in order to verify that those changes meet the requirements he’s mentioning. overall, i think we could make changes to the grant to reflect both brian’s and gergo’s comments, but i think we may want to wait until we’ve decided in which direction to move forward.
code review was always a very difficult hurdle to overcome. personally, i’m concerned about this thought that i left code review to the end; that is just simply not the case. initially i tried several times to “properly” place code in gerrit and to understand the process, but the comments i got were scarce and not helping me out. the gerrit workflow did finally gain progress, but only once we were able to pull a few foundation devs away from their regular projects. since that time i have always developed gwtoolset within the gerrit workflow. i think that gergo’s point about relying on foundation code review and erik’s suggestion regarding a second mw dev are the most important. i believe we do have such a dev considering working on this project, but i’m not sure of the details.
in regards to why other developers have not picked up the torch to help out, i personally don’t think it has to do with the gwtoolset codebase itself. i believe that if someone really wants to help out they will. i know that both david and i tried several times to find other developers, but during the entire project, except for one python programmer that expressed interest, and even before then, neither david and i, nor the steering committee were able to find anyone else who was interested in programming for the project and that continues to be a problem, yet i don’t think that means we should give up on it.
o dan
On Sun, Feb 22, 2015 at 3:15 AM, Erik Moeller erik@wikimedia.org wrote:
On Thu, Feb 19, 2015 at 10:18 AM, Maarten Dammers maarten@mdammers.nl wrote:
No, it doesn't need to exist with MediaWiki. We wanted to create a tool that ran on a production grade platform with production grade support.
What level of downtime per month / per year do you consider acceptable for this type of tool?
Thanks,
Erik
-- Erik Möller VP of Product & Strategy, Wikimedia Foundation
Glamtools mailing list Glamtools@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/glamtools
That phrase came from the fact that the Upload Wizard and the GWT were the only things listed in the "integrated" section of the page [[Commons:Upload Tools]]. However, I see that as of a couple of weeks ago someone's added a new item there too: https://commons.wikimedia.org/w/index.php?title=Commons:Upload_tools&dif...
I'll take your recommendation for how to better phrase this sentence to be both factually accurate, but also to try to emphasise that this tool is trying its best to be "of the system" rather than sitting outside of it.
Hmm, I guess that list is considering user scripts as integrated, which is arguable, but different from my definition (And I think different from what you were trying to get at).
I imagine commons doesn't include the mobile upload feature, as by and large they seem to hate it ;). Perhaps special:Upload isn't included because its seen less as an upload tool, and more as the base system.
Anyways, for the grant proposal, I would maybe use language like:
"It is the only “integrated upload tool” for Wikimedia commons allowing multiple files to be uploaded at once (aside from the WMF-developed Upload Wizard)"
as that makes it technically true, and also emphasizes the multiple file nature which is pretty key to gwtoolset.
--bawolff
On Thursday, February 12, 2015 at 9:11 AM, Liam Wyatt wrote:
Dear GLAMWiki-verse,
[tl;dr Seeking feedback. Please read the GWT 2.0 grant application draft before I submit it officially: https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset ]
Thank you very much for taking ownership of GLAMwiki Toolset—a huge project that the WMF is not interested in and no one else really has the resources for.
I consider this project central to Wikimedia DC’s strategic priorities and consider its funding to be essential. I am critical of a couple of points: * On the must/should/could/won’t scale, I would consider “improving documentation” to be a lot higher than simply “could.” A regular criticism I hear of the toolset is that the documentation is lousy, and if your goal is to encourage more people to use it without expert help, you need better documentation. I would consider elevating the priority for this. * You will get dinged on your labor rates. Is there a particular reason why it costs $100/hour to build an upload tool, a piece of software that is not terribly specialized—even when considering EU labor rates? This isn’t really a concern for me or Wikimedia DC since we’re not paying for it, but I would be prepared to discuss this should it come up.
Those points aside, I really look forward to seeing the improvements that come out of this project. Best of luck to you and Europeana!
Cheers, James
— James Hare President, Wikimedia DC http://wikimediadc.org @wikimediadc
On 13 February 2015 at 23:54, James Hare james.hare@wikidc.org wrote:
On Thursday, February 12, 2015 at 9:11 AM, Liam Wyatt wrote:
Dear GLAMWiki-verse,
[tl;dr Seeking feedback. Please read the GWT 2.0 grant application draft before I submit it officially: https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset ]
Thank you very much for taking ownership of GLAMwiki Toolset—a huge project that the WMF is not interested in and no one else really has the resources for.
I consider this project central to Wikimedia DC’s strategic priorities and consider its funding to be essential. I am critical of a couple of points:
- On the must/should/could/won’t scale, I would consider “improving
documentation” to be a *lot* higher than simply “could.” A regular criticism I hear of the toolset is that the documentation is lousy, and if your goal is to encourage more people to use it without expert help, you *need* better documentation. I would consider elevating the priority for this.
- You will get dinged on your labor rates. Is there a particular reason
why it costs $100/hour to build an upload tool, a piece of software that is not terribly specialized—even when considering EU labor rates? This isn’t really a concern for me or Wikimedia DC since we’re not paying for it, but I would be prepared to discuss this should it come up.
Those points aside, I really look forward to seeing the improvements that come out of this project. Best of luck to you and Europeana!
Cheers, James
— James Hare President, Wikimedia DC http://wikimediadc.org @wikimediadc
Thanks for the endorsement James :-)
[hint, hint... https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset#Endors... ]
I'm going to respond to the financial question right away because a couple of other people have asked me similar things offlist. It's a valid topic to raise so I might as well publicly reply-all :-) I'll paraphrase this issue as two questions:
Question 1: How do you justify the labor rates? Answer: To try to answer that, I have added a new paragraph in the grant application earlier today:
"The daily-rate of the different roles listed here is calculated based on the average hourly salary of the relevant *Europeana* employees. As a project-based organisation that frequently accepts funding tied to specific objectives, *Europeana* frequently operates by accounting for its staff obligations in 1 hour blocks. Therefore, the costs listed here are not invented specifically for a grant application, they are simply the normal hourly salary of that person *x* 8 hours = 1 daily rate." https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset#Total_...
I hope this answers the question sufficiently, although I do understand that it is not the detailed "justification" that you might like. In short, this is just how much the relevant staff cost.
Question 2: Could it be done cheaper elsewhere? Answer: Yeah - probably. No one is pretending that Europeana is the cheapest organisation for building software - especially since it does not normally build software for external organisation - but they're also the only ones with the technical ability, organisational willingness and availability right now. Some others might have the capacity, or the willingness, or the availability - but not all three.
As with the first question, I hope that this explains sufficiently. It is a fair question, and I've tried to give it as straightforward an answer as I can.
p.s. I may not be able to respond quickly to any further questions for the next couple of days (it being the weekend and all).
-Liam / Wittylama In my capacity as "*that Europeana guy*"
I would add that some things could be done by volunteers such as the user group; That includes improving the documentation, but not the core coding.
There is also the issue that software developers are not interchangeable. If an alternative bid for this were to come from a part of the world where programmers are paid much less, I would expect to see tooling up costs to get programmers up to speed with things like media wiki software. Also it is worth thinking of this sort of development as a bridge between the GLAM sector and Wikimedia, building it from one side or the other gives you the advantage of understanding the side you are working from, building it using an organisation that is new to both sides would I predict take longer as the developers would need to learn about both the GLAM sector and Wikimedia's quirks.
Jonathan Cardy GLAM (Galleries, Libraries, Archives & Museums) Organiser Wikimedia UK 020 7065 0921
(I'm normally in the office Tuesday's, Wednesdays and Fridays - Emails on Mondays and Thursdays wont usually be seen till the next day)
Wikimedia UK is a Company Limited by Guarantee registered in England and Wales, Registered No. 6741827. Registered Charity No.1144513. Registered Office 4th Floor, Development House, 56-64 Leonard Street, London EC2A 4LT. United Kingdom. Wikimedia UK is the UK chapter of a global Wikimedia movement. The Wikimedia projects are run by the Wikimedia Foundation (who operate Wikipedia, amongst other projects).
Wikimedia UK is an independent non-profit charity with no legal control over Wikipedia nor responsibility for its contents.
On 14 February 2015 at 00:11, Liam Wyatt liamwyatt@gmail.com wrote:
On 13 February 2015 at 23:54, James Hare james.hare@wikidc.org wrote:
On Thursday, February 12, 2015 at 9:11 AM, Liam Wyatt wrote:
Dear GLAMWiki-verse,
[tl;dr Seeking feedback. Please read the GWT 2.0 grant application draft before I submit it officially: https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset ]
Thank you very much for taking ownership of GLAMwiki Toolset—a huge project that the WMF is not interested in and no one else really has the resources for.
I consider this project central to Wikimedia DC’s strategic priorities and consider its funding to be essential. I am critical of a couple of points:
- On the must/should/could/won’t scale, I would consider “improving
documentation” to be a *lot* higher than simply “could.” A regular criticism I hear of the toolset is that the documentation is lousy, and if your goal is to encourage more people to use it without expert help, you *need* better documentation. I would consider elevating the priority for this.
- You will get dinged on your labor rates. Is there a particular reason
why it costs $100/hour to build an upload tool, a piece of software that is not terribly specialized—even when considering EU labor rates? This isn’t really a concern for me or Wikimedia DC since we’re not paying for it, but I would be prepared to discuss this should it come up.
Those points aside, I really look forward to seeing the improvements that come out of this project. Best of luck to you and Europeana!
Cheers, James
— James Hare President, Wikimedia DC http://wikimediadc.org @wikimediadc
Thanks for the endorsement James :-)
[hint, hint... https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset#Endors... ]
I'm going to respond to the financial question right away because a couple of other people have asked me similar things offlist. It's a valid topic to raise so I might as well publicly reply-all :-) I'll paraphrase this issue as two questions:
Question 1: How do you justify the labor rates? Answer: To try to answer that, I have added a new paragraph in the grant application earlier today:
"The daily-rate of the different roles listed here is calculated based on the average hourly salary of the relevant *Europeana* employees. As a project-based organisation that frequently accepts funding tied to specific objectives, *Europeana* frequently operates by accounting for its staff obligations in 1 hour blocks. Therefore, the costs listed here are not invented specifically for a grant application, they are simply the normal hourly salary of that person *x* 8 hours = 1 daily rate." https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset#Total_...
I hope this answers the question sufficiently, although I do understand that it is not the detailed "justification" that you might like. In short, this is just how much the relevant staff cost.
Question 2: Could it be done cheaper elsewhere? Answer: Yeah - probably. No one is pretending that Europeana is the cheapest organisation for building software - especially since it does not normally build software for external organisation - but they're also the only ones with the technical ability, organisational willingness and availability right now. Some others might have the capacity, or the willingness, or the availability - but not all three.
As with the first question, I hope that this explains sufficiently. It is a fair question, and I've tried to give it as straightforward an answer as I can.
p.s. I may not be able to respond quickly to any further questions for the next couple of days (it being the weekend and all).
-Liam / Wittylama In my capacity as "*that Europeana guy*"
Europeana-steering-group mailing list Europeana-steering-group@lists.wmnederland.nl
https://lists.wmnederland.nl/cgi-bin/mailman/listinfo/europeana-steering-gro...
Dear Cultural Partners, GLAMtools, and GWT v.1 steering group,
Following feedback from WMF executive[1] on the GLAMwikiToolset v.2 grant application, it has become clear that the grant’s request for the WMF to reserve code review time cannot be provided as non-monetary support. Nor is the GLAMwikiToolset considered “core site functionality”[1], which would warrant product ownership by the WMF in the long term. Both of these were key points listed in the “non-financial requirements” section of the grant request.[2]
To include these tasks as an enumerated costs in the grant would push the budget of the grant above the WMF's maximum allowable grant amount (an extra 20 hours a week was suggested for code review [1]; no amount of time was suggested for the other items mentioned in the non-financial requirements). Conversely, to reduce the scope of the grant in order to include these costs would reduce the expected benefit below what Europeana sees as helpful to its own non-profit mission. Therefore, I have confirmation from Europeana executive to say that we will be formally withdrawing the grant request and will go back to the beginning.
Specifically, the suggestion was made to “start over with a new external tool“ because the service that the GWT seeks to provide does “...not benefit dramatically from deep integration” in the WMF’s view.[1] We agree that there are significant advantages, both technical and organisational, to this “going independent” approach which make us quite enthusiastic about it.
Re-writing the GWT from scratch as an independent software project (rather than a mediawiki extension) is made possible by new functions (especially OAuth) which didn’t exist when we first started this project (as also mentioned in [1]). This should increase the speed and flexibility of development and hopefully lower the threshold of development knowledge so that developers with little MediaWiki framework knowledge can participate in the project. This should also allow us to attain feature-parity with the current system relatively quickly.
Therefore, soon we will be submitting a new grant proposal to the WMF for a “proof of concept” of an independently operating replacement to the GWT. This will be our way of properly testing if all of the necessary infrastructure (e.g. the API) is in place to do what the GWT needs. If that proof of concept is successful, we will return with a full grant request.
For the moment we suspect that the proof of concept would be deployed on Wikimedia Labs but would probably need to find a more stable home for its finished state. However that, and many other issues (like how to adjust the GWT user-right system to accommodate an external uploader bot; and the question of long term support), are operational questions that would be thought through as part of the proof of concept stage.
Sincerely, Liam Wyatt, on behalf of Europeana (in particular Dan Entous, lead GWT developer, and David Haskiya, Product Manager).
[1] This email, and subsequent emails in this thread: https://lists.wikimedia.org/pipermail/glamtools/2015-February/000343.html
[2] Points number 1 (“timely code review”) and 4 (“product ownership long term”): https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset#Non-fi...
Liam --
Thank you for this. I know this wasn't an easy pivot to ask for, and I really appreciate it. It's only fair that we do something in return, so ..
- I'll help review the revised version of the proposal (feel free to poke me directly); - We'll help to give you and Dan pointers (to the right people if needed) re: the use of OAuth, existing libraries, etc. - If we get underway, we'll help think through options for a long term hosting strategy with you (in Labs or beyond).
I'm overall very supportive of the aims of the project -- I think a truly specialized tool for GLAMs would go a long way to attract them to our movement. To start with, here are a few notes & pointers:
1) I checked with our product team if they see any blockers for GWT to use OAuth. Dan Garry, who PM'd the initial OAuth implementation, didn't. These especially relevant permissions are already available:
- uploadfile, which supports only uploading new files - uploadeditmovefile, which supports uploading, editing and moving files - highvolume, which allows making a large number of requests in a short timeframe
2) There are a number of existing libraries and implementations of MW OAuth that may serve as good examples. In Python we have http://pythonhosted.org/mwoauth/ , and Magnus' tools are also a great place to look.
3) Please note that the OAuth implementation is designed for _web_ use; it won't work yet for downloadable applications. (In short, a different security model applies when you're distributing code to the end user, and that requires a bit more work on our end and a later version of the protocol.) The overall OAuth experience should be reasonable; we're working through the set of issues tracked in https://phabricator.wikimedia.org/T86869 to make it really solid.
We _will_ help you with this if you encounter issues as you go.
Erik
Dear Erik,
Thanks for that very clear steer that the WMF is willing to help others build a mass upload tool for commons.
Two small points. Whilst many potential users of a mass upload tool are GLAMs, there are other opportunities out there for mass uploads, and I think it would be helpful if we remembered that and called it a mass upload tool that the GLAM community is the strongest supporter of rather than just a GLAM tool. The ability to upload millions of files while mapping other people's metadata into ours is important because it saves on volunteer time in uploading and categorising images in other ways. Commons has huge backlogs of uncategorised or poorly categorised files, and other projects especially wikipedia have vast backlogs in adding or upgrading their imagery. There has been recent research that articles with images get more viewers, so the implications of backing this aren't just the usual one that investing in the GLAM program is an investment in quality, there is also a good chance that investing in GLAM could increase readership.
Regards
Jonathan Cardy GLAM (Galleries, Libraries, Archives & Museums) Organiser Wikimedia UK 020 7065 0921
(I'm normally in the office Tuesday's, Wednesdays and Fridays - Emails on Mondays and Thursdays wont usually be seen till the next day)
Wikimedia UK is a Company Limited by Guarantee registered in England and Wales, Registered No. 6741827. Registered Charity No.1144513. Registered Office 4th Floor, Development House, 56-64 Leonard Street, London EC2A 4LT. United Kingdom. Wikimedia UK is the UK chapter of a global Wikimedia movement. The Wikimedia projects are run by the Wikimedia Foundation (who operate Wikipedia, amongst other projects).
Wikimedia UK is an independent non-profit charity with no legal control over Wikipedia nor responsibility for its contents.
On 26 February 2015 at 06:57, Erik Moeller erik@wikimedia.org wrote:
Liam --
Thank you for this. I know this wasn't an easy pivot to ask for, and I really appreciate it. It's only fair that we do something in return, so ..
- I'll help review the revised version of the proposal (feel free to
poke me directly);
- We'll help to give you and Dan pointers (to the right people if
needed) re: the use of OAuth, existing libraries, etc.
- If we get underway, we'll help think through options for a long term
hosting strategy with you (in Labs or beyond).
I'm overall very supportive of the aims of the project -- I think a truly specialized tool for GLAMs would go a long way to attract them to our movement. To start with, here are a few notes & pointers:
- I checked with our product team if they see any blockers for GWT to
use OAuth. Dan Garry, who PM'd the initial OAuth implementation, didn't. These especially relevant permissions are already available:
- uploadfile, which supports only uploading new files
- uploadeditmovefile, which supports uploading, editing and moving files
- highvolume, which allows making a large number of requests in a
short timeframe
- There are a number of existing libraries and implementations of MW
OAuth that may serve as good examples. In Python we have http://pythonhosted.org/mwoauth/ , and Magnus' tools are also a great place to look.
- Please note that the OAuth implementation is designed for _web_
use; it won't work yet for downloadable applications. (In short, a different security model applies when you're distributing code to the end user, and that requires a bit more work on our end and a later version of the protocol.) The overall OAuth experience should be reasonable; we're working through the set of issues tracked in https://phabricator.wikimedia.org/T86869 to make it really solid.
We _will_ help you with this if you encounter issues as you go.
Erik
-- Erik Möller VP of Product & Strategy, Wikimedia Foundation _______________________________________________ Europeana-steering-group mailing list Europeana-steering-group@lists.wmnederland.nl
https://lists.wmnederland.nl/cgi-bin/mailman/listinfo/europeana-steering-gro...
As a relevant update -- "During a planning sprint in Seattle, Sage and LiAnna laid the foundation for digital infrastructure improvements. Supported by an additional $300K grant provided by the Stanton Foundation, Wiki Ed will transform its Assignment Design Wizard and Dashboard into a fully functional course management system. This will be independent of the EducationProgram MediaWiki extension that we’ve used for course pages until now, and will let us move past current limitations of Wikipedia’s course pages."
http://wikiedu.org/blog/2015/03/23/2-2015-mr/
This is a real world example of an affiliate organization taking what's currently a integrated but poorly maintained MediaWiki extension and committing to re-thinking it as a stand-alone tool that uses our APIs. I'd love to see something similar happen with GWT.
Erik
Dear GLAMwikimedians, particularly those involved in the GLAMWiki Toolset,
Europeana has decided not to further invest in a new version of the GLAM-WIKI toolset (GWT). This includes developing a “version 2” of the current system as was proposed in the grant application [1], or as a standalone system using O-Auth as proposed by Erik earlier in this thread.
We have come to this conclusion after a lot of internal discussions, and talking with key stakeholders, notably by presenting several options at the European GLAMwiki Coordinators meeting. We have received an overwhelming consensus that investing our energy in other forms of co-operation was the best approach.[2] This is not a repudiation of the GWT in theory or in practice - the GWT is not going away and we are very supportive and proud of it.
As previously mentioned, it had become clear that the “version 2” grant request would never be approved on the basis that the WMF would not allocate code review nor would it support the project once build.[3] Redeveloping as an independent “stand alone” tool would solve the first problem, but conversely it would exacerbate the second problem.
It is universally agreed that moving to an independent tool would make for a more flexible, powerful and faster-to-develop system. However, consistent feedback from stakeholder consultation was that there is no demonstrable external investment in supporting a third-party mediawiki development ecosystem. The primary encouragement we received rebuild using O-Auth and the API was not for the tool itself, but rather as a test case for those systems. Europeana has therefore decided that, in the current environment, this would not be the best use of our resources.
Instead, we intend to focus our GLAMwiki resources in community event/activity support, rather than into software development. In particular, we are very interesting in “pivoting” our attentions towards Wikidata and the potential that holds for supporting digital cultural heritage.[4]
That said - this does NOT mean that the GWT as it currently stands is going away. It is, and will remain the most stable and integrated way to provide mass-uploads to Wikimedia Commons, probably for many years to come. We are aware that many Wikimedia Chapters have written GWT into their annual plans, and we intend to continue supporting training and related events. Also, given that there will not be an “all new” version of the GWT, we are also investigating asking for funding to close the major bugs that we had already triaged.[5]
Finally, we also hope to make greater use of the GWT for ourselves. There is a large and increasing amount of content on Europeana that has an open license and has quality metadata and image quality for it to be uploaded to Wikimedia. We hope to start sharing this material directly with Commons soon. Making the uploads ourselves would have the advantage for our many partner GLAMs to be able jump straight into the fun part of working with their local wikimedia community on editathons etc., rather than their time wrangling metadata.
Liam/Wittylama
In my capacity as Europeana GLAMwiki Coordinator (and the guy who gut Europeana into this project 4 years ago…)
p.s. In response to Erik’s comparison of the GWT to the WikiEd Foundation redeveloping the Education Extension as an independent tool [6]. The significant difference is that the WikiEdu Foundation’s organisational purpose is to run activities and maintain software for Wikipedia outreach to the education sector. Therefore the long term ‘ownership’ of software built for this goal, and of which they will be the primary user, is well within their mission. Neither was ever intended to be the case case for Europeana and the GWT.
[1] https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset
[2] Minutes of that section of the discussion are here: https://meta.wikimedia.org/wiki/2015_European_GLAMwiki_Coordinators_meeting#...
[3] https://lists.wikimedia.org/pipermail/glamtools/2015-February/000356.html
[4] In particular the “sum of all paintings” project https://www.wikidata.org/wiki/Wikidata:WikiProject_sum_of_all_paintings
[5] Which was ‘item 1’ in the grant application https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset#Identi...
[6] https://lists.wikimedia.org/pipermail/glamtools/2015-March/000365.html