On Feb 12, 2015 1:11 PM, "Liam Wyatt" <liamwyatt(a)gmail.com> wrote: draft before I submit it officially: https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset ] on building a major grant application to the Wikimedia Foundation to continue development of the GLAMwiki Toolset. shown that there is widespread interest in (and usage of) the tool. It's already become part of the "standard" things that we use when talking to potential partners. However, that there are lots of things that can still be improved in it. LOT of negotiation, behind the scenes that goes in to getting the document supported by all the relevant stakeholders, and this is no exception. [Heck - I chaired the first formal meeting about this project during Wikimania Haifa!!] grant application is almost ready to be submitted officially. construction or have been part of the consultation process about what gets prioritised and what doesn't. Now I would really appreciate if you could have another look at the [hopefully] finished application. This is the chance to identify any errors, missing elements, excellent examples I've forgotten, logical flaws, spelling mistakes... the bottom. talkpage, email, morse-code, carrier-pigeon etc...) switch" and formally submit the grant application. team to submit this application at this time, even thought it is during the 'inspire grants' campaign, because it was already so far down the track of negotiation with various stakeholders. First of all a nitpick: " It is the only “integrated upload tool” for Wikimedia commons (aside from the WMF-developed Upload Wizard)" Technically you also have the original special:upload, and also that mobile thing thats less popular. --- One thing i would like to see in future gwt development is bringing it further in line with mediawiki coding conventions. There are several parts that do things differently than the normal mediawiki way, and also some parts that duplicate mediawiki code, but is subtly different (e.g. some of the sanitation code stands out in that regard). Ancedotally i believe this inconsistency with MW core code conventions is significantly reducing the amount of patches that interested unaffiliated developers would otherwise contribute. I also think that more closely following mw core conventions would make it easier for a wmf staff (or volunteer) who isnt previously familar with the code to debug critical bugs in emergency situations (e.g. things like the hhvm job runner bug). --- On the subject of code review, i believe a major contributing factor to difficulties last time, was development outside of gerrit, and waiting until the end to do code review. I strongly urge that all development be done on wmf's version control (because people watch what happens on wmf version control, and even if nobody is providing code review, they may still point out things that could be significant problems later) . I know first hand that getting code review, especially for non-wmf projects can be extremely difficult. If at all possible, i strongly suggest code review be done as incremently as possible. Good luck with the grant. I really hope it goes through, gwt is an important project. --bawolff

On Thursday, February 12, 2015 at 9:11 AM, Liam Wyatt wrote: Thank you very much for taking ownership of GLAMwiki Toolset—a huge project that the WMF is not interested in and no one else really has the resources for. I consider this project central to Wikimedia DC’s strategic priorities and consider its funding to be essential. I am critical of a couple of points: * On the must/should/could/won’t scale, I would consider “improving documentation” to be a lot higher than simply “could.” A regular criticism I hear of the toolset is that the documentation is lousy, and if your goal is to encourage more people to use it without expert help, you need better documentation. I would consider elevating the priority for this. * You will get dinged on your labor rates. Is there a particular reason why it costs $100/hour to build an upload tool, a piece of software that is not terribly specialized—even when considering EU labor rates? This isn’t really a concern for me or Wikimedia DC since we’re not paying for it, but I would be prepared to discuss this should it come up. Those points aside, I really look forward to seeing the improvements that come out of this project. Best of luck to you and Europeana! Cheers, James — James Hare President, Wikimedia DC http://wikimediadc.org @wikimediadc

I would add that some things could be done by volunteers such as the user group; That includes improving the documentation, but not the core coding. There is also the issue that software developers are not interchangeable. If an alternative bid for this were to come from a part of the world where programmers are paid much less, I would expect to see tooling up costs to get programmers up to speed with things like media wiki software. Also it is worth thinking of this sort of development as a bridge between the GLAM sector and Wikimedia, building it from one side or the other gives you the advantage of understanding the side you are working from, building it using an organisation that is new to both sides would I predict take longer as the developers would need to learn about both the GLAM sector and Wikimedia's quirks. Jonathan Cardy GLAM (Galleries, Libraries, Archives & Museums) Organiser Wikimedia UK 020 7065 0921 (I'm normally in the office Tuesday's, Wednesdays and Fridays - Emails on Mondays and Thursdays wont usually be seen till the next day) Wikimedia UK is a Company Limited by Guarantee registered in England and Wales, Registered No. 6741827. Registered Charity No.1144513. Registered Office 4th Floor, Development House, 56-64 Leonard Street, London EC2A 4LT. United Kingdom. Wikimedia UK is the UK chapter of a global Wikimedia movement. The Wikimedia projects are run by the Wikimedia Foundation (who operate Wikipedia, amongst other projects). Wikimedia UK is an independent non-profit charity with no legal control over Wikipedia nor responsibility for its contents. On 14 February 2015 at 00:11, Liam Wyatt <liamwyatt(a)gmail.com> wrote:

Hmm, I guess that list is considering user scripts as integrated, which is arguable, but different from my definition (And I think different from what you were trying to get at). I imagine commons doesn't include the mobile upload feature, as by and large they seem to hate it ;). Perhaps special:Upload isn't included because its seen less as an upload tool, and more as the base system. Anyways, for the grant proposal, I would maybe use language like: "It is the only “integrated upload tool” for Wikimedia commons allowing multiple files to be uploaded at once (aside from the WMF-developed Upload Wizard)" as that makes it technically true, and also emphasizes the multiple file nature which is pretty key to gwtoolset. --bawolff

This is a high impact suggestion. But I do follow your reasoning. Maybe somebody from the GWT can try to discuss this question with Magnus? Best, Maarten

Good point James, thanks for raising this perspective. However building within the MW environment proved to be quite a challenge for the team in the first phase. So it’s also a question about what solution is most efficient. Best, Maarten

On 19 February 2015 at 14:37, MZMcBride <z(a)mzmcbride.com> wrote: Agree 100%, but there is no working procedure for doing this. Unfortunately when I attempted to get an important upload of 100,000 image on to Commons this way, timed to be just before last year's Wikimania in London, I failed dismally. The first disk was "lost" at the goods-in stage, the second disk was never acted on. After 3 months of going back and forth, I gave up and just uploaded the 300GB of files over my home broadband connection (which took around another 10 weeks, relied on my home-cooked upload scripts and made it darn hard to watch any on-line video streaming at home!) The results are great[1] however by the end of the process, I felt like a schoolboy being patted on the head and given platitudes but otherwise ignored. One of our most important GLAM partners is no doubt carefully considering the risks of working in the future with the somewhat unreliable Wikimedia community. P.S. The GWT does not do direct disk uploads, it is one of the things that needs a better solution and has been discussed several times, however my understanding is that it is not part of the current proposed project and most often we suggest uploading to an intermediary site or server without recommending any particular workflow. Links: 1. http://blog.wikimedia.org/2015/01/20/wellcome-library-donation/

On Thu, Feb 19, 2015 at 4:10 AM, James Hare <james.hare(a)wikidc.org> wrote: Magnus' OAuth uploader has been used to upload >600K files to Commons, nothing to sneeze at (GWT's count is at 375K): https://tools.wmflabs.org/magnustools/oauth_uploader.php User:Prolineserver's video conversion tool offers large file uploads and conversion with 7 day expiry: https://tools.wmflabs.org/videoconvert/ OAuth-created contributions can be tracked: https://commons.wikimedia.org/w/index.php?title=Special:RecentChanges&t… There's nothing that precludes external tools (web-based or not) from dealing with large sets of files and uploading them via the API. If we wanted such stashing and use Labs we'd need to clear the storage requirements with the Labs team, but it's certainly not an automatic blocker. There are also examples of tools that are dealing with the kind of complex metadata issues that GWT is dealing with, such as the Remixing Archival Metadata Project: https://tools.wmflabs.org/ramp/ Consider also how easy it is for others to contribute to the GWT. GWT right now is basically Dan's baby -- and almost necessarily so, because the intersection of "know how to write a MediaWiki extension" and "can figure out the complex GLAM metadata problems GWT solves for" is pretty small. If you can reduce the level of deep MW experience required for development, you may have a better chance of the project becoming self-sustaining in the long run, with active participation by GLAMs in Europeana's network. Erik -- Erik Möller VP of Product & Strategy, Wikimedia Foundation

I disagree that OAuth really represents a significant difference in the landscape. Building it as a bot/separate tool was always an option. I doubt having the uploader field being some user, vs "glam upload service" really makes a difference. From what I've seen, there's even been several instances with gwtoolset where the uploader is some random user not associated with the museum in question. There may still be arguments for an external tool, and I know that originally several people felt it should have been an external tool (well others disagreed), but I don't think there are any new arguments. As for our apis. With the exception of oAuth, nothing has changed on that front that's relavent. The last big change to the upload API was 2012, and for upload by url (which such a tool would likely use, that hasn't changed much since 2010. Marginal cost isn't really different (If one uses the upload by url api, its doing almost essentially the same thing). If you want to get marginal cost down further, you'd have to do it like MZMcBride said, and send in a hard disk. That's quite a bit quicker, but has a higher cost in preparing the hard disk properly (Needs to have all the information in a very specific format). Also requires time on the part of wmf staff to plug the disk in somewhere (I imagine that that works for one off requests, but would be tiresome if it happened all the time). Most people who have gone this route have had success (Fae being the exception, which from what I've read of the situation, was really beyond WMF's control). OTOH, typically external tools are also a single person's baby, and the exposure of being integrated into MW may help with attracting people. However I agree, it would probably be a good idea for the reasons why gwtoolset wants to be directly integrated to be explicitly enumerated. --bawolff

On Thu, Feb 19, 2015 at 10:18 AM, Maarten Dammers <maarten(a)mdammers.nl> wrote: What level of downtime per month / per year do you consider acceptable for this type of tool? Thanks, Erik -- Erik Möller VP of Product & Strategy, Wikimedia Foundation

Dear Cultural Partners, GLAMtools, and GWT v.1 steering group, Following feedback from WMF executive[1] on the GLAMwikiToolset v.2 grant application, it has become clear that the grant’s request for the WMF to reserve code review time cannot be provided as non-monetary support. Nor is the GLAMwikiToolset considered “core site functionality”[1], which would warrant product ownership by the WMF in the long term. Both of these were key points listed in the “non-financial requirements” section of the grant request.[2] To include these tasks as an enumerated costs in the grant would push the budget of the grant above the WMF's maximum allowable grant amount (an extra 20 hours a week was suggested for code review [1]; no amount of time was suggested for the other items mentioned in the non-financial requirements). Conversely, to reduce the scope of the grant in order to include these costs would reduce the expected benefit below what Europeana sees as helpful to its own non-profit mission. Therefore, I have confirmation from Europeana executive to say that we will be formally withdrawing the grant request and will go back to the beginning. Specifically, the suggestion was made to “start over with a new external tool“ because the service that the GWT seeks to provide does “...not benefit dramatically from deep integration” in the WMF’s view.[1] We agree that there are significant advantages, both technical and organisational, to this “going independent” approach which make us quite enthusiastic about it. Re-writing the GWT from scratch as an independent software project (rather than a mediawiki extension) is made possible by new functions (especially OAuth) which didn’t exist when we first started this project (as also mentioned in [1]). This should increase the speed and flexibility of development and hopefully lower the threshold of development knowledge so that developers with little MediaWiki framework knowledge can participate in the project. This should also allow us to attain feature-parity with the current system relatively quickly. Therefore, soon we will be submitting a new grant proposal to the WMF for a “proof of concept” of an independently operating replacement to the GWT. This will be our way of properly testing if all of the necessary infrastructure (e.g. the API) is in place to do what the GWT needs. If that proof of concept is successful, we will return with a full grant request. For the moment we suspect that the proof of concept would be deployed on Wikimedia Labs but would probably need to find a more stable home for its finished state. However that, and many other issues (like how to adjust the GWT user-right system to accommodate an external uploader bot; and the question of long term support), are operational questions that would be thought through as part of the proof of concept stage. Sincerely, Liam Wyatt, on behalf of Europeana (in particular Dan Entous, lead GWT developer, and David Haskiya, Product Manager). [1] This email, and subsequent emails in this thread: https://lists.wikimedia.org/pipermail/glamtools/2015-February/000343.html [2] Points number 1 (“timely code review”) and 4 (“product ownership long term”): https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset#Non-f… -- wittylama.com Peace, love & metadata

Dear Erik, Thanks for that very clear steer that the WMF is willing to help others build a mass upload tool for commons. Two small points. Whilst many potential users of a mass upload tool are GLAMs, there are other opportunities out there for mass uploads, and I think it would be helpful if we remembered that and called it a mass upload tool that the GLAM community is the strongest supporter of rather than just a GLAM tool. The ability to upload millions of files while mapping other people's metadata into ours is important because it saves on volunteer time in uploading and categorising images in other ways. Commons has huge backlogs of uncategorised or poorly categorised files, and other projects especially wikipedia have vast backlogs in adding or upgrading their imagery. There has been recent research that articles with images get more viewers, so the implications of backing this aren't just the usual one that investing in the GLAM program is an investment in quality, there is also a good chance that investing in GLAM could increase readership. Regards Jonathan Cardy GLAM (Galleries, Libraries, Archives & Museums) Organiser Wikimedia UK 020 7065 0921 (I'm normally in the office Tuesday's, Wednesdays and Fridays - Emails on Mondays and Thursdays wont usually be seen till the next day) Wikimedia UK is a Company Limited by Guarantee registered in England and Wales, Registered No. 6741827. Registered Charity No.1144513. Registered Office 4th Floor, Development House, 56-64 Leonard Street, London EC2A 4LT. United Kingdom. Wikimedia UK is the UK chapter of a global Wikimedia movement. The Wikimedia projects are run by the Wikimedia Foundation (who operate Wikipedia, amongst other projects). Wikimedia UK is an independent non-profit charity with no legal control over Wikipedia nor responsibility for its contents. On 26 February 2015 at 06:57, Erik Moeller <erik(a)wikimedia.org> wrote:

Dear GLAMwikimedians, particularly those involved in the GLAMWiki Toolset, Europeana has decided not to further invest in a new version of the GLAM-WIKI toolset (GWT). This includes developing a “version 2” of the current system as was proposed in the grant application [1], or as a standalone system using O-Auth as proposed by Erik earlier in this thread. We have come to this conclusion after a lot of internal discussions, and talking with key stakeholders, notably by presenting several options at the European GLAMwiki Coordinators meeting. We have received an overwhelming consensus that investing our energy in other forms of co-operation was the best approach.[2] This is not a repudiation of the GWT in theory or in practice - the GWT is not going away and we are very supportive and proud of it. As previously mentioned, it had become clear that the “version 2” grant request would never be approved on the basis that the WMF would not allocate code review nor would it support the project once build.[3] Redeveloping as an independent “stand alone” tool would solve the first problem, but conversely it would exacerbate the second problem. It is universally agreed that moving to an independent tool would make for a more flexible, powerful and faster-to-develop system. However, consistent feedback from stakeholder consultation was that there is no demonstrable external investment in supporting a third-party mediawiki development ecosystem. The primary encouragement we received rebuild using O-Auth and the API was not for the tool itself, but rather as a test case for those systems. Europeana has therefore decided that, in the current environment, this would not be the best use of our resources. Instead, we intend to focus our GLAMwiki resources in community event/activity support, rather than into software development. In particular, we are very interesting in “pivoting” our attentions towards Wikidata and the potential that holds for supporting digital cultural heritage.[4] That said - this does NOT mean that the GWT as it currently stands is going away. It is, and will remain the most stable and integrated way to provide mass-uploads to Wikimedia Commons, probably for many years to come. We are aware that many Wikimedia Chapters have written GWT into their annual plans, and we intend to continue supporting training and related events. Also, given that there will not be an “all new” version of the GWT, we are also investigating asking for funding to close the major bugs that we had already triaged.[5] Finally, we also hope to make greater use of the GWT for ourselves. There is a large and increasing amount of content on Europeana that has an open license and has quality metadata and image quality for it to be uploaded to Wikimedia. We hope to start sharing this material directly with Commons soon. Making the uploads ourselves would have the advantage for our many partner GLAMs to be able jump straight into the fun part of working with their local wikimedia community on editathons etc., rather than their time wrangling metadata. Liam/Wittylama In my capacity as Europeana GLAMwiki Coordinator (and the guy who gut Europeana into this project 4 years ago…) p.s. In response to Erik’s comparison of the GWT to the WikiEd Foundation redeveloping the Education Extension as an independent tool [6]. The significant difference is that the WikiEdu Foundation’s organisational purpose is to run activities and maintain software for Wikipedia outreach to the education sector. Therefore the long term ‘ownership’ of software built for this goal, and of which they will be the primary user, is well within their mission. Neither was ever intended to be the case case for Europeana and the GWT. [1] https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset [2] Minutes of that section of the discussion are here: https://meta.wikimedia.org/wiki/2015_European_GLAMwiki_Coordinators_meeting… [3] https://lists.wikimedia.org/pipermail/glamtools/2015-February/000356.html [4] In particular the “sum of all paintings” project https://www.wikidata.org/wiki/Wikidata:WikiProject_sum_of_all_paintings [5] Which was ‘item 1’ in the grant application https://meta.wikimedia.org/wiki/Grants:PEG/Europeana/GLAMwiki_Toolset#Ident… [6] https://lists.wikimedia.org/pipermail/glamtools/2015-March/000365.html