What was the publicising of the campaign prior to its launch?

It should be pretty apparent to people with experience within the movement that this would be both entirely novel and pretty controversial. I'd expect some amount of transparency around it (a phabricator ticket is not, in and of itself, transparency). To contrast, with search when we make /experimental/ modifications to the user experience of a tiny sample (through A/B testing) we not only list those changes in phabricator but also send explicit mailing list announcements - and those effect a smaller chunk of our user base on a platform.

On 1 September 2015 at 22:51, Greg Grossmeier greg@wikimedia.org wrote:

<quote name="MZMcBride" date="2015-09-01" time="19:47:39 -0400"> > It's been discussed previously both on this list and elsewhere, but for > better or worse the Wikimedia Foundation has an entire "Mobile apps" team > that pretty much exclusively works on closed platforms, as I understand > it. They've gone as far as to abandon Gerrit in favor of GitHub.

For the record, it's just the iOS team that moved to Github. Android is still in Gerrit. The iOS move was for CI reasons (summary: we (WMF RelEng) can't support the OSX platform for build and test cases with any ease, especially vis a vis other priorities).

https://gerrit.wikimedia.org/r/#/q/status:merged+project:apps/android/wikipe...

Greg

-- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |

---

Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Hi all --

I'm going to try to address as many of the issues mentioned in this thread and the Phabricator ticket[1] as I can. I'm going to preface this by explaining why we're doing this.

First of all, this is only a test in order to increase our understanding of how our readers interact with our content. It's limited to a relatively small but representative sample of our readers and is not permanent. The results of this test will inform our mobile strategy.

I'd also like to offer that in our community strategic consultation[3], mobile and apps was the single most commented upon topic, including many comments that we should build an app.

Specific Issues

1. We're moving people from an open platform to a closed platform: I think this is an oversimplification of the situation -- as has been noted before, the android app is 100% open source and while the data is not, in my opinion, comprehensive, it's inarguable that a large percentage of mobile traffic on the internet is from apps. It's not possible to fulfill our mission[4] if Wikipedia and sister project content is not available in widely used channels.

2. The campaign was not publicized before launch: We notified the Finnish community on their Village pump before the campaign began[5] and the campaign is detailed on the central notice page[6]. We felt this was appropriate considering the scope of the test.

3. Banners/Interstitials don't work/suck/etc: There's a difference between a forced install and letting users know that an app exists and our designers have worked hard to make the banners effective without being excessively intrusive. You can see the designs on the Phab ticket above. I don't generally place a great deal of faith in blog posts or other company's data -- the google study showing the ineffectiveness of interstitials has already been challenged by other similarly reputable sources [7,8]. For this and other reasons, I believe that we need to gather our own data.

4. We don't understand what success looks like: We are planning a meeting with our Research team[9] to assess the statistical validity of our results, but the basic question is if users read more content using the app than the mobile web. This information will help guide us on future product decisions and will be shared with the community.

-Toby

[1] https://phabricator.wikimedia.org/T103896 [2] https://meta.wikimedia.org/wiki/Research:Projects [3] http://blog.wikimedia.org/2015/08/27/strategy-potential-mobile-multimedia-tr... https://upload.wikimedia.org/wikipedia/commons/c/c7/2015_Strategy_Consultation_Report.pdf [4] https://wikimediafoundation.org/wiki/Mission_statement [5] https://fi.wikipedia.org/wiki/Wikipedia:Kahvihuone_%28uutiset%29#Running_ban... [6] https://meta.wikimedia.org/wiki/CentralNotice/Calendar [7] http://googlewebmastercentral.blogspot.com/2015/07/google-case-study-on-app-... [8] https://www.linkedin.com/pulse/all-app-promos-bad-googles-omar-restom [9] https://phabricator.wikimedia.org/T105561

On Wed, Sep 2, 2015 at 8:21 AM, Tilman Bayer tbayer@wikimedia.org wrote:

On Tue, Sep 1, 2015 at 8:10 PM, Oliver Keyes okeyes@wikimedia.org wrote:

...

What was the publicising of the campaign prior to its launch?

It should be pretty apparent to people with experience within the movement that this would be both entirely novel and pretty controversial.

As mentioned on the Phabricator ticked, this is by no means the first banner campaign inviting installation of an app.

In June/July last year, there was a global campaign announcing the launch of the new Android app (like now, shown on mobile web for Android devices only):

https://en.m.wikipedia.org/w/index.php?title=Special:Random&banner=Wpapp...

https://en.m.wikipedia.org/w/index.php?title=Special:Random&banner=Wpapp... (also ran in a few other languages besides English) I don't recall it being controversial back then.

And in 2013, the late Commons app was promoted in a similar campaign on desktop and mobile:

https://en.wikipedia.org/w/index.php?title=Special:Random&banner=Commons... (on desktop Wikipedia)

https://commons.wikimedia.org/w/index.php?title=Special:Random&banner=Co... (on Commons)

https://en.m.wikipedia.org/w/index.php?title=Special:Random&banner=Andro... (mobile Wikipedia on Android devices)

...

I'd expect some amount of transparency around it (a phabricator ticket is not, in and of itself, transparency).

For those not familiar with the existing processes around banners, WMF staff and community members who use this indeed highly prominent space have been coordinating for years on this page: https://meta.wikimedia.org/wiki/CentralNotice/Calendar Quite a lot of people who care about banner use are watching it for controversial or problematic uses ( https://meta.wikimedia.org/w/index.php?title=CentralNotice/Calendar&acti... ), discussion happens on the talk page there or is escalated to other venues. I see that the current banners were indeed listed there last week before the launch.

...

To contrast, with search when we make /experimental/ modifications to the user experience of a tiny sample (through A/B testing) we not only list those changes in phabricator but also send explicit mailing list announcements - and those effect a smaller chunk of our user base on a platform.

Perhaps you could post some advice at https://meta.wikimedia.org/wiki/Talk:CentralNotice about how people running banners could learn from the WMF Discovery team in that respect?

On Tue, Sep 1, 2015 at 8:30 AM, Ori Livneh ori@wikimedia.org wrote:

...

We appear to be running a banner campaign on the mobile web site, driving people to download the mobile app:

https://en.m.wikipedia.org/wiki/?banner=Aug2015_app_banner_2 https://en.m.wikipedia.org/wiki/?banner=Aug2015_app_banner_1

The links don't work for me (maybe because I'm not in Finland right now); you can append "force=1" to make them show regardless of targeting:

https://en.m.wikipedia.org/wiki/?banner=Aug2015_app_banner_2&force=1 https://en.m.wikipedia.org/wiki/?banner=Aug2015_app_banner_1&force=1

-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB

---

Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

On 4 September 2015 at 20:12, Brandon Harris bharris@gaijin.com wrote:

    So it seems to me that the apps are not required to fulfill the mission.  They feel like distractions, and - quite possibly - negatives to the mission (in that we can't convert Readers into Editors through the app).

It's worth stressing here that our app is really good for reading. I have kept a gaggle of 7-8yo children amused while they were playing "guess the animal" by calling up the animal in question on Wikipedia on my S4 Mini, complete with that picture in the header. It was better than trying to do the same on the website has been on the same phone in the past. So let's not forget to give the app at least some love :-)

- d.

Brandon, great to hear from you! I think you're working off of old information--like Matt said, you can edit via the app now. It's cool that you're inspired to bring up more questions, though, and I'm glad you're focusing on the design phase of the next experiment. Are you busy for the next ten years?

Another misconception or oversight I want to bring up is that Fundraising is the team pioneering the 2/3-page or full-page banners. We're driving readers from the website to completely closed and somewhat evil payments platforms. If there's any relevant or even irrelevant research about interstitials, please apply it to our work, cos we're about to have a huge impact on the English-speaking community in December. Any complaints about the Finnish mobile experiment and dogpiling on the awesome apps developers seem incredibly misplaced while I'm walking around with this "Kick Me" sign on my backside.

Thanks, Adam

On Fri, Sep 4, 2015 at 12:12 PM, Brandon Harris bharris@gaijin.com wrote:

...

On Sep 2, 2015, at 3:53 PM, Toby Negrin tnegrin@wikimedia.org wrote:

...

1. We're moving people from an open platform to a closed platform: I

think

...

this is an oversimplification of the situation -- as has been noted

before,

...

the android app is 100% open source and while the data is not, in my opinion, comprehensive, it's inarguable that a large percentage of mobile traffic on the internet is from apps. It's not possible to fulfill our mission[4] if Wikipedia and sister project content is not available in widely used channels.

    I'm not sure this makes a lot of sense.  The widest, most-open

content channel that the projects have is through the web interfaces: all phones, all devices, all computers can access the same content in the same manner. That is to say: 100% of our readers have the ability to use the web versions (either desktop or mobile web) where as only a subset can use the Android app, which is a different subset that can use iOS. (They also end up having fragmented experiences, which is sub-optimal.)

    So it seems to me that the apps are not required to fulfill the

mission. They feel like distractions, and - quite possibly - negatives to the mission (in that we can't convert Readers into Editors through the app).

    (Which, by the way, this whole "focus entirely on readers" shift

seems counter-intuitive to me. Having a billion readers doesn't mean anything if there aren't any editors anymore. It's a complete failure at that point.)

...

2. The campaign was not publicized before launch: We notified the Finnish

community on their Village pump before the campaign began[5] and the campaign is detailed on the central notice page[6]. We felt this was appropriate considering the scope of the test.

    Restricting the conversation to two very small, almost

impenetrable discussion areas seems unwise. It seems obvious to me that this idea and action would cause friction with the community. I don't think there's any bad-faith going on here, but this definitely feels like an oversight.

...

3. Banners/Interstitials don't work/suck/etc: There's a difference

between

...

a forced install and letting users know that an app exists and our designers have worked hard to make the banners effective without being excessively intrusive. You can see the designs on the Phab ticket above.

I

...

don't generally place a great deal of faith in blog posts or other company's data -- the google study showing the ineffectiveness of interstitials has already been challenged by other similarly reputable sources [7,8]. For this and other reasons, I believe that we need to

gather

...

our own data.

    Is "our own data" more important than the goodwill of our users or

developers? I think that's a big part of why people might be upset about this: it's a step away from what had classically been the principles underlying the movement's activities.

    Even that said, though:  this is the first anyone is saying "yes,

we did some research about interstitials". It seems to me that the Google study was something that could have been discussed ahead of time. I also don't understand why we can't do the whole Open Source thing and make use of other people's research, unless this indicates a further shift into "not invented here" territory.

...

4. We don't understand what success looks like: We are planning a meeting

with our Research team[9] to assess the statistical validity of our results, but the basic question is if users read more content using the

app

...

than the mobile web. This information will help guide us on future

product

...

decisions and will be shared with the community.

    An experiment without a box isn't an experiment.

            "We would like to determine if people read more through

the apps than through the web interface" is a _great_ question (but also one that could probably be answered just by looking at squid logs). I don't know that it needs an advertising campaign to create app users to do it (though I could be wrong, and often am, and would love to hear how if so). It further seems that advertising the mobile apps would create a biases in the research (if only that "newish" app users are likely to use it more often earlier in their

            "We would like to determine if people download the app

more often if they've been given an interstitial" is also an interesting question but it's got a secondary question that no one seems to care about: "How many readers have we put off from returning by showing them this interstitial?" I know that I often immediately shut windows and tabs when I'm told "download our app!"

    If this were brought to the wider community in a different manner,

there may have been a completely different response:

            "We do not believe that people are aware that there are

official Wikipedia apps. We would like to run an experiment to see how likely people are to switch to the app experience if they know it exists. Of those that switch, we would like to find out how many of them increase their usage of the content, and, ideally, we'd like to know which features of the app are the most popular and useful. Additionally, we'd like to know the drop-off counts. We want to do it in a controlled environment where we understand the patterns as they exist fairly well. We'll run this experiment for X days, and we know that there will be biases on W, Y, and Z."

    I could easily get behind that set of questions.

    I don't really expect a response to any of this, by the way.

---

Brandon Harris :: bharris@gaijin.com :: made of steel wool and whiskey

---

Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Hi all -- I wanted to follow up on this thread with the results of the test:

https://www.mediawiki.org/wiki/Wikimedia_Apps/Finland_Banner_Test

TLDR: While we saw a big increase in installs and opens, we found that the campaign did not significantly increase pageviews, leading us to conclude that readers are interested in using an app, but given the current experience would rather use the mobile web than the native app.

-Toby

On Wed, Sep 9, 2015 at 11:51 AM, Ryan Lane rlane32@gmail.com wrote:

On Wed, Sep 9, 2015 at 7:08 AM, Adam Wight awight@wikimedia.org wrote:

...

Another misconception or oversight I want to bring up is that Fundraising is the team pioneering the 2/3-page or full-page banners. We're driving readers from the website to completely closed and somewhat evil payments platforms. If there's any relevant or even irrelevant research about interstitials, please apply it to our work, cos we're about to have a

huge

...

impact on the English-speaking community in December. Any complaints

about

...

the Finnish mobile experiment and dogpiling on the awesome apps

developers

...

seem incredibly misplaced while I'm walking around with this "Kick Me"

sign

...

on my backside.

I'm couldn't fully grok what point this was making, but whether it's this mobile experiment or fundraising doing 2/3 or full-page banners, they're terrible. This has been a major complaint by numerous people over the past few years. Please tell me you're not considering interstitials, 2/3 page, or full page banners for fundraising? This may be the year I actually start boycotting the Wikimedia fundraiser, if so.

• Ryan

---

Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

On Tue, Sep 1, 2015 at 10:09 PM, Ori Livneh ori@wikimedia.org wrote:

Just in time! http://techcrunch.com/2015/09/01/death-to-app-install-interstitials/

Interstitials are full-page ads where you have to click a link to get to the actual content. These are normal banners. More importantly, as you can see in the Phabricator task, they are an experiment to measure if it is possible to make more people use the app. Experiments are good. For one thing, they can turn out negative, in which case we will have been spared a philosophical debate about openness.

For what it's worth, the line " For one thing, they can turn out negative, in which case we will have been spared a philosophical debate about openness." comes off as very snarky and also entirely the wrong approach. Whether something is /within our ethos/ should not be something we discuss after doing it, and even then, only if we find out that it's effective. To put that another way, "sure it might not be ethical by our standards but hey let's give it a whirl anyway". That's totally dissonant from our movement and organisation's principles.

On 2 September 2015 at 09:14, Oliver Keyes okeyes@wikimedia.org wrote:

On 2 September 2015 at 01:50, Gergo Tisza gtisza@wikimedia.org wrote:

...

On Tue, Sep 1, 2015 at 10:09 PM, Ori Livneh ori@wikimedia.org wrote:

...

Just in time! http://techcrunch.com/2015/09/01/death-to-app-install-interstitials/

Interstitials are full-page ads where you have to click a link to get to the actual content. These are normal banners. More importantly, as you can see in the Phabricator task, they are an experiment to measure if it is possible to make more people use the app. Experiments are good. For one thing, they can turn out negative, in which case we will have been spared a philosophical debate about openness.

Is this experiment also measuring what those users do on the app, versus what the same users (or a users with a similar background) did on the mobile web? Is it a formal A/B test?

We seem to be operating under the belief that merely switching users is, in and of itself, a victory. It's not; we still have the same number of users at the end. A victory is increased activity /due/ to the features on the app that cannot be created outside that closed ecosystem.

...

---

Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

-- Oliver Keyes Count Logula Wikimedia Foundation

On Sep 2, 2015, at 11:17 AM, Gergo Tisza gtisza@wikimedia.org wrote:

On Wed, Sep 2, 2015 at 6:19 AM, Oliver Keyes okeyes@wikimedia.org wrote:

...

For what it's worth, the line " For one thing, they can turn out negative, in which case we will have been spared a philosophical debate about openness." comes off as very snarky and also entirely the wrong approach.

Debates about the Wikimedia ethos tend to be highly subjective and thus costly both in terms of time and emotional resources. Measuring whether banners work is fairly simple and objective. It makes sense to perform the cheapest prerequisite checks first, to minimize total cost.

Part of the cost of business in being transparent and actually _having_ an ethos is that these conversations need to be had, regardless of their cost.

And I seriously doubt that there's any benefit to these banner ads at all. Converting a small number of people from using the web version to an app version is meaningless when operating at this scale. We're actually probably _reducing_ the number of readers overall because many will simply say "screw this if you're serving me interstitials".

This was a bad idea. It remains a bad idea. It looks bad on the movement.

--- Brandon Harris :: bharris@gaijin.com :: made of steel wool and whiskey

On 2 September 2015 at 14:17, Gergo Tisza gtisza@wikimedia.org wrote:

On Wed, Sep 2, 2015 at 6:19 AM, Oliver Keyes okeyes@wikimedia.org wrote:

...

For what it's worth, the line " For one thing, they can turn out negative, in which case we will have been spared a philosophical debate about openness." comes off as very snarky and also entirely the wrong approach.

Debates about the Wikimedia ethos tend to be highly subjective and thus costly both in terms of time and emotional resources. Measuring whether banners work is fairly simple and objective. It makes sense to perform the cheapest prerequisite checks first, to minimize total cost.

And without any answer to my question about whether this was an actual A/B test, and whether you're measuring overall user utility rather than 'did they download it', this is also highly subjective and costly both in terms of time and emotional resources.

But you're missing...well, two important points. First, as Brandon says, these debates /have to happen/. Identifying that something is a *right* thing to do, an *ethical* thing to do, cannot happen after that thing has been done. And second: costly in terms of time? Costly in terms of emotional resources? This thread is costly on both, and it is also an inevitable consequence of not having the discussion in advance.

Yes, having discussions takes time and energy. And sometimes you don't like the outcome. Those are a given outcome of talking to people. But they are things we do /regardless/ of whether we feel like not talking to people would be easier (not talking to people is always easier) and they are things that, nine times out of ten, are actually a massive saving on time and emotional energy. Because it means you can have conversations with people exploring the ethical costs and benefits of doing an action, and then do (or not do) that action, rather than do that action and then deal with /outraged/ people who are approaching the situation not as a hypothetical but as something that actually happened.

And it's apparent, from the replies to this thread, that this decision did not save on emotional energy - it just offloaded it. We have multiple staffers and volunteers sat here sending messages that boil down to "this does not represent me. This is not the movement I work towards". That's not a tremendously pleasant experience for us. We have an expectation on us, as human beings and movement members and staffers, that we will consider the /systemic/ impact of what we choose to do and not do. Describing talking about it in advance as too much of an emotional load makes it appear that that evaluation was not adequately performed.

---

Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

On 09/02/2015 03:55 PM, Ryan Lane wrote:

Even ignoring the "is it right and ethical" debate, there's a pretty large amount of research over the past 6 or so months that show this is a bad idea.

[citation needed]

Matt Flaschen