Awesome Oliver, thanks!
Having a look.

On Sat, Mar 21, 2015 at 4:32 PM, Oliver Keyes <okeyes@wikimedia.org> wrote:
Alrighty; existing implementation updated with
https://gerrit.wikimedia.org/r/#/c/198489/ which also exposes the
isAppPageview method and associates a UDF with it (Marcel, this means
you'll be able to do your stuff. Rockin'!)

Let me know when y'all have the necessary varnish and app changes
proposed/committed/merged/etc so I can work on the updates to this
method and the documentation in parallel.

On 20 March 2015 at 21:11, Oliver Keyes <okeyes@wikimedia.org> wrote:
> Okay. So, distinct header, registering refresh=1 or [nothing] in the
> x-analytics field when it hits the varnish layer? Rocking! Let me know
> what happens/where it goes/when it goes/etc.
>
> The RESTbase services, yeah; they're going to make an impact. Not just
> in format either (I have literally no idea how the caching setup there
> works, if there is a caching setup, or which varnish cluster any such
> caching setup would go through and so if we're even picking up those
> requests at all).
>
> On 20 March 2015 at 19:11, Bernd Sitzmann <bernd@wikimedia.org> wrote:
>> Adam: Thanks for including us Android devs.
>> I think a distinct header is probably preferable.
>>
>> Oliver:
>> I think you already got this, so just to confirm: for Android you can use
>> sections=0 for counting pageviews, and sections=all for counting saved page
>> refreshes.
>>
>> As a heads-up we're experimenting with Node.js/RESTBase services. Not sure
>> when and if those will be used in production. We're at an early stage. Just
>> wanted to mention that this since it will change the way we request pages
>> from the server significantly (actually it would be from different servers,
>> too). That's probably another pro for using HTTP request headers.
>>
>> The downside is that we're not using this special request header yet.
>>
>> Bernd
>>
>> On Fri, Mar 20, 2015 at 3:17 PM, Oliver Keyes <okeyes@wikimedia.org> wrote:
>>>
>>> Sounds good for me, although I'd just go for the first option. There's
>>> nothing contained in the second option not found in the first that we
>>> need (that is: yes, it has device version and OS and all of that,
>>> which we need for other things, but then so does the user agent, so
>>> it's probably extraneous work to include all of that a /second/ time)
>>>
>>> On 20 March 2015 at 15:42, Adam Baso <abaso@wikimedia.org> wrote:
>>> > Either use of a distinct header (e.g., X-WMF-Refresh: 1) or use of a
>>> > distinct parameter (e.g., wprov=rfsi1 for "refresh from saved pages on
>>> > iOS,
>>> > version 1") from the client would work. The use of a distinct parameter
>>> > is
>>> > much easier.
>>> >
>>> > Then the VCL code could be updated to look for the field and enrich the
>>> > X-Analytics header accordingly.
>>> >
>>> > Okay?
>>> >
>>> >
>>> > On Fri, Mar 20, 2015 at 11:37 AM, Oliver Keyes <okeyes@wikimedia.org>
>>> > wrote:
>>> >>
>>> >> Just a note that I'm started on a related patch now, using the (we now
>>> >> know, not-futureproof) logic of:
>>> >>
>>> >> 1. if it's got sections=0 it's a pageview;
>>> >> 2. if it's got sections=all and is from iOS it's a pageview.
>>> >>
>>> >> Would appreciate some feedback on the idea of sending refresh=1 in the
>>> >> x_analytics header, so we know what to expect there.
>>> >>
>>> >> On 20 March 2015 at 00:44, Oliver Keyes <okeyes@wikimedia.org> wrote:
>>> >> > In the long-term we'll just be relying on x_analytics, yes, because
>>> >> > the app will be sending through the pageID and namespace in the same
>>> >> > way as desktop and the mobile web do; so it'll be
>>> >> > pageid=50;ns=0;refresh-1 or pageid=50;ns=0, respectively. That's a
>>> >> > different patch. In the short-term I'm not seeing how building out a
>>> >> > complex ecosystem for this is a valuable use of time given that we
>>> >> > know what we'll be switching to (and that it's standardised not just
>>> >> > across apps, but also for desktop and the mobile web, to boot). All
>>> >> > we
>>> >> > care about distinguishing that we won't get for free as part of
>>> >> > already-scheduled work is refreshes from pageviews.
>>> >> >
>>> >> > On 19 March 2015 at 23:59, Gergo Tisza <gtisza@wikimedia.org> wrote:
>>> >> >> On Thu, Mar 19, 2015 at 8:06 PM, Oliver Keyes <okeyes@wikimedia.org>
>>> >> >> wrote:
>>> >> >>>
>>> >> >>> As I see it, we basically have two possibilities here:
>>> >> >>>
>>> >> >>> 1. Make the URLs distinguishable;
>>> >> >>> 2. Add additional metadata in a non-URL place
>>> >> >>>
>>> >> >>> 1 is undesirable because it ruins caching, and we like caching. So
>>> >> >>> we
>>> >> >>> look at 2, which realistically means the x_analytics field. Why
>>> >> >>> don't
>>> >> >>> we add a parameter there? refresh=1. And then, our app check boils
>>> >> >>> down to (in pseudocode):
>>> >> >>>
>>> >> >>> if(other_checks & urlContains("sections=(0|all)" &
>>> >> >>> !xAnalyticsContains("refresh")){
>>> >> >>>     return true;
>>> >> >>> }
>>> >> >>> return false;
>>> >> >>>
>>> >> >>> Nice and simple and easy. It'll require some coordination with
>>> >> >>> Ottomata because it means modifying the UDF parameters, and we're
>>> >> >>> using said UDF in production so it'll have to be synced to a change
>>> >> >>> in
>>> >> >>> the relevant Oozie job, but it should be totally doable, and I
>>> >> >>> can't
>>> >> >>> see an easier way of doing it.
>>> >> >>>
>>> >> >>> Thoughts, people?
>>> >> >>
>>> >> >>
>>> >> >> Why use anything other than X-Analytics at all? Source of the
>>> >> >> pageview
>>> >> >> is
>>> >> >> exactly the kind of information it is meant for. Just set
>>> >> >> source=AndroidAppPageView /  source=IosAppSectionView /
>>> >> >> source=AndroidAppSavedPageRefresh etc. (Or set up a mapping to
>>> >> >> three-letter
>>> >> >> acronyms if you want to be nice on the servers.) That puts logging
>>> >> >> completely in the hands of the app developers, so there are no
>>> >> >> information
>>> >> >> flow problems and less organizational overhead; it also makes rules
>>> >> >> more
>>> >> >> explicit (and thus harder to mess up / easier to spot errors).
>>> >> >>
>>> >> >> _______________________________________________
>>> >> >> Analytics mailing list
>>> >> >> Analytics@lists.wikimedia.org
>>> >> >> https://lists.wikimedia.org/mailman/listinfo/analytics
>>> >> >>
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Oliver Keyes
>>> >> > Research Analyst
>>> >> > Wikimedia Foundation
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Oliver Keyes
>>> >> Research Analyst
>>> >> Wikimedia Foundation
>>> >>
>>> >> _______________________________________________
>>> >> Analytics mailing list
>>> >> Analytics@lists.wikimedia.org
>>> >> https://lists.wikimedia.org/mailman/listinfo/analytics
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > Analytics mailing list
>>> > Analytics@lists.wikimedia.org
>>> > https://lists.wikimedia.org/mailman/listinfo/analytics
>>> >
>>>
>>>
>>>
>>> --
>>> Oliver Keyes
>>> Research Analyst
>>> Wikimedia Foundation
>>
>>
>
>
>
> --
> Oliver Keyes
> Research Analyst
> Wikimedia Foundation



--
Oliver Keyes
Research Analyst
Wikimedia Foundation

_______________________________________________
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics