Awesome Oliver, thanks!
Having a look.
On Sat, Mar 21, 2015 at 4:32 PM, Oliver Keyes <okeyes(a)wikimedia.org> wrote:
Alrighty; existing implementation updated with
https://gerrit.wikimedia.org/r/#/c/198489/ which also exposes the
isAppPageview method and associates a UDF with it (Marcel, this means
you'll be able to do your stuff. Rockin'!)
Let me know when y'all have the necessary varnish and app changes
proposed/committed/merged/etc so I can work on the updates to this
method and the documentation in parallel.
On 20 March 2015 at 21:11, Oliver Keyes <okeyes(a)wikimedia.org> wrote:
Okay. So, distinct header, registering refresh=1
or [nothing] in the
x-analytics field when it hits the varnish layer? Rocking! Let me know
what happens/where it goes/when it goes/etc.
The RESTbase services, yeah; they're going to make an impact. Not just
in format either (I have literally no idea how the caching setup there
works, if there is a caching setup, or which varnish cluster any such
caching setup would go through and so if we're even picking up those
requests at all).
On 20 March 2015 at 19:11, Bernd Sitzmann <bernd(a)wikimedia.org> wrote:
> Adam: Thanks for including us Android devs.
> I think a distinct header is probably preferable.
>
> Oliver:
> I think you already got this, so just to confirm: for Android you can
use
> sections=0 for counting pageviews, and
sections=all for counting saved
page
> refreshes.
>
> As a heads-up we're experimenting with Node.js/RESTBase services. Not
sure
> when and if those will be used in production.
We're at an early stage.
Just
> wanted to mention that this since it will
change the way we request
pages
> from the server significantly (actually it
would be from different
servers,
> too). That's probably another pro for
using HTTP request headers.
>
> The downside is that we're not using this special request header yet.
>
> Bernd
>
> On Fri, Mar 20, 2015 at 3:17 PM, Oliver Keyes <okeyes(a)wikimedia.org>
wrote:
>>
>> Sounds good for me, although I'd just go for the first option. There's
>> nothing contained in the second option not found in the first that we
>> need (that is: yes, it has device version and OS and all of that,
>> which we need for other things, but then so does the user agent, so
>> it's probably extraneous work to include all of that a /second/ time)
>>
>> On 20 March 2015 at 15:42, Adam Baso <abaso(a)wikimedia.org> wrote:
>> > Either use of a distinct header (e.g., X-WMF-Refresh: 1) or use of a
>> > distinct parameter (e.g., wprov=rfsi1 for "refresh from saved pages
on
>> > iOS,
>> > version 1") from the client would work. The use of a distinct
parameter
>> > is
>> > much easier.
>> >
>> > Then the VCL code could be updated to look for the field and enrich
the
> >
X-Analytics header accordingly.
> >
> > Okay?
> >
> >
> > On Fri, Mar 20, 2015 at 11:37 AM, Oliver Keyes <okeyes(a)wikimedia.org
>> > wrote:
>> >>
>> >> Just a note that I'm started on a related patch now, using the (we
now
>> >> know, not-futureproof) logic
of:
>> >>
>> >> 1. if it's got sections=0 it's a pageview;
>> >> 2. if it's got sections=all and is from iOS it's a pageview.
>> >>
>> >> Would appreciate some feedback on the idea of sending refresh=1 in
the
>> >> x_analytics header, so we know
what to expect there.
>> >>
>> >> On 20 March 2015 at 00:44, Oliver Keyes <okeyes(a)wikimedia.org>
wrote:
>> >> > In the long-term we'll
just be relying on x_analytics, yes,
because
>> >> > the app will be sending
through the pageID and namespace in the
same
>> >> > way as desktop and the
mobile web do; so it'll be
>> >> > pageid=50;ns=0;refresh-1 or pageid=50;ns=0, respectively.
That's a
>> >> > different patch. In the short-term I'm not seeing how building
out a
>> >> > complex ecosystem for this
is a valuable use of time given that we
>> >> > know what we'll be switching to (and that it's standardised
not
just
>> >> > across apps, but also for
desktop and the mobile web, to boot).
All
>> >> > we
>> >> > care about distinguishing that we won't get for free as part
of
>> >> > already-scheduled work is refreshes from pageviews.
>> >> >
>> >> > On 19 March 2015 at 23:59, Gergo Tisza <gtisza(a)wikimedia.org>
wrote:
>> >> >> On Thu, Mar 19, 2015 at
8:06 PM, Oliver Keyes <
okeyes(a)wikimedia.org>
>> >> >> wrote:
>> >> >>>
>> >> >>> As I see it, we basically have two possibilities here:
>> >> >>>
>> >> >>> 1. Make the URLs distinguishable;
>> >> >>> 2. Add additional metadata in a non-URL place
>> >> >>>
>> >> >>> 1 is undesirable because it ruins caching, and we like
caching.
So
>> >> >>> we
>> >> >>> look at 2, which realistically means the x_analytics field.
Why
>> >> >>> don't
>> >> >>> we add a parameter there? refresh=1. And then, our app
check
boils
>> >> >>> down to (in
pseudocode):
>> >> >>>
>> >> >>> if(other_checks &
urlContains("sections=(0|all)" &
>> >> >>> !xAnalyticsContains("refresh")){
>> >> >>> return true;
>> >> >>> }
>> >> >>> return false;
>> >> >>>
>> >> >>> Nice and simple and easy. It'll require some
coordination with
>> >> >>> Ottomata because it means modifying the UDF parameters, and
we're
>> >> >>> using said UDF in
production so it'll have to be synced to a
change
>> >> >>> in
>> >> >>> the relevant Oozie job, but it should be totally doable,
and I
>> >> >>> can't
>> >> >>> see an easier way of doing it.
>> >> >>>
>> >> >>> Thoughts, people?
>> >> >>
>> >> >>
>> >> >> Why use anything other than X-Analytics at all? Source of the
>> >> >> pageview
>> >> >> is
>> >> >> exactly the kind of information it is meant for. Just set
>> >> >> source=AndroidAppPageView / source=IosAppSectionView /
>> >> >> source=AndroidAppSavedPageRefresh etc. (Or set up a mapping to
>> >> >> three-letter
>> >> >> acronyms if you want to be nice on the servers.) That puts
logging
>> >> >> completely in the hands
of the app developers, so there are no
>> >> >> information
>> >> >> flow problems and less organizational overhead; it also makes
rules
>> >> more
>> >> explicit (and thus harder to mess up / easier to spot errors).
>> >>
>> >> _______________________________________________
>> >> Analytics mailing list
>> >> Analytics(a)lists.wikimedia.org
>> >>
https://lists.wikimedia.org/mailman/listinfo/analytics
>> >>
>> >
>> >
>> >
>> > --
>> > Oliver Keyes
>> > Research Analyst
>> > Wikimedia Foundation
>>
>>
>>
>> --
>> Oliver Keyes
>> Research Analyst
>> Wikimedia Foundation
>>
>> _______________________________________________
>> Analytics mailing list
>> Analytics(a)lists.wikimedia.org
>>
https://lists.wikimedia.org/mailman/listinfo/analytics
>
>
>
> _______________________________________________
> Analytics mailing list
> Analytics(a)lists.wikimedia.org
>
https://lists.wikimedia.org/mailman/listinfo/analytics
>
--
Oliver Keyes
Research Analyst
Wikimedia Foundation
--
Oliver Keyes
Research Analyst
Wikimedia Foundation
--
Oliver Keyes
Research Analyst
Wikimedia Foundation
_______________________________________________
Analytics mailing list
Analytics(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics