Re: [Analytics] top articles script

List overview All Threads
Download

newer

older

pageview API discrepancy

Short Hive and Oozie maintenance...

Kevin Leduc

20 Jan 2016 20 Jan '16

8:42 p.m.

+Analytics list so they can comment. I don't have such a script. It's a pretty intensive job to compile top articles especially over a month. The pageview API was supposed to have top articles per month per wiki but the job is so massive that it failed to run in Hive. Analytics knows there are better algorithms out there to solve this problem. So the pageview API just has top per day per wiki. I imagine that you are looking at some very specific wikis and countries... not all of them. Maybe someone on the list can make an example hive script (given a wiki and country) that gives the top for a day. On Wed, Jan 20, 2016 at 12:23 PM, Dan Foy <dfoy(a)wikimedia.org> wrote:

...

Hi Kevin, In your collection of scripts for Hive, do you have one that can act as a starting point for me to get the top N articles / URLs for Wikipedia in a country? Thanks, Dan

Attachments:

attachment.htm (text/html — 1.2 KB)

Show replies by date

Tilman Bayer

22 Jan 22 Jan

11:13 p.m.

New subject: top articles script

...

Hi Kevin, In your collection of scripts for Hive, do you have one that can act as a starting point for me to get the top N articles / URLs for Wikipedia in a country? Thanks, Dan

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB

Ryan Kaldari

11:17 p.m.

New subject: top articles script

Any idea why the most popular article in India is "-"? CCing Dan Garry of Discovery team. On Fri, Jan 22, 2016 at 5:13 PM, Tilman Bayer <tbayer(a)wikimedia.org> wrote:

...

+Analytics list so they can comment. I don't have such a script. It's a pretty intensive job to compile top articles especially over a month. The pageview API was supposed to have

top

articles per month per wiki but the job is so massive that it failed to

run

in Hive. Analytics knows there are better algorithms out there to solve this problem. So the pageview API just has top per day per wiki. I imagine that you are looking at some very specific wikis and

countries...

not all of them. Maybe someone on the list can make an example hive

script

(given a wiki and country) that gives the top for a day. On Wed, Jan 20, 2016 at 12:23 PM, Dan Foy <dfoy(a)wikimedia.org> wrote: > > Hi Kevin, > > In your collection of scripts for Hive, do you have one that can act as

> starting point for me to get the top N articles / URLs for Wikipedia in

country? Thanks, Dan

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Tilman Bayer

11:26 p.m.

New subject: top articles script

See also https://phabricator.wikimedia.org/T117945 and https://phabricator.wikimedia.org/T108867 for possibly related oddities in the top viewed pages. (And https://phabricator.wikimedia.org/T104755 : "Wikimedia's URL-routing logic straddles five layers ...") (switching CC to the intended Dan) On Fri, Jan 22, 2016 at 3:17 PM, Ryan Kaldari <rkaldari(a)wikimedia.org> wrote:

...

Any idea why the most popular article in India is "-"? CCing Dan Garry of Discovery team. On Fri, Jan 22, 2016 at 5:13 PM, Tilman Bayer <tbayer(a)wikimedia.org> wrote:

Hi Kevin, In your collection of scripts for Hive, do you have one that can act as a starting point for me to get the top N articles / URLs for Wikipedia in a country? Thanks, Dan

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

-- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB

Dan Garry

23 Jan 23 Jan

3:06 a.m.

New subject: top articles script

On 22 January 2016 at 15:17, Ryan Kaldari <rkaldari(a)wikimedia.org> wrote:

...

Any idea why the most popular article in India is "-"?

Oliver Keyes

4:32 a.m.

New subject: top articles script

Actually - is Hadoop's "nothing was provided in this field!" making it doubly confusing :/ On 22 January 2016 at 22:06, Dan Garry <dgarry(a)wikimedia.org> wrote:

...

On 22 January 2016 at 15:17, Ryan Kaldari <rkaldari(a)wikimedia.org> wrote:

Any idea why the most popular article in India is "-"?

That specific article often sees a lot of traffic. This is normally caused by a bot, spider, or other automaton. Unfortunately, by definition no method of detecting automated traffic is perfect, so things like this often slip through. Dan -- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

-- Oliver Keyes Count Logula Wikimedia Foundation

Toby Negrin

7:49 a.m.

New subject: top articles script

Is that a bug in the ETL? On Friday, January 22, 2016, Oliver Keyes <okeyes(a)wikimedia.org> wrote:

...

Actually - is Hadoop's "nothing was provided in this field!" making it doubly confusing :/ On 22 January 2016 at 22:06, Dan Garry <dgarry(a)wikimedia.org <javascript:;>> wrote:

On 22 January 2016 at 15:17, Ryan Kaldari <rkaldari(a)wikimedia.org

<javascript:;>> wrote:

Any idea why the most popular article in India is "-"?

That specific article often sees a lot of traffic. This is normally

caused

by a bot, spider, or other automaton. Unfortunately, by definition no

method

of detecting automated traffic is perfect, so things like this often slip through. Dan -- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org <javascript:;> https://lists.wikimedia.org/mailman/listinfo/analytics

-- Oliver Keyes Count Logula Wikimedia Foundation _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org <javascript:;> https://lists.wikimedia.org/mailman/listinfo/analytics

Dan Andreescu

11:36 a.m.

New subject: top articles script

Yes and no, it kind of depends whether we want to lose data. We've been talking about better ways to say "Unknown" but /wiki/Unknown is a page too :) We're just not focusing on this level of detail yet, bigger fish to fry, caveat emptor, etc. On Saturday, January 23, 2016, Toby Negrin <tnegrin(a)wikimedia.org> wrote:

...

Is that a bug in the ETL? On Friday, January 22, 2016, Oliver Keyes <okeyes(a)wikimedia.org <javascript:_e(%7B%7D,'cvml','okeyes@wikimedia.org');>> wrote:

Actually - is Hadoop's "nothing was provided in this field!" making it doubly confusing :/ On 22 January 2016 at 22:06, Dan Garry <dgarry(a)wikimedia.org> wrote:

On 22 January 2016 at 15:17, Ryan Kaldari <rkaldari(a)wikimedia.org>

wrote:

Any idea why the most popular article in India is "-"?

That specific article often sees a lot of traffic. This is normally

caused

by a bot, spider, or other automaton. Unfortunately, by definition no

method

of detecting automated traffic is perfect, so things like this often

slip

through. Dan -- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

-- Oliver Keyes Count Logula Wikimedia Foundation _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Toby Negrin

6:08 p.m.

New subject: top articles script

Thanks Dan -- I'm just concerned that we might be missing something (like the central notice banners back in the day) with a fairly large magnitude. -Toby On Sat, Jan 23, 2016 at 3:36 AM, Dan Andreescu <dandreescu(a)wikimedia.org> wrote:

...

Is that a bug in the ETL? On Friday, January 22, 2016, Oliver Keyes <okeyes(a)wikimedia.org> wrote:

Actually - is Hadoop's "nothing was provided in this field!" making it doubly confusing :/ On 22 January 2016 at 22:06, Dan Garry <dgarry(a)wikimedia.org> wrote:

On 22 January 2016 at 15:17, Ryan Kaldari <rkaldari(a)wikimedia.org>

wrote:

> > Any idea why the most popular article in India is "-"? That specific article often sees a lot of traffic. This is normally

caused

by a bot, spider, or other automaton. Unfortunately, by definition no

method

of detecting automated traffic is perfect, so things like this often

slip

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Oliver Keyes

8:55 p.m.

New subject: top articles script

+1. Could we look at the pageIDs rather than titles? Is that being passed through uniformly yet? On 23 January 2016 at 13:08, Toby Negrin <tnegrin(a)wikimedia.org> wrote:

...

Is that a bug in the ETL? On Friday, January 22, 2016, Oliver Keyes <okeyes(a)wikimedia.org> wrote: > > Actually - is Hadoop's "nothing was provided in this field!" making it > doubly confusing :/ > > On 22 January 2016 at 22:06, Dan Garry <dgarry(a)wikimedia.org> wrote: > > On 22 January 2016 at 15:17, Ryan Kaldari <rkaldari(a)wikimedia.org> > > wrote: > >> > >> Any idea why the most popular article in India is "-"? > > > > > > That specific article often sees a lot of traffic. This is normally > > caused > > by a bot, spider, or other automaton. Unfortunately, by definition no > > method > > of detecting automated traffic is perfect, so things like this often > > slip > > through. > > > > Dan > > > > -- > > Dan Garry > > Lead Product Manager, Discovery > > Wikimedia Foundation > > > > _______________________________________________ > > Analytics mailing list > > Analytics(a)lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/analytics > > > > > > -- > Oliver Keyes > Count Logula > Wikimedia Foundation > > _______________________________________________ > Analytics mailing list > Analytics(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/analytics

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

-- Oliver Keyes Count Logula Wikimedia Foundation

Dan Andreescu

24 Jan 24 Jan

4:04 a.m.

New subject: top articles script

Page Ids are still not coming in uniformly, but in this case should be there enough to figure out what - is, maybe. That's a good idea. On Saturday, January 23, 2016, Oliver Keyes <okeyes(a)wikimedia.org> wrote:

...

+1. Could we look at the pageIDs rather than titles? Is that being passed through uniformly yet? On 23 January 2016 at 13:08, Toby Negrin <tnegrin(a)wikimedia.org <javascript:;>> wrote:

Thanks Dan -- I'm just concerned that we might be missing something (like the central notice banners back in the day) with a fairly large

magnitude.

-Toby On Sat, Jan 23, 2016 at 3:36 AM, Dan Andreescu <dandreescu(a)wikimedia.org

<javascript:;>>

wrote: > > Yes and no, it kind of depends whether we want to lose data. We've been > talking about better ways to say "Unknown" but /wiki/Unknown is a page

too

> :) We're just not focusing on this level of detail yet, bigger fish to

fry,

> caveat emptor, etc. > > > On Saturday, January 23, 2016, Toby Negrin <tnegrin(a)wikimedia.org

<javascript:;>> wrote:

>> >> Is that a bug in the ETL? >> >> On Friday, January 22, 2016, Oliver Keyes <okeyes(a)wikimedia.org

<javascript:;>> wrote:

>>> >>> Actually - is Hadoop's "nothing was provided in this field!" making it >>> doubly confusing :/ >>> >>> On 22 January 2016 at 22:06, Dan Garry <dgarry(a)wikimedia.org

<javascript:;>> wrote:

>>> > On 22 January 2016 at 15:17, Ryan Kaldari <rkaldari(a)wikimedia.org

<javascript:;>>

>>> > wrote: >>> >> >>> >> Any idea why the most popular article in India is "-"? >>> > >>> > >>> > That specific article often sees a lot of traffic. This is normally >>> > caused >>> > by a bot, spider, or other automaton. Unfortunately, by definition

> > method > > of detecting automated traffic is perfect, so things like this often > > slip > > through. > > > > Dan > > > > -- > > Dan Garry > > Lead Product Manager, Discovery > > Wikimedia Foundation > > > > _______________________________________________ > > Analytics mailing list > > Analytics(a)lists.wikimedia.org <javascript:;> > > https://lists.wikimedia.org/mailman/listinfo/analytics > > > > > > -- > Oliver Keyes > Count Logula > Wikimedia Foundation > > _______________________________________________ > Analytics mailing list > Analytics(a)lists.wikimedia.org <javascript:;> > https://lists.wikimedia.org/mailman/listinfo/analytics

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org <javascript:;> https://lists.wikimedia.org/mailman/listinfo/analytics

3020

days inactive

3024

days old

analytics@lists.wikimedia.org

Manage subscription

10 comments

7 participants

tags (0)

participants (7)

Dan Andreescu
Dan Garry
Kevin Leduc
Oliver Keyes
Ryan Kaldari
Tilman Bayer
Toby Negrin