Were trending down on our page stats

List overview All Threads
Download

newer

older

Keeping up with changes to the...

Full screen search is out in beta

Tomasz Finc

29 Nov 2011 29 Nov '11

8:08 p.m.

I wanted to alert us to an alarming trend in our overall mobile page stats http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm We've been trending down over the last couple of months. This is not good. We need these stats to be growing not dropping. If any of us see any reason about why this may be happening then we need to speak up. I did some digging and found out that we may have another logging issue. Asher from our ops team informed me that *binasher <member:binasher>: *bad… i need to do something like make puppet monitor and ensure that our customized varnish logging daemon is always running *binasher <member:binasher>: *it died on one of the varnish servers, but not the other Which would account for the massive dips we are seeing. We can confirm that the logger died on 10/15 and that we've been under reporting for October and November. October is likely under reported by up to 25% while November is under reported by up to 50%. Now i'm happy that we have a simple operations related reason for this dip but we should have caught it sooner. With the foundations goal of 2 billion page views by July we have to stay on top of this. --tomasz

Attachments:

attachment.htm (text/html — 1.5 KB)

Show replies by date

Kul Wadhwa

29 Nov 29 Nov

8:11 p.m.

So basically you're saying that this is "just" a reporting issue and not a mobile user related issue, right? On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc <tfinc(a)wikimedia.org> wrote:

...

I wanted to alert us to an alarming trend in our overall mobile page stats http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm We've been trending down over the last couple of months. This is not good. We need these stats to be growing not dropping. If any of us see any reason about why this may be happening then we need to speak up. I did some digging and found out that we may have another logging issue. Asher from our ops team informed me that binasher: bad… i need to do something like make puppet monitor and ensure that our customized varnish logging daemon is always running binasher: it died on one of the varnish servers, but not the other Which would account for the massive dips we are seeing. We can confirm that the logger died on 10/15 and that we've been under reporting for October and November. October is likely under reported by up to 25% while November is under reported by up to 50%. Now i'm happy that we have a simple operations related reason for this dip but we should have caught it sooner. With the foundations goal of 2 billion page views by July we have to stay on top of this. --tomasz _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

-- Kul Wadhwa Wikimedia Foundation +1-415-839-6885 ext. 6603

Tomasz Finc

8:14 p.m.

...

So basically you're saying that this is "just" a reporting issue and not a mobile user related issue, right? On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc <tfinc(a)wikimedia.org> wrote:

I wanted to alert us to an alarming trend in our overall mobile page

stats

http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm We've been trending down over the last couple of months. This is not

good.

We need these stats to be growing not dropping. If any of us see any

reason

about why this may be happening then we need to speak up. I did some

digging

and found out that we may have another logging issue. Asher from our ops team informed me that binasher: bad… i need to do something like make puppet monitor and ensure that our customized varnish logging daemon is always running binasher: it died on one of the varnish servers, but not the other Which would account for the massive dips we are seeing. We can confirm

that

the logger died on 10/15 and that we've been under reporting for October

and

November. October is likely under reported by up to 25% while November is under reported by up to 50%. Now i'm happy that we have a simple operations related reason for this

dip

but we should have caught it sooner. With the foundations goal of 2

billion

page views by July we have to stay on top of this. --tomasz _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

-- Kul Wadhwa Wikimedia Foundation +1-415-839-6885 ext. 6603

Asher Feldman

8:18 p.m.

You can be fairly confident at having crossed 1 billion in October and being at 1.3+ billion for November to date. On Tue, Nov 29, 2011 at 11:14 AM, Tomasz Finc <tfinc(a)wikimedia.org> wrote:

...

So basically you're saying that this is "just" a reporting issue and not a mobile user related issue, right? On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc <tfinc(a)wikimedia.org> wrote:

I wanted to alert us to an alarming trend in our overall mobile page

stats

http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm We've been trending down over the last couple of months. This is not

good.

We need these stats to be growing not dropping. If any of us see any

reason

about why this may be happening then we need to speak up. I did some

digging

and found out that we may have another logging issue. Asher from our ops team informed me that binasher: bad… i need to do something like make puppet monitor and

ensure

that our customized varnish logging daemon is always running binasher: it died on one of the varnish servers, but not the other Which would account for the massive dips we are seeing. We can confirm

that

the logger died on 10/15 and that we've been under reporting for

October and

November. October is likely under reported by up to 25% while November

under reported by up to 50%. Now i'm happy that we have a simple operations related reason for this

dip

but we should have caught it sooner. With the foundations goal of 2

billion

-- Kul Wadhwa Wikimedia Foundation +1-415-839-6885 ext. 6603

_______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

Amit Kapoor

5 Dec 5 Dec

11:37 p.m.

Does anyone have an update if we can get this report fixed for mobile page views? some people outside mobile team have noticed the (erroneous) downward trend and are asking, so we need to fix before it spreads... On Tue, Nov 29, 2011 at 11:18 AM, Asher Feldman <afeldman(a)wikimedia.org>wrote;wrote:

...

You can be fairly confident at having crossed 1 billion in October and being at 1.3+ billion for November to date. On Tue, Nov 29, 2011 at 11:14 AM, Tomasz Finc <tfinc(a)wikimedia.org> wrote:

So basically you're saying that this is "just" a reporting issue and not a mobile user related issue, right? On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc <tfinc(a)wikimedia.org> wrote:

I wanted to alert us to an alarming trend in our overall mobile page

stats

http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm We've been trending down over the last couple of months. This is not

good.

We need these stats to be growing not dropping. If any of us see any

reason

about why this may be happening then we need to speak up. I did some

digging

and found out that we may have another logging issue. Asher from our ops team informed me that binasher: bad… i need to do something like make puppet monitor and

ensure

that our customized varnish logging daemon is always running binasher: it died on one of the varnish servers, but not the other Which would account for the massive dips we are seeing. We can confirm

that

the logger died on 10/15 and that we've been under reporting for

October and

November. October is likely under reported by up to 25% while November

under reported by up to 50%. Now i'm happy that we have a simple operations related reason for this

dip

but we should have caught it sooner. With the foundations goal of 2

billion

-- Kul Wadhwa Wikimedia Foundation +1-415-839-6885 ext. 6603

_______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

-- Amit Kapoor Mobile Partnerships Wikimedia Foundation

Philip Chang

6 Dec 6 Dec

6:26 a.m.

Hi Amit, This problem has been rectified in the immediate sense and is being worked on in the longer term sense. As with all our operations, resources are stretched thin, and so there are various issues with data collection, some more extreme and difficult than others. This one is major and should not happen again. The problem was that logging on one of the Varnish servers (there are two) had ceased, and so roughly half the data was missing. The logging has been turned back on and monitoring solutions are being put in place. Yes, the monitoring solutions should have been there but this is what happens when resources are stretched thin. Phil On Mon, Dec 5, 2011 at 2:37 PM, Amit Kapoor <akapoor(a)wikimedia.org> wrote:

...

You can be fairly confident at having crossed 1 billion in October and being at 1.3+ billion for November to date. On Tue, Nov 29, 2011 at 11:14 AM, Tomasz Finc <tfinc(a)wikimedia.org>wrote;wrote:

So basically you're saying that this is "just" a reporting issue and not a mobile user related issue, right? On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc <tfinc(a)wikimedia.org> wrote: > I wanted to alert us to an alarming trend in our overall mobile page stats > > http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm > > We've been trending down over the last couple of months. This is not good. > We need these stats to be growing not dropping. If any of us see any reason > about why this may be happening then we need to speak up. I did some digging > and found out that we may have another logging issue. > > Asher from our ops team informed me that > > binasher: bad… i need to do something like make puppet monitor and ensure > that our customized varnish logging daemon is always running > > binasher: it died on one of the varnish servers, but not the other > > Which would account for the massive dips we are seeing. We can confirm that > the logger died on 10/15 and that we've been under reporting for October and > November. October is likely under reported by up to 25% while November is > under reported by up to 50%. > > Now i'm happy that we have a simple operations related reason for this dip > but we should have caught it sooner. With the foundations goal of 2 billion > page views by July we have to stay on top of this. > > --tomasz > > _______________________________________________ > Mobile-l mailing list > Mobile-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > -- Kul Wadhwa Wikimedia Foundation +1-415-839-6885 ext. 6603

_______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

-- Amit Kapoor Mobile Partnerships Wikimedia Foundation _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

-- Phil Inje Chang Product Manager, Mobile Wikimedia Foundation 415-812-0854 m 415-882-7982 x 6810

Amit Kapoor

7:20 p.m.

Great, thanks Phil. Looks like since it's been fixed we're trending to 1.5 B page views in December -- really exciting. Do you know if the stats from Oct/Nov will forever be off due to the logging issue -- or is there a way to recover the data? On Mon, Dec 5, 2011 at 9:26 PM, Philip Chang <pchang(a)wikimedia.org> wrote:

...

You can be fairly confident at having crossed 1 billion in October and being at 1.3+ billion for November to date. On Tue, Nov 29, 2011 at 11:14 AM, Tomasz Finc <tfinc(a)wikimedia.org>wrote;wrote:

We at the very least have a reporting issue. I want all of us to be on top of these stats so that we can make our goals. We should be checking in on this regularly. --tomasz On Tue, Nov 29, 2011 at 11:11 AM, Kul Wadhwa <kwadhwa(a)wikimedia.org>wrote;wrote: > So basically you're saying that this is "just" a reporting issue and > not a mobile user related issue, right? > > On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc <tfinc(a)wikimedia.org> > wrote: > > I wanted to alert us to an alarming trend in our overall mobile page > stats > > > > http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm > > > > We've been trending down over the last couple of months. This is not > good. > > We need these stats to be growing not dropping. If any of us see any > reason > > about why this may be happening then we need to speak up. I did some > digging > > and found out that we may have another logging issue. > > > > Asher from our ops team informed me that > > > > binasher: bad… i need to do something like make puppet monitor and > ensure > > that our customized varnish logging daemon is always running > > > > binasher: it died on one of the varnish servers, but not the other > > > > Which would account for the massive dips we are seeing. We can > confirm that > > the logger died on 10/15 and that we've been under reporting for > October and > > November. October is likely under reported by up to 25% while > November is > > under reported by up to 50%. > > > > Now i'm happy that we have a simple operations related reason for > this dip > > but we should have caught it sooner. With the foundations goal of 2 > billion > > page views by July we have to stay on top of this. > > > > --tomasz > > > > _______________________________________________ > > Mobile-l mailing list > > Mobile-l(a)lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > > > > > -- > Kul Wadhwa > Wikimedia Foundation > +1-415-839-6885 ext. 6603 > _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

_______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

-- Phil Inje Chang Product Manager, Mobile Wikimedia Foundation 415-812-0854 m 415-882-7982 x 6810

-- Amit Kapoor Mobile Partnerships Wikimedia Foundation

Asher Feldman

8:15 p.m.

We know what percentage it's off by with a high level of confidence, so I think it's just a matter of manually updating the stats report. On Tue, Dec 6, 2011 at 10:20 AM, Amit Kapoor <akapoor(a)wikimedia.org> wrote:

...

You can be fairly confident at having crossed 1 billion in October and being at 1.3+ billion for November to date. On Tue, Nov 29, 2011 at 11:14 AM, Tomasz Finc <tfinc(a)wikimedia.org>wrote;wrote: > We at the very least have a reporting issue. I want all of us to be on > top of these stats so that we can make our goals. We should be checking in > on this regularly. > > --tomasz > > > > On Tue, Nov 29, 2011 at 11:11 AM, Kul Wadhwa <kwadhwa(a)wikimedia.org>wrote;wrote: > >> So basically you're saying that this is "just" a reporting issue and >> not a mobile user related issue, right? >> >> On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc <tfinc(a)wikimedia.org> >> wrote: >> > I wanted to alert us to an alarming trend in our overall mobile >> page stats >> > >> > http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm >> > >> > We've been trending down over the last couple of months. This is >> not good. >> > We need these stats to be growing not dropping. If any of us see >> any reason >> > about why this may be happening then we need to speak up. I did >> some digging >> > and found out that we may have another logging issue. >> > >> > Asher from our ops team informed me that >> > >> > binasher: bad… i need to do something like make puppet monitor and >> ensure >> > that our customized varnish logging daemon is always running >> > >> > binasher: it died on one of the varnish servers, but not the other >> > >> > Which would account for the massive dips we are seeing. We can >> confirm that >> > the logger died on 10/15 and that we've been under reporting for >> October and >> > November. October is likely under reported by up to 25% while >> November is >> > under reported by up to 50%. >> > >> > Now i'm happy that we have a simple operations related reason for >> this dip >> > but we should have caught it sooner. With the foundations goal of 2 >> billion >> > page views by July we have to stay on top of this. >> > >> > --tomasz >> > >> > _______________________________________________ >> > Mobile-l mailing list >> > Mobile-l(a)lists.wikimedia.org >> > https://lists.wikimedia.org/mailman/listinfo/mobile-l >> > >> >> >> >> -- >> Kul Wadhwa >> Wikimedia Foundation >> +1-415-839-6885 ext. 6603 >> > > > _______________________________________________ > Mobile-l mailing list > Mobile-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

-- Phil Inje Chang Product Manager, Mobile Wikimedia Foundation 415-812-0854 m 415-882-7982 x 6810

-- Amit Kapoor Mobile Partnerships Wikimedia Foundation

4518

days inactive

4525

days old

mobile-l@lists.wikimedia.org

Manage subscription

7 comments

5 participants

tags (0)

participants (5)

Amit Kapoor
Asher Feldman
Kul Wadhwa
Philip Chang
Tomasz Finc