On Mon, Dec 5, 2011 at 9:26 PM, Philip Chang <pchang@wikimedia.org> wrote:

Hi Amit,

This problem has been rectified in the immediate sense and is being worked on in the longer term sense. As with all our operations, resources are stretched thin, and so there are various issues with data collection, some more extreme and difficult than others.

This one is major and should not happen again. The problem was that logging on one of the Varnish servers (there are two) had ceased, and so roughly half the data was missing. The logging has been turned back on and monitoring solutions are being put in place. Yes, the monitoring solutions should have been there but this is what happens when resources are stretched thin.

Phil

On Mon, Dec 5, 2011 at 2:37 PM, Amit Kapoor <akapoor@wikimedia.org> wrote:

Does anyone have an update if we can get this report fixed for mobile page views? some people outside mobile team have noticed the (erroneous) downward trend and are asking, so we need to fix before it spreads...

On Tue, Nov 29, 2011 at 11:18 AM, Asher Feldman <afeldman@wikimedia.org> wrote:

You can be fairly confident at having crossed 1 billion in October and being at 1.3+ billion for November to date.

On Tue, Nov 29, 2011 at 11:14 AM, Tomasz Finc <tfinc@wikimedia.org> wrote:

We at the very least have a reporting issue. I want all of us to be on top of these stats so that we can make our goals. We should be checking in on this regularly.

--tomasz

On Tue, Nov 29, 2011 at 11:11 AM, Kul Wadhwa <kwadhwa@wikimedia.org> wrote:
So basically you're saying that this is "just" a reporting issue and
not a mobile user related issue, right?

On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc <tfinc@wikimedia.org> wrote:
> I wanted to alert us to an alarming trend in our overall mobile page stats
>
> http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
>
> We've been trending down over the last couple of months. This is not good.
> We need these stats to be growing not dropping. If any of us see any reason
> about why this may be happening then we need to speak up. I did some digging
> and found out that we may have another logging issue.
>
> Asher from our ops team informed me that
>
> binasher: bad… i need to do something like make puppet monitor and ensure
> that our customized varnish logging daemon is always running
>
> binasher: it died on one of the varnish servers, but not the other
>
> Which would account for the massive dips we are seeing. We can confirm that
> the logger died on 10/15 and that we've been under reporting for October and
> November. October is likely under reported by up to 25% while November is
> under reported by up to 50%.
>
> Now i'm happy that we have a simple operations related reason for this dip
> but we should have caught it sooner. With the foundations goal of 2 billion
> page views by July we have to stay on top of this.
>
> --tomasz
>

> _______________________________________________
> Mobile-l mailing list
> Mobile-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>

--
Kul Wadhwa
Wikimedia Foundation
+1-415-839-6885 ext. 6603

_______________________________________________
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

_______________________________________________
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

--
Amit Kapoor
Mobile Partnerships
Wikimedia Foundation

_______________________________________________
Mobile-l mailing list
Mobile-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l

--
Phil Inje Chang
Product Manager, Mobile
Wikimedia Foundation
415-812-0854 m
415-882-7982 x 6810