Hi Amit,
This problem has been rectified in the immediate sense and is being worked
on in the longer term sense. As with all our operations, resources are
stretched thin, and so there are various issues with data collection, some
more extreme and difficult than others.
This one is major and should not happen again. The problem was that logging
on one of the Varnish servers (there are two) had ceased, and so roughly
half the data was missing. The logging has been turned back on and
monitoring solutions are being put in place. Yes, the monitoring solutions
should have been there but this is what happens when resources are
stretched thin.
Phil
On Mon, Dec 5, 2011 at 2:37 PM, Amit Kapoor <akapoor(a)wikimedia.org> wrote:
Does anyone have an update if we can get this report
fixed for mobile page
views? some people outside mobile team have noticed the (erroneous)
downward trend and are asking, so we need to fix before it spreads...
On Tue, Nov 29, 2011 at 11:18 AM, Asher Feldman <afeldman(a)wikimedia.org>wrote;wrote:
You can be fairly confident at having crossed 1
billion in October and
being at 1.3+ billion for November to date.
On Tue, Nov 29, 2011 at 11:14 AM, Tomasz Finc <tfinc(a)wikimedia.org>wrote;wrote:
We at the very least have a reporting issue. I
want all of us to be on
top of these stats so that we can make our goals. We should be checking in
on this regularly.
--tomasz
On Tue, Nov 29, 2011 at 11:11 AM, Kul Wadhwa <kwadhwa(a)wikimedia.org>wrote;wrote:
So basically you're saying that this is
"just" a reporting issue and
not a mobile user related issue, right?
On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc <tfinc(a)wikimedia.org>
wrote:
> I wanted to alert us to an alarming trend in our overall mobile page
stats
>
>
http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
>
> We've been trending down over the last couple of months. This is not
good.
> We need these stats to be growing not dropping. If any of us see any
reason
> about why this may be happening then we need to speak up. I did some
digging
> and found out that we may have another logging issue.
>
> Asher from our ops team informed me that
>
> binasher: bad… i need to do something like make puppet monitor and
ensure
> that our customized varnish logging daemon is always running
>
> binasher: it died on one of the varnish servers, but not the other
>
> Which would account for the massive dips we are seeing. We can
confirm that
> the logger died on 10/15 and that we've been under reporting for
October and
> November. October is likely under reported by up to 25% while
November is
> under reported by up to 50%.
>
> Now i'm happy that we have a simple operations related reason for
this dip
> but we should have caught it sooner. With the foundations goal of 2
billion
> page views by July we have to stay on top of this.
>
> --tomasz
>
> _______________________________________________
> Mobile-l mailing list
> Mobile-l(a)lists.wikimedia.org
>
https://lists.wikimedia.org/mailman/listinfo/mobile-l
>
--
Kul Wadhwa
Wikimedia Foundation
+1-415-839-6885 ext. 6603
_______________________________________________
Mobile-l mailing list
Mobile-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l
_______________________________________________
Mobile-l mailing list
Mobile-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l
--
Amit Kapoor
Mobile Partnerships
Wikimedia Foundation
_______________________________________________
Mobile-l mailing list
Mobile-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mobile-l
--
Phil Inje Chang
Product Manager, Mobile
Wikimedia Foundation
415-812-0854 m
415-882-7982 x 6810