I wanted to alert us to an alarming trend in our overall mobile page stats
http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We've been trending down over the last couple of months. This is not good. We need these stats to be growing not dropping. If any of us see any reason about why this may be happening then we need to speak up. I did some digging and found out that we may have another logging issue.
Asher from our ops team informed me that
*binasher member:binasher: *bad… i need to do something like make puppet monitor and ensure that our customized varnish logging daemon is always running
*binasher member:binasher: *it died on one of the varnish servers, but not the other Which would account for the massive dips we are seeing. We can confirm that the logger died on 10/15 and that we've been under reporting for October and November. October is likely under reported by up to 25% while November is under reported by up to 50%.
Now i'm happy that we have a simple operations related reason for this dip but we should have caught it sooner. With the foundations goal of 2 billion page views by July we have to stay on top of this.
--tomasz