I wanted to alert us to an alarming trend in our overall mobile page stats
http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We've been trending down over the last couple of months. This is not good. We need these stats to be growing not dropping. If any of us see any reason about why this may be happening then we need to speak up. I did some digging and found out that we may have another logging issue.
Asher from our ops team informed me that
*binasher member:binasher: *bad… i need to do something like make puppet monitor and ensure that our customized varnish logging daemon is always running
*binasher member:binasher: *it died on one of the varnish servers, but not the other Which would account for the massive dips we are seeing. We can confirm that the logger died on 10/15 and that we've been under reporting for October and November. October is likely under reported by up to 25% while November is under reported by up to 50%.
Now i'm happy that we have a simple operations related reason for this dip but we should have caught it sooner. With the foundations goal of 2 billion page views by July we have to stay on top of this.
--tomasz
So basically you're saying that this is "just" a reporting issue and not a mobile user related issue, right?
On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc tfinc@wikimedia.org wrote:
I wanted to alert us to an alarming trend in our overall mobile page stats
http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We've been trending down over the last couple of months. This is not good. We need these stats to be growing not dropping. If any of us see any reason about why this may be happening then we need to speak up. I did some digging and found out that we may have another logging issue.
Asher from our ops team informed me that
binasher: bad… i need to do something like make puppet monitor and ensure that our customized varnish logging daemon is always running
binasher: it died on one of the varnish servers, but not the other
Which would account for the massive dips we are seeing. We can confirm that the logger died on 10/15 and that we've been under reporting for October and November. October is likely under reported by up to 25% while November is under reported by up to 50%.
Now i'm happy that we have a simple operations related reason for this dip but we should have caught it sooner. With the foundations goal of 2 billion page views by July we have to stay on top of this.
--tomasz
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
We at the very least have a reporting issue. I want all of us to be on top of these stats so that we can make our goals. We should be checking in on this regularly.
--tomasz
On Tue, Nov 29, 2011 at 11:11 AM, Kul Wadhwa kwadhwa@wikimedia.org wrote:
So basically you're saying that this is "just" a reporting issue and not a mobile user related issue, right?
On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc tfinc@wikimedia.org wrote:
I wanted to alert us to an alarming trend in our overall mobile page
stats
http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We've been trending down over the last couple of months. This is not
good.
We need these stats to be growing not dropping. If any of us see any
reason
about why this may be happening then we need to speak up. I did some
digging
and found out that we may have another logging issue.
Asher from our ops team informed me that
binasher: bad… i need to do something like make puppet monitor and ensure that our customized varnish logging daemon is always running
binasher: it died on one of the varnish servers, but not the other
Which would account for the massive dips we are seeing. We can confirm
that
the logger died on 10/15 and that we've been under reporting for October
and
November. October is likely under reported by up to 25% while November is under reported by up to 50%.
Now i'm happy that we have a simple operations related reason for this
dip
but we should have caught it sooner. With the foundations goal of 2
billion
page views by July we have to stay on top of this.
--tomasz
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Kul Wadhwa Wikimedia Foundation +1-415-839-6885 ext. 6603
You can be fairly confident at having crossed 1 billion in October and being at 1.3+ billion for November to date.
On Tue, Nov 29, 2011 at 11:14 AM, Tomasz Finc tfinc@wikimedia.org wrote:
We at the very least have a reporting issue. I want all of us to be on top of these stats so that we can make our goals. We should be checking in on this regularly.
--tomasz
On Tue, Nov 29, 2011 at 11:11 AM, Kul Wadhwa kwadhwa@wikimedia.orgwrote:
So basically you're saying that this is "just" a reporting issue and not a mobile user related issue, right?
On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc tfinc@wikimedia.org wrote:
I wanted to alert us to an alarming trend in our overall mobile page
stats
http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We've been trending down over the last couple of months. This is not
good.
We need these stats to be growing not dropping. If any of us see any
reason
about why this may be happening then we need to speak up. I did some
digging
and found out that we may have another logging issue.
Asher from our ops team informed me that
binasher: bad… i need to do something like make puppet monitor and
ensure
that our customized varnish logging daemon is always running
binasher: it died on one of the varnish servers, but not the other
Which would account for the massive dips we are seeing. We can confirm
that
the logger died on 10/15 and that we've been under reporting for
October and
November. October is likely under reported by up to 25% while November
is
under reported by up to 50%.
Now i'm happy that we have a simple operations related reason for this
dip
but we should have caught it sooner. With the foundations goal of 2
billion
page views by July we have to stay on top of this.
--tomasz
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Kul Wadhwa Wikimedia Foundation +1-415-839-6885 ext. 6603
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Does anyone have an update if we can get this report fixed for mobile page views? some people outside mobile team have noticed the (erroneous) downward trend and are asking, so we need to fix before it spreads...
On Tue, Nov 29, 2011 at 11:18 AM, Asher Feldman afeldman@wikimedia.orgwrote:
You can be fairly confident at having crossed 1 billion in October and being at 1.3+ billion for November to date.
On Tue, Nov 29, 2011 at 11:14 AM, Tomasz Finc tfinc@wikimedia.org wrote:
We at the very least have a reporting issue. I want all of us to be on top of these stats so that we can make our goals. We should be checking in on this regularly.
--tomasz
On Tue, Nov 29, 2011 at 11:11 AM, Kul Wadhwa kwadhwa@wikimedia.orgwrote:
So basically you're saying that this is "just" a reporting issue and not a mobile user related issue, right?
On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc tfinc@wikimedia.org wrote:
I wanted to alert us to an alarming trend in our overall mobile page
stats
http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We've been trending down over the last couple of months. This is not
good.
We need these stats to be growing not dropping. If any of us see any
reason
about why this may be happening then we need to speak up. I did some
digging
and found out that we may have another logging issue.
Asher from our ops team informed me that
binasher: bad… i need to do something like make puppet monitor and
ensure
that our customized varnish logging daemon is always running
binasher: it died on one of the varnish servers, but not the other
Which would account for the massive dips we are seeing. We can confirm
that
the logger died on 10/15 and that we've been under reporting for
October and
November. October is likely under reported by up to 25% while November
is
under reported by up to 50%.
Now i'm happy that we have a simple operations related reason for this
dip
but we should have caught it sooner. With the foundations goal of 2
billion
page views by July we have to stay on top of this.
--tomasz
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Kul Wadhwa Wikimedia Foundation +1-415-839-6885 ext. 6603
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Hi Amit,
This problem has been rectified in the immediate sense and is being worked on in the longer term sense. As with all our operations, resources are stretched thin, and so there are various issues with data collection, some more extreme and difficult than others.
This one is major and should not happen again. The problem was that logging on one of the Varnish servers (there are two) had ceased, and so roughly half the data was missing. The logging has been turned back on and monitoring solutions are being put in place. Yes, the monitoring solutions should have been there but this is what happens when resources are stretched thin.
Phil
On Mon, Dec 5, 2011 at 2:37 PM, Amit Kapoor akapoor@wikimedia.org wrote:
Does anyone have an update if we can get this report fixed for mobile page views? some people outside mobile team have noticed the (erroneous) downward trend and are asking, so we need to fix before it spreads...
On Tue, Nov 29, 2011 at 11:18 AM, Asher Feldman afeldman@wikimedia.orgwrote:
You can be fairly confident at having crossed 1 billion in October and being at 1.3+ billion for November to date.
On Tue, Nov 29, 2011 at 11:14 AM, Tomasz Finc tfinc@wikimedia.orgwrote:
We at the very least have a reporting issue. I want all of us to be on top of these stats so that we can make our goals. We should be checking in on this regularly.
--tomasz
On Tue, Nov 29, 2011 at 11:11 AM, Kul Wadhwa kwadhwa@wikimedia.orgwrote:
So basically you're saying that this is "just" a reporting issue and not a mobile user related issue, right?
On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc tfinc@wikimedia.org wrote:
I wanted to alert us to an alarming trend in our overall mobile page
stats
http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We've been trending down over the last couple of months. This is not
good.
We need these stats to be growing not dropping. If any of us see any
reason
about why this may be happening then we need to speak up. I did some
digging
and found out that we may have another logging issue.
Asher from our ops team informed me that
binasher: bad… i need to do something like make puppet monitor and
ensure
that our customized varnish logging daemon is always running
binasher: it died on one of the varnish servers, but not the other
Which would account for the massive dips we are seeing. We can
confirm that
the logger died on 10/15 and that we've been under reporting for
October and
November. October is likely under reported by up to 25% while
November is
under reported by up to 50%.
Now i'm happy that we have a simple operations related reason for
this dip
but we should have caught it sooner. With the foundations goal of 2
billion
page views by July we have to stay on top of this.
--tomasz
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Kul Wadhwa Wikimedia Foundation +1-415-839-6885 ext. 6603
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Amit Kapoor Mobile Partnerships Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Great, thanks Phil. Looks like since it's been fixed we're trending to 1.5 B page views in December -- really exciting.
Do you know if the stats from Oct/Nov will forever be off due to the logging issue -- or is there a way to recover the data?
On Mon, Dec 5, 2011 at 9:26 PM, Philip Chang pchang@wikimedia.org wrote:
Hi Amit,
This problem has been rectified in the immediate sense and is being worked on in the longer term sense. As with all our operations, resources are stretched thin, and so there are various issues with data collection, some more extreme and difficult than others.
This one is major and should not happen again. The problem was that logging on one of the Varnish servers (there are two) had ceased, and so roughly half the data was missing. The logging has been turned back on and monitoring solutions are being put in place. Yes, the monitoring solutions should have been there but this is what happens when resources are stretched thin.
Phil
On Mon, Dec 5, 2011 at 2:37 PM, Amit Kapoor akapoor@wikimedia.org wrote:
Does anyone have an update if we can get this report fixed for mobile page views? some people outside mobile team have noticed the (erroneous) downward trend and are asking, so we need to fix before it spreads...
On Tue, Nov 29, 2011 at 11:18 AM, Asher Feldman afeldman@wikimedia.orgwrote:
You can be fairly confident at having crossed 1 billion in October and being at 1.3+ billion for November to date.
On Tue, Nov 29, 2011 at 11:14 AM, Tomasz Finc tfinc@wikimedia.orgwrote:
We at the very least have a reporting issue. I want all of us to be on top of these stats so that we can make our goals. We should be checking in on this regularly.
--tomasz
On Tue, Nov 29, 2011 at 11:11 AM, Kul Wadhwa kwadhwa@wikimedia.orgwrote:
So basically you're saying that this is "just" a reporting issue and not a mobile user related issue, right?
On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc tfinc@wikimedia.org wrote:
I wanted to alert us to an alarming trend in our overall mobile page
stats
http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm
We've been trending down over the last couple of months. This is not
good.
We need these stats to be growing not dropping. If any of us see any
reason
about why this may be happening then we need to speak up. I did some
digging
and found out that we may have another logging issue.
Asher from our ops team informed me that
binasher: bad… i need to do something like make puppet monitor and
ensure
that our customized varnish logging daemon is always running
binasher: it died on one of the varnish servers, but not the other
Which would account for the massive dips we are seeing. We can
confirm that
the logger died on 10/15 and that we've been under reporting for
October and
November. October is likely under reported by up to 25% while
November is
under reported by up to 50%.
Now i'm happy that we have a simple operations related reason for
this dip
but we should have caught it sooner. With the foundations goal of 2
billion
page views by July we have to stay on top of this.
--tomasz
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Kul Wadhwa Wikimedia Foundation +1-415-839-6885 ext. 6603
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Amit Kapoor Mobile Partnerships Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Phil Inje Chang Product Manager, Mobile Wikimedia Foundation 415-812-0854 m 415-882-7982 x 6810
We know what percentage it's off by with a high level of confidence, so I think it's just a matter of manually updating the stats report.
On Tue, Dec 6, 2011 at 10:20 AM, Amit Kapoor akapoor@wikimedia.org wrote:
Great, thanks Phil. Looks like since it's been fixed we're trending to 1.5 B page views in December -- really exciting.
Do you know if the stats from Oct/Nov will forever be off due to the logging issue -- or is there a way to recover the data?
On Mon, Dec 5, 2011 at 9:26 PM, Philip Chang pchang@wikimedia.org wrote:
Hi Amit,
This problem has been rectified in the immediate sense and is being worked on in the longer term sense. As with all our operations, resources are stretched thin, and so there are various issues with data collection, some more extreme and difficult than others.
This one is major and should not happen again. The problem was that logging on one of the Varnish servers (there are two) had ceased, and so roughly half the data was missing. The logging has been turned back on and monitoring solutions are being put in place. Yes, the monitoring solutions should have been there but this is what happens when resources are stretched thin.
Phil
On Mon, Dec 5, 2011 at 2:37 PM, Amit Kapoor akapoor@wikimedia.orgwrote:
Does anyone have an update if we can get this report fixed for mobile page views? some people outside mobile team have noticed the (erroneous) downward trend and are asking, so we need to fix before it spreads...
On Tue, Nov 29, 2011 at 11:18 AM, Asher Feldman afeldman@wikimedia.orgwrote:
You can be fairly confident at having crossed 1 billion in October and being at 1.3+ billion for November to date.
On Tue, Nov 29, 2011 at 11:14 AM, Tomasz Finc tfinc@wikimedia.orgwrote:
We at the very least have a reporting issue. I want all of us to be on top of these stats so that we can make our goals. We should be checking in on this regularly.
--tomasz
On Tue, Nov 29, 2011 at 11:11 AM, Kul Wadhwa kwadhwa@wikimedia.orgwrote:
So basically you're saying that this is "just" a reporting issue and not a mobile user related issue, right?
On Tue, Nov 29, 2011 at 7:08 PM, Tomasz Finc tfinc@wikimedia.org wrote: > I wanted to alert us to an alarming trend in our overall mobile page stats > > http://stats.wikimedia.org/EN/TablesPageViewsMonthlyMobile.htm > > We've been trending down over the last couple of months. This is not good. > We need these stats to be growing not dropping. If any of us see any reason > about why this may be happening then we need to speak up. I did some digging > and found out that we may have another logging issue. > > Asher from our ops team informed me that > > binasher: bad… i need to do something like make puppet monitor and ensure > that our customized varnish logging daemon is always running > > binasher: it died on one of the varnish servers, but not the other > > Which would account for the massive dips we are seeing. We can confirm that > the logger died on 10/15 and that we've been under reporting for October and > November. October is likely under reported by up to 25% while November is > under reported by up to 50%. > > Now i'm happy that we have a simple operations related reason for this dip > but we should have caught it sooner. With the foundations goal of 2 billion > page views by July we have to stay on top of this. > > --tomasz > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l >
-- Kul Wadhwa Wikimedia Foundation +1-415-839-6885 ext. 6603
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Amit Kapoor Mobile Partnerships Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Phil Inje Chang Product Manager, Mobile Wikimedia Foundation 415-812-0854 m 415-882-7982 x 6810
-- Amit Kapoor Mobile Partnerships Wikimedia Foundation