It's true that we're hoping to re-focus on Limn. It hasn't seen any work since last March. However, I think this issue has little to do with Limn and more with how we're generating data for it. I would love to talk about how we're currently generating this data and how we can improve, I've been thinking a lot about the problem and I think Toby is making solutions to it an upcoming priority.
On Wed, Nov 13, 2013 at 8:00 PM, Toby Negrin tnegrin@wikimedia.org wrote:
Hi all --
Limn is indeed the child of the Analytics team and we are happy to work with you to resolve these issues. Always feel free to reach out to this list if you have any questions/issues.
A couple of specific comments -- right now Limn will graph what you tell it to so while we can certainly help optimize the processing, the scripts themselves are custom to the data being graphed.
We have hired an engineer to focus on Limn specifically who will start in December. We're in the process of putting together some requirements and use cases for her to work on and I will circulate this broadly once I have incorporated feedback from our quarterly reviews.
We'll also reach out to Mobile directly.
-Toby
On Wed, Nov 13, 2013 at 4:14 PM, Arthur Richards arichards@wikimedia.orgwrote:
+analytics
On Wed, Nov 13, 2013 at 5:09 PM, Jon Robson jrobson@wikimedia.orgwrote:
Thanks so much Juliusz for exploring this and great work fixing the schema (apologies for me not predicting that might be an issue) and sorry for all the pain this must have caused you.
We can't be the only teams using Limn in the Foundation. It might be worth pulling everyone together. Am I right in thinking that Limn is a child of the analytics team? Maybe we should at least spend some with them getting our use case resolved.. I guess this is why we have an analytics department? I can raise this issue in the next Scrum of Scrums if it is not resolved by then.
On Wed, Nov 13, 2013 at 3:54 PM, Juliusz Gonera jgonera@wikimedia.org wrote:
For the past few days (or more) graphs at http://mobile-reportcard.wmflabs.org/ stopped updating. The dashboard consists of two parts: Limn, which displays the data, and backend
scripts
that generate the graph data based on Event Logging data. The issue was caused by two independent problems in the second component:
- A change of MobileWebEditing schema was incorrectly addressed in the
scripts' config and caused the script to throw an exception. 2. Backend scripts are stupid and not optimized at all.
The first thing is fixed. To work around the second thing I had to
disable
updates of "Editors registered on mobile who made 5+ edits on enwiki (mobile+desktop)" graph [1] for now (the query was timing out and
causing an
exception too) and removed the performance graph, since we'll be using ganglia (and soon graphite) for that [2]. Graphs should get updated
soon.
So why are those backend scripts stupid? Because they run every hour
and
recalculate _all_ the values for every single graph. For example, even though total unique editors for June 2013 will never change, they are
still
recalculated every hour. This was a quick and easy solution for
generating
graphs, but as Event Logging tables keep growing, we add more graphs
and
those graphs show more and more data, it's no longer performing.
I discussed this briefly with Ori and I think we agree on the general direction. We should definitely schedule some time for working on
this. We
could start with a spike investigating if there is a framework for aggregating the sums that we could use and asking what other teams in
the
foundation use for generating their graph data. The results of this
spike
and possible following work could be useful not only for the mobile
team.
http://ganglia.wikimedia.org/latest/?r=month&cs=&ce=&tab=v&v...
-- Juliusz
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics