Hi Mobile,
Analyzing EventLogging logs we percieved that a significant share of MobileWikiAppSavedPages, MobileWikiAppArticleSuggestions and MobileWikiAppShareAFact events are failing validation.
*1) MobileWikiAppShareAFact: 1.5% not validating* In this schema, the field "text" stores long fractions of text sometimes. This exceeds the size limitation of EL, specially when the text contains special characters, like chinese, greek, etc.
*2) MobileWikiAppArticleSuggestions: 1% not validating* In this case, it's the field "readMoreList" that is sometimes very long, specially when it contains special characters. This, again, exceeds the log size limit.
*3) MobileWikiAppSavedPages: 1% not validating* Some events do not contain the required field "appInstallID".
In cases 1) and 2) the percentage is not big overall, but it can be that for a given language, a lot of events are lost.
EventLogging performance is not compromised by these validation errors, but we are receiving monitoring alerts, and would like to maintain the validation rate close to 100%.
Is it possible for you to somehow reduce the size of the logs of 1) and 2)? If so, have in mind that the log size limit is 1k, and that the highest priority for us would be 2).
Thank you!
Marcel
Thanks for reporting this, Marcel! I've created a task for us to correct the behavior of our EL funnels: https://phabricator.wikimedia.org/T99276
Is the actual character limit of EL messages specified somewhere?
-Dmitry
On Fri, May 15, 2015 at 3:41 PM, Marcel Ruiz Forns mforns@wikimedia.org wrote:
Hi Mobile,
Analyzing EventLogging logs we percieved that a significant share of MobileWikiAppSavedPages, MobileWikiAppArticleSuggestions and MobileWikiAppShareAFact events are failing validation.
*1) MobileWikiAppShareAFact: 1.5% not validating* In this schema, the field "text" stores long fractions of text sometimes. This exceeds the size limitation of EL, specially when the text contains special characters, like chinese, greek, etc.
*2) MobileWikiAppArticleSuggestions: 1% not validating* In this case, it's the field "readMoreList" that is sometimes very long, specially when it contains special characters. This, again, exceeds the log size limit.
*3) MobileWikiAppSavedPages: 1% not validating* Some events do not contain the required field "appInstallID".
In cases 1) and 2) the percentage is not big overall, but it can be that for a given language, a lot of events are lost.
EventLogging performance is not compromised by these validation errors, but we are receiving monitoring alerts, and would like to maintain the validation rate close to 100%.
Is it possible for you to somehow reduce the size of the logs of 1) and 2)? If so, have in mind that the log size limit is 1k, and that the highest priority for us would be 2).
Thank you!
Marcel
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
[brain fart] it's in your email. :( Thanks!
On Fri, May 15, 2015 at 3:59 PM, Dmitry Brant dbrant@wikimedia.org wrote:
Thanks for reporting this, Marcel! I've created a task for us to correct the behavior of our EL funnels: https://phabricator.wikimedia.org/T99276
Is the actual character limit of EL messages specified somewhere?
-Dmitry
On Fri, May 15, 2015 at 3:41 PM, Marcel Ruiz Forns mforns@wikimedia.org wrote:
Hi Mobile,
Analyzing EventLogging logs we percieved that a significant share of MobileWikiAppSavedPages, MobileWikiAppArticleSuggestions and MobileWikiAppShareAFact events are failing validation.
*1) MobileWikiAppShareAFact: 1.5% not validating* In this schema, the field "text" stores long fractions of text sometimes. This exceeds the size limitation of EL, specially when the text contains special characters, like chinese, greek, etc.
*2) MobileWikiAppArticleSuggestions: 1% not validating* In this case, it's the field "readMoreList" that is sometimes very long, specially when it contains special characters. This, again, exceeds the log size limit.
*3) MobileWikiAppSavedPages: 1% not validating* Some events do not contain the required field "appInstallID".
In cases 1) and 2) the percentage is not big overall, but it can be that for a given language, a lot of events are lost.
EventLogging performance is not compromised by these validation errors, but we are receiving monitoring alerts, and would like to maintain the validation rate close to 100%.
Is it possible for you to somehow reduce the size of the logs of 1) and 2)? If so, have in mind that the log size limit is 1k, and that the highest priority for us would be 2).
Thank you!
Marcel
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Dmitry,
you're totally right, it was in the email, but it should have been in the wikis. I added some documentation on it here: https://wikitech.wikimedia.org/wiki/EventLogging#Log_size_limit
Thanks!
On Fri, May 15, 2015 at 10:00 PM, Dmitry Brant dbrant@wikimedia.org wrote:
[brain fart] it's in your email. :( Thanks!
On Fri, May 15, 2015 at 3:59 PM, Dmitry Brant dbrant@wikimedia.org wrote:
Thanks for reporting this, Marcel! I've created a task for us to correct the behavior of our EL funnels: https://phabricator.wikimedia.org/T99276
Is the actual character limit of EL messages specified somewhere?
-Dmitry
On Fri, May 15, 2015 at 3:41 PM, Marcel Ruiz Forns mforns@wikimedia.org wrote:
Hi Mobile,
Analyzing EventLogging logs we percieved that a significant share of MobileWikiAppSavedPages, MobileWikiAppArticleSuggestions and MobileWikiAppShareAFact events are failing validation.
*1) MobileWikiAppShareAFact: 1.5% not validating* In this schema, the field "text" stores long fractions of text sometimes. This exceeds the size limitation of EL, specially when the text contains special characters, like chinese, greek, etc.
*2) MobileWikiAppArticleSuggestions: 1% not validating* In this case, it's the field "readMoreList" that is sometimes very long, specially when it contains special characters. This, again, exceeds the log size limit.
*3) MobileWikiAppSavedPages: 1% not validating* Some events do not contain the required field "appInstallID".
In cases 1) and 2) the percentage is not big overall, but it can be that for a given language, a lot of events are lost.
EventLogging performance is not compromised by these validation errors, but we are receiving monitoring alerts, and would like to maintain the validation rate close to 100%.
Is it possible for you to somehow reduce the size of the logs of 1) and 2)? If so, have in mind that the log size limit is 1k, and that the highest priority for us would be 2).
Thank you!
Marcel
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Marcel, thanks for the links and Dmitry thanks for coordinating this.
A couple notes from my memory:
iOS: EventLogging.h https://git.wikimedia.org/blob/apps%2Fios%2Fwikipedia.git/HEAD/Wikipedia%2FEventLogging%2FEventLoggingFunnel.h#L12 defines WMFEventLoggingMaxStringLength_Snippet and WMFEventLoggingMaxStringLength_General, which are only currently used for Share a Fact in WMFShareFunnel.m https://git.wikimedia.org/blob/apps%2Fios%2Fwikipedia.git/HEAD/Wikipedia%2FView%20Controllers%2FShareCard%2FWMFShareFunnel.m . Not sure if URL-encoding on the request path or actual database bytes (Unicode?) occupied causes overflows despite this.
As I recall, Android does something similar on Share a Fact ( ShareAFactFunnel.java https://git.wikimedia.org/blob/apps%2Fandroid%2Fwikipedia.git/HEAD/wikipedia%2Fsrc%2Fmain%2Fjava%2Forg%2Fwikipedia%2Fanalytics%2FShareAFactFunnel.java#L19). Same thing here with URL-encoding and actual database bytes (Unicode?).
-Adam
On Fri, May 15, 2015 at 1:21 PM, Marcel Ruiz Forns mforns@wikimedia.org wrote:
Dmitry,
you're totally right, it was in the email, but it should have been in the wikis. I added some documentation on it here: https://wikitech.wikimedia.org/wiki/EventLogging#Log_size_limit
Thanks!
On Fri, May 15, 2015 at 10:00 PM, Dmitry Brant dbrant@wikimedia.org wrote:
[brain fart] it's in your email. :( Thanks!
On Fri, May 15, 2015 at 3:59 PM, Dmitry Brant dbrant@wikimedia.org wrote:
Thanks for reporting this, Marcel! I've created a task for us to correct the behavior of our EL funnels: https://phabricator.wikimedia.org/T99276
Is the actual character limit of EL messages specified somewhere?
-Dmitry
On Fri, May 15, 2015 at 3:41 PM, Marcel Ruiz Forns <mforns@wikimedia.org
wrote:
Hi Mobile,
Analyzing EventLogging logs we percieved that a significant share of MobileWikiAppSavedPages, MobileWikiAppArticleSuggestions and MobileWikiAppShareAFact events are failing validation.
*1) MobileWikiAppShareAFact: 1.5% not validating* In this schema, the field "text" stores long fractions of text sometimes. This exceeds the size limitation of EL, specially when the text contains special characters, like chinese, greek, etc.
*2) MobileWikiAppArticleSuggestions: 1% not validating* In this case, it's the field "readMoreList" that is sometimes very long, specially when it contains special characters. This, again, exceeds the log size limit.
*3) MobileWikiAppSavedPages: 1% not validating* Some events do not contain the required field "appInstallID".
In cases 1) and 2) the percentage is not big overall, but it can be that for a given language, a lot of events are lost.
EventLogging performance is not compromised by these validation errors, but we are receiving monitoring alerts, and would like to maintain the validation rate close to 100%.
Is it possible for you to somehow reduce the size of the logs of 1) and 2)? If so, have in mind that the log size limit is 1k, and that the highest priority for us would be 2).
Thank you!
Marcel
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Marcel, do you have breakdown per platform? Not sure if each of these issues is shared between iOS and Android or specific to one or the other. Would be nice to know as we check things out.
On Fri, May 15, 2015 at 4:38 PM, Adam Baso abaso@wikimedia.org wrote:
Marcel, thanks for the links and Dmitry thanks for coordinating this.
A couple notes from my memory:
iOS: EventLogging.h https://git.wikimedia.org/blob/apps%2Fios%2Fwikipedia.git/HEAD/Wikipedia%2FEventLogging%2FEventLoggingFunnel.h#L12 defines WMFEventLoggingMaxStringLength_Snippet and WMFEventLoggingMaxStringLength_General, which are only currently used for Share a Fact in WMFShareFunnel.m https://git.wikimedia.org/blob/apps%2Fios%2Fwikipedia.git/HEAD/Wikipedia%2FView%20Controllers%2FShareCard%2FWMFShareFunnel.m . Not sure if URL-encoding on the request path or actual database bytes (Unicode?) occupied causes overflows despite this.
As I recall, Android does something similar on Share a Fact ( ShareAFactFunnel.java https://git.wikimedia.org/blob/apps%2Fandroid%2Fwikipedia.git/HEAD/wikipedia%2Fsrc%2Fmain%2Fjava%2Forg%2Fwikipedia%2Fanalytics%2FShareAFactFunnel.java#L19). Same thing here with URL-encoding and actual database bytes (Unicode?).
-Adam
On Fri, May 15, 2015 at 1:21 PM, Marcel Ruiz Forns mforns@wikimedia.org wrote:
Dmitry,
you're totally right, it was in the email, but it should have been in the wikis. I added some documentation on it here: https://wikitech.wikimedia.org/wiki/EventLogging#Log_size_limit
Thanks!
On Fri, May 15, 2015 at 10:00 PM, Dmitry Brant dbrant@wikimedia.org wrote:
[brain fart] it's in your email. :( Thanks!
On Fri, May 15, 2015 at 3:59 PM, Dmitry Brant dbrant@wikimedia.org wrote:
Thanks for reporting this, Marcel! I've created a task for us to correct the behavior of our EL funnels: https://phabricator.wikimedia.org/T99276
Is the actual character limit of EL messages specified somewhere?
-Dmitry
On Fri, May 15, 2015 at 3:41 PM, Marcel Ruiz Forns < mforns@wikimedia.org> wrote:
Hi Mobile,
Analyzing EventLogging logs we percieved that a significant share of MobileWikiAppSavedPages, MobileWikiAppArticleSuggestions and MobileWikiAppShareAFact events are failing validation.
*1) MobileWikiAppShareAFact: 1.5% not validating* In this schema, the field "text" stores long fractions of text sometimes. This exceeds the size limitation of EL, specially when the text contains special characters, like chinese, greek, etc.
*2) MobileWikiAppArticleSuggestions: 1% not validating* In this case, it's the field "readMoreList" that is sometimes very long, specially when it contains special characters. This, again, exceeds the log size limit.
*3) MobileWikiAppSavedPages: 1% not validating* Some events do not contain the required field "appInstallID".
In cases 1) and 2) the percentage is not big overall, but it can be that for a given language, a lot of events are lost.
EventLogging performance is not compromised by these validation errors, but we are receiving monitoring alerts, and would like to maintain the validation rate close to 100%.
Is it possible for you to somehow reduce the size of the logs of 1) and 2)? If so, have in mind that the log size limit is 1k, and that the highest priority for us would be 2).
Thank you!
Marcel
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Just to be clear… not looking for exact percentages, just a note of “iOS”, Android” or “both".
On Fri, May 15, 2015 at 4:49 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Marcel, do you have breakdown per platform? Not sure if each of these issues is shared between iOS and Android or specific to one or the other. Would be nice to know as we check things out.
On Fri, May 15, 2015 at 4:38 PM, Adam Baso abaso@wikimedia.org wrote:
Marcel, thanks for the links and Dmitry thanks for coordinating this.
A couple notes from my memory:
iOS: EventLogging.h https://git.wikimedia.org/blob/apps%2Fios%2Fwikipedia.git/HEAD/Wikipedia%2FEventLogging%2FEventLoggingFunnel.h#L12 defines WMFEventLoggingMaxStringLength_Snippet and WMFEventLoggingMaxStringLength_General, which are only currently used for Share a Fact in WMFShareFunnel.m https://git.wikimedia.org/blob/apps%2Fios%2Fwikipedia.git/HEAD/Wikipedia%2FView%20Controllers%2FShareCard%2FWMFShareFunnel.m . Not sure if URL-encoding on the request path or actual database bytes (Unicode?) occupied causes overflows despite this.
As I recall, Android does something similar on Share a Fact ( ShareAFactFunnel.java https://git.wikimedia.org/blob/apps%2Fandroid%2Fwikipedia.git/HEAD/wikipedia%2Fsrc%2Fmain%2Fjava%2Forg%2Fwikipedia%2Fanalytics%2FShareAFactFunnel.java#L19). Same thing here with URL-encoding and actual database bytes (Unicode?).
-Adam
On Fri, May 15, 2015 at 1:21 PM, Marcel Ruiz Forns mforns@wikimedia.org wrote:
Dmitry,
you're totally right, it was in the email, but it should have been in the wikis. I added some documentation on it here: https://wikitech.wikimedia.org/wiki/EventLogging#Log_size_limit
Thanks!
On Fri, May 15, 2015 at 10:00 PM, Dmitry Brant dbrant@wikimedia.org wrote:
[brain fart] it's in your email. :( Thanks!
On Fri, May 15, 2015 at 3:59 PM, Dmitry Brant dbrant@wikimedia.org wrote:
Thanks for reporting this, Marcel! I've created a task for us to correct the behavior of our EL funnels: https://phabricator.wikimedia.org/T99276
Is the actual character limit of EL messages specified somewhere?
-Dmitry
On Fri, May 15, 2015 at 3:41 PM, Marcel Ruiz Forns < mforns@wikimedia.org> wrote:
Hi Mobile,
Analyzing EventLogging logs we percieved that a significant share of MobileWikiAppSavedPages, MobileWikiAppArticleSuggestions and MobileWikiAppShareAFact events are failing validation.
*1) MobileWikiAppShareAFact: 1.5% not validating* In this schema, the field "text" stores long fractions of text sometimes. This exceeds the size limitation of EL, specially when the text contains special characters, like chinese, greek, etc.
*2) MobileWikiAppArticleSuggestions: 1% not validating* In this case, it's the field "readMoreList" that is sometimes very long, specially when it contains special characters. This, again, exceeds the log size limit.
*3) MobileWikiAppSavedPages: 1% not validating* Some events do not contain the required field "appInstallID".
In cases 1) and 2) the percentage is not big overall, but it can be that for a given language, a lot of events are lost.
EventLogging performance is not compromised by these validation errors, but we are receiving monitoring alerts, and would like to maintain the validation rate close to 100%.
Is it possible for you to somehow reduce the size of the logs of 1) and 2)? If so, have in mind that the log size limit is 1k, and that the highest priority for us would be 2).
Thank you!
Marcel
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Since today 2015-05-15 at 08:25h the validation errors for MobileWikiAppShareAFact schema are distributed like:
553 iPhone 261 Android
On Fri, May 15, 2015 at 10:51 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Just to be clear… not looking for exact percentages, just a note of “iOS”, Android” or “both".
On Fri, May 15, 2015 at 4:49 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Marcel, do you have breakdown per platform? Not sure if each of these issues is shared between iOS and Android or specific to one or the other. Would be nice to know as we check things out.
On Fri, May 15, 2015 at 4:38 PM, Adam Baso abaso@wikimedia.org wrote:
Marcel, thanks for the links and Dmitry thanks for coordinating this.
A couple notes from my memory:
iOS: EventLogging.h https://git.wikimedia.org/blob/apps%2Fios%2Fwikipedia.git/HEAD/Wikipedia%2FEventLogging%2FEventLoggingFunnel.h#L12 defines WMFEventLoggingMaxStringLength_Snippet and WMFEventLoggingMaxStringLength_General, which are only currently used for Share a Fact in WMFShareFunnel.m https://git.wikimedia.org/blob/apps%2Fios%2Fwikipedia.git/HEAD/Wikipedia%2FView%20Controllers%2FShareCard%2FWMFShareFunnel.m . Not sure if URL-encoding on the request path or actual database bytes (Unicode?) occupied causes overflows despite this.
As I recall, Android does something similar on Share a Fact ( ShareAFactFunnel.java https://git.wikimedia.org/blob/apps%2Fandroid%2Fwikipedia.git/HEAD/wikipedia%2Fsrc%2Fmain%2Fjava%2Forg%2Fwikipedia%2Fanalytics%2FShareAFactFunnel.java#L19). Same thing here with URL-encoding and actual database bytes (Unicode?).
-Adam
On Fri, May 15, 2015 at 1:21 PM, Marcel Ruiz Forns <mforns@wikimedia.org
wrote:
Dmitry,
you're totally right, it was in the email, but it should have been in the wikis. I added some documentation on it here: https://wikitech.wikimedia.org/wiki/EventLogging#Log_size_limit
Thanks!
On Fri, May 15, 2015 at 10:00 PM, Dmitry Brant dbrant@wikimedia.org wrote:
[brain fart] it's in your email. :( Thanks!
On Fri, May 15, 2015 at 3:59 PM, Dmitry Brant dbrant@wikimedia.org wrote:
Thanks for reporting this, Marcel! I've created a task for us to correct the behavior of our EL funnels: https://phabricator.wikimedia.org/T99276
Is the actual character limit of EL messages specified somewhere?
-Dmitry
On Fri, May 15, 2015 at 3:41 PM, Marcel Ruiz Forns < mforns@wikimedia.org> wrote:
> Hi Mobile, > > Analyzing EventLogging logs we percieved that a significant share of > MobileWikiAppSavedPages, MobileWikiAppArticleSuggestions and > MobileWikiAppShareAFact events are failing validation. > > *1) MobileWikiAppShareAFact: 1.5% not validating* > In this schema, the field "text" stores long fractions of text > sometimes. > This exceeds the size limitation of EL, specially when the text > contains special characters, like chinese, greek, etc. > > *2) MobileWikiAppArticleSuggestions: 1% not validating* > In this case, it's the field "readMoreList" that is sometimes very > long, > specially when it contains special characters. > This, again, exceeds the log size limit. > > *3) MobileWikiAppSavedPages: 1% not validating* > Some events do not contain the required field "appInstallID". > > In cases 1) and 2) the percentage is not big overall, but it can be > that for a given language, a lot of events are lost. > > EventLogging performance is not compromised by these validation > errors, but we are receiving monitoring alerts, and would like to maintain > the validation rate close to 100%. > > Is it possible for you to somehow reduce the size of the logs of 1) > and 2)? > If so, have in mind that the log size limit is 1k, and that the > highest priority for us would be 2). > > Thank you! > > Marcel > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > >
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
On Android, we restrict logging of the text field to the first 99 characters. But, of course, there are many other fields. If there is a long title, maybe the text is in a multi-byte code set, then it could be that the total message gets too long. Ideally, we would look at the total message length, but that's much harder to implement.
Marcel, do you know by how much we are over the limit?
Bernd
On Fri, May 15, 2015 at 11:11 PM, Marcel Ruiz Forns mforns@wikimedia.org wrote:
Since today 2015-05-15 at 08:25h the validation errors for MobileWikiAppShareAFact schema are distributed like:
553 iPhone 261 AndroidOn Fri, May 15, 2015 at 10:51 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Just to be clear… not looking for exact percentages, just a note of “iOS”, Android” or “both".
On Fri, May 15, 2015 at 4:49 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Marcel, do you have breakdown per platform? Not sure if each of these issues is shared between iOS and Android or specific to one or the other. Would be nice to know as we check things out.
On Fri, May 15, 2015 at 4:38 PM, Adam Baso abaso@wikimedia.org wrote:
Marcel, thanks for the links and Dmitry thanks for coordinating this.
A couple notes from my memory:
iOS: EventLogging.h https://git.wikimedia.org/blob/apps%2Fios%2Fwikipedia.git/HEAD/Wikipedia%2FEventLogging%2FEventLoggingFunnel.h#L12 defines WMFEventLoggingMaxStringLength_Snippet and WMFEventLoggingMaxStringLength_General, which are only currently used for Share a Fact in WMFShareFunnel.m https://git.wikimedia.org/blob/apps%2Fios%2Fwikipedia.git/HEAD/Wikipedia%2FView%20Controllers%2FShareCard%2FWMFShareFunnel.m . Not sure if URL-encoding on the request path or actual database bytes (Unicode?) occupied causes overflows despite this.
As I recall, Android does something similar on Share a Fact ( ShareAFactFunnel.java https://git.wikimedia.org/blob/apps%2Fandroid%2Fwikipedia.git/HEAD/wikipedia%2Fsrc%2Fmain%2Fjava%2Forg%2Fwikipedia%2Fanalytics%2FShareAFactFunnel.java#L19). Same thing here with URL-encoding and actual database bytes (Unicode?).
-Adam
On Fri, May 15, 2015 at 1:21 PM, Marcel Ruiz Forns < mforns@wikimedia.org> wrote:
Dmitry,
you're totally right, it was in the email, but it should have been in the wikis. I added some documentation on it here: https://wikitech.wikimedia.org/wiki/EventLogging#Log_size_limit
Thanks!
On Fri, May 15, 2015 at 10:00 PM, Dmitry Brant dbrant@wikimedia.org wrote:
[brain fart] it's in your email. :( Thanks!
On Fri, May 15, 2015 at 3:59 PM, Dmitry Brant dbrant@wikimedia.org wrote:
> Thanks for reporting this, Marcel! > I've created a task for us to correct the behavior of our EL > funnels: https://phabricator.wikimedia.org/T99276 > > Is the actual character limit of EL messages specified somewhere? > > > -Dmitry > > > On Fri, May 15, 2015 at 3:41 PM, Marcel Ruiz Forns < > mforns@wikimedia.org> wrote: > >> Hi Mobile, >> >> Analyzing EventLogging logs we percieved that a significant share >> of MobileWikiAppSavedPages, MobileWikiAppArticleSuggestions and >> MobileWikiAppShareAFact events are failing validation. >> >> *1) MobileWikiAppShareAFact: 1.5% not validating* >> In this schema, the field "text" stores long fractions of text >> sometimes. >> This exceeds the size limitation of EL, specially when the text >> contains special characters, like chinese, greek, etc. >> >> *2) MobileWikiAppArticleSuggestions: 1% not validating* >> In this case, it's the field "readMoreList" that is sometimes very >> long, >> specially when it contains special characters. >> This, again, exceeds the log size limit. >> >> *3) MobileWikiAppSavedPages: 1% not validating* >> Some events do not contain the required field "appInstallID". >> >> In cases 1) and 2) the percentage is not big overall, but it can be >> that for a given language, a lot of events are lost. >> >> EventLogging performance is not compromised by these validation >> errors, but we are receiving monitoring alerts, and would like to maintain >> the validation rate close to 100%. >> >> Is it possible for you to somehow reduce the size of the logs of 1) >> and 2)? >> If so, have in mind that the log size limit is 1k, and that the >> highest priority for us would be 2). >> >> Thank you! >> >> Marcel >> >> _______________________________________________ >> Mobile-l mailing list >> Mobile-l@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mobile-l >> >> >
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Bernd,
No, I can't give you the original sixze, the earliest register of the logs is already truncated. But you're right, the problem is the multi-byte code set, which when URI-decoded gets really long.
On Fri, May 15, 2015 at 11:38 PM, Bernd Sitzmann bernd@wikimedia.org wrote:
On Android, we restrict logging of the text field to the first 99 characters. But, of course, there are many other fields. If there is a long title, maybe the text is in a multi-byte code set, then it could be that the total message gets too long. Ideally, we would look at the total message length, but that's much harder to implement.
Marcel, do you know by how much we are over the limit?
Bernd
On Fri, May 15, 2015 at 11:11 PM, Marcel Ruiz Forns mforns@wikimedia.org wrote:
Since today 2015-05-15 at 08:25h the validation errors for MobileWikiAppShareAFact schema are distributed like:
553 iPhone 261 AndroidOn Fri, May 15, 2015 at 10:51 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Just to be clear… not looking for exact percentages, just a note of “iOS”, Android” or “both".
On Fri, May 15, 2015 at 4:49 PM, Corey Floyd cfloyd@wikimedia.org wrote:
Marcel, do you have breakdown per platform? Not sure if each of these issues is shared between iOS and Android or specific to one or the other. Would be nice to know as we check things out.
On Fri, May 15, 2015 at 4:38 PM, Adam Baso abaso@wikimedia.org wrote:
Marcel, thanks for the links and Dmitry thanks for coordinating this.
A couple notes from my memory:
iOS: EventLogging.h https://git.wikimedia.org/blob/apps%2Fios%2Fwikipedia.git/HEAD/Wikipedia%2FEventLogging%2FEventLoggingFunnel.h#L12 defines WMFEventLoggingMaxStringLength_Snippet and WMFEventLoggingMaxStringLength_General, which are only currently used for Share a Fact in WMFShareFunnel.m https://git.wikimedia.org/blob/apps%2Fios%2Fwikipedia.git/HEAD/Wikipedia%2FView%20Controllers%2FShareCard%2FWMFShareFunnel.m . Not sure if URL-encoding on the request path or actual database bytes (Unicode?) occupied causes overflows despite this.
As I recall, Android does something similar on Share a Fact ( ShareAFactFunnel.java https://git.wikimedia.org/blob/apps%2Fandroid%2Fwikipedia.git/HEAD/wikipedia%2Fsrc%2Fmain%2Fjava%2Forg%2Fwikipedia%2Fanalytics%2FShareAFactFunnel.java#L19). Same thing here with URL-encoding and actual database bytes (Unicode?).
-Adam
On Fri, May 15, 2015 at 1:21 PM, Marcel Ruiz Forns < mforns@wikimedia.org> wrote:
Dmitry,
you're totally right, it was in the email, but it should have been in the wikis. I added some documentation on it here: https://wikitech.wikimedia.org/wiki/EventLogging#Log_size_limit
Thanks!
On Fri, May 15, 2015 at 10:00 PM, Dmitry Brant dbrant@wikimedia.org wrote:
> [brain fart] it's in your email. :( Thanks! > > On Fri, May 15, 2015 at 3:59 PM, Dmitry Brant dbrant@wikimedia.org > wrote: > >> Thanks for reporting this, Marcel! >> I've created a task for us to correct the behavior of our EL >> funnels: https://phabricator.wikimedia.org/T99276 >> >> Is the actual character limit of EL messages specified somewhere? >> >> >> -Dmitry >> >> >> On Fri, May 15, 2015 at 3:41 PM, Marcel Ruiz Forns < >> mforns@wikimedia.org> wrote: >> >>> Hi Mobile, >>> >>> Analyzing EventLogging logs we percieved that a significant share >>> of MobileWikiAppSavedPages, MobileWikiAppArticleSuggestions and >>> MobileWikiAppShareAFact events are failing validation. >>> >>> *1) MobileWikiAppShareAFact: 1.5% not validating* >>> In this schema, the field "text" stores long fractions of text >>> sometimes. >>> This exceeds the size limitation of EL, specially when the text >>> contains special characters, like chinese, greek, etc. >>> >>> *2) MobileWikiAppArticleSuggestions: 1% not validating* >>> In this case, it's the field "readMoreList" that is sometimes very >>> long, >>> specially when it contains special characters. >>> This, again, exceeds the log size limit. >>> >>> *3) MobileWikiAppSavedPages: 1% not validating* >>> Some events do not contain the required field "appInstallID". >>> >>> In cases 1) and 2) the percentage is not big overall, but it can >>> be that for a given language, a lot of events are lost. >>> >>> EventLogging performance is not compromised by these validation >>> errors, but we are receiving monitoring alerts, and would like to maintain >>> the validation rate close to 100%. >>> >>> Is it possible for you to somehow reduce the size of the logs of >>> 1) and 2)? >>> If so, have in mind that the log size limit is 1k, and that the >>> highest priority for us would be 2). >>> >>> Thank you! >>> >>> Marcel >>> >>> _______________________________________________ >>> Mobile-l mailing list >>> Mobile-l@lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >>> >>> >> >
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
-- Corey Floyd Software Engineer Mobile Apps / iOS Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l