Following up on this, data is been re-ingested into druid so turnilo should
not show up any holes related to this event.
Thanks,
Nuria
On Thu, Nov 15, 2018 at 8:45 AM Marcel Ruiz Forns <mforns(a)wikimedia.org>
wrote:
Not all data sources are populated at the same time,
the data on Druid is
ingested twice, once per hour and once daily
looking 4 days back. Data
should appear once daily job runs for the "holes" missing.
+1 The EL2Druid daily loading job will cover up the holes for the 12th
and 13th in 1 or 2 days.
On Thu, Nov 15, 2018 at 5:03 PM Nuria Ruiz <nuria(a)wikimedia.org> wrote:
> Hello,
>
> Not all data sources are populated at the same time, the data on Druid is
ingested twice, once per hour and once daily
looking 4 days back. Data
should appear once daily job runs for the "holes" missing.
>
> Thanks,
>
> Nuria
>
> On Thu, Nov 15, 2018 at 7:49 AM Andrew Otto <otto(a)wikimedia.org> wrote:
>
>> > Does "fixed" mean that the missing data already been backfilled?
I’m
>> seeing gaps (zero events) in Turnilo for Druid-ingested EL data, for the
>> timespans between around 6am-16pm on November 13, and 7am-10am on November
>> 12.
>>
>> Hm. Fixed means the data has been refined to Hive. I didn’t check on
>> Druid EL data. Marcel, can you check up on that? This sounds related, we
>> might need to figure out how to re run the Druid ingestion if/when base
>> data partitions change.
>>
>>
>>
>> On Thu, Nov 15, 2018 at 10:17 AM Tilman Bayer <tbayer(a)wikimedia.org>
>> wrote:
>>
>>> Hi Andrew,
>>>
>>> PS, offtopic: reminder that data-analysts(a)wikimedia.org has been
>>> deprecated/retired a while ago; our team can be reached at
>>> product-analytics(a)wikimedia.org (CCed)
>>>
>>> On Thu, Nov 15, 2018 at 7:13 AM Tilman Bayer <tbayer(a)wikimedia.org>
>>> wrote:
>>>
>>>> Does "fixed" mean that the missing data already been
backfilled? I'm
>>>> seeing gaps (zero events) in Turnilo for Druid-ingested EL data, for the
>>>> timespans between around 6am-16pm on November 13, and 7am-10am on
November
>>>> 12.
>>>>
>>>> On Thu, Nov 15, 2018 at 6:51 AM Andrew Otto <otto(a)wikimedia.org>
>>>> wrote:
>>>>
>>>>> OH I’m sorry! There is a Phab task, and this is fixed.
>>>>>
https://phabricator.wikimedia.org/T209407
>>>>>
>>>>> Very sorry, I should have updated and linked the task.
>>>>>
>>>>> On Thu, Nov 15, 2018 at 9:50 AM Gilles Dubuc
<gilles(a)wikimedia.org>
>>>>> wrote:
>>>>>
>>>>>> Is this issue still ongoing? Is there a corresponding
Phabricator
>>>>>> task to follow?
>>>>>>
>>>>>> On Tue, Nov 13, 2018 at 6:27 PM Andrew Otto
<otto(a)wikimedia.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Yesterday we upgraded the Hadoop cluster to a newer version.
It
>>>>>>> seems that along the way the job that imports EventLogging
data into Hive
>>>>>>> has started failing for some EventLogging datasets. I’m
still
>>>>>>> investigating, but it seems that tables (I believe ones that
need to have
>>>>>>> their schema modified during refinement) are failing
refinement. The list
>>>>>>> of affected tables at this moment is large, and will likely
only grow as
>>>>>>> time goes on.
>>>>>>>
>>>>>>> We’re looking into this and will try to have things fixed
ASAP.
>>>>>>> For now you might expect Hive EventLogging data to be
delayed.
>>>>>>>
>>>>>>> Sorry for the trouble, and we’ll keep you updated!
>>>>>>> -Andrew Otto
>>>>>>> Systems Engineer, WMF
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The list of currently affected tables is:
>>>>>>>
>>>>>>> AdvancedSearchRequest
>>>>>>> CentralAuth
>>>>>>> CentralNoticeBannerHistory
>>>>>>> CentralNoticeImpression
>>>>>>> CentralNoticeTiming
>>>>>>> ChangesListFilterGrouping
>>>>>>> ChangesListFilters
>>>>>>> CitationUsage
>>>>>>> CitationUsagePageLoad
>>>>>>> ContentTranslation
>>>>>>> ContentTranslationAbuseFilter
>>>>>>> ContentTranslationCTA
>>>>>>> ContentTranslationError
>>>>>>> ContentTranslationSuggestion
>>>>>>> CpuBenchmark
>>>>>>> EchoInteraction
>>>>>>> EchoMail
>>>>>>> EditAttemptStep
>>>>>>> EditConflict
>>>>>>> EditorActivation
>>>>>>> EUCCVisit
>>>>>>> EventError
>>>>>>> FlowReplies
>>>>>>> GettingStartedRedirectImpression
>>>>>>> GuidedTourButtonClick
>>>>>>> GuidedTourExited
>>>>>>> GuidedTourExternalLinkActivation
>>>>>>> GuidedTourGuiderHidden
>>>>>>> GuidedTourGuiderImpression
>>>>>>> LandingPageImpression
>>>>>>> MediaViewer
>>>>>>> MediaWikiPingback
>>>>>>> MobileAppCategorizationAttempts
>>>>>>> MobileAppUploadAttempts
>>>>>>> MobileWebMainMenuClickTracking
>>>>>>> MobileWebSearch
>>>>>>> MobileWebUIClickTracking
>>>>>>> MobileWikiAppAppearanceSettings
>>>>>>> MobileWikiAppArticleSuggestions
>>>>>>> MobileWikiAppCreateAccount
>>>>>>> MobileWikiAppDailyStats
>>>>>>> MobileWikiAppEdit
>>>>>>> MobileWikiAppFeed
>>>>>>> MobileWikiAppFeedConfigure
>>>>>>> MobileWikiAppFindInPage
>>>>>>> MobileWikiAppInstallReferrer
>>>>>>> MobileWikiAppIntents
>>>>>>> MobileWikiAppiOSFeed
>>>>>>> MobileWikiAppiOSLoginAction
>>>>>>> MobileWikiAppiOSReadingLists
>>>>>>> MobileWikiAppiOSSearch
>>>>>>> MobileWikiAppiOSSessions
>>>>>>> MobileWikiAppiOSSettingAction
>>>>>>> MobileWikiAppiOSUserHistory
>>>>>>> MobileWikiAppLangSelect
>>>>>>> MobileWikiAppLanguageSearching
>>>>>>> MobileWikiAppLanguageSettings
>>>>>>> MobileWikiAppLinkPreview
>>>>>>> MobileWikiAppLogin
>>>>>>> MobileWikiAppMediaGallery
>>>>>>> MobileWikiAppNavMenu
>>>>>>> MobileWikiAppNotificationInteraction
>>>>>>> MobileWikiAppNotificationPreferences
>>>>>>> MobileWikiAppOfflineLibrary
>>>>>>> MobileWikiAppOnboarding
>>>>>>> MobileWikiAppOnThisDay
>>>>>>> MobileWikiAppPageScroll
>>>>>>> MobileWikiAppProtectedEditAttempt
>>>>>>> MobileWikiAppRandomizer
>>>>>>> MobileWikiAppReadingLists
>>>>>>> MobileWikiAppSavedPages
>>>>>>> MobileWikiAppSearch
>>>>>>> MobileWikiAppSessions
>>>>>>> MobileWikiAppShareAFact
>>>>>>> MobileWikiAppStuffHappens
>>>>>>> MobileWikiAppTabs
>>>>>>> MobileWikiAppToCInteraction
>>>>>>> MobileWikiAppWidgets
>>>>>>> MobileWikiAppWiktionaryPopup
>>>>>>> MultimediaViewerAttribution
>>>>>>> MultimediaViewerDimensions
>>>>>>> MultimediaViewerDuration
>>>>>>> MultimediaViewerNetworkPerformance
>>>>>>> NavigationTiming
>>>>>>> PageIssues
>>>>>>> Popups
>>>>>>> PrefUpdate
>>>>>>> Print
>>>>>>> QuickSurveyInitiation
>>>>>>> QuickSurveysResponses
>>>>>>> ReadingDepth
>>>>>>> ResourceTiming
>>>>>>> SaveTiming
>>>>>>> SearchSatisfaction
>>>>>>> SearchSatisfactionErrors
>>>>>>> ServerSideAccountCreation
>>>>>>> TemplateWizard
>>>>>>> TestSearchSatisfaction2
>>>>>>> TranslationRecommendationAPIRequests
>>>>>>> TranslationRecommendationUIRequests
>>>>>>> TranslationRecommendationUserAction
>>>>>>> TwoColConflictConflict
>>>>>>> UniversalLanguageSelector
>>>>>>> UploadWizardErrorFlowEvent
>>>>>>> UploadWizardExceptionFlowEvent
>>>>>>> UploadWizardFlowEvent
>>>>>>> UploadWizardStep
>>>>>>> UploadWizardTutorialActions
>>>>>>> UploadWizardUploadFlowEvent
>>>>>>> VirtualPageView
>>>>>>> WikidataCompletionSearchClicks
>>>>>>> WikimediaBlogVisit
>>>>>>> WikipediaPortal
>>>>>>> WikipediaZeroUsage
>>>>>>> WMDEBannerEvents
>>>>>>> WMDEBannerSizeIssue
>>>>>>> _______________________________________________
>>>>>>> Analytics mailing list
>>>>>>> Analytics(a)lists.wikimedia.org
>>>>>>>
https://lists.wikimedia.org/mailman/listinfo/analytics
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Analytics mailing list
>>>>>> Analytics(a)lists.wikimedia.org
>>>>>>
https://lists.wikimedia.org/mailman/listinfo/analytics
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Tilman Bayer
>>>> Senior Analyst
>>>> Wikimedia Foundation
>>>> IRC (Freenode): HaeB
>>>>
>>>
>>>
>>> --
>>> Tilman Bayer
>>> Senior Analyst
>>> Wikimedia Foundation
>>> IRC (Freenode): HaeB
>>>
>> _______________________________________________
>> Analytics mailing list
>> Analytics(a)lists.wikimedia.org
>>
https://lists.wikimedia.org/mailman/listinfo/analytics
>>
> _______________________________________________
> Analytics mailing list
> Analytics(a)lists.wikimedia.org
>
https://lists.wikimedia.org/mailman/listinfo/analytics
>
--
*Marcel Ruiz Forns*
Analytics Developer
Wikimedia Foundation
_______________________________________________
Analytics mailing list
Analytics(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics