Not all data sources are populated at the same time, the data on Druid is ingested twice, once per hour and once daily looking 4 days back. Data should appear once daily job runs for the "holes" missing.+1 The EL2Druid daily loading job will cover up the holes for the 12th and 13th in 1 or 2 days.On Thu, Nov 15, 2018 at 5:03 PM Nuria Ruiz <nuria@wikimedia.org> wrote:Hello,Not all data sources are populated at the same time, the data on Druid is ingested twice, once per hour and once daily looking 4 days back. Data should appear once daily job runs for the "holes" missing.Thanks,Nuria_______________________________________________On Thu, Nov 15, 2018 at 7:49 AM Andrew Otto <otto@wikimedia.org> wrote:> Does "fixed" mean that the missing data already been backfilled? I’m seeing gaps (zero events) in Turnilo for Druid-ingested EL data, for the timespans between around 6am-16pm on November 13, and 7am-10am on November 12.Hm. Fixed means the data has been refined to Hive. I didn’t check on Druid EL data. Marcel, can you check up on that? This sounds related, we might need to figure out how to re run the Druid ingestion if/when base data partitions change._______________________________________________On Thu, Nov 15, 2018 at 10:17 AM Tilman Bayer <tbayer@wikimedia.org> wrote:Hi Andrew,PS, offtopic: reminder that data-analysts@wikimedia.org has been deprecated/retired a while ago; our team can be reached at product-analytics@wikimedia.org (CCed)On Thu, Nov 15, 2018 at 7:13 AM Tilman Bayer <tbayer@wikimedia.org> wrote:Does "fixed" mean that the missing data already been backfilled? I'm seeing gaps (zero events) in Turnilo for Druid-ingested EL data, for the timespans between around 6am-16pm on November 13, and 7am-10am on November 12.On Thu, Nov 15, 2018 at 6:51 AM Andrew Otto <otto@wikimedia.org> wrote:OH I’m sorry! There is a Phab task, and this is fixed.Very sorry, I should have updated and linked the task.On Thu, Nov 15, 2018 at 9:50 AM Gilles Dubuc <gilles@wikimedia.org> wrote:Is this issue still ongoing? Is there a corresponding Phabricator task to follow?_______________________________________________On Tue, Nov 13, 2018 at 6:27 PM Andrew Otto <otto@wikimedia.org> wrote:_______________________________________________Hi all,Yesterday we upgraded the Hadoop cluster to a newer version. It seems that along the way the job that imports EventLogging data into Hive has started failing for some EventLogging datasets. I’m still investigating, but it seems that tables (I believe ones that need to have their schema modified during refinement) are failing refinement. The list of affected tables at this moment is large, and will likely only grow as time goes on.We’re looking into this and will try to have things fixed ASAP. For now you might expect Hive EventLogging data to be delayed.Sorry for the trouble, and we’ll keep you updated!-Andrew OttoSystems Engineer, WMFThe list of currently affected tables is:AdvancedSearchRequestCentralAuthCentralNoticeBannerHistoryCentralNoticeImpressionCentralNoticeTimingChangesListFilterGroupingChangesListFiltersCitationUsageCitationUsagePageLoadContentTranslationContentTranslationAbuseFilterContentTranslationCTAContentTranslationErrorContentTranslationSuggestionCpuBenchmarkEchoInteractionEchoMailEditAttemptStepEditConflictEditorActivationEUCCVisitEventErrorFlowRepliesGettingStartedRedirectImpressionGuidedTourButtonClickGuidedTourExitedGuidedTourExternalLinkActivationGuidedTourGuiderHiddenGuidedTourGuiderImpressionLandingPageImpressionMediaViewerMediaWikiPingbackMobileAppCategorizationAttemptsMobileAppUploadAttemptsMobileWebMainMenuClickTrackingMobileWebSearchMobileWebUIClickTrackingMobileWikiAppAppearanceSettingsMobileWikiAppArticleSuggestionsMobileWikiAppCreateAccountMobileWikiAppDailyStatsMobileWikiAppEditMobileWikiAppFeedMobileWikiAppFeedConfigureMobileWikiAppFindInPageMobileWikiAppInstallReferrerMobileWikiAppIntentsMobileWikiAppiOSFeedMobileWikiAppiOSLoginActionMobileWikiAppiOSReadingListsMobileWikiAppiOSSearchMobileWikiAppiOSSessionsMobileWikiAppiOSSettingActionMobileWikiAppiOSUserHistoryMobileWikiAppLangSelectMobileWikiAppLanguageSearchingMobileWikiAppLanguageSettingsMobileWikiAppLinkPreviewMobileWikiAppLoginMobileWikiAppMediaGalleryMobileWikiAppNavMenuMobileWikiAppNotificationInteractionMobileWikiAppNotificationPreferencesMobileWikiAppOfflineLibraryMobileWikiAppOnboardingMobileWikiAppOnThisDayMobileWikiAppPageScrollMobileWikiAppProtectedEditAttemptMobileWikiAppRandomizerMobileWikiAppReadingListsMobileWikiAppSavedPagesMobileWikiAppSearchMobileWikiAppSessionsMobileWikiAppShareAFactMobileWikiAppStuffHappensMobileWikiAppTabsMobileWikiAppToCInteractionMobileWikiAppWidgetsMobileWikiAppWiktionaryPopupMultimediaViewerAttributionMultimediaViewerDimensionsMultimediaViewerDurationMultimediaViewerNetworkPerformanceNavigationTimingPageIssuesPopupsPrefUpdateQuickSurveyInitiationQuickSurveysResponsesReadingDepthResourceTimingSaveTimingSearchSatisfactionSearchSatisfactionErrorsServerSideAccountCreationTemplateWizardTestSearchSatisfaction2TranslationRecommendationAPIRequestsTranslationRecommendationUIRequestsTranslationRecommendationUserActionTwoColConflictConflictUniversalLanguageSelectorUploadWizardErrorFlowEventUploadWizardExceptionFlowEventUploadWizardFlowEventUploadWizardStepUploadWizardTutorialActionsUploadWizardUploadFlowEventVirtualPageViewWikidataCompletionSearchClicksWikimediaBlogVisitWikipediaPortalWikipediaZeroUsageWMDEBannerEventsWMDEBannerSizeIssue
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics
--Tilman Bayer
Senior Analyst
Wikimedia Foundation
IRC (Freenode): HaeB
--Tilman Bayer
Senior Analyst
Wikimedia Foundation
IRC (Freenode): HaeB
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics
--_______________________________________________Marcel Ruiz FornsAnalytics DeveloperWikimedia Foundation
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics