Hi all,
Yesterday we upgraded the Hadoop cluster to a newer version. It seems that along the way the job that imports EventLogging data into Hive has started failing for some EventLogging datasets. I’m still investigating, but it seems that tables (I believe ones that need to have their schema modified during refinement) are failing refinement. The list of affected tables at this moment is large, and will likely only grow as time goes on.
We’re looking into this and will try to have things fixed ASAP. For now you might expect Hive EventLogging data to be delayed.
Sorry for the trouble, and we’ll keep you updated! -Andrew Otto Systems Engineer, WMF
The list of currently affected tables is:
AdvancedSearchRequest CentralAuth CentralNoticeBannerHistory CentralNoticeImpression CentralNoticeTiming ChangesListFilterGrouping ChangesListFilters CitationUsage CitationUsagePageLoad ContentTranslation ContentTranslationAbuseFilter ContentTranslationCTA ContentTranslationError ContentTranslationSuggestion CpuBenchmark EchoInteraction EchoMail EditAttemptStep EditConflict EditorActivation EUCCVisit EventError FlowReplies GettingStartedRedirectImpression GuidedTourButtonClick GuidedTourExited GuidedTourExternalLinkActivation GuidedTourGuiderHidden GuidedTourGuiderImpression LandingPageImpression MediaViewer MediaWikiPingback MobileAppCategorizationAttempts MobileAppUploadAttempts MobileWebMainMenuClickTracking MobileWebSearch MobileWebUIClickTracking MobileWikiAppAppearanceSettings MobileWikiAppArticleSuggestions MobileWikiAppCreateAccount MobileWikiAppDailyStats MobileWikiAppEdit MobileWikiAppFeed MobileWikiAppFeedConfigure MobileWikiAppFindInPage MobileWikiAppInstallReferrer MobileWikiAppIntents MobileWikiAppiOSFeed MobileWikiAppiOSLoginAction MobileWikiAppiOSReadingLists MobileWikiAppiOSSearch MobileWikiAppiOSSessions MobileWikiAppiOSSettingAction MobileWikiAppiOSUserHistory MobileWikiAppLangSelect MobileWikiAppLanguageSearching MobileWikiAppLanguageSettings MobileWikiAppLinkPreview MobileWikiAppLogin MobileWikiAppMediaGallery MobileWikiAppNavMenu MobileWikiAppNotificationInteraction MobileWikiAppNotificationPreferences MobileWikiAppOfflineLibrary MobileWikiAppOnboarding MobileWikiAppOnThisDay MobileWikiAppPageScroll MobileWikiAppProtectedEditAttempt MobileWikiAppRandomizer MobileWikiAppReadingLists MobileWikiAppSavedPages MobileWikiAppSearch MobileWikiAppSessions MobileWikiAppShareAFact MobileWikiAppStuffHappens MobileWikiAppTabs MobileWikiAppToCInteraction MobileWikiAppWidgets MobileWikiAppWiktionaryPopup MultimediaViewerAttribution MultimediaViewerDimensions MultimediaViewerDuration MultimediaViewerNetworkPerformance NavigationTiming PageIssues Popups PrefUpdate Print QuickSurveyInitiation QuickSurveysResponses ReadingDepth ResourceTiming SaveTiming SearchSatisfaction SearchSatisfactionErrors ServerSideAccountCreation TemplateWizard TestSearchSatisfaction2 TranslationRecommendationAPIRequests TranslationRecommendationUIRequests TranslationRecommendationUserAction TwoColConflictConflict UniversalLanguageSelector UploadWizardErrorFlowEvent UploadWizardExceptionFlowEvent UploadWizardFlowEvent UploadWizardStep UploadWizardTutorialActions UploadWizardUploadFlowEvent VirtualPageView WikidataCompletionSearchClicks WikimediaBlogVisit WikipediaPortal WikipediaZeroUsage WMDEBannerEvents WMDEBannerSizeIssue
Is this issue still ongoing? Is there a corresponding Phabricator task to follow?
On Tue, Nov 13, 2018 at 6:27 PM Andrew Otto otto@wikimedia.org wrote:
Hi all,
Yesterday we upgraded the Hadoop cluster to a newer version. It seems that along the way the job that imports EventLogging data into Hive has started failing for some EventLogging datasets. I’m still investigating, but it seems that tables (I believe ones that need to have their schema modified during refinement) are failing refinement. The list of affected tables at this moment is large, and will likely only grow as time goes on.
We’re looking into this and will try to have things fixed ASAP. For now you might expect Hive EventLogging data to be delayed.
Sorry for the trouble, and we’ll keep you updated! -Andrew Otto Systems Engineer, WMF
The list of currently affected tables is:
AdvancedSearchRequest CentralAuth CentralNoticeBannerHistory CentralNoticeImpression CentralNoticeTiming ChangesListFilterGrouping ChangesListFilters CitationUsage CitationUsagePageLoad ContentTranslation ContentTranslationAbuseFilter ContentTranslationCTA ContentTranslationError ContentTranslationSuggestion CpuBenchmark EchoInteraction EchoMail EditAttemptStep EditConflict EditorActivation EUCCVisit EventError FlowReplies GettingStartedRedirectImpression GuidedTourButtonClick GuidedTourExited GuidedTourExternalLinkActivation GuidedTourGuiderHidden GuidedTourGuiderImpression LandingPageImpression MediaViewer MediaWikiPingback MobileAppCategorizationAttempts MobileAppUploadAttempts MobileWebMainMenuClickTracking MobileWebSearch MobileWebUIClickTracking MobileWikiAppAppearanceSettings MobileWikiAppArticleSuggestions MobileWikiAppCreateAccount MobileWikiAppDailyStats MobileWikiAppEdit MobileWikiAppFeed MobileWikiAppFeedConfigure MobileWikiAppFindInPage MobileWikiAppInstallReferrer MobileWikiAppIntents MobileWikiAppiOSFeed MobileWikiAppiOSLoginAction MobileWikiAppiOSReadingLists MobileWikiAppiOSSearch MobileWikiAppiOSSessions MobileWikiAppiOSSettingAction MobileWikiAppiOSUserHistory MobileWikiAppLangSelect MobileWikiAppLanguageSearching MobileWikiAppLanguageSettings MobileWikiAppLinkPreview MobileWikiAppLogin MobileWikiAppMediaGallery MobileWikiAppNavMenu MobileWikiAppNotificationInteraction MobileWikiAppNotificationPreferences MobileWikiAppOfflineLibrary MobileWikiAppOnboarding MobileWikiAppOnThisDay MobileWikiAppPageScroll MobileWikiAppProtectedEditAttempt MobileWikiAppRandomizer MobileWikiAppReadingLists MobileWikiAppSavedPages MobileWikiAppSearch MobileWikiAppSessions MobileWikiAppShareAFact MobileWikiAppStuffHappens MobileWikiAppTabs MobileWikiAppToCInteraction MobileWikiAppWidgets MobileWikiAppWiktionaryPopup MultimediaViewerAttribution MultimediaViewerDimensions MultimediaViewerDuration MultimediaViewerNetworkPerformance NavigationTiming PageIssues Popups PrefUpdate Print QuickSurveyInitiation QuickSurveysResponses ReadingDepth ResourceTiming SaveTiming SearchSatisfaction SearchSatisfactionErrors ServerSideAccountCreation TemplateWizard TestSearchSatisfaction2 TranslationRecommendationAPIRequests TranslationRecommendationUIRequests TranslationRecommendationUserAction TwoColConflictConflict UniversalLanguageSelector UploadWizardErrorFlowEvent UploadWizardExceptionFlowEvent UploadWizardFlowEvent UploadWizardStep UploadWizardTutorialActions UploadWizardUploadFlowEvent VirtualPageView WikidataCompletionSearchClicks WikimediaBlogVisit WikipediaPortal WikipediaZeroUsage WMDEBannerEvents WMDEBannerSizeIssue _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
OH I’m sorry! There is a Phab task, and this is fixed. https://phabricator.wikimedia.org/T209407
Very sorry, I should have updated and linked the task.
On Thu, Nov 15, 2018 at 9:50 AM Gilles Dubuc gilles@wikimedia.org wrote:
Is this issue still ongoing? Is there a corresponding Phabricator task to follow?
On Tue, Nov 13, 2018 at 6:27 PM Andrew Otto otto@wikimedia.org wrote:
Hi all,
Yesterday we upgraded the Hadoop cluster to a newer version. It seems that along the way the job that imports EventLogging data into Hive has started failing for some EventLogging datasets. I’m still investigating, but it seems that tables (I believe ones that need to have their schema modified during refinement) are failing refinement. The list of affected tables at this moment is large, and will likely only grow as time goes on.
We’re looking into this and will try to have things fixed ASAP. For now you might expect Hive EventLogging data to be delayed.
Sorry for the trouble, and we’ll keep you updated! -Andrew Otto Systems Engineer, WMF
The list of currently affected tables is:
AdvancedSearchRequest CentralAuth CentralNoticeBannerHistory CentralNoticeImpression CentralNoticeTiming ChangesListFilterGrouping ChangesListFilters CitationUsage CitationUsagePageLoad ContentTranslation ContentTranslationAbuseFilter ContentTranslationCTA ContentTranslationError ContentTranslationSuggestion CpuBenchmark EchoInteraction EchoMail EditAttemptStep EditConflict EditorActivation EUCCVisit EventError FlowReplies GettingStartedRedirectImpression GuidedTourButtonClick GuidedTourExited GuidedTourExternalLinkActivation GuidedTourGuiderHidden GuidedTourGuiderImpression LandingPageImpression MediaViewer MediaWikiPingback MobileAppCategorizationAttempts MobileAppUploadAttempts MobileWebMainMenuClickTracking MobileWebSearch MobileWebUIClickTracking MobileWikiAppAppearanceSettings MobileWikiAppArticleSuggestions MobileWikiAppCreateAccount MobileWikiAppDailyStats MobileWikiAppEdit MobileWikiAppFeed MobileWikiAppFeedConfigure MobileWikiAppFindInPage MobileWikiAppInstallReferrer MobileWikiAppIntents MobileWikiAppiOSFeed MobileWikiAppiOSLoginAction MobileWikiAppiOSReadingLists MobileWikiAppiOSSearch MobileWikiAppiOSSessions MobileWikiAppiOSSettingAction MobileWikiAppiOSUserHistory MobileWikiAppLangSelect MobileWikiAppLanguageSearching MobileWikiAppLanguageSettings MobileWikiAppLinkPreview MobileWikiAppLogin MobileWikiAppMediaGallery MobileWikiAppNavMenu MobileWikiAppNotificationInteraction MobileWikiAppNotificationPreferences MobileWikiAppOfflineLibrary MobileWikiAppOnboarding MobileWikiAppOnThisDay MobileWikiAppPageScroll MobileWikiAppProtectedEditAttempt MobileWikiAppRandomizer MobileWikiAppReadingLists MobileWikiAppSavedPages MobileWikiAppSearch MobileWikiAppSessions MobileWikiAppShareAFact MobileWikiAppStuffHappens MobileWikiAppTabs MobileWikiAppToCInteraction MobileWikiAppWidgets MobileWikiAppWiktionaryPopup MultimediaViewerAttribution MultimediaViewerDimensions MultimediaViewerDuration MultimediaViewerNetworkPerformance NavigationTiming PageIssues Popups PrefUpdate Print QuickSurveyInitiation QuickSurveysResponses ReadingDepth ResourceTiming SaveTiming SearchSatisfaction SearchSatisfactionErrors ServerSideAccountCreation TemplateWizard TestSearchSatisfaction2 TranslationRecommendationAPIRequests TranslationRecommendationUIRequests TranslationRecommendationUserAction TwoColConflictConflict UniversalLanguageSelector UploadWizardErrorFlowEvent UploadWizardExceptionFlowEvent UploadWizardFlowEvent UploadWizardStep UploadWizardTutorialActions UploadWizardUploadFlowEvent VirtualPageView WikidataCompletionSearchClicks WikimediaBlogVisit WikipediaPortal WikipediaZeroUsage WMDEBannerEvents WMDEBannerSizeIssue _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Does "fixed" mean that the missing data already been backfilled? I'm seeing gaps (zero events) in Turnilo for Druid-ingested EL data, for the timespans between around 6am-16pm on November 13, and 7am-10am on November 12.
On Thu, Nov 15, 2018 at 6:51 AM Andrew Otto otto@wikimedia.org wrote:
OH I’m sorry! There is a Phab task, and this is fixed. https://phabricator.wikimedia.org/T209407
Very sorry, I should have updated and linked the task.
On Thu, Nov 15, 2018 at 9:50 AM Gilles Dubuc gilles@wikimedia.org wrote:
Is this issue still ongoing? Is there a corresponding Phabricator task to follow?
On Tue, Nov 13, 2018 at 6:27 PM Andrew Otto otto@wikimedia.org wrote:
Hi all,
Yesterday we upgraded the Hadoop cluster to a newer version. It seems that along the way the job that imports EventLogging data into Hive has started failing for some EventLogging datasets. I’m still investigating, but it seems that tables (I believe ones that need to have their schema modified during refinement) are failing refinement. The list of affected tables at this moment is large, and will likely only grow as time goes on.
We’re looking into this and will try to have things fixed ASAP. For now you might expect Hive EventLogging data to be delayed.
Sorry for the trouble, and we’ll keep you updated! -Andrew Otto Systems Engineer, WMF
The list of currently affected tables is:
AdvancedSearchRequest CentralAuth CentralNoticeBannerHistory CentralNoticeImpression CentralNoticeTiming ChangesListFilterGrouping ChangesListFilters CitationUsage CitationUsagePageLoad ContentTranslation ContentTranslationAbuseFilter ContentTranslationCTA ContentTranslationError ContentTranslationSuggestion CpuBenchmark EchoInteraction EchoMail EditAttemptStep EditConflict EditorActivation EUCCVisit EventError FlowReplies GettingStartedRedirectImpression GuidedTourButtonClick GuidedTourExited GuidedTourExternalLinkActivation GuidedTourGuiderHidden GuidedTourGuiderImpression LandingPageImpression MediaViewer MediaWikiPingback MobileAppCategorizationAttempts MobileAppUploadAttempts MobileWebMainMenuClickTracking MobileWebSearch MobileWebUIClickTracking MobileWikiAppAppearanceSettings MobileWikiAppArticleSuggestions MobileWikiAppCreateAccount MobileWikiAppDailyStats MobileWikiAppEdit MobileWikiAppFeed MobileWikiAppFeedConfigure MobileWikiAppFindInPage MobileWikiAppInstallReferrer MobileWikiAppIntents MobileWikiAppiOSFeed MobileWikiAppiOSLoginAction MobileWikiAppiOSReadingLists MobileWikiAppiOSSearch MobileWikiAppiOSSessions MobileWikiAppiOSSettingAction MobileWikiAppiOSUserHistory MobileWikiAppLangSelect MobileWikiAppLanguageSearching MobileWikiAppLanguageSettings MobileWikiAppLinkPreview MobileWikiAppLogin MobileWikiAppMediaGallery MobileWikiAppNavMenu MobileWikiAppNotificationInteraction MobileWikiAppNotificationPreferences MobileWikiAppOfflineLibrary MobileWikiAppOnboarding MobileWikiAppOnThisDay MobileWikiAppPageScroll MobileWikiAppProtectedEditAttempt MobileWikiAppRandomizer MobileWikiAppReadingLists MobileWikiAppSavedPages MobileWikiAppSearch MobileWikiAppSessions MobileWikiAppShareAFact MobileWikiAppStuffHappens MobileWikiAppTabs MobileWikiAppToCInteraction MobileWikiAppWidgets MobileWikiAppWiktionaryPopup MultimediaViewerAttribution MultimediaViewerDimensions MultimediaViewerDuration MultimediaViewerNetworkPerformance NavigationTiming PageIssues Popups PrefUpdate Print QuickSurveyInitiation QuickSurveysResponses ReadingDepth ResourceTiming SaveTiming SearchSatisfaction SearchSatisfactionErrors ServerSideAccountCreation TemplateWizard TestSearchSatisfaction2 TranslationRecommendationAPIRequests TranslationRecommendationUIRequests TranslationRecommendationUserAction TwoColConflictConflict UniversalLanguageSelector UploadWizardErrorFlowEvent UploadWizardExceptionFlowEvent UploadWizardFlowEvent UploadWizardStep UploadWizardTutorialActions UploadWizardUploadFlowEvent VirtualPageView WikidataCompletionSearchClicks WikimediaBlogVisit WikipediaPortal WikipediaZeroUsage WMDEBannerEvents WMDEBannerSizeIssue _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics