Hola Xabriel,
The new release is there and the DAG is updated to point to it. Not deployed yet: I'll be out Thu and Fri, so I thought it would be safer to do it when I'm back, as we have some important changes in the new release.
Now the story is: it wasn't planned to deploy the DAG *before* the release, but I fear that the latest hotfix hijacked the plan. As a result, the currently failing run is likely to break the search indices deltas. I think we should clear it, delete the relevant partitions, and re-run with the new deployed release. I just paused the DAG to be extra safe, so that the current up-to-reschedule run doesn't harm.
Could you plz coordinate with Cormac while I'm off?
Thanks a ton! Marco
On Wed, May 17, 2023 at 7:41 PM Marco Fossati mfossati@wikimedia.org wrote:
I suspect the latest DAG was deployed, but I still have to make the new image suggestions release!
On Wed, May 17, 2023 at 6:41 PM Xabriel Collazo Mojica < xcollazo@wikimedia.org> wrote:
This one seems unrelated to https://phabricator.wikimedia.org/T336800.
Stack:
Error in query: cannot resolve '`section_index`' given input columns: [spark_catalog.analytics_platform_eng.image_suggestions_suggestions.confidence, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.found_on, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.id, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.image, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.kind, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.origin_wiki, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.page_id, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.page_rev, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.section_heading, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.snapshot, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.wiki]; line 1 pos 156;
'InsertIntoStatement 'UnresolvedRelation [aqs, image_suggestions, suggestions], [], false, false, false
+- 'Repartition 6, false
+- 'Project [wiki#10, page_id#0L, id#1, image#2, origin_wiki#3, confidence#4, found_on#5, kind#6, page_rev#7L, section_heading#8, 'section_index, 'page_qid]
+- Filter (snapshot#9 = 2023-05-01) +- SubqueryAliasspark_catalog.analytics_platform_eng.image_suggestions_suggestions
+-Relation[page_id#0L,id#1,image#2,origin_wiki#3,confidence#4,found_on#5,kind#6,page_rev#7L,section_heading#8,snapshot#9,wiki#10] parquet
On Wed, May 17, 2023 at 2:31 PM airflow-platform_eng@an-airflow1004.eqiad.wmnet wrote:
Try 6 out of 6 Exception: SkeinHook Airflow SparkSkeinSubmitHook skein launcher image_suggestions__hive_to_cassandra_suggestions__20230501 application_1678266962370_405987 Log: Link http://localhost:8080/log?execution_date=2023-05-01T00%3A00%3A00%2B00%3A00&task_id=hive_to_cassandra_suggestions&dag_id=image_suggestions&map_index=-1 Host: an-airflow1004.eqiad.wmnet Mark success: Link http://localhost:8080/confirm?task_id=hive_to_cassandra_suggestions&dag_id=image_suggestions&dag_run_id=scheduled__2023-05-01T00%3A00%3A00%2B00%3A00&upstream=false&downstream=false&state=success _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
-- -xabriel _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/