This one seems unrelated to https://phabricator.wikimedia.org/T336800.
Stack:
Error in query: cannot resolve '`section_index`' given input columns: [spark_catalog.analytics_platform_eng.image_suggestions_suggestions.confidence, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.found_on, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.id, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.image, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.kind, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.origin_wiki, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.page_id, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.page_rev, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.section_heading, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.snapshot, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.wiki]; line 1 pos 156;
'InsertIntoStatement 'UnresolvedRelation [aqs, image_suggestions, suggestions], [], false, false, false
+- 'Repartition 6, false
+- 'Project [wiki#10, page_id#0L, id#1, image#2, origin_wiki#3, confidence#4, found_on#5, kind#6, page_rev#7L, section_heading#8, 'section_index, 'page_qid]
+- Filter (snapshot#9 = 2023-05-01)
+- SubqueryAlias spark_catalog.analytics_platform_eng.image_suggestions_suggestions
+- Relation[page_id#0L,id#1,image#2,origin_wiki#3,confidence#4,found_on#5,kind#6,page_rev#7L,section_heading#8,snapshot#9,wiki#10] parquet
On Wed, May 17, 2023 at 2:31 PM airflow-platform_eng@an-airflow1004.eqiad.wmnet wrote:
Try 6 out of 6 Exception: SkeinHook Airflow SparkSkeinSubmitHook skein launcher image_suggestions__hive_to_cassandra_suggestions__20230501 application_1678266962370_405987 Log: Link http://localhost:8080/log?execution_date=2023-05-01T00%3A00%3A00%2B00%3A00&task_id=hive_to_cassandra_suggestions&dag_id=image_suggestions&map_index=-1 Host: an-airflow1004.eqiad.wmnet Mark success: Link http://localhost:8080/confirm?task_id=hive_to_cassandra_suggestions&dag_id=image_suggestions&dag_run_id=scheduled__2023-05-01T00%3A00%3A00%2B00%3A00&upstream=false&downstream=false&state=success _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
I suspect the latest DAG was deployed, but I still have to make the new image suggestions release!
On Wed, May 17, 2023 at 6:41 PM Xabriel Collazo Mojica < xcollazo@wikimedia.org> wrote:
This one seems unrelated to https://phabricator.wikimedia.org/T336800.
Stack:
Error in query: cannot resolve '`section_index`' given input columns: [spark_catalog.analytics_platform_eng.image_suggestions_suggestions.confidence, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.found_on, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.id, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.image, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.kind, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.origin_wiki, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.page_id, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.page_rev, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.section_heading, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.snapshot, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.wiki]; line 1 pos 156;
'InsertIntoStatement 'UnresolvedRelation [aqs, image_suggestions, suggestions], [], false, false, false
+- 'Repartition 6, false
+- 'Project [wiki#10, page_id#0L, id#1, image#2, origin_wiki#3, confidence#4, found_on#5, kind#6, page_rev#7L, section_heading#8, 'section_index, 'page_qid]
+- Filter (snapshot#9 = 2023-05-01) +- SubqueryAliasspark_catalog.analytics_platform_eng.image_suggestions_suggestions
+-Relation[page_id#0L,id#1,image#2,origin_wiki#3,confidence#4,found_on#5,kind#6,page_rev#7L,section_heading#8,snapshot#9,wiki#10] parquet
On Wed, May 17, 2023 at 2:31 PM airflow-platform_eng@an-airflow1004.eqiad.wmnet wrote:
Try 6 out of 6 Exception: SkeinHook Airflow SparkSkeinSubmitHook skein launcher image_suggestions__hive_to_cassandra_suggestions__20230501 application_1678266962370_405987 Log: Link http://localhost:8080/log?execution_date=2023-05-01T00%3A00%3A00%2B00%3A00&task_id=hive_to_cassandra_suggestions&dag_id=image_suggestions&map_index=-1 Host: an-airflow1004.eqiad.wmnet Mark success: Link http://localhost:8080/confirm?task_id=hive_to_cassandra_suggestions&dag_id=image_suggestions&dag_run_id=scheduled__2023-05-01T00%3A00%3A00%2B00%3A00&upstream=false&downstream=false&state=success _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
-- -xabriel _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
Hola Xabriel,
The new release is there and the DAG is updated to point to it. Not deployed yet: I'll be out Thu and Fri, so I thought it would be safer to do it when I'm back, as we have some important changes in the new release.
Now the story is: it wasn't planned to deploy the DAG *before* the release, but I fear that the latest hotfix hijacked the plan. As a result, the currently failing run is likely to break the search indices deltas. I think we should clear it, delete the relevant partitions, and re-run with the new deployed release. I just paused the DAG to be extra safe, so that the current up-to-reschedule run doesn't harm.
Could you plz coordinate with Cormac while I'm off?
Thanks a ton! Marco
On Wed, May 17, 2023 at 7:41 PM Marco Fossati mfossati@wikimedia.org wrote:
I suspect the latest DAG was deployed, but I still have to make the new image suggestions release!
On Wed, May 17, 2023 at 6:41 PM Xabriel Collazo Mojica < xcollazo@wikimedia.org> wrote:
This one seems unrelated to https://phabricator.wikimedia.org/T336800.
Stack:
Error in query: cannot resolve '`section_index`' given input columns: [spark_catalog.analytics_platform_eng.image_suggestions_suggestions.confidence, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.found_on, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.id, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.image, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.kind, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.origin_wiki, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.page_id, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.page_rev, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.section_heading, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.snapshot, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.wiki]; line 1 pos 156;
'InsertIntoStatement 'UnresolvedRelation [aqs, image_suggestions, suggestions], [], false, false, false
+- 'Repartition 6, false
+- 'Project [wiki#10, page_id#0L, id#1, image#2, origin_wiki#3, confidence#4, found_on#5, kind#6, page_rev#7L, section_heading#8, 'section_index, 'page_qid]
+- Filter (snapshot#9 = 2023-05-01) +- SubqueryAliasspark_catalog.analytics_platform_eng.image_suggestions_suggestions
+-Relation[page_id#0L,id#1,image#2,origin_wiki#3,confidence#4,found_on#5,kind#6,page_rev#7L,section_heading#8,snapshot#9,wiki#10] parquet
On Wed, May 17, 2023 at 2:31 PM airflow-platform_eng@an-airflow1004.eqiad.wmnet wrote:
Try 6 out of 6 Exception: SkeinHook Airflow SparkSkeinSubmitHook skein launcher image_suggestions__hive_to_cassandra_suggestions__20230501 application_1678266962370_405987 Log: Link http://localhost:8080/log?execution_date=2023-05-01T00%3A00%3A00%2B00%3A00&task_id=hive_to_cassandra_suggestions&dag_id=image_suggestions&map_index=-1 Host: an-airflow1004.eqiad.wmnet Mark success: Link http://localhost:8080/confirm?task_id=hive_to_cassandra_suggestions&dag_id=image_suggestions&dag_run_id=scheduled__2023-05-01T00%3A00%3A00%2B00%3A00&upstream=false&downstream=false&state=success _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
-- -xabriel _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
All right, taking care of this via https://phabricator.wikimedia.org/T336958 .
Cormac and I discussed this, and we decided that rather than delete data, it was easier to revert the changes to the DAG prior to my hotfix, and just let the DAG finish with v0.11.0 code.
Later we can revert the revert and deploy v0.12.0 code.
Let's continue work/conversation on the ticket.
On Wed, May 17, 2023 at 7:40 PM Marco Fossati mfossati@wikimedia.org wrote:
Hola Xabriel,
The new release is there and the DAG is updated to point to it. Not deployed yet: I'll be out Thu and Fri, so I thought it would be safer to do it when I'm back, as we have some important changes in the new release.
Now the story is: it wasn't planned to deploy the DAG *before* the release, but I fear that the latest hotfix hijacked the plan. As a result, the currently failing run is likely to break the search indices deltas. I think we should clear it, delete the relevant partitions, and re-run with the new deployed release. I just paused the DAG to be extra safe, so that the current up-to-reschedule run doesn't harm.
Could you plz coordinate with Cormac while I'm off?
Thanks a ton! Marco
On Wed, May 17, 2023 at 7:41 PM Marco Fossati mfossati@wikimedia.org wrote:
I suspect the latest DAG was deployed, but I still have to make the new image suggestions release!
On Wed, May 17, 2023 at 6:41 PM Xabriel Collazo Mojica < xcollazo@wikimedia.org> wrote:
This one seems unrelated to https://phabricator.wikimedia.org/T336800.
Stack:
Error in query: cannot resolve '`section_index`' given input columns: [spark_catalog.analytics_platform_eng.image_suggestions_suggestions.confidence, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.found_on, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.id, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.image, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.kind, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.origin_wiki, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.page_id, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.page_rev, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.section_heading, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.snapshot, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.wiki]; line 1 pos 156;
'InsertIntoStatement 'UnresolvedRelation [aqs, image_suggestions, suggestions], [], false, false, false
+- 'Repartition 6, false
+- 'Project [wiki#10, page_id#0L, id#1, image#2, origin_wiki#3, confidence#4, found_on#5, kind#6, page_rev#7L, section_heading#8, 'section_index, 'page_qid]
+- Filter (snapshot#9 = 2023-05-01) +- SubqueryAliasspark_catalog.analytics_platform_eng.image_suggestions_suggestions
+-Relation[page_id#0L,id#1,image#2,origin_wiki#3,confidence#4,found_on#5,kind#6,page_rev#7L,section_heading#8,snapshot#9,wiki#10] parquet
On Wed, May 17, 2023 at 2:31 PM airflow-platform_eng@an-airflow1004.eqiad.wmnet wrote:
Try 6 out of 6 Exception: SkeinHook Airflow SparkSkeinSubmitHook skein launcher image_suggestions__hive_to_cassandra_suggestions__20230501 application_1678266962370_405987 Log: Link http://localhost:8080/log?execution_date=2023-05-01T00%3A00%3A00%2B00%3A00&task_id=hive_to_cassandra_suggestions&dag_id=image_suggestions&map_index=-1 Host: an-airflow1004.eqiad.wmnet Mark success: Link http://localhost:8080/confirm?task_id=hive_to_cassandra_suggestions&dag_id=image_suggestions&dag_run_id=scheduled__2023-05-01T00%3A00%3A00%2B00%3A00&upstream=false&downstream=false&state=success _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/
-- -xabriel _______________________________________________ Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/