Hola Xabriel,

The new release is there and the DAG is updated to point to it.

Not deployed yet: I'll be out Thu and Fri, so I thought it would be safer to do it when I'm back, as we have some important changes in the new release.

Now the story is: it wasn't planned to deploy the DAG *before* the release, but I fear that the latest hotfix hijacked the plan.

As a result, the currently failing run is likely to break the search indices deltas.

I think we should clear it, delete the relevant partitions, and re-run with the new deployed release.

I just paused the DAG to be extra safe, so that the current up-to-reschedule run doesn't harm.

Could you plz coordinate with Cormac while I'm off?

Thanks a ton!

Marco

On Wed, May 17, 2023 at 7:41 PM Marco Fossati <mfossati@wikimedia.org> wrote:

I suspect the latest DAG was deployed, but I still have to make the new image suggestions release!

On Wed, May 17, 2023 at 6:41 PM Xabriel Collazo Mojica <xcollazo@wikimedia.org> wrote:

This one seems unrelated to https://phabricator.wikimedia.org/T336800.

Stack:

Error in query: cannot resolve '`section_index`' given input columns: [spark_catalog.analytics_platform_eng.image_suggestions_suggestions.confidence, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.found_on, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.id, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.image, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.kind, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.origin_wiki, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.page_id, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.page_rev, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.section_heading, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.snapshot, spark_catalog.analytics_platform_eng.image_suggestions_suggestions.wiki]; line 1 pos 156;

'InsertIntoStatement 'UnresolvedRelation [aqs, image_suggestions, suggestions], [], false, false, false

+- 'Repartition 6, false

   +- 'Project [wiki#10, page_id#0L, id#1, image#2, origin_wiki#3, confidence#4, found_on#5, kind#6, page_rev#7L, section_heading#8, 'section_index, 'page_qid]

      +- Filter (snapshot#9 = 2023-05-01)

         +- SubqueryAlias spark_catalog.analytics_platform_eng.image_suggestions_suggestions

            +- Relation[page_id#0L,id#1,image#2,origin_wiki#3,confidence#4,found_on#5,kind#6,page_rev#7L,section_heading#8,snapshot#9,wiki#10] parquet

On Wed, May 17, 2023 at 2:31 PM <airflow-platform_eng@an-airflow1004.eqiad.wmnet> wrote:

Try 6 out of 6
Exception:
SkeinHook Airflow SparkSkeinSubmitHook skein launcher image_suggestions__hive_to_cassandra_suggestions__20230501 application_1678266962370_405987
Log: Link
Host: an-airflow1004.eqiad.wmnet
Mark success: Link
_______________________________________________
Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/

--

-xabriel

_______________________________________________
Sd-alerts mailing list -- sd-alerts@lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/sd-alerts.lists.wikimedia.org/