Hello all!

The Elasticsearch upgrade is now completed. During the migration process, we've seen a few issues:

* Search now caps total results count at 10k because of elasticsearch 7 upgrade [1], which is now fixed.

* Some performance degradation was reported on en-wiki village pump [2]. This was observed during the migration when our Search was running on a single datacenter, with added cross-datacenter latency. Our metrics currently show a very minor performance degradation compared to before the upgrade, with the median request time going from between 11 and 12 milliseconds to between 12 and 14 milliseconds [3]. Individual queries might behave differently.

Please let us know if you spot any outstanding issues, preferably via a Phabricator task [4] tagged with [Discovery-Search].

Have fun!

Guillaume

[1] https://phabricator.wikimedia.org/T317374

[2] https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Search_totals_capped_at_10,000

[3] https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1&refresh=1m&from=now-30d&to=now&var-cirrus_group=eqiad&var-cluster=elasticsearch&var-exported_cluster=production-search&var-smoothing=15&viewPanel=21

[4] https://phabricator.wikimedia.org/

On Tue, 23 Aug 2022 at 15:46, Guillaume Lederrey <glederrey@wikimedia.org> wrote:

Hello all!

For the last 2 quarters, the Search Platform team has been working on upgrading our Elasticsearch clusters to version 7.10.2 [1]. Keeping our software up to date is part of the usual project hygiene, allowing us to benefit from bugs and security fixes, performance improvements, and new features. In our case, upgrading to Elasticsearch 7.10.2 is also a required step towards a potential move to OpenSearch [2].

After much testing, fixes and validations, we are now ready to start the final migration process. We are anticipating a 3-week migration process, starting on August 29 2022. You can follow along on Phabricator [3].

What does this mean for you?

For users of Special:Search, Special:MediaSearch and other user-facing Search interfaces, the upgrade should be fully seamless, and should not cause any disruptions to normal usage.

For users of Cloudelastic [4] who are accessing the Elasticsearch API directly, there might be minor API changes that could affect your queries. Please review the documented breaking changes [5]. Most of the breaking changes are not related to queries, so it is unlikely that any client code will break with this upgrade.

If you have any questions about this process, you can find us in #wikimedia-search on IRC, or at discovery@lists.wikimedia.org. Have fun!

The Search Platform team

[1] https://phabricator.wikimedia.org/T263142
[2] https://phabricator.wikimedia.org/T280482
[3] https://phabricator.wikimedia.org/T308676
[4] https://wikitech.wikimedia.org/wiki/Help:CirrusSearch_elasticsearch_replicas
[5] https://www.elastic.co/guide/en/elasticsearch/reference/7.17/breaking-changes-7.0.html

--
Guillaume Lederrey (he/him)
Engineering Manager
Wikimedia Foundation