Hello all!
We had an interesting discussion yesterday with David about the way we do sharding of our indices on elasticsearch. Here are a few notes for whoever finds the subject interesting and wants to jump in the discussion:
Context:
We recently activated row aware shard allocation on our elasticsearch search clusters. This means that we now have one additional constraint on shard allocation: spread copies of shards across multiple datacenter rows, so that if we loose a full row, we still have a copy of all the data. During an upgrade of elasticsearch, another constraint comes into play: a shard can move from a node with an older version of elasticsearch to a node with a newer version, but not the other way around. This leads to elasticsearch struggling to allocate all shards during the recent codfw upgrade to elasticsearch 2.3.5. While it is not the end of the world (we can still server traffic if some indices don't have all shards allocated), this is something we need to improve.
Number of shards / number of replicas:
An elasticsearch index is split at creation in a number of shards. A number of replica per shard is configured [1]. The total number of shards for an index is "number_of_shards * (number_of_replicas + 1)". Increasing the number of shards per index allow to execute read operation in parallel over the different shards and aggregate the results at the end, improving response time Increasing the number of replicas allow to distribute the read load over more nodes (and provides some redundancy in case we loose one server). As term frequency [2] is calculated over a shard and not over the full index,
There is some black magic involved in how we shard our indices, but most of it is documented [3]
The enwiki_content example:
enwiki_content index is configured to have 6 shards and 3 replicas, for a total number of 24 shards. It also has the additional constraint that there is at most 1 enwiki_content per node. This ensures a maximum spread of enwiki_content shards over the cluster. Since enwiki_content is one of the index with the most traffic, this ensure that the load is well distributed over the cluster.
Now the bad news: for codfw, which is a 24 node cluster, it means that reaching this perfect equilibrium of 1 shard per node is a serious challenge if you take into account the other constraint in place. Even with relaxing the constraint to 2 enwiki shards per node, we have seen unassigned shards during elasticsearch upgrade.
Potential improvements:
While ensuring that a large index has a number of shards close to the number of nodes in the cluster allows for optimally spreading load over the cluster, it degrade fast if all the stars are not aligned perfectly. There are 2 opposite solutions
1) decrease the number of shards to leave some room to move them around 2) increase the number of shards and allow multiple shards of the same index to be allocated on the same node
1) is probably impractical on our large indices, enwiki_content shards are already ~30Gb and this makes it impractical to move them around during relocation and recovery
2) is probably our best bet. More smaller shards means that a single query load will be spread over more nodes, potentially improving response time. Increasing number of shards for enwiki_content from 6 to 20 (total shards = 80) means we have 80 / 24 = 3.3 shards per node. Removing the 1 shards per node constraint and letting elasticsearch spread the shards as best as it can means that in case 1 node is missing, or during an upgrade, we still have the ability to move shards around. Increasing this number even more might help keep the load evenly spread across the cluster (the difference between 8 or 9 shards per node is smaller than the difference between 3 or 4 shards per node).
David is going to do some tests to validate that those smaller shards don't impact the scoring (smaller shards mean worse frequency analysis).
I probably forgot a few points, but this email is more than long enough already...
Thanks to all of you who kept reading until the end!
MrG
[1] https://www.elastic.co/guide/en/elasticsearch/reference/current/_basic_conce... [2] https://www.elastic.co/guide/en/elasticsearch/guide/current/scoring-theory.h... [3] https://wikitech.wikimedia.org/wiki/Search#Estimating_the_number_of_shards_r...