Hi Amir & others,
I’m glad we are making changes to improve DB storage/query efficiency. I wanted to express my agreement with Tacsipacsi that dropping the data before the migration has completed is a really bad outcome. Now tool maintainers need to deal with multiple migrations depending on the wikis they query or add more code complexity. And there is little time to make the changes for those of us who had planned to wait until the new data was available.
I totally understand the frustration. In my volunteer capacity, I also maintain numerous tools and they break every now and then because of changes.
> Commons has grown to 1.8TB already
That’s a big number yes, but it doesn’t really answer the question — is the database actually about to fill up?
It's a bit more nuanced. We are not hitting limits on the storage. But the memory for data cache on each replica is about 350GB and we need to serve almost everything from memory since the disk is 1000 times slower than memory. If we read too much from disk, reads start to pile up, leading the appserver requests starting to pile up and general outage happening (which has happened before with wikidata's database). You can have a 3TB database with only 100GB of "hot" data and you'd be fine but Commons is both big and very heavily read and across its tables and rows. Ratio-wise, Commons database is already reading twice as much as English Wikipedia from disk.
How much time do you have until that happens and how much time until s1/s8 finish their migration?
The database is already in the "fragile" and "high risk" state. I can't give you an exact date when it'll go down but due to reasons mentioned above I can tell you that even now with any noticeable increase in its traffic or sudden shift in its read patterns it will go down and bring all wikis down with it. There are already user-facing parts in commons that shouldn't be slow but they are due to excessive read from disk.
Also, for the case of Wikidata, it might take a long time, possibly three more months, to finish due to its unique pagelinks usage pattern because of scholarly articles.
Is there a reason you can share why this work wasn’t started earlier if the timing is so close?
We have been constantly working to reduce its size in the past several years, templatelinks migration, externallinks redesign, and so on has been done back to back (started in 2021. We even bumped priority of externallinks migration because of Commons only) but at the same time, the wiki has been growing way too fast. (Emphasizing that the growth doesn't have much to do with images being uploaded, the image table is only 100GB, the problem is the overly large links tables, including templatelinks being 270GB, categorylinks being 200GB, pagelinks being 190GB and so on.). This has put us into a
red queen situation with no easy way out.
> so you need to use the old way for the list of thirty-ish wikis (s1, s2, s6, s7, s8) and for any wiki not a member of that, you can just switch to the new system
IMO, a likely outcome is some tools/bots will simply be broken on a subset of wikis until the migtation is completed across all DBs.
The most urgent one is Commons. What about only dropping it from Commons to reduce the risk of outage and leave the rest until the all are finished (or all except Wikidata)? You'd have to write something for the new schema regardless.
Thanks for all your work on this task so far.
Thank you and sorry for the inconvenience.
Ben / Earwig