On Wed, Oct 29, 2025 at 2:37 PM Finn Årup Nielsen via Cloud cloud@lists.wikimedia.org wrote:
Our Scholia webapp on Toolforge is struggling with the request load at https://scholia.toolforge.org/
According to toolviews (also displayed at https://people.compute.dtu.dk/faan/scholia-page-view-statistics.html) we had yesterday almost 700,000 daily hit. Monitoring the log (tail -f uwsgi.log) I see quite a lot of various "troubling" hits, e.g., to the static files. I do not know what it is but I imagine it is some kind of GenAI wannabees with a Playwright script crawling Scholia.
Sorry to hear that you are having this difficulty. You are likely correct that some poorly behaved bots are adding to your load.
Am I correct that async on Toolforge will buy us a bit of extra performance?
It is possible, yes. Using async python typically helps with total throughput by allowing other things to get done while a request is waiting on I/O like a database query or web API request.
Depending on your bottlenecks, you may also see quick relief by just adding more CPU or RAM to the deployment with the `--cpu` and `--mem` flags to `webservice`.
If your app will tolerate load balancing between multiple instances you might be able to try scaling up your deployment as way to handle more requests in parallel. The `--replicas REPLICAS` argument to `webservice` sets the number of copies of your application container to run in parallel. If you are using the default `--mem` and `--cpu` settings there should be room for up to 8 replicas in the default quota (16 CPU cores, 8Gi memory).
Your tool is currently running on Python 3.9 via the shared image system where all of the python application code and pip managed dependencies are loaded from NFS mounts. It is possible that moving to a custom built container [0] could improve your total throughput. Python containers built with the `--use-latest-versions` option can get Python 3.13, but not yet Python 3.14 [1]. These buildpack containers are also capable of running without NFS mounts which can improve I/O for serving static files and loading python code in general.
[0]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_container_images [1]: https://phabricator.wikimedia.org/T408108
Bryan