Our Scholia webapp on Toolforge is struggling with the request load at https://scholia.toolforge.org/
According to toolviews (also displayed at https://people.compute.dtu.dk/faan/scholia-page-view-statistics.html) we had yesterday almost 700,000 daily hit. Monitoring the log (tail -f uwsgi.log) I see quite a lot of various "troubling" hits, e.g., to the static files. I do not know what it is but I imagine it is some kind of GenAI wannabees with a Playwright script crawling Scholia.
In the current Scholia webapp code, we have embarrassments:
1) Serving static files in the Flask webapp.
2) Blocking requests to Wikidata API or the SPARQL endpoint here and there in the code (most Scholia requests are client-side SPARQL requests though).
I have been moving some static file requests to tools-static.wmflabs.org, but still need to do some more that is embedded in Bootstrap.
I am thinking about moving from Flask to an async framework. I am gaining some experience with FastAPI for web services.
Am I correct that async on Toolforge will buy us a bit of extra performance?
Are there other Toolforge users that have struggling webapps and, if yes, then what do you do? If it continue we could do login I suppose.
In the Scholia repo we have this issue: https://github.com/WDscholia/scholia/issues/2727
best regards Finn Årup Nielsen https://people.compute.dtu.dk/faan/
On Wed, Oct 29, 2025 at 2:37 PM Finn Årup Nielsen via Cloud cloud@lists.wikimedia.org wrote:
Our Scholia webapp on Toolforge is struggling with the request load at https://scholia.toolforge.org/
According to toolviews (also displayed at https://people.compute.dtu.dk/faan/scholia-page-view-statistics.html) we had yesterday almost 700,000 daily hit. Monitoring the log (tail -f uwsgi.log) I see quite a lot of various "troubling" hits, e.g., to the static files. I do not know what it is but I imagine it is some kind of GenAI wannabees with a Playwright script crawling Scholia.
Sorry to hear that you are having this difficulty. You are likely correct that some poorly behaved bots are adding to your load.
Am I correct that async on Toolforge will buy us a bit of extra performance?
It is possible, yes. Using async python typically helps with total throughput by allowing other things to get done while a request is waiting on I/O like a database query or web API request.
Depending on your bottlenecks, you may also see quick relief by just adding more CPU or RAM to the deployment with the `--cpu` and `--mem` flags to `webservice`.
If your app will tolerate load balancing between multiple instances you might be able to try scaling up your deployment as way to handle more requests in parallel. The `--replicas REPLICAS` argument to `webservice` sets the number of copies of your application container to run in parallel. If you are using the default `--mem` and `--cpu` settings there should be room for up to 8 replicas in the default quota (16 CPU cores, 8Gi memory).
Your tool is currently running on Python 3.9 via the shared image system where all of the python application code and pip managed dependencies are loaded from NFS mounts. It is possible that moving to a custom built container [0] could improve your total throughput. Python containers built with the `--use-latest-versions` option can get Python 3.13, but not yet Python 3.14 [1]. These buildpack containers are also capable of running without NFS mounts which can improve I/O for serving static files and loading python code in general.
[0]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_container_images [1]: https://phabricator.wikimedia.org/T408108
Bryan