Hi everyone,
I'm seeing strange things with resource consumption with toolforge. My resource consumption reported by the quota is way above the consumption of my currently running job, preventing other jobs from starting. Where does the extra consumption come from ? And how to clean this ?
More detailed data below
Any help appreciated, NicoV
My tool account is wpcleaner on toolforge.
Only one job is currently running :
tools.wpcleaner@tools-bastion-15:~$ toolforge jobs list +------------------------+--------------------+------------------------------------------+ | Job name: | Job type: | Status: | +------------------------+--------------------+------------------------------------------+ | wpcleaner-cs-weekly | schedule: @weekly | Last schedule time: 2025-12-03T05:17:00Z | | wpcleaner-en-list | schedule: @weekly | Unable to start, out of quota for memory | | wpcleaner-fr-dab | schedule: @weekly | Unable to start, out of quota for memory | | wpcleaner-fr-daily | schedule: @daily | Last schedule time: 2025-12-07T03:58:00Z | | wpcleaner-fr-list | schedule: @weekly | Running for 2d4h10m | | wpcleaner-fr-weekly | schedule: @weekly | Last schedule time: 2025-12-04T05:03:00Z | | wpcleaner-meta-list | schedule: @monthly | Last schedule time: 2025-12-04T04:25:00Z | | wpcleaner-meta-monthly | schedule: @monthly | Last schedule time: 2025-11-18T06:05:00Z | +------------------------+--------------------+------------------------------------------+
The currently running job defines 3G and 1 CPU for resources :
tools.wpcleaner@tools-bastion-15:~$ toolforge jobs show wpcleaner-fr-list +---------------+-----------------------------------------------------------------+ | Job name: | wpcleaner-fr-list | +---------------+-----------------------------------------------------------------+ | Command: | /data/project/wpcleaner/tools/scripts/fr_ListCheckWiki_List.sh | +---------------+-----------------------------------------------------------------+ | Job type: | schedule: @weekly | +---------------+-----------------------------------------------------------------+ | Image: | jdk17 | +---------------+-----------------------------------------------------------------+ | Port: | none | +---------------+-----------------------------------------------------------------+ | File log: | no | +---------------+-----------------------------------------------------------------+ | Output log: | | +---------------+-----------------------------------------------------------------+ | Error log: | | +---------------+-----------------------------------------------------------------+ | Emails: | onfailure | +---------------+-----------------------------------------------------------------+ | Resources: | mem: 3.0Gi, cpu: 1.0 | +---------------+-----------------------------------------------------------------+ | Replicas: | | +---------------+-----------------------------------------------------------------+ | Mounts: | none | +---------------+-----------------------------------------------------------------+ | Retry: | no | +---------------+-----------------------------------------------------------------+ | Timeout: | no | +---------------+-----------------------------------------------------------------+ | Health check: | none | +---------------+-----------------------------------------------------------------+ | Status: | Running for 2d4h11m | +---------------+-----------------------------------------------------------------+ | Hints: | Last run at 2025-12-03T20:36:14Z. Pod in 'Running' phase. State | | | 'running'. Started at '2025-12-03T20:36:15Z'. | +---------------+-----------------------------------------------------------------+
But the quota command says I'm consuming 2.5 CPU and 6.5G, which is a lot more than what the only running job defines.
tools.wpcleaner@tools-bastion-15:~$ toolforge jobs quota Running jobs Used Limit -------------------------------------------- ------ ------- Total running jobs at once (Kubernetes pods) 3 16 Running one-off and cron jobs 4 15 CPU 2.5 16.0 Memory 6.5Gi 8.0Gi
Per-job limits Used Limit ---------------- ------ ------- CPU 3.0 Memory 6.0Gi
Job definitions Used Limit ---------------------------------------- ------ ------- Cron jobs 8 50 Continuous jobs (including web services) 1 16