What's a good way of rotating logs on Toolforge? I haven't found anything in the documentation on Wikitech. For context, in this case I'm looking to use it for logs from Celery workers. I figured I can create a scheduled job that runs logrotate in the tools home directory. Is there any other, recommended or better way to do this?
*Sebastian Berlin* Utvecklare/*Developer* Wikimedia Sverige (WMSE)
E-post/*E-Mail*: sebastian.berlin@wikimedia.se Telefon/*Phone*: (+46) 0707 - 92 03 84
On Thu, Jul 20, 2023 at 3:53 AM Sebastian Berlin sebastian.berlin@wikimedia.se wrote:
What's a good way of rotating logs on Toolforge? I haven't found anything in the documentation on Wikitech. For context, in this case I'm looking to use it for logs from Celery workers. I figured I can create a scheduled job that runs logrotate in the tools home directory. Is there any other, recommended or better way to do this?
Your instinct to use a scheduled job and logrotate is likely as good a solution as exists today. Maybe someone on the list has an example of what they do that they can share?
There have been many Phabricator tickets over the years related to a need for better log management solutions in Toolforge [0][1][2][3], but to date none of them have lead to major feature work or technological breakthroughs.
One of the big challenges has turned out to be finding a FOSS log aggregation system that has strong support for multi-tenancy. In the Toolforge environment we would not typically want the logs of a given tool to be readable by non-maintainers. The main reason for this concern is the potential exposure of secrets like wiki account passwords and OAuth tokens. It has been quite a while, probably nearly 5 years, since I looked deeply into the options in this space. There may be more featureful solutions today.
I think this is an area where the Toolforge admins would welcome help from technical folks who have the time and energy to review available options, do exploratory testing, or even plan and execute a complete deployment of a solution.
[0]: https://phabricator.wikimedia.org/T152235 [1]: https://phabricator.wikimedia.org/T97861 [2]: https://phabricator.wikimedia.org/T127367 [3]: https://phabricator.wikimedia.org/T50846
Bryan
*> Maybe someone on the list has an example of what they do that they can share?*
Here is mine
toolforge-jobs run prune-old-logs --command "php7.4 /path/to/my/executable/prune-old-logs.php" --image tf-php74 --no-filelog --schedule "@daily"
More info here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Creating_s...)
On Thu, Jul 20, 2023 at 11:31 AM Bryan Davis bd808@wikimedia.org wrote:
On Thu, Jul 20, 2023 at 3:53 AM Sebastian Berlin sebastian.berlin@wikimedia.se wrote:
What's a good way of rotating logs on Toolforge? I haven't found
anything in the documentation on Wikitech. For context, in this case I'm looking to use it for logs from Celery workers. I figured I can create a scheduled job that runs logrotate in the tools home directory. Is there any other, recommended or better way to do this?
Your instinct to use a scheduled job and logrotate is likely as good a solution as exists today. Maybe someone on the list has an example of what they do that they can share?
There have been many Phabricator tickets over the years related to a need for better log management solutions in Toolforge [0][1][2][3], but to date none of them have lead to major feature work or technological breakthroughs.
One of the big challenges has turned out to be finding a FOSS log aggregation system that has strong support for multi-tenancy. In the Toolforge environment we would not typically want the logs of a given tool to be readable by non-maintainers. The main reason for this concern is the potential exposure of secrets like wiki account passwords and OAuth tokens. It has been quite a while, probably nearly 5 years, since I looked deeply into the options in this space. There may be more featureful solutions today.
I think this is an area where the Toolforge admins would welcome help from technical folks who have the time and energy to review available options, do exploratory testing, or even plan and execute a complete deployment of a solution.
Bryan
Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808 _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/