On Thu, Jul 20, 2023 at 3:53 AM Sebastian Berlin
<sebastian.berlin@wikimedia.se> wrote:
>
> What's a good way of rotating logs on Toolforge? I haven't found anything in the documentation on Wikitech. For context, in this case I'm looking to use it for logs from Celery workers. I figured I can create a scheduled job that runs logrotate in the tools home directory. Is there any other, recommended or better way to do this?

Your instinct to use a scheduled job and logrotate is likely as good a
solution as exists today. Maybe someone on the list has an example of
what they do that they can share?

There have been many Phabricator tickets over the years related to a
need for better log management solutions in Toolforge [0][1][2][3],
but to date none of them have lead to major feature work or
technological breakthroughs.

One of the big challenges has turned out to be finding a FOSS log
aggregation system that has strong support for multi-tenancy. In the
Toolforge environment we would not typically want the logs of a given
tool to be readable by non-maintainers. The main reason for this
concern is the potential exposure of secrets like wiki account
passwords and OAuth tokens. It has been quite a while, probably nearly
5 years, since I looked deeply into the options in this space. There
may be more featureful solutions today.

I think this is an area where the Toolforge admins would welcome help
from technical folks who have the time and energy to review available
options, do exploratory testing, or even plan and execute a complete
deployment of a solution.

[0]: https://phabricator.wikimedia.org/T152235
[1]: https://phabricator.wikimedia.org/T97861
[2]: https://phabricator.wikimedia.org/T127367
[3]: https://phabricator.wikimedia.org/T50846


Bryan
--
Bryan Davis              Technical Engagement      Wikimedia Foundation
Principal Software Engineer                               Boise, ID USA
[[m:User:BDavis_(WMF)]]                                      irc: bd808
_______________________________________________
Cloud mailing list -- cloud@lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/