Yesterday, I reported some log messages in T318479 https://phabricator.wikimedia.org/T318479:

logs/django/django.log.2022-09-27:2022-09-28 16:43:39,873 [76e999afc82c10fb99b6c9bf76448d1a] INFO tools_app.middleware: IndexView() logs/django/django.log.2022-09-27:2022-09-28 16:59:18,903 [76e999afc82c10fb99b6c9bf76448d1a] ERROR tools_app.redis: Redis ConnectionError: Error while reading from tools-redis.svc.eqiad.wmflabs:6379 : (110, 'Connection timed out') logs/django/django.log.2022-09-27:2022-09-28 16:59:19,196 [76e999afc82c10fb99b6c9bf76448d1a] INFO tools_app.middleware: request took 0:15:39.323408

When I went to look at this again today, the messages were gone. After a bit of head-scratching, I discovered they were now in a .nfs file:

(venv) spi-tools-dev [django] grep 76e999afc82c10fb99b6c9bf76448d1a .nfs0000000005f910c800000388 2022-09-28 16:43:39,873 [76e999afc82c10fb99b6c9bf76448d1a] INFO tools_app.middleware: IndexView() 2022-09-28 16:59:18,903 [76e999afc82c10fb99b6c9bf76448d1a] ERROR tools_app.redis: Redis ConnectionError: Error while reading from tools-redis.svc.eqiad.wmflabs:6379 : (110, 'Connection timed out') 2022-09-28 16:59:19,196 [76e999afc82c10fb99b6c9bf76448d1a] INFO tools_app.middleware: request took 0:15:39.323408

These log files are created by Python's TimedRotatingFileHandler. So it looks like something was holding the file open at the time it was rotated. In theory, I should be able to find what process has them open using lsof, but that doesn't work when I run it on tools-sgebastion-11:

lsof .nfs0000000005f910c800000388 lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing Output information may be incomplete.

and if I shell into the krb instance, I just get:

bash: lsof: command not found

So how do I figure out what's going on?