This is insane in my opinion. There sure is a better reason to cause service
disription for lots of hardworking volunteers in a way that there is almost no
way to find out whats going on.
Toolserver users genrally don't work on their tools every day. I just got home
and after getting stuff running I hear reports that people can't log into my
php tools. The fact that I have time to look into this right away is probably
an exception compared to the average ts-user. Two of the MMPs I maintain are
broken, and several irc bots are down.
I see:
* No automatic e-mails or anything
* Nothing in php errors
* No mysql errors (since the error report I got mentinoed that users couldn't
log into the tool)
* No obvious thread on toolserver-l (lots of noise there anyway, maybe we need
a separate toolserver-announce-l for stuff that actually matters that likely
need users to do something?)
Now, in the mean time I think I know what caused it. But just so you know, here
is a short summary of how I've spend the last 3 hours trying to figure out what
the hell is going on. And hopefully will encourage ts-admins to act more
carefully or at least better communicate.
One of the MMPs is CVN. The IRC bots timed out earlier today and those that I
granted access to start them from a web control panel couldn't log in. Turns
out that the PHP sessions were the issue. For some reason whenever the session
was modified, it was emptied and the user was as if there is no active session.
Whenever a new session is created, it appears to work fine, until you look up
the data in the next request and find it is gone.
errors and then ssh-ing into my account and trying to access the database
directly, it turns out everything looks fine.
I opened TS-1422[1], and worked on a test case to reproduce it in a plain .php
file. Tried to upload it to /home/krinkle/public_html/tmp and everything seems
to have gone fine. No errors or anything out of the ordinary.
Then when I try to access that file from the web, I get a blank 200 OK reponse.
Looking it up in SSH shows me it is chmod 000 and size 0 bytes. So I opened up
TS-1423[2].
Then I'm reading up on toolserver-l and see that the quotas are finally going
to enforced. I welcome that. DaB tells us we have the weekend to make sure we
are either under the quota or have requested a bigger quota. This sounds
reasonable to me.
I did not connect the problems I was hearing about from all over with the quota
that was going to come after the weekend. The reason being that I did not get
any emails regarding limits being reached on /home/krinkle (or the home of the
MMPs) or any errors when trying to write to a file.
e.g. $ echo "Hello World" > test.txt
.. works fine without errors. But looking it up shows it is size 0 bytes. If
this is indeed being done by the quota system, then I'd recommend getting a
better quota system or configuring it differenly. Allowing empty files to be
created is one thing. Silently ignoring non-empty write attempts and turning
them into empty files without any form of response is quite another. Obviously
I'd rather have no file at all, then a broken file without any indication that
it is broken.
Also, $ quota -v; gives me this rather useless response:
cvn@willow:~$ quota -v cvn;
Disk quotas for cvn (uid 8153):
Filesystem usage quota limit timeleft files quota limit timeleft
Looks like something is missing there?
Connect that to DaB's mail, and I'd say this means the quota will come, but is
not yet started/activated. So I spend another hour trying to find out the
"real" cause (which, obviously, I didn't find since it is indeed caused by the
quota). And tried to temporarily disabled a few things only to find out that
the files I modified are gone:
For example:
* /home/krinkle/public_html/wikimedia-svn-search/header.php - 0 bytes
* /home/krinkle/public_html/tmp/session-test.php - 0 bytes
And then I see your message that (albeit it not appearing so) the quota has
indeed been enabled for everyone now. Why? Now I can't even try to clean up,
because I can't even edit a big file and replace it with "Temporarily
disabled". I can't remove 100MB to add a small .ini file. I can't comment out
things that are breaking stuff. My account is completely locked and anything I
try to touch is immediately wiped. Error/warnings are absent.
On IRC it was pointed out that logging in would tell me if the limit is
reached. Looking again, it does say "Block limit reached on /home". But
considering it makes no mention of "quota" and no mention of "/home/krinkle". I
didn't notice it. And also, it was placed in no particularly attention-grabby
way. Just on the bottom of the welcome screen.
And then there is the fact that that is only for personal accounts. For MMPs
there is no welcome screen. So for MMPs this information is not expressed in
any place I know of.
So, afraid of touching anything else, I'll log out, and wait for things to be
fixed on your end. So I can then fix things on my end.
Thanks,
-- Krinkle