[Toolserver-l] Server switch for s3/s4/s6, Monday morning UTC

[River Tarnell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Mashiah Davidson:
> > Hm... I knew I forget to mention something.  Yes,
> > maximum_max_heap_table_size is now set to 128MB.
 
> This disables my tool to be functional.
 
> > 4GB is *way* too much; we don't have even close to that much free memory
> > available on the MySQL servers.
 
> As far as I've seen during last two years, it was ok.

I don't think so.  Over the last year or so, we've seen a lot of
problems with MySQL using too much memory for no apparent reason, and
sometimes running out of memory and crashing.  (This is not something
that's just been happening in the last couple of months.)  I wasn't able
to find anything that might cause this, but I did not consider that
someone would create 4GB of MEMORY tables.

> As you remember, we've experienced issues with memory during last few
> weeks. 

We've been having this issue a lot longer than that.  It was simply that
in the last month or so, it caused more MySQL crashes than previously.

> Sorry if something looks too emotional in this message; indeed too
> much efforts were spent to implement it and I am really not sure "way
> too much" statement is well proved to cause the restrictions just
> introduced.

I understand this seems like we're breaking something that previously
worked.  However, that is not the case.  This has _never_ worked
properly, we just weren't able to find the cause before.  This excessive
memory use is causing significant problems for all Toolservers users, in
the form of unreliable MySQL servers and slow performance (= slow
queries, higher replication lag).

The statement is trivial to prove: we do not have 4GB free memory on
MySQL servers.  The reason we buy servers with 32GB RAM is so we can
*use* 32GB of memory.  There is no free memory (beyond a little bit that
the OS needs to function).  If we use 32GB, and you use 4GB, we are
using 36GB.  The server only has 32G.  The only possibly result of that
is either it will start swapping (which completely kills performance),
or it runs out of memory and crashes.

Still, since I have no absolutely certainly proof that golem causes this
problem, I will monitor memory use on cassia over the next month or so.
If we see the same issue again, even though MEMORY tables are limited to
128MB, I'll consider that something else might be the cause.

	- river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (HP-UX)

iEYEARECAAYFAkuxCCgACgkQIXd7fCuc5vIf+QCeLQr6Irm1KEIryEySn/ZXCnzO
6fYAniCuVG46EYc1xhGQ2IFtkx5o4Xsj
=Feh0
-----END PGP SIGNATURE-----