php-fpm worker crashes when editing article - MediaWiki-l

19 Apr 2019

After updating MediaWiki, PHP, and MySQL, I'm fairly consistently getting 
php-fpm worker crashes when I try to edit articles on my wiki.

I think I can rule out database corruption. mysql_check says everything is OK, 
and even after dropping the database and restoring from a mysqldump backup, 
the crashes continue.

I've enabled error reporting & logging in php.ini, php-fpm.conf, and my.cnf.

The MySQL log reveals that the crash, if it happens (it doesn't always) is 
always right after a certain MySQL query runs. Meanwhile, php_error.log says 
nothing (but is working), and php-fpm.log says nothing of note, just "[pool 
www] child 17491 exited on signal 11 (SIGSEGV - core dumped) after 0.000125 
seconds from start".

After adjusting some kernel settings and rebuilding PHP with --enable-debug, I 
managed to finally get some core dumps with symbols. Using gdb, I saw in one 
of them that it looks like pcre could be to blame:

---------------------------------------------------------------------------
[...]
Reading symbols from /usr/local/sbin/php-fpm...done.
[New LWP 100226]
Core was generated by `php-fpm: pool www'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x26641354 in ?? ()
(gdb) bt
#0  0x26641354 in ?? ()
#1  0x2065b634 in ?? () from /usr/local/lib/libpcre.so.1
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
---------------------------------------------------------------------------

However, even after updating pcre, PHP and all its extensions, and nginx, the 
crashes persist, again at the same spot. And now, for some reason, the 
backtrace is no longer mentioning pcre by name:

---------------------------------------------------------------------------
[...]
Reading symbols from /usr/local/sbin/php-fpm...done.
[New LWP 100136]
Core was generated by `php-fpm: pool www'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x268b5412 in ?? ()
(gdb) bt
#0  0x268b5412 in ?? ()
#1  0x26564d64 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
---------------------------------------------------------------------------

Anyway, here is the last message in the MySQL log just before the crash:

---------------------------------------------------------------------------
SELECT /* LCStoreDB::get  */  lc_value  FROM `mw_l10n_cache`    WHERE lc_lang =
'en' AND lc_key = 'magicWords'  LIMIT 1
---------------------------------------------------------------------------

And here is what the next message is, those times when there is no crash:

---------------------------------------------------------------------------
SELECT /* WikiPage::pageData  */ 
page_id,page_namespace,page_title,page_restrictions,page_is_redirect,page_is_new,page_random,page_touched,page_links_updated,page_latest,page_len,page_content_model
 FROM `mw_page`    WHERE page_namespace = '2' AND page_title =
'Mjb/FreeBSD_on_BeagleBone_Black/Additional_software'  LIMIT 1;
---------------------------------------------------------------------------

Both of these queries are benign; I can run them from the MySQL client or from 
a simple PHP CGI test webpage with no problem.

Does anyone have any idea what could be going wrong or how to further 
diagnose?

Thanks,
Mike

(Info about my system follows:)
---------------------------------------------------------------------------
FreeBSD 12.0-RELEASE-p3 running on a BeagleBone Black.

The MySQL connections are via sockets, not TCP.

selected output from 'pkg info':

mediawiki132-php72-1.32.0      Wiki engine used by Wikipedia
mysql57-client-5.7.25          Multithreaded SQL database (client)
mysql57-server-5.7.25          Multithreaded SQL database (server)
nginx-1.14.2_13,2              Robust and small WWW server
pcre-8.43_1                    Perl Compatible Regular Expressions library
php72-7.2.17_2                 PHP Scripting Language
php72-ctype-7.2.17_2           The ctype shared extension for php
php72-curl-7.2.17_2            The curl shared extension for php
php72-dom-7.2.17_2             The dom shared extension for php
php72-fileinfo-7.2.17_2        The fileinfo shared extension for php
php72-hash-7.2.17_2            The hash shared extension for php
php72-iconv-7.2.17_2           The iconv shared extension for php
php72-intl-7.2.17_2            The intl shared extension for php
php72-json-7.2.17_2            The json shared extension for php
php72-mbstring-7.2.17_2        The mbstring shared extension for php
php72-mysqli-7.2.17_2          The mysqli shared extension for php
php72-opcache-7.2.17_2         The opcache shared extension for php
php72-openssl-7.2.17_2         The openssl shared extension for php
php72-pcntl-7.2.17_2           The pcntl shared extension for php
php72-pdo-7.2.17_2             The pdo shared extension for php
php72-pdo_mysql-7.2.17_2       The pdo_mysql shared extension for php
php72-posix-7.2.17_2           The posix shared extension for php
php72-readline-7.2.17_2        The readline shared extension for php
php72-session-7.2.17_2         The session shared extension for php
php72-xml-7.2.17_2             The xml shared extension for php
php72-xmlreader-7.2.17_2       The xmlreader shared extension for php
php72-xmlrpc-7.2.17_2          The xmlrpc shared extension for php
php72-zlib-7.2.17_2            The zlib shared extension for php
---------------------------------------------------------------------------