After updating MediaWiki, PHP, and MySQL, I'm fairly consistently getting
php-fpm worker crashes when I try to edit articles on my wiki.
I think I can rule out database corruption. mysql_check says everything is OK,
and even after dropping the database and restoring from a mysqldump backup,
the crashes continue.
I've enabled error reporting & logging in php.ini, php-fpm.conf, and my.cnf.
The MySQL log reveals that the crash, if it happens (it doesn't always) is
always right after a certain MySQL query runs. Meanwhile, php_error.log says
nothing (but is working), and php-fpm.log says nothing of note, just "[pool
www] child 17491 exited on signal 11 (SIGSEGV - core dumped) after 0.000125
seconds from start".
After adjusting some kernel settings and rebuilding PHP with --enable-debug, I
managed to finally get some core dumps with symbols. Using gdb, I saw in one
of them that it looks like pcre could be to blame:
---------------------------------------------------------------------------
[...]
Reading symbols from /usr/local/sbin/php-fpm...done.
[New LWP 100226]
Core was generated by `php-fpm: pool www'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x26641354 in ?? ()
(gdb) bt
#0 0x26641354 in ?? ()
#1 0x2065b634 in ?? () from /usr/local/lib/libpcre.so.1
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
---------------------------------------------------------------------------
However, even after updating pcre, PHP and all its extensions, and nginx, the
crashes persist, again at the same spot. And now, for some reason, the
backtrace is no longer mentioning pcre by name:
---------------------------------------------------------------------------
[...]
Reading symbols from /usr/local/sbin/php-fpm...done.
[New LWP 100136]
Core was generated by `php-fpm: pool www'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x268b5412 in ?? ()
(gdb) bt
#0 0x268b5412 in ?? ()
#1 0x26564d64 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
---------------------------------------------------------------------------
Anyway, here is the last message in the MySQL log just before the crash:
---------------------------------------------------------------------------
SELECT /* LCStoreDB::get */ lc_value FROM `mw_l10n_cache` WHERE lc_lang = 'en' AND lc_key = 'magicWords' LIMIT 1
---------------------------------------------------------------------------
And here is what the next message is, those times when there is no crash:
---------------------------------------------------------------------------
SELECT /* WikiPage::pageData */ page_id,page_namespace,page_title,page_restrictions,page_is_redirect,page_is_new,page_random,page_touched,page_links_updated,page_latest,page_len,page_content_model FROM `mw_page` WHERE page_namespace = '2' AND page_title = 'Mjb/FreeBSD_on_BeagleBone_Black/Additional_software' LIMIT 1;
---------------------------------------------------------------------------
Both of these queries are benign; I can run them from the MySQL client or from
a simple PHP CGI test webpage with no problem.
Does anyone have any idea what could be going wrong or how to further
diagnose?
Thanks,
Mike
(Info about my system follows:)
---------------------------------------------------------------------------
FreeBSD 12.0-RELEASE-p3 running on a BeagleBone Black.
The MySQL connections are via sockets, not TCP.
selected output from 'pkg info':
mediawiki132-php72-1.32.0 Wiki engine used by Wikipedia
mysql57-client-5.7.25 Multithreaded SQL database (client)
mysql57-server-5.7.25 Multithreaded SQL database (server)
nginx-1.14.2_13,2 Robust and small WWW server
pcre-8.43_1 Perl Compatible Regular Expressions library
php72-7.2.17_2 PHP Scripting Language
php72-ctype-7.2.17_2 The ctype shared extension for php
php72-curl-7.2.17_2 The curl shared extension for php
php72-dom-7.2.17_2 The dom shared extension for php
php72-fileinfo-7.2.17_2 The fileinfo shared extension for php
php72-hash-7.2.17_2 The hash shared extension for php
php72-iconv-7.2.17_2 The iconv shared extension for php
php72-intl-7.2.17_2 The intl shared extension for php
php72-json-7.2.17_2 The json shared extension for php
php72-mbstring-7.2.17_2 The mbstring shared extension for php
php72-mysqli-7.2.17_2 The mysqli shared extension for php
php72-opcache-7.2.17_2 The opcache shared extension for php
php72-openssl-7.2.17_2 The openssl shared extension for php
php72-pcntl-7.2.17_2 The pcntl shared extension for php
php72-pdo-7.2.17_2 The pdo shared extension for php
php72-pdo_mysql-7.2.17_2 The pdo_mysql shared extension for php
php72-posix-7.2.17_2 The posix shared extension for php
php72-readline-7.2.17_2 The readline shared extension for php
php72-session-7.2.17_2 The session shared extension for php
php72-xml-7.2.17_2 The xml shared extension for php
php72-xmlreader-7.2.17_2 The xmlreader shared extension for php
php72-xmlrpc-7.2.17_2 The xmlrpc shared extension for php
php72-zlib-7.2.17_2 The zlib shared extension for php
---------------------------------------------------------------------------