Hi,
We've had some complaints off and on about diffs taking a long time. This happens at random times, doesn't happen often and clears itself up after awhile. When this happens, the load on the Apache servers seem low and the database seems ok.
We did seem to have this problem if users had the Google toolbar installed a few months ago, but it appears that the problem is back and unrelated.
I wonder if the job queue has anything to do with it, sometimes it get's up to over 10,000 on us, but usually is at 0. I haven't been able to check what it's at when users experience this problem.
Does anyone have any ideas on how to debug, or what the possible causes might be? We're still running 1.6.7.
Thanks, Travis
Travis Derouin wrote:
Hi,
We've had some complaints off and on about diffs taking a long time. This happens at random times, doesn't happen often and clears itself up after awhile. When this happens, the load on the Apache servers seem low and the database seems ok.
We did seem to have this problem if users had the Google toolbar installed a few months ago, but it appears that the problem is back and unrelated.
I wonder if the job queue has anything to do with it, sometimes it get's up to over 10,000 on us, but usually is at 0. I haven't been able to check what it's at when users experience this problem.
Does anyone have any ideas on how to debug, or what the possible causes might be? We're still running 1.6.7.
The PHP difference engine is slow, especially where lots of distinct changes were made. Processing time can easily run into the minutes. It could be that the complaints arise from such diffs, and that it appears to be random because when you view the diff again, it is cached and thus fast.
The solution to this is to install the C++ extension "wikidiff2", which is fast for any input text.
The other possibility is that diffs are slow for reasons not directly related to diffing. Display of diff pages requires loading the revision text, which is uncached in 1.6.7, and also rendering the whole page to HTML for display underneath the diff, which can take several seconds. However, if this were the problem, you would expect other operations such as page saves to suffer as well.
-- Tim Starling
Thanks Tim.
Is there any documentation for wikidiff2 other than the readme file? I'm having issues compiling (wikidiff2.i:13: Syntax error in input.) and am not sure how to configure it - what settings do I change to enable it?
Thanks, Travis
On 10/19/06, Tim Starling t.starling@physics.unimelb.edu.au wrote:
Travis Derouin wrote:
Hi,
We've had some complaints off and on about diffs taking a long time. This happens at random times, doesn't happen often and clears itself up after awhile. When this happens, the load on the Apache servers seem low and the database seems ok.
We did seem to have this problem if users had the Google toolbar installed a few months ago, but it appears that the problem is back and unrelated.
I wonder if the job queue has anything to do with it, sometimes it get's up to over 10,000 on us, but usually is at 0. I haven't been able to check what it's at when users experience this problem.
Does anyone have any ideas on how to debug, or what the possible causes might be? We're still running 1.6.7.
The PHP difference engine is slow, especially where lots of distinct changes were made. Processing time can easily run into the minutes. It could be that the complaints arise from such diffs, and that it appears to be random because when you view the diff again, it is cached and thus fast.
The solution to this is to install the C++ extension "wikidiff2", which is fast for any input text.
The other possibility is that diffs are slow for reasons not directly related to diffing. Display of diff pages requires loading the revision text, which is uncached in 1.6.7, and also rendering the whole page to HTML for display underneath the diff, which can take several seconds. However, if this were the problem, you would expect other operations such as page saves to suffer as well.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@wikimedia.org http://mail.wikipedia.org/mailman/listinfo/wikitech-l
Travis Derouin wrote:
Thanks Tim.
Is there any documentation for wikidiff2 other than the readme file?
I doubt it.
I'm having issues compiling (wikidiff2.i:13: Syntax error in input.)
We seem to be using swig version 1.3.21, you might want to try something similar.
and am not sure how to configure it - what settings do I change to enable it?
"make install" will install the shared library into the PHP extensions directory. Then in LocalSettings.php, put:
$wgExternalDiffEngine = 'wikidiff2';
That will tell MW to use wikidiff2, and if necessary, to do a dl() to load the extension on demand. Apparently dl() is deprecated these days, so to be completely pure and true you might want to avoid it by putting:
extension=php_wikidiff2.so
in php.ini.
-- Tim Starling
Upgrading swig seemed to get me by the first part. Which version of GCC did you use? I'm getting compilation errors as well:
g++ -O3 `php-config --includes` -shared -fPIC -o php_wikidiff2.so wikidiff2.cpp wikidiff2_wrap.cpp DiffEngine.h: In member function `void _DiffEngine<T>::diff(const std::vector<T, std::allocator<_CharT> >&, const std::vector<T, std::allocator<_CharT> >&, Diff<T>&) [with T = std::string]': DiffEngine.h:560: instantiated from `Diff<T>::Diff(const std::vector<T, std::allocator<_CharT> >&, const std::vector<T, std::allocator<_CharT> >&) [with T = std::string]' wikidiff2.cpp:15: instantiated from here DiffEngine.h:196: error: `assert' undeclared (first use this function) DiffEngine.h:196: error: (Each undeclared identifier is reported only once for each function it appears in.) DiffEngine.h: In member function `void _DiffEngine<T>::diff(const std::vector<T, std::allocator<_CharT> >&, const std::vector<T,
Thanks.
On 10/25/06, Tim Starling tstarling@wikimedia.org wrote:
Travis Derouin wrote:
Thanks Tim.
Is there any documentation for wikidiff2 other than the readme file?
I doubt it.
I'm having issues compiling (wikidiff2.i:13: Syntax error in input.)
We seem to be using swig version 1.3.21, you might want to try something similar.
and am not sure how to configure it - what settings do I change to enable it?
"make install" will install the shared library into the PHP extensions directory. Then in LocalSettings.php, put:
$wgExternalDiffEngine = 'wikidiff2';
That will tell MW to use wikidiff2, and if necessary, to do a dl() to load the extension on demand. Apparently dl() is deprecated these days, so to be completely pure and true you might want to avoid it by putting:
extension=php_wikidiff2.so
in php.ini.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@wikimedia.org http://mail.wikipedia.org/mailman/listinfo/wikitech-l
Hey Tim,
It seems like if
#include <assert.h>
is added to DiffEngine.h, it takes care of this problem.
Thanks, Travis
On 10/25/06, Travis Derouin travis@wikihow.com wrote:
Upgrading swig seemed to get me by the first part. Which version of GCC did you use? I'm getting compilation errors as well:
g++ -O3 `php-config --includes` -shared -fPIC -o php_wikidiff2.so wikidiff2.cpp wikidiff2_wrap.cpp DiffEngine.h: In member function `void _DiffEngine<T>::diff(const std::vector<T, std::allocator<_CharT> >&, const std::vector<T, std::allocator<_CharT> >&, Diff<T>&) [with T = std::string]': DiffEngine.h:560: instantiated from `Diff<T>::Diff(const std::vector<T, std::allocator<_CharT> >&, const std::vector<T, std::allocator<_CharT> >&) [with T = std::string]' wikidiff2.cpp:15: instantiated from here DiffEngine.h:196: error: `assert' undeclared (first use this function) DiffEngine.h:196: error: (Each undeclared identifier is reported only once for each function it appears in.) DiffEngine.h: In member function `void _DiffEngine<T>::diff(const std::vector<T, std::allocator<_CharT> >&, const std::vector<T,
Thanks.
On 10/25/06, Tim Starling tstarling@wikimedia.org wrote:
Travis Derouin wrote:
Thanks Tim.
Is there any documentation for wikidiff2 other than the readme file?
I doubt it.
I'm having issues compiling (wikidiff2.i:13: Syntax error in input.)
We seem to be using swig version 1.3.21, you might want to try something similar.
and am not sure how to configure it - what settings do I change to enable it?
"make install" will install the shared library into the PHP extensions directory. Then in LocalSettings.php, put:
$wgExternalDiffEngine = 'wikidiff2';
That will tell MW to use wikidiff2, and if necessary, to do a dl() to load the extension on demand. Apparently dl() is deprecated these days, so to be completely pure and true you might want to avoid it by putting:
extension=php_wikidiff2.so
in php.ini.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@wikimedia.org http://mail.wikipedia.org/mailman/listinfo/wikitech-l
Hey Tim,
I have some more questions about wikidiff2. I installed it, added it to LocalSettings.php, and ran some benchmarks on diff pages, it doesn't seem to give me any improvement, at least on our development server. Loading 183 diff consecutive pages takes about 5 minutes using both wikidiff2 and the default diff.
Is there any way to tell wikidiff2 is being used and is configured properly? Or is there a better way benchmark the module? I seem to be able to call wikidiff2_do_diff manually from a test page.
Or is it possible that the benefits are only realized when there's some substantial load on the server?
Any ideas would be appreciated. Thanks.
Travis
On 11/2/06, Travis Derouin travis@wikihow.com wrote:
Hey Tim,
It seems like if
#include <assert.h>
is added to DiffEngine.h, it takes care of this problem.
Thanks, Travis
On 10/25/06, Travis Derouin travis@wikihow.com wrote:
Upgrading swig seemed to get me by the first part. Which version of GCC did you use? I'm getting compilation errors as well:
g++ -O3 `php-config --includes` -shared -fPIC -o php_wikidiff2.so wikidiff2.cpp wikidiff2_wrap.cpp DiffEngine.h: In member function `void _DiffEngine<T>::diff(const std::vector<T, std::allocator<_CharT> >&, const std::vector<T, std::allocator<_CharT> >&, Diff<T>&) [with T = std::string]': DiffEngine.h:560: instantiated from `Diff<T>::Diff(const std::vector<T, std::allocator<_CharT> >&, const std::vector<T, std::allocator<_CharT> >&) [with T = std::string]' wikidiff2.cpp:15: instantiated from here DiffEngine.h:196: error: `assert' undeclared (first use this function) DiffEngine.h:196: error: (Each undeclared identifier is reported only once for each function it appears in.) DiffEngine.h: In member function `void _DiffEngine<T>::diff(const std::vector<T, std::allocator<_CharT> >&, const std::vector<T,
Thanks.
On 10/25/06, Tim Starling tstarling@wikimedia.org wrote:
Travis Derouin wrote:
Thanks Tim.
Is there any documentation for wikidiff2 other than the readme file?
I doubt it.
I'm having issues compiling (wikidiff2.i:13: Syntax error in input.)
We seem to be using swig version 1.3.21, you might want to try something similar.
and am not sure how to configure it - what settings do I change to enable it?
"make install" will install the shared library into the PHP extensions directory. Then in LocalSettings.php, put:
$wgExternalDiffEngine = 'wikidiff2';
That will tell MW to use wikidiff2, and if necessary, to do a dl() to load the extension on demand. Apparently dl() is deprecated these days, so to be completely pure and true you might want to avoid it by putting:
extension=php_wikidiff2.so
in php.ini.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@wikimedia.org http://mail.wikipedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org