I believe there was a configuration error this morning which got some redirects stuck in cache; they should be on the way to being fixed...

-- brion

On Tue, Oct 22, 2013 at 11:56 AM, Eugene Zelenko eugene.zelenko@gmail.comwrote:

Hi!

I experienced weird problem today when accessing Commons (addresses like https://commons.wikimedia.org/wiki/Special:Watchlist and http://commopns.wikimedia.org/wiki/Category:Yuri_Gagarin): I was redirected to same pages on http://wikimediafoundation.org.

Eugene.

---

Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Hi,

here's a more detailed report:

As part of an ongoing effort to simplify, cleanup and reduce code lines of the Apache cluster config, we we're planning to unify a lot of document roots for the "www"-portals into a single docroot, like here:

https://gerrit.wikimedia.org/r/#/c/90669/

looking at that we saw that the www.wikimedia.org portal wasn't handled by this, unlike the other portals, so to further unify this we merged:

https://gerrit.wikimedia.org/r/#/c/91195/

which "just" moved the existing config for www.wikimedia.org to the wwwportals.conf file.

But because this had an old " ServerAlias *.wikimedia.org" in it, which now changed in the order Apache goes through the config, it caused *.wikimedia.org URLs to redirect to wikimediafoundation.org

.. ServerAlias *.wikimedia.org .. RewriteRule ^/wiki/(.*)$ http://wikimediafoundation.org/wiki/$1 [R=301,L]

We reverted both of the recent changes after just a few minutes and synced Apache config and restarted them.

Users still reported problems though due to caching. So we started some Squid purging and Mark banned php content-type in varnish.

Additionally it turned out several Apaches didn't get restarted properly by apache-graceful-all, and using apache-fast-test with the pybal option we found some more that needed manual restarts a little while later.

For futher fixes Brandon banned by object size: did < bblack> !log varnish: banned 'obj.http.content-length == 33518' on text varnishes everywhere (extract2.php leakage). .. Reedy already prepared new patches to fix the root cause ..and apache-fast-test should check eqiad now instead of Tampa: https://gerrit.wikimedia.org/r/#/c/91270/

Sorry for breakage! Thank you for help!