We just finished deploying a new SSL certificate to the sites. Now all *.m and *. certificates are included in a single certificate, except mediawiki.org. Unfortunately we somehow forgot mediawiki.org when we ordered the updated cert. We'll be replacing this soon with another cert that had mediawiki.org included.
This should fix any certificate errors that folks have been seeing on non-wikipedia m. domains.
- Ryan
On Tue, Mar 12, 2013 at 3:43 PM, Ryan Lane rlane32@gmail.com wrote:
We just finished deploying a new SSL certificate to the sites. Now all *.m and *. certificates are included in a single certificate, except mediawiki.org. Unfortunately we somehow forgot mediawiki.org when we ordered the updated cert. We'll be replacing this soon with another cert that had mediawiki.org included.
This should fix any certificate errors that folks have been seeing on non-wikipedia m. domains.
Thanks guys!
-- brion
On Tue, Mar 12, 2013 at 4:47 PM, Brion Vibber brion@pobox.com wrote:
Thanks guys!
-- brion
Don't thank us too quick. We needed to revert this for mobile. Seems *. m.wikipedia.org was also missing from the cert. Needless to say I'll be writing a script that can be run against a cert to ensure it's not missing anything. We'll also be adding monitoring to check for invalid certificates for any top level domain.
- Ryan
On 12 March 2013 21:15, Ryan Lane rlane32@gmail.com wrote:
On Tue, Mar 12, 2013 at 4:47 PM, Brion Vibber brion@pobox.com wrote:
Thanks guys!
-- brion
Don't thank us too quick. We needed to revert this for mobile. Seems *. m.wikipedia.org was also missing from the cert. Needless to say I'll be writing a script that can be run against a cert to ensure it's not missing anything. We'll also be adding monitoring to check for invalid certificates for any top level domain.
I think it might also be missing some of the small/private wikis. I got "bad certificate" messages for the English Wikipedia Arbcom wiki tonight.
But thanks for working on this.
Risker/Anne
You mean: https://arbcom.en.wikipedia.org ?
Our certificates have never covered that. That's a sub-sub domain, and our certs only cover single subdomains. We really need to rename all of our sub-sub domains to single subdomains for them to be covered (or we need to include every sub-subdomain in the unified cert, but that's going to bloat it).
On Tue, Mar 12, 2013 at 9:18 PM, Risker risker.wp@gmail.com wrote:
On 12 March 2013 21:15, Ryan Lane rlane32@gmail.com wrote:
On Tue, Mar 12, 2013 at 4:47 PM, Brion Vibber brion@pobox.com wrote:
Thanks guys!
-- brion
Don't thank us too quick. We needed to revert this for mobile. Seems *. m.wikipedia.org was also missing from the cert. Needless to say I'll be writing a script that can be run against a cert to ensure it's not
missing
anything. We'll also be adding monitoring to check for invalid
certificates
for any top level domain.
I think it might also be missing some of the small/private wikis. I got "bad certificate" messages for the English Wikipedia Arbcom wiki tonight.
But thanks for working on this.
Risker/Anne _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Yes, that's the wiki I mean. And I can see your point about all those sub-subdomains; there must be a stack of them. The domain name was changed fairly recently and we got the bad cert messages then, and added our exceptions. Tonight we got the messages again. Perhaps it was because the subdomain's cert changed.
Risker/Anne
On 13 March 2013 00:30, Ryan Lane rlane32@gmail.com wrote:
You mean: https://arbcom.en.wikipedia.org ?
Our certificates have never covered that. That's a sub-sub domain, and our certs only cover single subdomains. We really need to rename all of our sub-sub domains to single subdomains for them to be covered (or we need to include every sub-subdomain in the unified cert, but that's going to bloat it).
On Tue, Mar 12, 2013 at 9:18 PM, Risker risker.wp@gmail.com wrote:
On 12 March 2013 21:15, Ryan Lane rlane32@gmail.com wrote:
On Tue, Mar 12, 2013 at 4:47 PM, Brion Vibber brion@pobox.com wrote:
Thanks guys!
-- brion
Don't thank us too quick. We needed to revert this for mobile. Seems *. m.wikipedia.org was also missing from the cert. Needless to say I'll
be
writing a script that can be run against a cert to ensure it's not
missing
anything. We'll also be adding monitoring to check for invalid
certificates
for any top level domain.
I think it might also be missing some of the small/private wikis. I got "bad certificate" messages for the English Wikipedia Arbcom wiki tonight.
But thanks for working on this.
Risker/Anne _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Tue, Mar 12, 2013 at 9:45 PM, Risker risker.wp@gmail.com wrote:
Yes, that's the wiki I mean. And I can see your point about all those sub-subdomains; there must be a stack of them. The domain name was changed fairly recently and we got the bad cert messages then, and added our exceptions. Tonight we got the messages again. Perhaps it was because the subdomain's cert changed.
Ah. Yes. You're likely to get another round of messages when we get the new cert in as well. All of the arbcom wikis, and other sub-subdomain wikis are in the same cluster and use the same IP addresses and as such use the same certificates. I wonder if we have a bug in about renaming sub-subdomain wikis....
- Ryan
On Wed, Mar 13, 2013 at 12:50 AM, Ryan Lane rlane32@gmail.com wrote:
I wonder if we have a bug in about renaming sub-subdomain wikis....
Let's use Bug 31335 - https://bugzilla.wikimedia.org/31335
-Jeremy
On 13 March 2013 00:50, Ryan Lane rlane32@gmail.com wrote:
On Tue, Mar 12, 2013 at 9:45 PM, Risker risker.wp@gmail.com wrote:
Yes, that's the wiki I mean. And I can see your point about all those sub-subdomains; there must be a stack of them. The domain name was
changed
fairly recently and we got the bad cert messages then, and added our exceptions. Tonight we got the messages again. Perhaps it was because
the
subdomain's cert changed.
Ah. Yes. You're likely to get another round of messages when we get the new cert in as well. All of the arbcom wikis, and other sub-subdomain wikis are in the same cluster and use the same IP addresses and as such use the same certificates. I wonder if we have a bug in about renaming sub-subdomain wikis....
Well, if you need to rename the wiki again, a bit of notice would be appreciated; it was rather a shock when folks went to log in using bookmarks, only to find they no longer worked. We still haven't finished cleaning up links. :-)
But as to security certificates, I do want to thank the team - this does make a difference for a lot of users.
Risker/Anne
On Tue, Mar 12, 2013 at 10:06 PM, Risker risker.wp@gmail.com wrote:
Well, if you need to rename the wiki again, a bit of notice would be appreciated; it was rather a shock when folks went to log in using bookmarks, only to find they no longer worked. We still haven't finished cleaning up links. :-)
Hm. Kind of annoying that they were renamed, but weren't renamed to something that would solve the certificate issues. :(
If we rename them to solve the certificate issues, I'll make sure the community is notified well in advance.
But as to security certificates, I do want to thank the team - this does make a difference for a lot of users.
Great. Glad to hear it!
- Ryan
----- Original Message -----
From: "Ryan Lane" rlane32@gmail.com
We just finished deploying a new SSL certificate to the sites. Now all *.m and *. certificates are included in a single certificate, except mediawiki.org. Unfortunately we somehow forgot mediawiki.org when we ordered the updated cert. We'll be replacing this soon with another cert that had mediawiki.org included.
This should fix any certificate errors that folks have been seeing on non-wikipedia m. domains.
Hey, Ryan; did you see, perhaps on outages-discussion, the after action report from Microsoft about how their Azure SSL cert expiration screwup happened?
Cheers, -- jra
On Wed, Mar 13, 2013 at 8:12 PM, Jay Ashworth jra@baylink.com wrote:
----- Original Message -----
From: "Ryan Lane" rlane32@gmail.com
We just finished deploying a new SSL certificate to the sites. Now all *.m and *. certificates are included in a single certificate, except mediawiki.org. Unfortunately we somehow forgot mediawiki.org when we ordered the updated cert. We'll be replacing this soon with another cert that had mediawiki.org included.
This should fix any certificate errors that folks have been seeing on non-wikipedia m. domains.
Hey, Ryan; did you see, perhaps on outages-discussion, the after action report from Microsoft about how their Azure SSL cert expiration screwup happened?
What's the relevance here?
- Ryan
----- Original Message -----
From: "Ryan Lane" rlane32@gmail.com
Hey, Ryan; did you see, perhaps on outages-discussion, the after action report from Microsoft about how their Azure SSL cert expiration screwup happened?
What's the relevance here?
"Does ops have a procedure for avoiding unexpected SSL cert expirations, and does this affect it in any way other than making it easier to implement?", I would think...
Cheers, -- jra
On Wed, Mar 13, 2013 at 9:24 PM, Jay Ashworth jra@baylink.com wrote:
----- Original Message -----
From: "Ryan Lane" rlane32@gmail.com
Hey, Ryan; did you see, perhaps on outages-discussion, the after action report from Microsoft about how their Azure SSL cert expiration screwup happened?
What's the relevance here?
"Does ops have a procedure for avoiding unexpected SSL cert expirations, and does this affect it in any way other than making it easier to implement?", I would think...
We didn't have a certificate expiration. We replaced all individual certificates, delivered by different top level domains, with a single unified certificate. This change was to fix certificate errors being shown on all non-wikipedia domains for HTTPS mobile users, who were being delivered the *.wikipedia.org certificate for all domains.
The unified certificate was missing 6 Subject Alternative Names: mediawiki.org, *.mediawiki.org, m.mediawiki.org, *.m.mediawiki.org, m.wikipedia.org and *.m.wikipedia.org. Shortly after deploying the certificate we noticed it was bad and reverted the affected services ( mediawiki.org and mobile) back to their individual certificates. The change only affected a small portion of users for a short period of time.
If you notice, I've already mentioned how we'll avoid and more quickly detect problems like this in the future:
"Needless to say I'll be writing a script that can be run against a cert to ensure it's not missing anything. We'll also be adding monitoring to check for invalid certificates for any top level domain."
- Ryan
----- Original Message -----
From: "Ryan Lane" rlane32@gmail.com
On Wed, Mar 13, 2013 at 9:24 PM, Jay Ashworth jra@baylink.com wrote:
----- Original Message -----
From: "Ryan Lane" rlane32@gmail.com
Hey, Ryan; did you see, perhaps on outages-discussion, the after action report from Microsoft about how their Azure SSL cert expiration screwup happened?
What's the relevance here?
"Does ops have a procedure for avoiding unexpected SSL cert expirations, and does this affect it in any way other than making it easier to implement?", I would think...
We didn't have a certificate expiration. We replaced all individual certificates, delivered by different top level domains, with a single unified certificate. This change was to fix certificate errors being shown on all non-wikipedia domains for HTTPS mobile users, who were being delivered the *.wikipedia.org certificate for all domains.
The unified certificate was missing 6 Subject Alternative Names: mediawiki.org, *.mediawiki.org, m.mediawiki.org, *.m.mediawiki.org, m.wikipedia.org and *.m.wikipedia.org. Shortly after deploying the certificate we noticed it was bad and reverted the affected services ( mediawiki.org and mobile) back to their individual certificates. The change only affected a small portion of users for a short period of time.
If you notice, I've already mentioned how we'll avoid and more quickly detect problems like this in the future:
"Needless to say I'll be writing a script that can be run against a cert to ensure it's not missing anything. We'll also be adding monitoring to check for invalid certificates for any top level domain."
I don't really think it was necessary to be this defensive, do you?
Well, clearly, you do. My apologies for trying to be helpful in making sure you saw an analysis with useful information in it.
Cheers, -- jra
On Mar 13, 2013 11:13 PM, "Jay Ashworth" jra@baylink.com wrote:
Hey, Ryan; did you see, perhaps on outages-discussion, the after action report from Microsoft about how their Azure SSL cert expiration screwup happened?
Can you just link to the discussion archive?
----- Original Message -----
From: "Jeremy Baron" jeremy@tuxmachine.com
Can you just link to the discussion archive?
Was a posting:
http://blogs.msdn.com/b/windowsazure/archive/2013/03/01/details-of-the-febru...
Cheers, -- jra
wikitech-l@lists.wikimedia.org