[QA] Delivery Status Notification (Failure)

4df53dd0 at opayq.com 4df53dd0 at opayq.com
Sat Feb 16 05:08:02 UTC 2019 

** Recipient inbox full **

Your message couldn't be delivered to curtiskwong101 at gmail.com. Their inbox is full, or it's getting too much mail right now.



The response was:

Storage quota exceeded
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20190215/dcc2c9e3/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: icon.png
Type: image/png
Size: 2187 bytes
Desc: not available
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20190215/dcc2c9e3/attachment.png>
-------------- next part --------------
hR1t3+4Zjypot9ywLg/JU91
X-Received: by 2002:a50:a95d:: with SMTP id m29mr10033481edc.271.1550293680=
707;
        Fri, 15 Feb 2019 21:08:00 -0800 (PST)
ARC-Seal: i=3D1; a=3Drsa-sha256; t=3D1550293680; cv=3Dnone;
        d=3Dgoogle.com; s=3Darc-20160816;
        b=3DC0lot7ya/1yG+4y6wAzS0DO6kDTOIet5HwyUmoPFp2gJUUcAO7qGhyrhYn14WyX=
9vA
         vR4c2gOYvHU/QW2GDqDUmQNMeMnuLQhplRLUolSV/3p7tGHf+O1jAbI5i72M3vap6V=
1U
         Td7EO0kN1hRaMiZbQbhrtUUMP2V6bNgs/YLxbYKFARElj2n5jEUSRukU0HIdtNhtxA=
AW
         S0a9HhqRGMkKkVAqwonNdQrL/nBl5SdWxpyvO17oDelEY7i+XwgwNq+FBXSEAPrwaN=
Um
         M44P+AJCdwmLYJPpQz5ABCruuA8FJcIp3WHIZMJvwnXkswWVCr2Gr8BJc7szhTXmTf=
SL
         AAHw=3D=3D
ARC-Message-Signature: i=3D1; a=3Drsa-sha256; c=3Drelaxed/relaxed; d=3Dgoog=
le.com; s=3Darc-20160816;
        h=3Derrors-to:list-subscribe:list-help:list-post:list-archive
         :list-unsubscribe:list-unsubscribe:list-id:precedence:subject
         :user-agent:content-disposition:message-id:date:to:reply-to:sender
         :from:content-transfer-encoding:mime-version:dkim-signature
         :delivery-date;
        bh=3DaFoSvu/8qbYpoXJuNJnrgmlIDhXhVfCWUW2UUnngYRs=3D;
        b=3DiSrXn+8fwDeAA3uxSg8pKAdRYAEYh8whL0YeYI5ZLlTpJdJARPveZYQOEJ/Bdhh=
Kx9
         FTXaOpDYvriOj2BXFjNjDUET47kVuYDAYIrPjJu9Mzf56ohvu7+J7MqMzUJIEg4dCJ=
0h
         dmhfvEVELhVFi7ee4Qr0Dz0GfH3AGaJCq4Elfl40VREIHe2js6HQL55D1vSpk29wR1=
Jc
         /eHkxI0VV/mEHTVFSAjuNJmKCdp9sGnXQ9yeO2G3YA+v75GuHBJ7x8J061kW97jljq=
wL
         A0us9HLjo4g72AHnAJiswXtj8IFldb0vBNrWbEDLEPihVd5nnIyJdTOTRnYGR2Tfs9=
0a
         lYog=3D=3D
ARC-Authentication-Results: i=3D1; mx.google.com;
       dkim=3Dpass header.i=3D at opayq.com header.s=3Dabine header.b=3Db0KSkY=
Qc;
       spf=3Dpass (google.com: domain of fwd.47tblvy4rnnv at opayq.com designa=
tes 184.105.182.73 as permitted sender) smtp.mailfrom=3DFWD.47tblvy4rnnv at op=
ayq.com;
       dmarc=3Dpass (p=3DNONE sp=3DNONE dis=3DNONE) header.from=3Dopayq.com
Return-Path: <FWD.47tblvy4rnnv at opayq.com>
Received: from 129-he.filtered.junkemailfilter.com (retry2.ctyme.com. [184.=
105.182.73])
        by mx.google.com with ESMTPS id c1si1845444edw.35.2019.02.15.21.08.=
00
        for <curtiskwong101 at gmail.com>
        (version=3DTLS1_2 cipher=3DECDHE-RSA-AES128-GCM-SHA256 bits=3D128/1=
28);
        Fri, 15 Feb 2019 21:08:00 -0800 (PST)
Received-SPF: pass (google.com: domain of fwd.47tblvy4rnnv at opayq.com design=
ates 184.105.182.73 as permitted sender) client-ip=3D184.105.182.73;
Authentication-Results: mx.google.com;
       dkim=3Dpass header.i=3D at opayq.com header.s=3Dabine header.b=3Db0KSkY=
Qc;
       spf=3Dpass (google.com: domain of fwd.47tblvy4rnnv at opayq.com designa=
tes 184.105.182.73 as permitted sender) smtp.mailfrom=3DFWD.47tblvy4rnnv at op=
ayq.com;
       dmarc=3Dpass (p=3DNONE sp=3DNONE dis=3DNONE) header.from=3Dopayq.com
Delivery-date: Fri, 15 Feb 2019 21:08:00 -0800
Received: from 133-he.filtered.junkemailfilter.com ([184.105.182.133]:33151=
) helo=3D133-he.filtered.junkemailfilter.com
	by retry2.junkemailfilter.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SH=
A384:256)
	(Exim 4.90_1)
	id 1gtx8X-0004q7-L0 on interface=3D184.105.182.73
	for curtiskwong101 at gmail.com; Wed, 13 Feb 2019 08:11:49 -0800
Received: from smtp3.opayq.com ([23.23.126.237]:37008) helo=3D[23.23.126.23=
7]
	by opayq-outbound.junkemailfilter.com with esmtps (TLSv1.2:AES256-SHA256:2=
56)
	(Exim 4.90)
	id 1gtx8P-0005gG-Dl on interface=3D184.105.182.150
	for curtiskwong101 at gmail.com; Wed, 13 Feb 2019 08:11:41 -0800
DKIM-Signature: v=3D1; d=3Dopayq.com; t=3D1550074293; b=3Db0KSkYQcd7mg5IIuC=
wewnAfqDlvq+KBXcTg+erSiNgSCi6iSCa+fmlt2993k/PME53A35vJRfrSI04wWDF/OU7l/KL5i=
vda0PVUpohM5slv2AfhhvDSAp9jd9kWzvi0I9Jy8jflbcACNirX0unfm8ACar7ie45y+KMM4Gwl=
OOMs=3D; s=3Dabine; c=3Drelaxed/relaxed; a=3Drsa-sha256; bh=3DaFoSvu/8qbYpo=
XJuNJnrgmlIDhXhVfCWUW2UUnngYRs=3D; h=3DDate:From:Reply-To:Subject:To:List-U=
nsubscribe;
MIME-Version: 1.0
Content-Type: text/plain; charset=3D"utf-8"
Content-Transfer-Encoding: quoted-printable
X-GetAbine-Processed: 1
From: "Greg Grossmeier [Masked]" <FWD.47tblvy4rnnv at opayq.com>
Sender: "Greg Grossmeier [Masked]" <FWD.47tblvy4rnnv at opayq.com>
Reply-To: FWD.47tblvy4rnnv at opayq.com
To: 4df53dd0 at opayq.com
X-GetAbine-Sender: qa-bounces at lists.wikimedia.org
X-GetAbine-Disposable: 4df53dd0 at opayq.com
X-GetAbine-Host-Address: 23.23.126.237
X-Google-DKIM-Signature: v=3D1; a=3Drsa-sha256; c=3Drelaxed/relaxed;
 d=3D1e100.net; s=3D20161025;
 h=3Dx-gm-message-state:date:from:to:subject:message-id:mime-version
 :content-disposition:content-transfer-encoding:user-agent;
 bh=3D9VmEp0o32IBM2KMmQyivwklxx51LJTH5VGpOIz3Jelc=3D;
 b=3DrTN8kOFtbcI1S2siN8+XaZrdMoSqxFdVwKiHF8IgICbZNuOqdRebQ9mBKvSTDgksrK
 63D5RYKU2HjmwM3+USeVPsCa4myQ04dZVkSkxv3pcfIRqRxhRMtdGaym2yAVspNxCVJo
 6SjZYPWyusZ1CpzbphpQ9rbx2RnDDYFEwZK28UIh4Y9ck/BaDX2T7d5yZ70f3qgsFDAW
 K3FLF/8oJm0x1bi61MXG8Zou7uXRrA13fPjoNLeY3HxsPWEreKg/HxtOo55YeUup/hhp
 0j7Q6DtWW6uNIuT8VEBBt3c4h4mXrLIs/GFVMZIlNB8oRebLznfKt6Pvu84Bm3u/MA3t
 G+ng=3D=3D
X-Gm-Message-State: AHQUAuaQNR7rPHf62OWQsWZw5tTwbqgopgc7GSaXHrLd//mKdQfJfQ+=
Y
 gyjZbrHmY3vRAD0NrWwMx0bidI8z5eAg/XvD
X-Received: by 2002:a63:2209:: with SMTP id i9mr1102768pgi.325.155007428053=
6;=20
 Wed, 13 Feb 2019 08:11:20 -0800 (PST)
Date: Wed, 13 Feb 2019 08:11:17 -0800
Message-ID: <20190213161117.zps2njttchhatnsx at x1>
Content-Disposition: inline
X-PGP-Key: https://sks-keyservers.net/pks/lookup?op=3Dget&search=3D0x11388E=
47FAC81C7D
User-Agent: NeoMutt/20170113 (1.7.2)
Subject: [QA] Beta Cluster and CI issues (was Fwd: [Cloud] [Cloud-announce]
 VPS hardware failure -- downtime ongoing)
X-BeenThere: qa at lists.wikimedia.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: "QA for Wikimedia projects." <qa.lists.wikimedia.org>
List-Unsubscribe: <https://lists.wikimedia.org/mailman/options/qa>,
 <mailto:qa-request at lists.wikimedia.org?subject=3Dunsubscribe>
List-Unsubscribe: <mailto:unsubscribe-388062454-4df53dd0 at opayq.com>, <https=
://emails.abine.com/disableDisposable?fwd=3DFWD.47tblvy4rnnv at opayq.com>
List-Archive: <https://lists.wikimedia.org/pipermail/qa/>
List-Post: <mailto:qa at lists.wikimedia.org>
List-Help: <mailto:qa-request at lists.wikimedia.org?subject=3Dhelp>
List-Subscribe: <https://lists.wikimedia.org/mailman/listinfo/qa>,
 <mailto:qa-request at lists.wikimedia.org?subject=3Dsubscribe>
X-Sender-Domain: wikimedia.org
X-Spamfilter-host: plato.junkemailfilter.com - http://www.junkemailfilter.c=
om
X-Key-ID: NGRmNTNkZDBAb3BheXEuY29tIHFhLWJvdW5jZXNAbGlzdHMud2lraW1lZGlhLm9yZ=
yAyMDE5LTAyLTEzIDA4OjExOjMwLjM2MiAxZ3R4OEQtMDAwMXZBLU9I
X-Content-flags: 76char affecting aging android automation compiler current=
ly dashes date design dev01 down downtime email email-adr experience follow=
ing formerly forwarded fwd guard https integration learn link manager messa=
ge monitoring month number-text order planet pro prod project projects quot=
atest quoted re reply responses sciencesource services web woman working
X-Domain-list: wikimedia.org
X-Sender-Host-Address: 208.80.154.21
X-Sender-Host-Name: lists.wikimedia.org
Errors-to: <>
X-Key-ID: Y3VydGlza3dvbmcxMDFAZ21haWwuY29tIGZ3ZC40N3RibHZ5NHJubnZAb3BheXEuY=
29tIDIwMTktMDItMTMgMDg6MTE6NDEuNTg0IDFndHg4UC0wMDA1Z0ctRGw=3D
X-Sender-Domain: junkemailfilter.com


-------------------------Blur-------------------------
This email is forwarded from a MASKED EMAIL you created using Blur.  (https=
=3D
://dnt.abine.com/#help/faq/faq-whataremaskedemails).
IF THIS IS SPAM, CLICK HERE TO BLOCK:
https://dnt.abine.com/#/block_email/4df53dd0@opayq.com/FWD.47tblvy4rnnv@opa=
=3D
yq.com

Want to shop safely and privately online? Go Premium: https://dnt.abine.com=
=3D
/?pk_campaign=3D3DmaskHeader#premium
-------------------------by Abine-------------------------


This WMCS outage is affecting Beta Cluster (aka: deployment-pre VPS
project) and our CI build/test servers (aka: the integration project).

Apologies for any downtime and/or slow CI responses while this is being
sorted.

Greg

----- Forwarded message from Andrew Bogott <abogott at wikimedia.org> -----

> Date: Wed, 13 Feb 2019 07:25:39 -0600
> From: Andrew Bogott <abogott at wikimedia.org>
> To: Cloud-announce at lists.wikimedia.org
> Subject: [Cloud] [Cloud-announce] VPS hardware failure -- downtime ongoin=
=3D
g
> Reply-To: cloud at lists.wikimedia.org
>=3D20
> We're currently experiencing a mysterious hareware failure in our datacen=
=3D
ter
> -- three different SSDs failed overnight, two of them in cloudvirt1018 an=
=3D
d
> one of them in cloudvirt1024.=3DC2=3DA0 The VMs on 1018 are down entirely=
.=3DC2=3D
=3DA0 We may
> move those on 1024 to another host shortly in order to guard against
> additional drive failure.
>=3D20
> There's some possibility that we will experience permanent data loss on
> cloudvirt1018, but everyone is working hard to avoid this.
>=3D20
> The following VMs are on cloudvirt1018:
>=3D20
>=3D20
> a11y=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA=
0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3D
=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3D
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0 | reading-web-staging
> abogott-scapserver=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=
=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3D
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0 | testlabs
> af-puppetdb01=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3D
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=
=3DA0=3DC2=3DA0 | automation-framewo=3D
rk
> api=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3D
=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3D
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0 | openocr
> asdf=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA=
0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3D
=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3D
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0 | quotatest
> bastion-eqiad1-02=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=
=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3D
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0 | bastion
> clm-test-01=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3D
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=
=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0 | commun=3D
ity-labs-monitoring
> compiler1002=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3D
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=
=3DA0=3DC2=3DA0=3DC2=3DA0 | puppet-diffs
> cyberbot-exec-iabot-01=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3D
=3DA0=3DC2=3DA0 | cyberbot
> deployment-db03=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3D=
A0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3D
=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=
 | deployment-prep
> deployment-db04=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3D=
A0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3D
=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=
 | deployment-prep
> deployment-memc05=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=
=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3D
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0 | deployment-p=
rep
> deployment-pdfrender02=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3D
=3DA0=3DC2=3DA0 | deployment-prep
> deployment-sca01=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=
=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3D
=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0 | deploym=
ent-prep
> design-lsg3=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3D
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=
=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0 | design
> eventmetrics-dev01=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=
=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3D
=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0 | eventmetrics
> fridolin=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=
=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3D
=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3DC2=3DA0=3D
----- Message truncated -----

More information about the QA mailing list