[Labs-l] Network changes to Labs

[Ryan Lane]

Seems I'm now also getting reports that some instances are showing as down.
It looks like DHCP on instances isn't acting properly and has made them
inaccessible. Rebooting the instances does bring them back up. So, you can
reboot if you are having this issue. I'll be changing the network driver
now, and rebooting any non-responsive instances.


On Mon, Mar 4, 2013 at 4:52 PM, Ryan Lane <rlane at wikimedia.org> wrote:

> Today we bonded three NICs on the network node to increase network
> capacity between instances and resources outside of Labs (including
> glusterfs). This resulted in a short network outage while the router was
> reconfigured and again when we restarted the network services on the
> network node.
>
> If you had any bots that have issues with lost network connections, you'll
> need to restart them.
>
> We've made these changes for a couple reasons:
>
> 1. We were occasionally (but rarely) saturating the current link
> 2. This allows us to change the network driver we use on the virtual
> machines to the virtio driver, which changes the NICs from 100Mb to 1Gb.
>
> I've set virtio as the default for all new instance creations, and am in
> the process of changing the driver for existing instances. After I've
> changed the driver, instances will need to be rebooted for this to take
> affect. I'll send out another mail when this has occurred.
>
> - Ryan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/labs-l/attachments/20130304/6d6a478f/attachment.html>