ARM servers - Wikitech-l - lists.wikimedia.org

List overview All Threads
Download

ARM servers

RfC: Should we support MP4 Video...

FOSDEM coordination

James Salsman

12 Jan 2014 12 Jan '14

10:05 a.m.

Nicolas Charbonnier's "Latest ARM Server solutions booths tour" may be of some interest for those of you interested in low power server hardware: http://armdevices.net/2013/12/30/latest-arm-server-solutions-booths-tour/ Mitac isn't represented there, but he did an interview of them a year and a half ago: http://armdevices.net/2012/06/07/mitac-gfx-arm-server/

Reply

Show replies by date

Eugene Zelenko

12 Jan 12 Jan

4:38 p.m.

Hi! ARM servers is definitely worth to look at, but please be aware that technology is not mainstream and sad things may happens: http://calxeda.com (one of exhibitors of ARM TechCon). OS support if not mature yet, especially for ARMv8 (64 bit). Eugene. On Sun, Jan 12, 2014 at 1:05 AM, James Salsman <jsalsman(a)gmail.com> wrote:

Nicolas Charbonnier's "Latest ARM Server solutions booths tour" may be of some interest for those of you interested in low power server hardware: http://armdevices.net/2013/12/30/latest-arm-server-solutions-booths-tour/ Mitac isn't represented there, but he did an interview of them a year and a half ago: http://armdevices.net/2012/06/07/mitac-gfx-arm-server/ _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Reply

James Salsman

13 Jan 13 Jan

10:47 a.m.

Eugene wrote:

... OS support if not mature yet, especially for ARMv8 (64 bit).

Does someone have an exhaustive list of packages which we depend on for production but aren't available as arm binaries yet? We could try to build those. As for development, I understand that Oracle's JDK isn't on arm yet, but am not sure why openjdk wouldn't be strongly preferred. Maybe the multimedia team wants a 64 bit JRE for Adobe Flash C++ CrossBridge (formerly Alchemy) as per http://adobe-flash.github.io/crossbridge/ but I assume they probably have Macs, or can get them if they need to compile Flash applets for non-webrtc compliant client browser support. Can someone more familiar with the Foundation's server infrastructure needs than I please create a page somewhere with a checklist of packages, modules, tools, etc., which need to be on arm but aren't yet? Jasper mentioned that we need virtualization for Labs but aren't using Zen. It would be great to see what the developers for the virtualization that Labs uses say about prospects for arm builds. Best regards, James On Sun, Jan 12, 2014 at 5:05 PM, James Salsman <jsalsman(a)gmail.com> wrote:

Nicolas Charbonnier's "Latest ARM Server solutions booths tour" may be of some interest for those of you interested in low power server hardware: http://armdevices.net/2013/12/30/latest-arm-server-solutions-booths-tour/ Mitac isn't represented there, but he did an interview of them a year and a half ago: http://armdevices.net/2012/06/07/mitac-gfx-arm-server/

Reply

Faidon Liambotis

11:14 a.m.

On Mon, Jan 13, 2014 at 05:47:12PM +0800, James Salsman wrote:

Can someone more familiar with the Foundation's server infrastructure needs than I please create a page somewhere with a checklist of packages, modules, tools, etc., which need to be on arm but aren't yet?

Before we do that, we need to find a use case for ARM servers and we need to find quality ARM hardware (= not first generation) in reasonable prices for their performance. We've thought a bit about it in the past, but couldn't come up with a use case that made technical or financial sense. We have dozens of x86 servers e.g. just for MediaWiki; having thousands of ARM servers for the same purpose instead doesn't sound very appealing. It'll increase our problems (e.g. scalability), it will slow down individual requests and it's unlikely to provide a technical or financial benefit. Other uses cases are similar: even our storage servers are too busy CPU-wise for ARM to make sense. Maybe if ARM gets sufficiently fast (e.g. with arm64) and they've been proven in the field by the early adopters, it might sense for us in the long run. But I don't foresee us becoming one of these early adopters anytime soon. I'd love to be convinced otherwise, though :) Regards, Faidon

Reply

Tim Starling

14 Jan 14 Jan

12:33 a.m.

On 13/01/14 21:14, Faidon Liambotis wrote:

We've thought a bit about it in the past, but couldn't come up with a use case that made technical or financial sense. We have dozens of x86 servers e.g. just for MediaWiki; having thousands of ARM servers for the same purpose instead doesn't sound very appealing. It'll increase our problems (e.g. scalability), it will slow down individual requests and it's unlikely to provide a technical or financial benefit.

In fact, it would slow down individual requests by a factor of 7, judging by the benchmarks of Calxeda and Xeon CPUs at http://www.eembc.org/coremark/index.php So instead of a 10s parse time, you would have 70s. Obviously that's not tolerable. According to Intel's benchmarks, some Xeons have better performance per watt than ARM, so there's a question of whether ARM is an appropriate choice for a fully utilised CPU cluster, even if the load is parallelizable.

Other uses cases are similar: even our storage servers are too busy CPU-wise for ARM to make sense.

It may make sense for memcached, maybe some misc servers. But I think it would make more sense to have a bare metal provisioning process for misc servers which allowed smaller numbers of Intel cores per server, where that fits the application. That would improve energy efficiency without the need to deploy a new architecture. -- Tim Starling

Reply

George Herbert

12:55 a.m.

On Mon, Jan 13, 2014 at 3:33 PM, Tim Starling <tstarling(a)wikimedia.org>wrote;wrote:

In fact, it would slow down individual requests by a factor of 7, judging by the benchmarks of Calxeda and Xeon CPUs at http://www.eembc.org/coremark/index.php So instead of a 10s parse time, you would have 70s. Obviously that's not tolerable.

Question - is that 10s linear CPU core time for a parse, or 10s of average response time given our workloads? If it is the linear one-core parse processing time, how much of that is dependencies on DB lookups and the like, externalities within the infrastructure rather than the straight-line CPU time needed for the parse itself? Amdahl's law works both ways... -- -george william herbert george.herbert(a)gmail.com

Reply

Tim Starling

1:40 a.m.

On 14/01/14 10:55, George Herbert wrote:

On Mon, Jan 13, 2014 at 3:33 PM, Tim Starling <tstarling(a)wikimedia.org>wrote;wrote:

In fact, it would slow down individual requests by a factor of 7, judging by the benchmarks of Calxeda and Xeon CPUs at http://www.eembc.org/coremark/index.php So instead of a 10s parse time, you would have 70s. Obviously that's not tolerable.

Question - is that 10s linear CPU core time for a parse, or 10s of average response time given our workloads?

Just an arbitrary number chosen to be within the range of CPU times for slower articles. On average, it is much faster than that. For actual data, you could look at: http://tstarling.com/stuff/featured-parse-boxplot.png

If it is the linear one-core parse processing time, how much of that is dependencies on DB lookups and the like, externalities within the infrastructure rather than the straight-line CPU time needed for the parse itself?

WikitextContent::getParserOutput() profiles at around 1.25s real and 1.17s CPU. -- Tim Starling

Reply

Eugene Zelenko

2:46 a.m.

Hi! I think will be good idea to try to get access to real hardware. For example, Boston (http://www.boston.co.uk) produces Calxeda-based servers and well as HP has experimental Calxeda and X-Gene based cartridges for Moonshot servers (http://www.hp.com/moonshot). Both provide remote access to own servers for trials. Eugene. On Mon, Jan 13, 2014 at 4:40 PM, Tim Starling <tstarling(a)wikimedia.org> wrote:

On 14/01/14 10:55, George Herbert wrote:

On Mon, Jan 13, 2014 at 3:33 PM, Tim Starling <tstarling(a)wikimedia.org>wrote;wrote:

In fact, it would slow down individual requests by a factor of 7, judging by the benchmarks of Calxeda and Xeon CPUs at http://www.eembc.org/coremark/index.php So instead of a 10s parse time, you would have 70s. Obviously that's not tolerable.

Question - is that 10s linear CPU core time for a parse, or 10s of average response time given our workloads?

Just an arbitrary number chosen to be within the range of CPU times for slower articles. On average, it is much faster than that. For actual data, you could look at: http://tstarling.com/stuff/featured-parse-boxplot.png

If it is the linear one-core parse processing time, how much of that is dependencies on DB lookups and the like, externalities within the infrastructure rather than the straight-line CPU time needed for the parse itself?

WikitextContent::getParserOutput() profiles at around 1.25s real and 1.17s CPU. -- Tim Starling _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Reply

Martijn Hoekstra

8:42 a.m.

On Jan 14, 2014 2:47 AM, "Eugene Zelenko" <eugene.zelenko(a)gmail.com> wrote:

Hi! I think will be good idea to try to get access to real hardware. For example, Boston (http://www.boston.co.uk) produces Calxeda-based servers and well as HP has experimental Calxeda and X-Gene based cartridges for Moonshot servers (http://www.hp.com/moonshot). Both provide remote access to own servers for trials. Eugene.

I don't know much about this bare metal stuff, but everyone who does seems to be saying the same thing on this list: ARM architecture is awesome, and also a bad fit for us as it will harm performance, and possibly not even save energy by doing so. Can't we just thank James for coming up with an idea that seemed good, and drop the thread because it turns out it won't work for us? Martijn h.

On Mon, Jan 13, 2014 at 4:40 PM, Tim Starling <tstarling(a)wikimedia.org>

wrote:

On 14/01/14 10:55, George Herbert wrote: > On Mon, Jan 13, 2014 at 3:33 PM, Tim Starling <tstarling(a)wikimedia.org

wrote: >>> >>> In fact, it would slow down individual requests by a factor of 7, >>> judging by the benchmarks of Calxeda and Xeon CPUs at >>> >>> http://www.eembc.org/coremark/index.php >>> >>> So instead of a 10s parse time, you would have 70s. Obviously that's >>> not tolerable. >> >> >> Question - is that 10s linear CPU core time for a parse, or 10s of

average

>> response time given our workloads? > > Just an arbitrary number chosen to be within the range of CPU times > for slower articles. On average, it is much faster than that. > > For actual data, you could look at: > > http://tstarling.com/stuff/featured-parse-boxplot.png > >> If it is the linear one-core parse processing time, how much of that is >> dependencies on DB lookups and the like, externalities within the >> infrastructure rather than the straight-line CPU time needed for the

parse

itself?

WikitextContent::getParserOutput() profiles at around 1.25s real and 1.17s CPU. -- Tim Starling _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Reply

Tim Starling

15 Jan 15 Jan

5:12 a.m.

I wrote:

But I think it would make more sense to have a bare metal provisioning process for misc servers which allowed smaller numbers of Intel cores per server, where that fits the application. That would improve energy efficiency without the need to deploy a new architecture.

Actually, after looking into this, I don't think it would help very much. I had a look at ssl1, which is an idle PowerEdge 410. It has two Intel Xeon X5650 packages, i.e. 12 cores in total. According to powertop, one package is in the C6 state about 95% of the time, and the other is about 99% C6. According to the datasheet [1], this processor model should use 10W per package in C6, i.e. 20W total. Meanwhile, the server's RAC is reporting a system power usage of 160W, which would imply that the non-CPU parts of the system are using 88% of the idle power. I don't know where all the power goes, but it looks like it isn't to the processor. [1] http://www.intel.com/content/www/us/en/processors/xeon/xeon-5600-vol-1-data… -- Tim Starling

Reply

Federico Leva (Nemo)

11:40 p.m.

160 W measured at what point, before or after they're converted to the end tension needed? PSU performance deteriorates a lot at low usage ratio and their stock PSU is not particularly brilliant. But yes, it's easy to reach 140 W just with one hard disk, some RAM sticks and a few non-disabled powered ports/slots. Nemo

Reply

Tim Starling

16 Jan 16 Jan

2:56 a.m.

On 16/01/14 09:40, Federico Leva (Nemo) wrote:

160 W measured at what point, before or after they're converted to the end tension needed?

I am pretty sure it is 160W measured at the AC input. The failure threshold for this metric is set to 679W on a power supply rated at 680W AC input and 500W DC output.

PSU performance deteriorates a lot at low usage ratio and their stock PSU is not particularly brilliant.

This server has the redundant PSU option. According to the R410 spec sheet, it is 80+ Gold, which means it should use about 16W. -- Tim Starling

Reply

Matthew Flaschen

13 Jan 13 Jan

10:58 p.m.

On 01/13/2014 04:47 AM, James Salsman wrote:

Can someone more familiar with the Foundation's server infrastructure needs than I please create a page somewhere with a checklist of packages, modules, tools, etc., which need to be on arm but aren't yet? Jasper mentioned that we need virtualization for Labs but aren't using Zen.

I think you mean Xen. It uses OpenStack, a free and open source (Apache 2.0) virtualization suite. OpenStack supports multiple hypervisors (Xen, KVM, etc.). Coren let me know that we're using KVM. https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine says there is an ARM port for KVM in the 3.9 version of the Linux kernel. Matt Flaschen

Reply

3753

days inactive

3757

days old

wikitech-l@lists.wikimedia.org

Manage subscription

12 comments

8 participants

tags (0)

participants (8)

Eugene Zelenko
Faidon Liambotis
Federico Leva (Nemo)
George Herbert
James Salsman
Martijn Hoekstra
Matthew Flaschen
Tim Starling