Thoughts on compact server system

[dZeus]

Thread update:

The i5-750 was Dead-on-Arrival; since I sourced various components second hand from various sources with no spares to test, I got rid of it all and bought a Haswell Celeron G1820 + microATX mobo as a temporary solution.

Against expectations, the Haswell Celeron (dual core w/12GB RAM) runs virtualization needs (RHEL/CentOS 6 KVM) extremely well for the price (<100 usd for mobo + cpu).

However, I want to both consolidate all my current server needs (NAS, virtualization of home lab and various production systems such as a Samba4 Active Directory Domain Controller for home domain) as well as being extremely portable (mini-ITX).

For this purpose, I acquired a compact mini-ITX case (MS-TECH CI-70) and a SuperMicro 8-core Intel Avoton/C2750 motherboard with 4 DIMM slots (32GB max, 64GB possible if 16GB ECC DDR3 UDIMM sticks ever become affordable which I doubt as no Xeon supports them).

The reason for choosing the Avoton over a Xeon system (at a similar price, as Avoton is really overpriced), is fitting 32GB RAM or more into a compact mini-ITX case.

It's not ready yet (still waiting for case fans, 2x 2TB 2.5" SATA in mirror RAID, 1x 500GB mSATA for host OS and guest VMs, TPM module), and I'll post an update with pics when it's been completed.

The 60W max. DC-DC of the MS-TECH CI-70 (aka Iwill HT-60) appears to be of Chinese low-grade quality, for the long run it will be replaced with either another DC-DC such as a picoPSU or by using the SuperMicro on-board's DC-DC if that's feasible. Don't want to fry expensive server hardware due to a poor quality power supply blowing up.

[Jammrock]

The Avoton boards are generally all server grade stuff. 2-4 Gb Ethernet ports, often Intel I354 network chip. dKVM for remote headless management (remote KVM over Ethernet stuff with ICMP controls). 32GB-64GB RAM support, including ECC support. Some come with SAS RAID controllers. Then add the 8-core server grade Atom processor on top and it's a pretty good deal for ~$350-$400 USD.

Avoton's aren't as good as Xeon's in general computing, but performance per watt and price is fairly comparable. I'm been looking at an Avoton based solution myself. In home server applications it should be more than enough processing power.

http://www.servethehome.com/Server-detail/supermicro-a1sai-2750f-review/

The cheapest Haswell Xeon E3 and motherboard combo with dKVM, quad Gb Ethernet runs about $500+. So unless you really need Xeon grade processing for a SQL database the Avoton should work well for you.

[dZeus]

true, but in performance/$, the Xeon E3 (even the Lv3 models) perform better than Avoton (I'm estimating a $500 E3 solution will likely perform twice as fast as the $400 Avoton solution).

If you don't need IPMI/BMC/iKVM or ECC, and can live with relatively limited (but still fairly respectable) CPU performance, nothing beats the Haswell Celeron solution performance/price ratio.

However, if you want 32+ GB RAM, it's only possible to go the micro-ATX (or larger) route for the Xeon E3 which I was intend on avoiding.

anyway... within a week or 2 I should be receiving my HDDs and I'll start reporting on how the Avoton setup is working (using the motherboard you linked to, i.e. the now fan-less version of the SuperMicro a1sai-2570f, as the Asrock C2750D4I motherboard has many users reporting as showing many issues. )

[Evildead666]

Being as Avoton is an 8 core Atom (Silvermont core ?), i'm very interested in a comparison with the Celeron, if possible.

And mentioning PSU's, my external 90W power brick for the cooling of my PC (Water pumps and fans) failed a couple of weeks ago.
Luckily I noticed it, as the fans seemed to be going slower than usual. Didn't see the cpu pump was red and not running though, or the gpu pump.
Noticed something was really wrong when i blackscreened opening a youtube vid. Reboot, black screen. Panic.
Have replaced it with an overkill 200W sff PSU shunted to start when its plugged in.
Nothing died thank goodness HAdn't pushed the OC too far this time.

So, yes, a good PSU is a must, especially for expensive hardware

[Jammrock]

I spotted this the other day.

http://www.servethehome.com/Server-detail/supermicro-x9skv-1125-review-embedded-communications-platform/

http://www.supermicro.com/products/motherboard/Gladden/DH8903/X9SKV-1125.cfm

It's a mATX with an embedded Xeon E3, 6 NIC ports, 4 SoDIMMs (32GB max) and a PCIe 2.0 x8 slot. It's $800 USD though. I think it's too specialty though.

Let me know what you think of the Avoton once you've had time to play with is dZeus.

[dZeus]

I spotted this the other day.

http://www.servethehome.com/Server-detail/supermicro-x9skv-1125-review-embedded-communications-platform/

http://www.supermicro.com/products/motherboard/Gladden/DH8903/X9SKV-1125.cfm

It's a mATX with an embedded Xeon E3, 6 NIC ports, 4 SoDIMMs (32GB max) and a PCIe 2.0 x8 slot. It's $800 USD though. I think it's too specialty though.

Let me know what you think of the Avoton once you've had time to play with is dZeus.

when I was still considering micro-atx, I had my eye on the following board:
SuperMicro X10SLL-F

It's got 'only' 3 LAN ports (one of which for IPMI) but 4 more SATA ports.

My main issue with micro ATX is that I haven't been able to find really compact server cases that can host at least 2x 2.5" drives. The smallest I found were around 8.5 litres, while the MS-TECH CI-70 mini ITX case is only 3.4 litres (and has 2x 2.5" drive mounting options + I'm doing a ghetto-tech mod to add a mSATA bracket for the system SSD drive).

Anyway, the fans and hard drives should arrive somewhere next week, and then I'll start the testing. The base system will run CentOS 7 with KVM as hypervisor, even though I'm not fully acquainted with RHEL/CentOS 7 yet (systemd, firewalld, etc.).

One thing that surprises me is that Windows guests VMs use about 2-5% cpu when idling on QEMU/KVM, while *NIX guests VMs only use 0-0.5% when idling. I wonder how ESXi, Hyper-V and Xen fare on this aspect.

edit: of course, if you need more than two dedicated networks you'd need to start creating VLANs on the same physical interface; I think this functionality is finally integrated on Microsoft Windows 8+ / Windows Server 2012+ in the form of a 'Hyper-V virtual switch device', and you will need to use it with a VLAN-capable switch.
For this purpose I use a simple Netgear ProSafe 8-port gigabit switch, but I can't recommend it as it need propriety client software to configure it (which uses Adobe AIR), and the management connection is possible on any port of the switch (poor security).

[dZeus]

Finally received the case fan today (earlier packet got lost in the post).

List of hardware:
- MS Tech CI-70 mini-itx case w/crappy 60W brick + DC-DC
- SuperMicro A1SAi-2750F avoton mini-itx server board with IPMI
- SuperMicro TPM module
- 2 x 8GB ECC SODIMM (2x 8GB to be added)
- 2x 2TB Samsung m9t 2.5" 9mm HDD
- 1x Samsung 500GB EVO mSATA SSD
- 1x NoiseBlocker BlackSilent 50 mm fan

Here are some pics

lid.jpg mSATA.jpg bottle.jpg

The Samsung 500GB EVO mSATA drive resides in an mSATA to SATA convertor which I've mounted to the fan-mounting position on the right side of the case. The non-angular power and SATA cable block mounting a case fan on the same side, so I've mounted it on the opposite side.

Unfortunately, the case is not very well thought out, and the 'feet' of the lid (where the screws are tightened) are much too large and make it impossible to mount the case fan anywhere other than in the centre, and make it very tough to mount things on both sides at the same time.

On the other hand, maybe I'm pushing it a bit by trying to put 2x 2.5" HDD + mSATA into this tiny case.

With a bit of luck I'll be able to start the config + OS install (CentOS 7) onto this server tonight.

[Jammrock]

Looks nice. Keep us posted. I'm very interested in hearing your impressions of the Avoton in action.

[dZeus]

I'll need a bit of time to get accustomed to the IPMI BMC... however I can see it allows me to mount iso disks over the network to install the base operating system, meaning I don't need to burn CDs or write to USB sticks, nor hook up a monitor to the server for the initial installation.

My power consumption reader says 21 Watts in use when in BIOS/IPMI configuration interface. I expect 2 extra LDDR3 SO-DIMMs to add another 2-4 watts to this, and cpu load figures should be substantially higher too.

edit: it's possible to stop/start/reset the machine from the BMC, which of course always keeps running. In 'powered down' state, the power consumption is still 6 Watt, which represents the consumption of the IPMI BMC plus possibly a bit extra for WOL. Which also means that the base power consumption of the Avoton system w/o BMC is a mere 15 Watts in BIOS! In case of the SuperMicro board this is not that relevant, as it is not possible to deactivate the IPMI if desired (at least, I haven't found any obvious way to do so).

update 1:
Post installation in CentOS 7, running with all C-states enabled in the BIOS for the CPU & power management enabled as suggested by powertop.

Power figures:
- 8 instances of BurnP6 (part of cpuburn package): 31 Watt
- idle, both mechanical drives spun down: 13 Watt

Some notes:
- I had to break off the cpuburn test after about 15 minutes as cpu temp kept rising and all cores were at about 90 degrees C. I suppose a second fan for the case or directly on the cpu heatsink would not be a luxury, and even better would be to find 4-pin PWM fans with 50x50x10mm dimensions to minimize noise during regular workloads, as the Supermicro board has plenty of PWM headers. I'm kind of disappointed that SuperMicro removed the heatsinkfan and replaced it by a heatsink in later models (including the one I received).
- The power figures are with the DC-DC and 12V brick that came with the case. Since I don't trust the cheap Chinese capacitors these use for my not-so-cheap hardware, I'm intending to switch to a high-efficiency/high-quality 12V brick and use the integrated DC-DC on the SuperMicro board. I'm expecting to switch these somewhere over the next few months
- I'll update performance impression over the next few months, after intensively using KVM to run a relatively substantial size of virtual machines (mostly *NIX)

[Paddy]

Bit late to the party, but I use mac minis for my server needs. SSD drive, low power and headless. They are cheap and easy to come by and fit nicely on top of the rack mounted switches. I have a synology NAS for storage. my only issue was having to splice the power cable so I could connect it to a UPS.