redundancy...

[VJ]

The C236M-WS is mATX, not mini ITX...

http://www.asrockrack.com/general/productdetail.asp?Model=C236M%20WS#Specifications

I've put my quest on hold until I can measure properly what is possible: what space is available near my patch panel and switch, and what could be the possibilities with the big tower. I'll be in Warsaw in December/January, so I can do it then.

I quickly glanced at the ZFS presentation but have the same reservations about ZFS. Finally, this is one of the first explanations where they properly explain it. Most present ZFS as the solution to add redundancy, but it is not. You need to add the redundancy in the VDEVs. That was a bit the point of me starting the thread, as it confused me. ZFS just pools it.

[UtwigMU]

I was split between mATX and mini ITX. Mini ITX allows for really tiny computer and as I'm renting and may move, this has benefits. The drawbacks are that there are few motherboards that meet requirements of 32GB ECC mem support, IMPI and many SATA slots. The reason is ZFS requires 8GB +1GB/TB storage. 16GB is good for start, 32GB is for big arrays.
I've found:
ASRock Rack E3C232D2I, ASRock Rack E3C236D2I, and Supermicro boards with embedded Xeon such as Super Micro X10SDV-4C-TLN2F.
Asrock: cheaper, has 6 SATA + onboard USB to boot OS, need LSI SAS card such as LSI9240-8i SAS HBA (50€) for expansion. Supermicro can be had with dual 10Gb NICs and has PCIe M2 which means you can have SSD boot drive (more reliable than USB) + 6 SATA drives. Problem is it's 500€-ish and if it gets fried you need to replace entire board.

The I in RAID stands for inexpensive and I think a NAS while overall expensive should be made of inexpensive parts. Another point is that if in 5 years motherboard dies and you're constrained to mini ITX it might be hard to find board that meets your needs. If you need to go bigger you will have to throw away the case. So this is why I was against mini ITX. For example when Gigabyte/Sempron board's caps in my 2003 server died. I bought HP desktop board/CPU/Mobo/RAM combo, popped it in, installed a few drivers and then server ran for 4 years. Then after 7-8 years of operation the OS hosed itself and I decided to just build another server instead of troubleshooting the old one.

Another reason is that I liked 804 vs 304 more as it has more room for drives. I'm not against not having hot-swap but having to lift drive cages with drives and cables hanging in small case is something I don't like. I take care of one Proliant ML150 which has front cage to mount drives. You have to pull cage out everytime you want to swap or add drives. Say your RAID fails, you replace a drive and undo one of the other cables in the process. I had a few high adrenaline moments with that server (running for 6 years - all 4 drives are different than the original ones) Then I found Silverstone DS380 which is 8-bay hot swap mini ITX. I like that case a lot and I've started to plan my build around it.

ZFS and BTRFS are supposed to make FS aware of underlying disks and enable the FS to be able to calculate and correct parity. Problem is that errors per amount of read/written data have stayed same order of magnitude since 1990s while disks got bigger. Now probability of encountering one is much greater than then.

BTRFS is OK for RAID1 as RAID5/6 are not yet production grade.
ZFS is similar to Linux logical volumes which is what you would use if you'd create software RAID in Linux.
On Linux you have physical volumes which you combine into volume group. You can use RAID. You partition volume group in logical volumes which are presented to system as disks and you can then format them in any supported file system.
Example would be 2GB 4GB 4GB drives which are in single 10GB volume group which is then partitioned in 1GB boot 2GB swap 3GB root and 4GB home partitions
The advantage is that if you run out of space you can just add more space to volume group. You can also move data from drive in volume group and remove the drive. You can also resize filesystems (xfs cannot be shrunk, ext3/4 can be, don't know about btrfs). I'm using ext4 on this laptop that I'm typing this on. Also you can snapshot logical volumes. LVMs are cool but the GUI tools for them sometimes crash. You will want to learn pv/vg/lv create display reduce commands. I'm fairly familiar with this from my RH exams and I imagine ZFS is similar.

On ZFS you put disks in vdevs and set raid at vdev level. I would say unless you have big enteprise grade server or storage array I would just keep one vdev per zpool and have multiple zpools. Enterprises usually just add arrays to have one big space and redundancy is taken care at the hardware levle. For home use it's better to have either 1 4-10 drive vdev in 1 zpool or multiple 1-3 drive vdevs in multiple (1vedv/per pool) zpools. The benefit of ZFS is that it knowsh where is data and which space is empty. It also allows for deduplication (say 100 users backup winword.exe to ZFS, ZFS will know this and write only one copy of the file - this is CPU intensive and only useful for enterprises), encryption. So if you need to rebuild after a drive failure, ZFS will only rebuild used space.

Another benefit is also scrubbing and resilvering. This is the only FS that has this and is reliable and good for general use. On scrub ZFS will go through all drives and recalculate parity and correct any bit-rot errors. Resilvering is like rebuilding RAID but it will only rebuild used parts not entire drives.

On my ESX (HP Proliant ML350G6) it took ~2 days to sync 2x4TB hw RAID1. One server which uses Windows 2008R2 dynamic disk mirror takes around 10-15h to resync. around 10-20TB will take days. Also while rebuilding array of old drives with one missing there is a chance of another drive breaking, so RAID 6 is recommended over RAID5. Another chance is read errors during rebuild.

http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

This is why all people on Arstechnica have been building ZFS NASes for some years now and the thread is 75-pages long.

Another benefit of ZFS is copy on write. with each write the ZFS writes new copy of file. If power outage occurs there is still original file intact.

See here for quick comparison of ZFS BTRFS and ReFS

https://en.wikipedia.org/w/index.php?title=ReFS#Performance_and_competitor_comparisons

[VJ]

I get the benefits of ZFS, my point is just that in many forums or posts, they use ZFS as the solution for adding redundancy. But they don't elaborate on which redundancy to use in the VDEVS. As you say, for home use, a single vdev in a zpool makes sense, but in that case I'm worried of the added complexity: if there is a single vdev in the pool, I might as well use the vdev directly (as in: typical raid1 use for example). I understand that I would not have some of the zfs features, but at least I have a system I would understand fully.

Remember that the Silverstone DS380 is for 2.5" harddisks!
EDIT: Sorry, I confused it with the DS280, which is for 2.5". The DS380 is for 3.5", but gets bad reviews for lack of cooling ability when populated with a lot of disks.

Streacom has a chassis that supports many disks:

http://www.streacom.com/products/f12c-chassis/

They claim up to 12, while still having a case of fairly limited outer dimensions. I'm just not sure I trust harddisk mounts that just attach on one side of the disk and leave the disk hanging that way. Seems dangerous for vibrations to me...

Good point about the limited options with mini-ITX, although there are quite some options both server (with many sata ports) and end user (few sata ports) and I have the impression more are available. I know the limitations of going with a soc, but with all the computers I've had, I've never done a CPU upgrade (at best I increased the memory). So if the mITX case is cheap enough, one could consider it a disposable item. Or repurpose it for e.g. some htpc use.

Intel Denverton is coming, and both Gigabyte and Asrock have announced mainboards (they are in their news section, but not yet in the list of products). It may drop the price of the Avontons a bit, and then Asrock has some nice options.

[UtwigMU]

I would use either cheap Zpool1 (Mirror (disk1 disk2)) Zpool2 (Mirror (disk3 disk4))
pros: can add disks in pairs (couple of 100€ you can absorb, you may not always absorb 500€ for 4 drives
cons: noncontinuous spaces (but with 2+TB disks it's really not that problematic)
need some copying

If not ZFS for cheap I'd also do for example 1 RAID1 2x4, then add 2x6, then add 2x8 decomision 2x4. My previous server was 2x2TB + 1x1 (1TB was only for VMs and backups of important data from 2x2).

For contiguous space I'd go 6xdisk RAIDZ2 (similar to RAID6). More expensive, single contigous space.

IMO ZFS is only OK if you have copies of important data somewhere else as well as if it fails data recovery is nearly impossible. On another computer, on USB hard drive, on another location. It was really not designed for recovery as in corporate environment they would have online mirrors and offline backups.

If not ZFS I'd still consider RAID6 for 4+TB drives if there is budget for 6 drives.
ZFS has
JBOD
MIRROR (RAID1)
RAIDZ1 (RAID5 n-1 usable space)
RAIDZ2 (RAID6 n-2 usable space)
It's like RAID but at FS level, not hardware or OS level.
You can only expand by replacing all drives with bigger drives or by adding another zpool or vdev to a zpool.

I would only use multiple vdevs in zpool if you need more than 50-100GB contigous space as if single vdev fails it will bring entire zpool down. Some data from other vdevs may be recovered.

I will start playing with ZFS and freenas on ESX. (give machine some small drives, simulate drive failure).

If your Xeons can run ESX you can put storage server in ESX and then give it direct access to drives. I'm using this to back up my Windows server: give it access to PCIE USB3.0 card to which external drive is connected. That drive then gets taken off site.

[VJ]

My Xeons are old, generation Prestonia, 32 bit. The biggest problem it has are the drivers for the storage hosts: it uses an onboard Adaptec U320 (it boots from a U320 disk) and a PCI-X Promise SX4300 as sata controller. The latter has no drivers for recent OS (Promise basically killed support for these adapters a few years after I bought it), so it is stuck running Windows. It is a 14 year old system: not power efficient and loud. It is not that slow (surprisingly), and I have Virtualbox on it, but being only 32 bit limits the possibilities now. I know that if the u320 hdd fails, I may have a problem booting it: it will boot from the sata controller but it will need an OS that supports it (so again old Windows, or I would have to find an IDE disk), I'm not sure it boots from usb.

I will start initially with 2 big disks in raid1. It would be my first storage server and I want to start small and perhaps add stuff later. I don't need the contiguous space and will keep data mirrored. All data at first (during my learning phase), but later only the data I would really hate to lose.

However, you made me wonder about the mainboard... it has 4 sata ports, which is a bit tight (2x raid1 and it is full). It also has just one memory slot, allowing for up to 16 GB of SO-DIMM. However, it has a PCIe x1 slot so it would be possible to add a sata host, OS could be on a M2 disk. Still, I feel I should not think about it as a "computer", but rather as a "device" : if I would buy some 4-bay QNap, it would also not be expandable. And I should see this build more as a QNap alternative, but it is difficult to look at it that way. The mainboard can be found for as little as 100 euro, so-dimm memory costs 50 euro / 4 GB, even the smallest m2 disk would suffice so for just over 360 euro, I can build it (without the case and psu). Add a case and it puts it in the price range of the lower range 4-bay QNap NAS's. But it should allow much more flexibility (it could run VMs, and later I could re-purpose the mainboard as a media client or so).
I have an idea for a temporary case but need to measure some things first. As we will have to do renovations in about 5 years time (after the building gets renovated), perhaps that may be the best moment to look for a more "permanent" storage server location and enclosure.

I still have one other option: repurpose the htpc as file server (Asus z97-a mainboard with an i7). It has 6 sata ports and as a full atx has enough possibilities for adding sata hosts. I don't want to go too low power for my htpc and am planning to add a video card to it to make it suitable for gaming. So while using that as file server could be the more logical option (but also more costly), I'm not sure I want to go that route. It would not make it a low power server and it is overkill for my purpose.

[VJ]

Still thinking about other options...

I could just get DrivePool (which Umfriend is suggesting all the time ), add a few harddisks to the HTPC (it has space) and optimize that one as media server.

Or I could just retire the Xeon, and get a new mainboard, cpu and memory. The HTPC's mainboard has PCI, so I can plugin the scsi card for my scanner. And then get a new system as HTPC.

[Umfriend]

Don't take DP. It is to simple, versatile and resilient for the likes of you! :d

But seriously, have a look at their Scanner application as well (if you go Windows, that is), it's the combination that is great.

Personally, I have been drooling over the Lian Li PC-D8000 as a result of this thread and that is all your fault. It is almost a perfect case for me.

[VJ]

There also is the Thermaltake Core W200 which can be teamed up with the P200 for even more space... Coolermaster has something similar with their HAF series... Though the LianLi looks the best option if you want drives....

[VJ]

Some questions about Drivepool...

Suppose you have 3 physical disks (d:, e:, f: ), each with just one partition. Let's say you define 3 pools: one with no redundancy (lets call it x: ), one with double redundancy (y: ) and one with triple redundancy (z: ).

Am I right with the following assumptions:
1. if you put files on x:, they will be on one of the three disks, which one depends on the balancing mechanism used.
2. if you put files on y:, they will be on two of the disks, again which ones depends on the balancing
3. if you put files on z:, they will be on all three disks.
4. you can still put files on the disks directly (so on d:, e: and f: ) if you want. This may affect the balancing when files are put on pool x: or pool y:

Now, what if a disk fails? Suppose disk f: fails. Of course you loose the data that you put on f:, and the data that you put on x: that was stored on f:. But what with the double/triple redundancy data? Will it automatically fill the disk that you put as a replacement for f: ? Or do you have to do special things then?

[Umfriend]

Sure, make it complicated...

You can not add a single HDD (or partition, really) to more than one Pool. And yes, you can still place data on a HDD that is part of a Pool. It is basically like this:
D:\root -> Here you can place data on HDD D: which will not be in a Pool. D:\Documents\Data\TrumpIsATurdPics\Pic01.jpg will also not be part of a Pool.
D:\Poolpart.* folder -> Here DP will place files that are saved in a Pool, say X:\

You *can* add a Pool to a Pool. You could, for instance create Pool Y: with
1. E:
2. X: (what happens is that D:\Poolpart.* folder will get a nested D:\Poolpart.*\Poolpart.* folder. Files witten to X: and up at first level, written to Y: at second level.)

Then set x2 duplication on Pool Y: and a copy will reside on E and another on D: (through use of X

Next, create Pool Z: with
1. F:
2. Y:

And set x2(!) duplication on Z:

Z will duplicate on F: and Y:. Y: in turn will duplicate to E: and X: and finally X: will write to D:

Of course, it is questionable if this is a good setup. Should D: fail then, although you can recover everything that was duplicated, all your Pools will, I am pretty sure, go into read-only mode as all Pools are affected.

It would be better to setup one Pool consisting of D to F and then tell DP for which folders you want x1, x2 or x3 duplication (You can set a default duplication level on a Pool and adjust at the folder level which inherits downwards where you can adjust again AFAIK).

If a disk fails then DP will reduplicate if neccessary and if sufficient HDDs are present, i.e., it can't do x3 on a Pool with 2 HDDs. If only 2 HDDs remain and you add a third then yes, it will reestablish the x3 duplication. The only caveat is that the Pool may go into read-only mode. I am not entirely sure when that happens and what it does then. I think this happens when a HDD gets disconnected but that may happen with a catastrophic failure too (e.g. controller blows out, no connection). It may be that in that case user intervention is required first.

In any case, I believe it is advisable to have a Pool consist of at least one HDD over the highest duplication level. Not because of risk of loss but because of uptime considerations.

[VJ]

Sure, make it complicated...

Well, it usually helps me to understand things.
Thanks for answering!

You can not add a single HDD (or partition, really) to more than one Pool.

Ok, that I did not know... But at least this limitation keeps the complexity down.

You *can* add a Pool to a Pool. You could, for instance create Pool Y: with
...
Of course, it is questionable if this is a good setup. Should D: fail then, although you can recover everything that was duplicated, all your Pools will, I am pretty sure, go into read-only mode as all Pools are affected.

That I read, but it seemed confusing to me...

It would be better to setup one Pool consisting of D to F and then tell DP for which folders you want x1, x2 or x3 duplication (You can set a default duplication level on a Pool and adjust at the folder level which inherits downwards where you can adjust again AFAIK).

Wait, so you set the duplication level per folder? So it is not a setting of the entire pool?

So if I have four hdds (lets assume one partition per hdd), can I define a pool over those that for some folders has 1x duplication but for others 2x, 3x or even 4x duplication?

If a disk fails then DP will reduplicate if neccessary and if sufficient HDDs are present, i.e., it can't do x3 on a Pool with 2 HDDs. If only 2 HDDs remain and you add a third then yes, it will reestablish the x3 duplication. The only caveat is that the Pool may go into read-only mode. I am not entirely sure when that happens and what it does then. I think this happens when a HDD gets disconnected but that may happen with a catastrophic failure too (e.g. controller blows out, no connection). It may be that in that case user intervention is required first.

Interesting.

In any case, I believe it is advisable to have a Pool consist of at least one HDD over the highest duplication level. Not because of risk of loss but because of uptime considerations.

Basically it acts a bit as a hot-spare...

Any experience with their new cloud-pool thing?

[UtwigMU]

Guess what I found for 300€
Supermicro X9SCM-F
3.2GHz Pentium
32GB ECC
LSI 8-port SAS controller

The board has dual LAN + IPMI and best of all: Matrox G200
Now I only need 2 drives, SSD, PSU and a case.

[Umfriend]

@VJ: Yes, so I have x2 duplication on the Pool which ensures, well, all files are duplicated once. However, I can adjust at the folder level, set some to x4 and others to x1 for instance and x3 for yet others. Personally, I do not do that because I like KISS.

And yes, it is a sort of a hot-spare idea. Not really as that HDD will be used as opposed to stay idle and in reserve but at least your Pool will be able to repair itself to the max set duplication level after loss of one HDD. Scanner really comes into play here as it will evacuate HDDs once SMART errors occur.

I have no exeperience with their CloudDrive product and I do not intend to use it. I *think* it is still beta and it has known its issues but they may be sorted out mostly. The *one* thing I really like about CloudDrive is that it encrypts at the client level, i.e, you can have a virtual HDD on Google, Dropbox or some other cloud-provider (they support quite a few) and the provider is not able to see what is stored on there. On the other hand, I guess it makes it hard for such providers to provide backup-servicer. For example, we use DropBox pro or somesort and we can restore older versions and delete files etc.

[VJ]

Guess what I found for 300€

Looks nice...

@VJ: Yes, so I have x2 duplication on the Pool which ensures, well, all files are duplicated once. However, I can adjust at the folder level, set some to x4 and others to x1 for instance and x3 for yet others. Personally, I do not do that because I like KISS.

Yes, that can get complicated... But so it is possible to use the same disks for different configurations (not in different pools as I called it earlier, but with different levels).
Is there any way to see which folders have which duplication, outside of the drivepool software (eg like subversion on Windows puts overlays on the folder icons)?

One option to keep it simple (and to make it look like the example I had in the previous post) could be to make a folder on the pool that you give e.g. 2x duplication, a second folder that you give 3x duplication and then assign drive letters to those folders.

And yes, it is a sort of a hot-spare idea. Not really as that HDD will be used as opposed to stay idle and in reserve but at least your Pool will be able to repair itself to the max set duplication level after loss of one HDD. Scanner really comes into play here as it will evacuate HDDs once SMART errors occur.

True, not a full hot spare, but if everything is duplicated at least 2x and you use no disk directly, then any one disk can fail and the system will repair automatically.
If this happens, and you then add a disk, does it rebalance the data over the disks? Or can you force it to do that? [/QUOTE]

I have no exeperience with their CloudDrive product and I do not intend to use it.

From what I read, it also works with network mounts. So you could probably use a nas as one of the duplication targets. Would be interesting to see how they deal with performance differences of the different interfaces.

I now feel that getting just the disks and putting them in the htpc could be the easiest option: there is room in the case for up to 5 harddisks (2 installed). The downside is of course I cannot configure it now and will only be able to do it when I return to Warsaw.

[Umfriend]

Yes, that can get complicated... But so it is possible to use the same disks for different configurations (not in different pools as I called it earlier, but with different levels).
Is there any way to see which folders have which duplication, outside of the drivepool software (eg like subversion on Windows puts overlays on the folder icons)?

Not that I know of. Windows itself will be unaware of duplicates.

One option to keep it simple (and to make it look like the example I had in the previous post) could be to make a folder on the pool that you give e.g. 2x duplication, a second folder that you give 3x duplication and then assign drive letters to those folders.

I guess that would workm, yes.

True, not a full hot spare, but if everything is duplicated at least 2x and you use no disk directly, then any one disk can fail and the system will repair automatically.
If this happens, and you then add a disk, does it rebalance the data over the disks? Or can you force it to do that?

That depends. With default settings, it will do this automatically within a certain time after you add the disk. You can turn if off or set it to immediate.

From what I read, it also works with network mounts. So you could probably use a nas as one of the duplication targets. Would be interesting to see how they deal with performance differences of the different interfaces.

Perhaps, you could check/ask at their website/forum.

I now feel that getting just the disks and putting them in the htpc could be the easiest option: there is room in the case for up to 5 harddisks (2 installed). The downside is of course I cannot configure it now and will only be able to do it when I return to Warsaw.[/QUOTE]