redundancy...

[VJ]

Hello,

What is the current best practice to have redundancy of your data?
Typical Raid5 is considered risky with big harddisks due to the risk of read errors (which statistically are very likely to occur on big disks), preventing a rebuild.

I'm reading up on many things, such as SnapRaid, ZFS, Greyhole, ...

SnapRaid seems interesting with benefits (every data disk is readable) and downsides (redundancy is not instant but snapshot based). One benefit is actually that you can recover data that was accidentally deleted if the SnaprRaid has not been synchronized.

But I have a bit of problems getting my head around ZFS. Basically you have disks, which you group in some redundant way (mirror, raidz) in vdev, and then you group different vdev in a zfs pool. If I understand correctly, the redundancy is within the vdev; loose one vdev and you loose the zfs pool.
So why would one add multiple vdev to a zfs pool? Is the only reason drive pooling?
And how is raidz1 better than raid5: doesn't it suffer the same rebuild risk?

I like the system on DrivePool on Windows, and I suspect Greyhole comes closest to it on Linux... The main attraction of these systems (to me) is that in case of an emergency you can just plug the disk in another system and access the data. It basically seems like an intelligent way of mirroring, so it does waste more storage.

Any comments?

Jörg

[Umfriend]

I use DP (DrivePool, not Double P.... ) on my server and like it very much. No clue about Linux.

Having said that, DP is more like a file based RAID 1 if you want redundancy and it does not protect against accidental deletions. It is not a backup.

[VJ]

Yes, I know raid is not a backup... but after the typical old-school raid systems, it became such a mess to figure out good policies for redundancy now.

I found a nice explanation on why raidz1 does not have the same issue as raid5here

ZFS mitigates this failure to some extent. If a RAID5 device can't be reconstructed, you are pretty much out of luck; copy your (remaining) data off and rebuild from scratch. With ZFS, on the other hand, it will reconstruct all but the bad chunk, and let the administrator "clear" the errors. You'll lose a file/portion of a file, but you won't lose the entire array.

Does this make sense?

The thing I like about DrivePool is that you can easily access all the data by just putting the disks in a different system. You may have to figure out where exactly the mirror of a file on a damaged disk was put, but you won't have to worry about a rebuild or so before you can access the data. A second thing I like is that you can maximize the harddisk space, e.g. use the excess space on a bigger harddisk as non-mirrored storage.

It seems to me that GreyHole does the something similar using samba shares:

https://github.com/gboudreau/Greyhole/wiki/HowGreyholeWorks

Snapraid also protects on file level, but you need to rebuild to access the data on a failed disk. All other data is directly accessible still.

Unraid is also often mentioned, but I still fail to see how it adds redundancy. It just seems like a combined hypervisor/docker platform.

The main reason I inquire is because I'm feeling I need to better organize my data. I have multiple copies and so, but it just is a mess as it is not done systematically enough. And combining my girlfriends data with it also makes it more complicated. I'm planning on adding some storage server when I move back to Poland, but it usually takes me quite some time to decide on a system. Now I started playing around with different systems on VirtualBox, just to see how it all works (and it is quite easy to simulate a failed harddisk :-)). The easiest would probably be some NAS (e.g. Qnap), but I'm not always that sure of a black box system...

[UtwigMU]

You want ZFS.

Check this out, this is what I'm building next instead of Covfefe Lake gaming rig I want. (Data is more important).

https://arstechnica.com/civis/viewtopic.php?f=8&t=1365955&hilit=zfs+nas

https://arstechnica.com/civis/viewtopic.php?f=11&t=37779

ZFS with freenas is not so hard - see threads I linked.

Our Synologies at work work nicely (but I use them to back up servers to) and they use EXT4. Only problem was a failed fan after 5+ years of use but I was able to MacGywer in a fan from some old desktop P4 to get it going again and then I ordered a replacement fan.

If loudness and power draw is not a problem you can just buy LFF drive Proliant ML350 G6 (I did this for my ESX hypervisor hosting my email server) and then add some RAID intended drives. It's a tower server and is not too loud in another room but not for living room.

Then another thing is offline backup in case of fire, theft, flood, major disaster, cryptolockers (they can lock mapped network shares from your Windows machines). I'm using two 3TB Toshiba USB 2.5" drives to back up my server and I take them offsite and rotate them around once a month.

You could also use two NASes on two locations and use rsync. I did Synology to Synology VPN rsync setup once for some company.

[VJ]

But the thing is: ZFS in itself is not adding redundancy, it is just pooling multiple VDEVs.

You need to add redundancy within each of the VDEVs, but the question is what to use at the level.

From what I gather now, ZFS on top of a single VDEV that uses RaidZ seems to be the way to go... Other VDEVs can be added to increase the size of the pool (or you increase the size of each harddisk in the existing VDEV).

[UtwigMU]

Probably something like 6-bay case. Hotswap is not needed. I'm thinking something ultra quiet (5400RPM drives, high end CPU cooler). Then you use 4 4TB drives in one VDEV and leave 2 spares so you can add 2 larger drives in future to migrate data there when you want to upgrade the main 4 drives. Use cheap small SSD for OS.

I haven't researched this thoroughly yet as I'm still waiting for budget. 4-drive self built NAS is around 1200€ ballpark.

[VJ]

I'm still first trying to figure out which standard to use for the redundancy of a VDEV. Currently I'm playing with the idea of one system with important data that would be stored on a ZFS which has a single 2-drive mirror VDEV and a second system for less important data that also serves as backup of the important data, where data would be on a ZFS which would have a RAIDZ1 built on 2 or 3 harddisks. The second system could even be my current computer.

As far as hardware goes, I came across these servers, which I think are interesting:

https://www.hpe.com/us/en/product-catalog/servers/proliant-servers/pip.hpe-proliant-microserver-gen10.1009955118.html

Especially if you consider the price of 220 euro:

https://www.idealo.de/preisvergleich/Typ/4549821091225.html

(no storage included)
But I need to read more about them regarding linux compatibility. From the Marvel bios manual, it seems it only supports RAID0 and RAID1, not JBOD, which would be a big downside... Then again... If I would use RAID0 for my important data, I would not have to worry about software mirroring.

Still hardware comes later, first I want to have a good plan on how to organize it all. And be sure I can use the system to recover from failures.

[UtwigMU]

That HP mini is interesting. I wonder how quiet is it. Before I had Gigabyte board with 64-bit Sempron cooled with Ninja mini and 2 WD green 2TB drives running Server 2003. That thing was super quiet - CPU fan never spun up because temps never reached 60°C.

[UtwigMU]

I found thread with issues about installing FreeNAS on MicroServer Gen10. Also note that ECC is desired for data protection and ZFS likes lots of RAM. 16GB DDR4 ECC is currently around 200€.

There is also a nice SuperMicro case but I don't know how quiet it is.

https://www.supermicro.com/products/chassis/tower/721/SC721TQ-250B

[VJ]

On VirtualBox, I'm now testing several linux based systems: OpenMediaVault, Ubuntu Server, ... I most likely will need to replace my TV tuner and am considering to combine it with the storage server. But as BSD does not have good support for tuners, I'm looking at linux solutions. My first worry is how to set the data, which is why I'm playing around on VirtualBox: it easily allows me to see what is possible. At the moment, Ubuntu Server looks the most promising, as everything can be installed and with Webmin there is an easy web-based control.

Just to get back on the raidZ1: is the difference between raid5 and raidz1 as described in my earlier post correct?

I've also been looking at hardware, which is how I came across that HP (funny that 16 GB of memory costs almost as much as the entire system with 8 GB). A bit surprised about the AMD, which is why I need to read more. And also the fact that you cannot switch off the raid seems annoying. Of course it as always possible to sacrifice one of the PCIe slots for a sata controller, but I'd prefer not to have to do that.
For custom built, there are some interesting mainboards based on the Intel C3000 series, particularly from Asrock and Supermicro. Just annoying that it is so difficult to find a decent enclosure. In my situation, I would install the server on a high shelf, so I would not want it to be too big or to heavy. But as that rules out a tower, choices are very limited. While looking for smaller cases (check U-NAS and CFI), I found the HP, which costs almost the same as some of the empty cases...

[UtwigMU]

The case experience: HP, Supermicro cases are generally designed to be in server rack or noisy office at best and not in the home. So I assume they will be audible or will need to be hacked with better fans like Noctua. Another thing is the motherboard standard size. With my previous server motherboard died half way into it's 10-year life and I just bought some desktop motherboard+CPU+RAM, screwed it in the case and off I went. You generally would want to upgrade the motherboard without having to replace the case. Even with that HP micro server the whole project is still 900 Euros + VAT with 4 4TB drives, so spending a bit more on case, PSU and motherboard is not that bad. (you need RAM, SSD)

Popular case seems Fractal design Node 304. There is no hot swap but at home you can power down to swap a drive and you will open it when you add or replace drives. The hot swap cases add noise because there are more parts which can rattle and airflow is tighter with fans behind drives.

http://www.fractal-design.com/home/product/cases/node-series/node-304-black

RAIDZ is a bit like RAID5

https://en.wikipedia.org/wiki/Non-standard_RAID_levels#RAID-Z

As far as I know (haven't researched this thoroghly) it's not easy to resize ZFS - you need to back up and rebuild or build another array or another box. I like virtualization - running servers virtualized at work but storage is where I think another layer of complexity might be bad for rebuilds and data recovery unless you give VM full access to physical drives. Thus combining storage and TV tuner might not be optimal. You could use NAS with iSCSI and then run VMs off it on another physical host.

[VJ]

The case experience: HP, Supermicro cases are generally designed to be in server rack or noisy office at best and not in the home. So I assume they will be audible or will need to be hacked with better fans like Noctua. Another thing is the motherboard standard size. With my previous server motherboard died half way into it's 10-year life and I just bought some desktop motherboard+CPU+RAM, screwed it in the case and off I went. You generally would want to upgrade the motherboard without having to replace the case. Even with that HP micro server the whole project is still 900 Euros + VAT with 4 4TB drives, so spending a bit more on case, PSU and motherboard is not that bad. (you need RAM, SSD)

The noise thing is indeed true. So it is strange that no mainstream manufacturer came with a standard case, for server use. A micro ATX board should fit with some cold swap front accessible 3.5" hdd trays and an ATX PSU in something the size of a short depth 2U-3U rack-size case (LianLi used to have something, but not anymore; Silverstone now has something with 3 drive bays IIRC; Chenbro has a nice 4-bay mini tower but they are 2.5"). From what I read, the previous generation of that HP miniserver was relatively quiet.
To some extent I agree on the motherboard replacement thing, but on the other hand if the whole setup is cheap enough, the server itself is almost disposable: rather than upgrade the mainboard, upgrade the server. If you would buy a regular NAS, you also cannot upgrade it. And as it serves a single main purpose, if it is suitable for this purpose now, it should remain suitable for a long time (my brother had a Netgear NAS, and it just kept going) - a bit different from a normal computer where the software you run tends to get more demanding. I always count the price without harddisks, as these you will always need, independent of the solution.

Popular case seems Fractal design Node 304. There is no hot swap but at home you can power down to swap a drive and you will open it when you add or replace drives.

Yes, but it is more difficult to keep track of the harddisks: if one fails, you have to be sure to know which one. Where I would want to install the server, I would have difficult access to it, and I would like to avoid having to move it too much. U-NAS has a nice case but at 200 $ for the 8-bay one, it is as much as the entire HP server. CFI makes relatively cheap mini-itx cases with 4-5 hotswap bays, and those seem like an option.

http://www.mini-itx.com/store/?c=42#cfi-a7879

http://www.mini-itx.com/store/?c=42#CFI-A6039

One could even consider putting two of those next to each other and run cables from one to the other (not so elegant, but doable). And if you can DIY a bit, perhaps even making something that takes a couple of 5.25 to 3.5 backplanes could work:

http://www.icydock.de/category.php?id=51

A few of those, a motherboard tray, a backpanel recycled from an old case, some metal or wooden sidepanels... and things start to take shape.

RAIDZ is a bit like RAID5

https://en.wikipedia.org/wiki/Non-standard_RAID_levels#RAID-Z

As far as I know (haven't researched this thoroghly) it's not easy to resize ZFS - you need to back up and rebuild or build another array or another box. I like virtualization - running servers virtualized at work but storage is where I think another layer of complexity might be bad for rebuilds and data recovery unless you give VM full access to physical drives. Thus combining storage and TV tuner might not be optimal. You could use NAS with iSCSI and then run VMs off it on another physical host.

Yes, but I have not found confirmation on whether or not the RAIDZ also is lost in the even of an unrecoverable read error. In the case of raid5, the rebuild fails. The quote I posted earlier states that this is not the case for raidz, and that there just a sector is lost. I would love to find confirmation on that, as that makes a huge difference.
In many media-oriented forums, people seem to combine TV server with their storage. I'm still wondering what to do... I have the computer in my livingroom which is currently used to record stuff, so I can keep the TV tuner in that one and make it record to a nas, but if the same could be achieved with having just one computer powered on, it seems like a better solution.

[VJ]

I'm still worried about using systems I don't fully grasp. This post on raidz is a bit worrying:

https://forums.freenas.org/index.php?threads/raidz-and-unrecoverable-bit-errors.27014/

My biggest fear is that I would not be able to recover from a device failure due to lack of knowledge either when configuring and setting it up, or when things go bad.

The more I look at it, the more the HP microserver with raid1 configuration through its controller makes sense. I would not have to worry about recovery, nor about software configurations to manage the redundancy. If by comparison you go with e.g. 3 hdd in a raidz1 configuration, you have 66% usable space. take 4 disks in a raidz2 configuration and you have 50% usable space... exactly the same as if you would mirror 2 harddisks. Ok, the raidz2 protects against any 2 harddisk failures whereas if both disks of the same mirror dies, you lose that mirror.

Looking at the cost, adding more memory to be able to run a raidz2 which I'm not sure I am knowledgeable enough to recover or adding an extra disk so I can mirror is more or less the same...

I currently installed Ubuntu Server as a test case on VirtualBox, and added Webmin to it (web based management). It all seems to work very well, and as Ubuntu server has no GUI, it is very lightweight. In addition, I could focus on having Ubuntu serve the files through different protocols and not have to set it for redundancy as this is dealt with at a lower level. The whole purpose is to have peace of mind, and not being sure I'd be able to recover from a failure does not give me peace of mind.. :-)

[UtwigMU]

Problem with server RAID management is notification of drive failure, unless you have a fancy controller and ILO or at least lights on the front. Drive may fail but you may not notice it for a month. Also more often disk drops out of RAID than it fails. For example one SBS I manage has all new drives now. First both large RAID1 data drives failed (some bad sectors) but i was able to clone them and check file system. Then the small system RAID1 drives started dropping out of RAID so I replaced them one by one. If going with Linux I'd set up LVM and RAID in Linux on entry level box. Also the ESX with 8 1TB WD black drives we had at one place in 2009 kept dropping out drives of hardware RAID on adaptec. I once had to put laptop drive to motherboard SATA, install windows on it to see what's going on with controller.

The only benefit of ZFS is protection against data corruption. If not, I think it's better to just buy a small Synology or QNAP (essentially same as Linux with webmin but more functionality and easier to manage). The newer faster CPU Synologies now have BTRFS.

[VJ]

Yes... but then: would you trust raid5 on a NAS? I mean, if the disks are too big, you risk failing the rebuild. As most simpler models only offer 4 bays, the only alternative is mirroring.
Detection of the failed harddisk is indeed an issue with server raid management, but don't forget that I'm considering a home environment; I won't have to manage many of them. I was looking at NAS initially, but then considered that for that price I could actually build a computer... And hence the problem of the disk organization... And somewhere along the road I found that HP miniserver, which would be perfect if the Marvell Storage Utility would run on Ubuntu (they have it for ClearOS)... While I realize that the current NAS boxes are really small computers (even with virtualization/docker/...), I'm still weary of an expensive black box... a bit irrational; I know that a Qnap could do everything I need... (even the TV server part with optional software and a compatible dvb tuner).