Intel storage configuration

[VJ]

Ok, so at least I get how it works...

Currently, I'm thinking of going with 2x 4TB to get 4 TB net storage. I will try SnapRAID first: if that one works well, it should suit my purpose of some low cost redundancy. The reason to go for the 4TB would be price: same price per GB as 2 and 3 TB, and the price will likely go down: two disks now, one more later (e.g. next year) to double storage to 8 TB, perhaps one more later to even go to 12 TB.

I was planning to get the system after summer, but it is really itching... beginning of august currently looks plausible.

[Umfriend]

Just to be sure that I get how DrivePool works.

You put 3 harddisks (D:, E:, F: ), let's say each of them is 3TB.
You define the pool, on those 3 disks.
- D, E and F stay as drives
- new virtual drive G is added
- hidden folder is created in D, E and F
- everything written to D, E or F is just put on it

Is the folder structure you put on G represented in the hidden folder on the disks?

Sry, missed this one.
It is as you say. Although:

- everything written to G is written to the hidden folder in D, E or F, but you don't know which disk (due balancing); this impacts free space on D E and/or F.

Correct but a new balancer plug-in does allow you to control to which drives data is being written, assuming enough space, any excess will spill over to another drive. So you could enforce, I think, to have G:/Movies to be written to D: and E: only (x2 duplication). As said, I won't use it so I am not 100% sure, but the forum there is very helpful.

If you enable duplication on the pool, all files put on G are written to the hidden folder on 2 disks.
If you enable duplication on a folder on G, it is written to 2 or 3 disks, depending on the duplication level.

Correct. To be precise, with pool duplication, it is written to 2 or 3 hidden folders depening on x2 or x3 duplication. Same with folder duplication.

Yes, the folder structure on G is represented in the hidden folders. One thing I am not 100% sure about though is, say a folder happens to be only written to D: and E:, then the hidden folder on drive F: may not contain that folder or may contain it but without content.

So, assume everything on the system gets screwed but the three drives and you had x2 duplication (so net storage of 4.5TB) and you had a 6TB drive. You could connect all four drives to a NTFS-capable system, copy from each individual 3TB drive to the 6TB drive. It'll ask to copy/replace, rename, don't copy and if you select the first or third option, the 6TB drive would be a copy of the Pool (without duplicates of course but at least avoiding the same files ending up double in a different folder tree).

[VJ]

Well how about this:
2 x 2 TB drives, Pooled, full duplication. Say them drives are E: and F: and the Pool gets drive letter P:
Anything you store on P: is duplicated.
You can, however also put folders on E: and/or F:, these would not be duplicated. That's where media goes.
That is the easy solution. You can also tell DP to duplicate only _some_ folders in the Pool (I don't do that becuase I almost had a real SNAFU that way, entirely my bad). Duplicate all, just not the media folders and you're good to go + all accesible through P:/.

Yes... I'll have to consider what data I need protected and how much. I would like to have everything protected for a single disk failure, but with media and full duplication, it needs a lot of space. The SnapRAID might be good for media though.
I don't mind a large number of drive letters, so everything accessible through P is not a necessity. For things such as photos, it makes sense to have the best possible redundancy. But for music and movies, I'm not so sure. It will be a choice I have to make...

[Umfriend]

Well how about this:
2 x 2 TB drives, Pooled, full duplication. Say them drives are E: and F: and the Pool gets drive letter P:
Anything you store on P: is duplicated.
You can, however also put folders on E: and/or F:, these would not be duplicated. That's where media goes.

That is the easy solution. You can also tell DP to duplicate only _some_ folders in the Pool (I don't do that becuase I almost had a real SNAFU that way, entirely my bad). Duplicate all, just not the media folders and you're good to go + all accesible through P:/.

[VJ]

Yes, I also read that about Storage Spaces.

From what I read, I'm currently looking at either DrivePool or SnapRAID.
Full duplication seems a bit overkill for media files, considering that I also backup them. Really important files I put on multiple computers... The reason to have the disk protection is more comfort (faster than putting back backups if a disk fails).

[Umfriend]

As I read it, Storage Spaces has its issues and many get snafus with it. DP has recently introduced the Folder Placement Balance add-in. I categorically refuse to use it but others are satisfied by it and get a lot of control over placement. Some use it with SnapRAID.

[VJ]

Just to be sure that I get how DrivePool works.

You put 3 harddisks (D:, E:, F: ), let's say each of them is 3TB.
You define the pool, on those 3 disks.
- D, E and F stay as drives
- new virtual drive G is added
- hidden folder is created in D, E and F
- everything written to D, E or F is just put on it
- everything written to G is written to the hidden folder in D, E or F, but you don't know which disk (due balancing); this impacts free space on D E and/or F.
If you enable duplication on the pool, all files put on G are written to the hidden folder on 2 disks.
If you enable duplication on a folder on G, it is written to 2 or 3 disks, depending on the duplication level.

Is the folder structure you put on G represented in the hidden folder on the disks?

[VJ]

Well, zfs is not supported in Windows, so it doesn't change much.

Windows now indeed has what they call Storage Spaces (from Windows 8), and it is what you describe in the second point. However, it would not be accessible with a different OS.

Umfriend's DrivePool looks interesting, though I still like to keep some control on where the files go. But perhaps this is my old way of thinking. FlexRAID looks to be a bit similar (but their website appears quite superficial). And I also found SnapRAID.

SnapRAID has an interesting concept... You assign one disk to be the parity disk, and it takes the parity info of a number of datadisks (and you can increase this number). It is not realtime, so you need to sync the disks, and everything is safe only after synchronizing (so they recommend it for files that do not change frequently, e.g. a media library). You might be able to recover files that were deleted before a sync, but this depends on some factors.
However, the datadisks are just written to normally (no special structures or file systemes: they can be read anywhere). With one parity disk, you are protected for 1 disk failure. If a data disk fails, you loose access to it. So it is not realtime, recovering is a manual action (replace the failed disk, run a command) during which the disks should not be written to.

It ticks a number of my boxes:
+ hardware independent (even individual disks are accessible)
+ os independent (individual disks are accessible)
+ synchronization should even work on multiboot (but I don't need this)
+ same economy as raid5
In addition, only the disk that is accessed needs to be spinning, so it lowers overall noise. Downsides would be:
- not real time (but for a media library, things do not change too often; sync can be scheduled in Window or manually done)
- synchronizing is important: changing files on one disk of the snapraid may put files on other disks in unprotected state. So it is really only suitable if the data on each data disk does not change frequently. EDIT: Got confirmation that adding files on one disk does not infuence the protection of files on other disks.

Basically, a bit more hassle in daily use, requiring more discipline when putting files. (you can exclude folders from the snapraid, so that may be an option)

[dZeus]

to expand your horizon a bit (and make your choice even harder):

- You can chose to use RAID-like functionality implemented at filesystem level, such as with ZFS (Z-raid). You will need to use an operating system that supports ZFS, and ZFS by itself is rather RAM intensive. ZFS seems to have a fair share of followers, as well as people attacking it as its benefits being both over-hyped and over-stated. It sure seems to be memory hungry to manage though.
- You can chose to use RAID-like functionality implemented between the physical and the filesystem layer; modern operating systems usually provide this in the form of Logical Volume Management (LVM), with various types of RAID-like features integrated. Again, you'd need an OS that supports the exact mechanism used and portability would be limited to other OSes implementing exactly the same LVM type.

[Umfriend]

Yes! That is one thing that is absolutely great about DP, it is very flexible with respect to different drive sizes (which matters only if you want duplication).

In a 1x1TB + 1x2TB situation for instance, it will/can only duplicate 1TB of data. That makes sense of course (and it takes care not to duplicate on the same physical HDD, you cannot fool it by partitioning etc.).

But as long as all other drives have the same aggregate capacity as the single largest drive, you are good to go.

Add a disk and you;ve gained 50% of the additional space (only 50% so that is a drawback but that comes with duplication other than on an parity basis I guess). Want to replace a smaller drive with a larger one and you have no extra slot but enough space on the remaining drives? Simply kick one out of the Pool, DP will move/reduplicate. Then replace the physical HDD, add the new one to the Pool and it will balance HDD usage again.

Sure, there are disadvantages. E.g., say you have a folder for Videos. There may not be a single drive that contains them all, the may be scattered over N-1 drives (where N is the total number of HDDs in the Pool). They are working on/have realised a plug-in that actually allows you a lot of control on where specific folders are stored/duplicated. Personally, I think it is nonsensical but apparantly some users want it.

[VJ]

Reading back, they should be paying me cause I'm such a good salesman! Just love the software, that's all. And nothing I've read sofar makes me think any RAID would be better. Having said that, I *think* RAID might be a bit faster, especially with HW-controlled I/O and, yes, with parity you would save on the gross space you'd need.

I'll admit I'm a bit too stuck with the raid thing, and am rarely considering non-standard tools for basic stuff. I'll have to read more on the drive pooling thing in general. I know that some NAS systems for instance manage to optimize storage over different sized disks, while keeping redundancy: the ability to plug in an extra disk and just gaining more net storage without configuring is nice. Or replacing one disk and have the system repair itself and use the extra storage is super convenient. And perhaps DP can do this, I never thought of looking for tools that would offer the same on a computer.

Anyway, I'll stop plugging DP for now ;-)

No problem, it opened my eyes to different options, and that is never a bad thing.

Would using HDD/SDD hybrids meet your goal? Not sure it makes sense cost-wise but if it works, who knows?

Not sure how it behaves or how it decides how to cache. But it is not that important: the system should be quiet enough.

[Umfriend]

Reading back, they should be paying me cause I'm such a good salesman! Just love the software, that's all. And nothing I've read sofar makes me think any RAID would be better. Having said that, I *think* RAID might be a bit faster, especially with HW-controlled I/O and, yes, with parity you would save on the gross space you'd need.

Anyway, I'll stop plugging DP for now ;-)

Would using HDD/SDD hybrids meet your goal? Not sure it makes sense cost-wise but if it works, who knows?

[VJ]

There are people that run external enclosures and have like 20 HDDs in a single Pool. Not sure how they back that up though, perhaps there are tape-drives that allow for 40-80TB?

There are such back things, but they are expensive... I would change my backup policy from DVD to blueray, it now is just cheaper than a harddisk (cheapest harddisk is 0.03 Euro/GB, bluerays can be just below that). I know that many people rather backup on harddisk, and it has benefits (faster, easier), but it feels a bit too much eggs in a backup basket. :-)

edit: What do you mean with a second mirrored system? With DP, you'd just add one HDD to the Pool and it'd be done. They need not be in pairs or anything.

Yes, I keep forgetting you are talking about the drive pool software... I'm more thinking in the classic raid terms...

[Umfriend]

DP would, of course, also be in critical state after the loss of 1 HDD (assuming x2 duplication) but it will re-duplicate automatically given enough drives (at least 2 need to remain) and space. Another nice thing is if you complement it with Scanner. Not only does it read SMART-details, it also peforms surface scans. If a drive becomes suspect, DP will automatically offload the files to the other drives (given enough drives and space again off course).

There are people that run external enclosures and have like 20 HDDs in a single Pool. Not sure how they back that up though, perhaps there are tape-drives that allow for 40-80TB?

edit:

expand with a second mirrored system when needed

What do you mean with a second mirrored system? With DP, you'd just add one HDD to the Pool and it'd be done. They need not be in pairs or anything.

[VJ]

How would that work with RAID5 exactly? I always thought that RAID5 was block-level striping/duplication and that downsides are:
- Lose the controller / MB, lose the data?
- Lose one drive and I/O becomes terribly slow until the array has been rebuild?

I've only had experience with my hardware raid. But yes: you cannot move the disks to a controller with a different chipset, so the disks are tied to the controller. I only once experienced a raid failure, and I/O was not terribly slow. But rebuilding takes time (several hours), and it depends how much priority you give to the rebuild. As long as it is not rebuilt, the raid is in critical state (one more disk failure, and there is data loss). On a pure software raid, I would guess rebuilding takes even more time and cpu cycles (slowing down the whole system).

But if you consider that you can have e.g. 4 disks, and have the net storage of 3 disks, it is a nice balance between redundancy and cost.

I may be terribly confused on this. I like things simple and DP does that for me. And yeah, upgrading, either adding disks or, once the case or connectors are full, replacing with larger disks is so easy. In case of the latter, all you need is to have one port and bracket spare. You can even do without but then it becomes a bit more complicated.

Upgrading a raid5 is more of a pain though. Most raid5 nas systems offer some way of increasing the raid by changing the disks to bigger disks (one at a time, and rebuild after each disks), but they generally use a software raid. If you stick with standard hardware raid, I'm not aware of upgrade paths that allow you to keep the data. Perhaps if you do it the same way, and have an option to extend the volume... but e.g. my old raid controller does not allow that.

I always use to have this (bad) tactic of buying a nice system that is intended to be more future proof. So in that tactic, a raid5 that is too big seems like a good idea. But prices of storage drop so fast, that it is not a good tactic: a few years from now, the raid will be too small, with no upgrade path left. And then to keep the same thing, it means again buying 3-4 disks, and now way to keep the old ones in the case (perhaps as external disks or so). I'm really beginning to like the mirroring approach. Sufficient storage for the near future, expand with a second mirrored system when needed. Replace the oldest one when needed, etc....