redundancy...

[VJ]

Reviving this thread as my stay in Spain is nearing its end (just half a year to go... amazing how time flies) and I'll soon be back in Warsaw taking care of organizing my data properly.


Umfriend: I have some questions regarding Drive Pool. Is it possible to do something like this:

2 x 2TB harddisk (let's call them 2a and 2b), 1x 4TB harddisk (let's call it 4), 1x 8 TB harddisk (let's call it 8)

Duplicate a folder on 2a + 4 + 8 and another folder on 2a + 8 (I would do something similar with 2b+4+8, 2b+8 and 4+8). If the 8 TB disk is slower does it affect the overal write speed? If so, is there a way of making it duplicate in the background? I think the plugin "SSD optimizer" seems to allow for that, or maybe the "Ordered file placement". Would this work?

Can you set a network mount as a harddisk, or does this need CloudDrive? What happens when it is not available? (thinking of using my old computer to hold some duplicates, but it may be just easier to use some synchronization rather than DrivePool).

Thanks!

[UtwigMU]

I bought camere before drives (have case, mobo, cpu, RAM) and then got hit with taxes, car service, so this project is still on a shelf.

[Evildead666]

Looks like I'm in the right place, just one year late

I built a NAS in May-June this year. Had been thinking about it for quite some time (about a year), having data spread over multiple hard disks in a mini-itx sff case.
I wanted some sort of redundancy in case a HDD failed, and was really wanting something to counter bit-rot as well, but that got put aside.
I looked at FreeNAS, NAS4Free, Windows Server, Linux, and also ZFS, BTRFS, ReiserFS, CoW, all of that......
ZFS - Requires ECC RAM, and Lots of it, which also requires Server grade Mobo, etc, Costs were too high for a home system.
BTRFS - not stable in RAID5/6.

Ended up making my own Synology NAS, with XPenology.
I had an ASUS Itx board in the SFF PC, with Win7. Re-used the Asus P8H77-I (6xSata), the i3-2120, and a single 4GB DDR3 stick (the other one died...).
Bought a Silverstone DS380 case, I liked the 8 3.5in drive bays, with 4x2.5in bays also.
I've got 4x4TB WD RED drives in there now, used the western digital diag tool to check the drives on arrival as the shipping from every company I bought them from was terrible. One was DOA. Exchanged within 24Hrs thankfully.
I looked at the Synology Raid FS, and BTRFS Raid, which this is capable of, but the unstable nature of BTRFS with any type of RAID 5/6 put me off that, and the Synology Raid was very user friendly, as you can add disks of varying sizes to the Raid.
In the end I went with a simple RAID6 with EXT4, so two disks can fail. It leaves me with a little over 7TB of storage.
I also added a 120GB SSD as read cache (just because i could. Don't need it at all).
The XPenology boots from a USB drive, and is installed on a dedicated SSD.
The 4 NAS drives are connected on the 4xIntel SATA ports, and the two SSD's are on the additional ASMedia SATA controller.
I also have a relatively cheap 4xSATA controller installed to add another 4 drives down the road.
The i3 has the stock Intel cooler i got with a faster i5.

Basically I get 100MB/s reads and writes to it, it maxes out the GBit LAN no problem.
It did end up costing a little bit, but i'm very very happy with it, and you control it through the Web interface.
All I bought were the drives (the biggest cost at 4x150€) the DS380 case, ~200€, an SFF 450W PSU (dinky little thing), and the PCIe SATA controller, 50€.

As for triple backups, i have a SCSI GigaMO back in my main PC, its a Low Profile Adaptec PCI card, on a PCIEx1 to PCI adapter. I only use the external 50-pin SCSI interface. That can only take 1.1GB MO disks though, and i only have about 30 of them. they are everlasting though, its the drives that dies first lol anything of extreme inportance goes to that.
( i have a backup drive still in its anti-stat bag).
Next step would be LTO storage, as with LTFS you can now use them in standard Windows 10. 50€ for 1.5TB of long term storage.
2000€ for the drive though...

Still, have a look at XPenology, and maybe even give it a try ? only need to boot to usb to check if it works on your hardware. Thats what i did, and then just went from there.

https://xpenology.com/forum/

one restriction, it would seem the last XPenology compatible version of the Synology DSM software is 6.1.x

[Umfriend]

Reviving this thread as my stay in Spain is nearing its end (just half a year to go... amazing how time flies) and I'll soon be back in Warsaw taking care of organizing my data properly.


Umfriend: I have some questions regarding Drive Pool. Is it possible to do something like this:

2 x 2TB harddisk (let's call them 2a and 2b), 1x 4TB harddisk (let's call it 4), 1x 8 TB harddisk (let's call it 8)

Duplicate a folder on 2a + 4 + 8 and another folder on 2a + 8 (I would do something similar with 2b+4+8, 2b+8 and 4+8). If the 8 TB disk is slower does it affect the overal write speed? If so, is there a way of making it duplicate in the background? I think the plugin "SSD optimizer" seems to allow for that, or maybe the "Ordered file placement". Would this work?

Can you set a network mount as a harddisk, or does this need CloudDrive? What happens when it is not available? (thinking of using my old computer to hold some duplicates, but it may be just easier to use some synchronization rather than DrivePool).

Thanks!

Missed this! No, a network mount can not be part of a Pool AFAIK. I don;t understand the 2a + 4 + 8 and 2a + 8 bit. How many duplicates do you want and am I right that you have 4HDDs in total? Yes, if the 8TB HDD is slow than that will govern write speeds. SSD Optimizer may help though, that is what someone does to feed from faster HDDs to 8TB Seagate Archive drives, which can be slow (if you write more than 20GB in a short time).

I am not familiar with CloudDrive. It may help using network mounts but I think you'd need some sort of API like the cloud storage providers have. Best to ask at their forums.

[VJ]

Thanks!
When I return to Poland, the plan is to fully make order in our computers and data storage (data are currently spread over 2 computers with in total 6 internal disks and over 6 external disks). One of the computers is the old dual Xeon, which I would not like to have permanently on as storage server (noise + power).

I know that at the moment a lot of data is duplicated, but lacking a proper strategy and automation, it is quite a mess... We mainly have four types of data: own made photos/videos (for which we want proper redundancy and backup), work related files (articles, source code; which I also don't want to lose), various support files (software/manuals, usually can be downloaded again but I like to have my own offline copy - has saved me a couple of times) and then our music/video collection (for which redundancy is less important although I'd hate to have to re-rip it all over again). I'm currently examining the data flows we have (e.g. the difference with how photos/videos from cameras and mobiles are treated).

I am thinking of re-purposing one of the external disks as a storage connected to our router as sort of a private cloud; its main purpose would be to serve as universal entry point for data that comes from mobile devices (e.g. photos from phones or laptops): the main computer could do a one-way sync with it in order to integrate the data. The reason for using the router as a private cloud is that there is mobile phone software that integrates with it easily. It would also be an easy point to upload data using e.g. a laptop. It would not be the fastest entry point though, but for mobile data it should suffice. Other data (e.g. from my camera) would go directly to the main computer. Redundancy depends on the type of data. And then at regular times this could be synchronized with the other computer (as a type of backup).

[UtwigMU]

I've built ZFS NAS.
Fractal Define Mini + additional Fractal fan (2 intake, 1 exhaust)
Corsair RM650x modular gold, silent mode on low load
Supermicro X9SCM-F - onboard Matrox G200
32GB ECC
Pentium G
Noctua NH-U12S
IBM (LSI) 8 port SAS flashed to LSI and IT mode - flashing this in UEFI shell was a bit of work
4x8TB WD RED
2x Samsung 860 (tried Sandisk SSDPlus at first 120GB but boot volume would become corrupt - read that this drive has problems with FreeBSD)

While web interface is not streamlined everywhere (you need to create dataset and set permissions there and share on top of it where you can't set permissions) and BSD is a bit quirky (sometimes it would not get IP), it's nice overall. I tested simulated drive failure and replacement, volume import, config restore, boot RAID, email alerts.

Resilver (equivalent of RAID rebuild) with about 1TB of data on volume completed in about half an hour which is much better than traditional RAID where with this drive size it would take 8-20h. You can also schedule scrubs (read all data, check for errors and correct from parity). I would say ZFS is for you if you know Unix/Linux CLI and have basic understanding of Unix/Linux administration. While you don't need it to set up, you will probably need it if something goes wrong. It's also good to know what GUI is abstracting from you and what goes on in the background (Unix permissions, users, groups).

I'm tempted to add 2 more drives in the future (will need to back up and rebuild) for RAIDZ2 (RAID6 equivalent), while I'll exceed 1GB RAM / 1TB storage, I'm not running jails and don't have multiple users or VMs.

You can upgrade where you replace all drives with larger ones (you can leave old drive in until end of rebuild if you have enough SAS/SATA ports). Once all drives have been replaced, extra space becomes visible.

Windows shares and setting share permissions from Windows Explorer (provided you're share owner) works perfectly, it's using SMBv3. I hit 100MB/s over single Gb NIC with large files and 30MB/s with MP3s.

If you have a spare box or hypervisor with 8GB free you can try FreeNAS (not for production - just for testing, for production you want bare metal and ECC) - the system install from USB takes under 3 minutes.

I'm copying data from my old 2k3 server which has been sitting powered off in the corner. Then I will copy data from laptops and external drives, SD Cards and sort it.

I set shares:
backup where I have RW - here I will dump my own created files from my laptops
archive where I have read only (created another user with RW for storing data) - here I will move finished projects and photos
media (movies, music, pdfs, isos) - here I also have read only

I plan to buy couple 5TB external drives. One for media and other for archive + backup. I'll take these off site (like I do with my esx hosting mail server - 2 drives, daily backup, rotate drives about once a month)

[UtwigMU]

No problems after 7 days of uptime.
I can plug USB drive into the NAS box, create zfs, copy data using ssh, rsync or mc and then plug drive in my Linux box. Data on zfs is readable by Linux.

[VJ]

Nice!

I'm planning to reinstall my media computer and organizing the storage better, most likely using drivepool. At the moment, the system has 2x 3TB harddisks (apart from the boot SSD), which is getting tight. But a lot of data is duplicated, both locally, on external disks and on a third computer. My plan is to restructure the data to gain a better overview of what is where, improve on the datamanagement using drivepool and then if necessary add a harddisk to it.

I also have the plan of using Docker to provide various ways of accessing the data: e.g. a NextCloud, or various media servers. I've been experimenting with Docker on a raspberry and it looks quite interesting. So I plan to test it on the media computer (could not test it on my laptop as it does not support hyper-v).

[UtwigMU]

Just of note: on freenas you can run jails and virtualization (if CPU allows). You can one-click install many plugins into jails. Nextcloud and owncloud are available

https://www.ixsystems.com/documentation/freenas/11/plugins.html#available-plugins

[VJ]

Interesting... However, the computer doubles as a htpc, so it runs Windows. The main reason I want to go with Docker for the different services is not to "pollute" the system installation: I could easily enable/disable services by starting/stopping the container and I could experiment (add/remove) without it leaving traces in the system that may slow it down or give conflicts over time. The nextcloud or owncloud (will try which one I like best) would be for example to easily transfer files from/to mobile devices (they have a client). It would also allow me to test with different media servers (I need several specific servers; docker allows me to keep them isolated).

[UtwigMU]

What I'll do next is subsonic. You install subsonic and give it access to music. Then all music is available indexed and searchable in browser.

There is also a jukebox mode - I could bring an old Thinkpad to work, connect it to speakers and load it with MP3s so that colleagues in office can play music on the server itself from their browsers.

[VJ]

I am using Logitech Media Server, as I have a number of Squeezeboxes to stream music (and it exists on Docker!). I would also add some video server and a tv receiving server (currently I use NextPVR, but I would like to change it to a TVHeadEnd). Something I planned for some time is also automated conversion of tv recordings: they pointlessly take up a lot of diskspace; but it is possible to convert them to a better format without a big impact on quality. I've already learned that they all exist on Docker, so I have a feeling that will allow me to keep a clean system while being able to experiment.

So, quite some things to configure to get different workflows up and running. Will be a busy month of December. I have already a list of 6-7 Docker containers I want to use, range from the media stuff over cloud stuff to even a user interface for it all (Portainer).