Any opinions on RAID 0 being detrimental to capture?

[Brian Ellis]

Formatting is quicker than defragmenting. Erasing/deleting does nothing of the kind. All it does is to change the first character of the filename so that it is rendered invisible, and permits it to be overwritten. The file remains on the disk and, if you defrag, the "deleted" file is also part of the defrag process. This means that you can have deleted files in the middle of wanted files. Goodness knows what happens to them if you delete all: chaos, I suspect. A full format is the only way to get any semblance of order- and even then, it's not 100% sure as I have been able to recover data from a formatted disk (difficult, but possible). Probably the only 100% sure, sure, sure way is to delete all, using a software which will replace all data with a single character half-a-dozen times (used for cleaning disks with sensitive info on them), then reformatting them (maybe even low-level). However, I find that full formatting is the best way to get reasonable results quickly after delete all.

[landrover]

Brian,

I agree when it comes to deleting, but not to erasing. As far as I know, erased files will be overwritten by valid files during defragmentation. Erased files are not involved in the defragmentation proces. If you "shift+delete" all contents of a hard drive or RAID array and start defragmentation, I can assure you the process will be completed very quickly! As only deleted files are the ones with the modified first character, the DOS command "undelete" is still in my memory, I can´t remember hearing or reading about "unerase"

[Flying dutchman]

Hoi Landrover,

"Erasing" and "Deleting" a file is exactly the same thing. In fact the keywords "DEL" and "ERASE" are synonymous in MSDOS and in the command window of Windows. I know there are are lots of security tools that will fill a file with nulls prior to deleting/erasing it and call that "erase" or "wipe" but that's a different thing.

The old "Dos" Method of marking a file deleted by setting the first character of the file name to null in the file allocation table (FAT) never prevented the operating system from re-using the occupied space at all. Maybe not immediately, but if you copy new files to the hard drive or defragment the drive I can assure you the deleted files will be overwritten. Only in the case where the hook to the root block of a cluster chain is lost you will get an orphaned "chain" that must be freed using a tool like Scandisk or chkdsk.

I think you confuse "deleting" with Window's "trashcan" functionality which simply renames the files instead of deleting them - this renaming process puts the file into a hidden trashcan directory and it only really gets deleted when this folder exceeds a certain size or if the user tells the system to empty the trashcan. It has nothing at all to do with "real" file deletion. By the way, if you type "DEL somefile.txt" in a command window, the file does NOT wind up in the trashcan. It gets deleted...

[landrover]

Sorry, my mistake....but would a drive / array be slowed down by the fact that there are already data on it that are allowed to be overwritten?

[Brian Ellis]

Oh! But yes. If it is reformatted, the files are recorded from the start (the fastest part), in order, without fragmentation (at least, until some manipulation starts fragmentation.). If there are already files present, whether they are visible or not, the system has to hunt for the first free space. Even if files are able to be overwritten, it does not mean there are not spaces which are not available, such as lost clusters, slack and so on. If you take a large disk or array on which you have been working for some time, erase the lot or do a quick format, then check the space available, it can be anything up to, say, 10% less than the disk capacity. This will be recovered by full reformatting. That space taken may cause fragmentation, which can slow down transfers in both directions.

[landrover]

The prosecution rests, your Honour...

[Snake-Eyes]

Originally posted by Brian Ellis
Oh! But yes. If it is reformatted, the files are recorded from the start (the fastest part), in order, without fragmentation (at least, until some manipulation starts fragmentation.). If there are already files present, whether they are visible or not, the system has to hunt for the first free space. Even if files are able to be overwritten, it does not mean there are not spaces which are not available, such as lost clusters, slack and so on. If you take a large disk or array on which you have been working for some time, erase the lot or do a quick format, then check the space available, it can be anything up to, say, 10% less than the disk capacity. This will be recovered by full reformatting. That space taken may cause fragmentation, which can slow down transfers in both directions.

ScanDisk, THEN Defrag. ScanDisk fixes the lost clusters, etc. Defrag will put anything that's left in a nice neat contiguous order.

No need to reformat, unless the drive's so severely screwed up that even ScanDisk (or in WinXP's case, the equivalent utility CheckDisk) can't fix it.

[Brian Ellis]

Sorry, cannot agree. You can scan and defrag until the cows come home, but you can still lose a sizeable portion of disk capacity AND the placement of files, which will correct itself ONLY by full reformat. Been there, done that (in fact, I do it frequently).

[Dr Mordrid]

My 2 cents:

Defragging takes too long and is nowhere near as effective at restoring disk performance as reformatting.

Dr. Mordrid

[Eddy]

On a FAT partition, deleting all files (including hidden and system files) should result in the same as a (quick) format. But I'm not sure whether this also applies to NTFS. In any case, if there are a lot of files on the disk, (quick) format is a lot faster. If there is stuff on the partition, writing to the disk can be slower than on an empty disk. First of all because the transfer rate decreases as the heads move inwards. And when the free space is fragmented, the occupied blocks have to be skipped, causing delays and lower throughput.

So it would be best to capture to an empty partition. I have never heard that partitions can cause problems, but I don't have experience with IDE RAID arrays. So if you want to keep other stuff on the RAID disks, you could create a separate partition for it. But make sure that the capture partition is the first one, as that part of the disk can sustain the highest transfer rates. And make sure that no processes are accessing data on the other partition. You should disable indexing on the capture disk.

The only reason I can think of that could cause RAID to perform worse during capture, is when the RAID controller chip is connected via the PCI bus. The onboard IDE controllers are in the I/O controller chip, which has a high bandwidth connection to the memory controller. So the RAID controller might have to share the limited PCI bandwith (max 133 MB/s) with the other PCI devices, including the capture card, and the integrated IDE controllers have their own connection. But even then, there should be enough bandwidth on the PCI bus for capture and writing the data to the harddisk in uncompressed form.

It seems strange to me that your system starts dropping frames when you increase the quality above 17. On my old PII 450 with an old 37GB 5400 RPM disk, I can capture full res PAL with quality 18 with very few dropped frames. Above that the disk probably can't handle the data stream anymore.

Capturing in MJPEG should not cause any difficulties for your system, but it is strange that it only happens when the quality setting is increased. Have you tried other codecs? Your disks should be able to sustain a much higher transfer rate than what is needed for MJPEG, so HuffYUV or even uncompressed should be possible. If any of these works, it is not a transfer rate issue, as they require much more space than MJPEG.

Eddy.

[Will Hay]

Thanks for the post Eddy.
I'll do some testing over the next few days and report back.
I'm in the process of backing up the 50gb or so of data I had on there and will then format, if only to see how that compares.
I've done so much moving, transferring and deleting on this drive it never occurred to me it would be best to leave this device as empty.
My problem is, as you may have gleamed from my earlier posts is finances prohibit another large drive for storage, and I need some space for exactly this purpose.
I think I'll try explore the option of the partition, but I ain't entirely sure how I ensure the capture partition is the first, is this simply a case of configuring that first?
Also, would I right in thinking I create the partition using the management software that came with the card?
I expect so
Assuming I manage to devote say a third of the drive capacity for capture, say 50gb for capture, would a quick format suffice before each capture?
I don't want a scenario where I have to wait for a half hour format each time I want to capture (which is going to be quite frequent).
I guess a format will suffice, no need for defrag etc. right?
Thanks again for the input, all you folks.
Will Hay

[KRSESQ]

I'm in the process of backing up the 50gb or so of data I had on there

You're gonna fill a lot of CDRs or even DVDs with all that. Time to invest in a removable drive bay and a couple of extra hard drives to swap out for storage!

Kevin

[Will Hay]

Originally posted by KRSESQ
You're gonna fill a lot of CDRs or even DVDs with all that. Time to invest in a removable drive bay and a couple of extra hard drives to swap out for storage!

I used five dvd-ram 4.3gb disks and put the rest on my primary hdd (temporarily), didn't take long
Will

[Eddy]

Will,

The standard tools for partitioning create the first partition at the beginning of the disk. But there are some that give you the option of specifying a location. Tools like partition magic can even move a partition with data on it (they do advice to create a backup first though). As far as I know, after you have created the array using the configuration tools of the controller, the OS will just treat it as a single disk and you should be able to use the standard disk management tools of the OS. But I have no experience with IDE RAID controllers, so please check for yourself. When you first installed it, at some point a single partition must have been created before you could format the drive. You probably can use the same method to delete the single partition and create two new ones.

When formatting, a full format is better the first time, to make sure there are no problems. The only difference with a quick format is that it checks all the sectors on the logical drive for errors. So when you just want to delete all files, a quick format is fine. If there are just a few files in the root directory, deleting all of them is probably just as good (as long as they don't end up in the recycle bin).

When you have multiple partitions, it is often not very efficient to move or copy files between the two. Because in that case, the disks have to constantly seek to switch between the partitions. So when encoding the captured video it is better to put the output on the separate disk instead of on the other RAID partition.

And ofcourse, don't put the OS, a swap file or applications on a partition on the array.

Eddy.

[Brian Ellis]

A quick format does the same as delete and empty the recycle bin in one operation. I don't think it will recover lost space. I don't think it's a good idea to partition a RAID array used for video capture but I agree that, if it has to be, make sure that you use the first one for video capture. The problem with partitioning is that each operation has to refer to a partition directory, as well as a file directory, before the heads are moved to where they have to go, taking an extra couple of millisecs or so.

The advantage of a full reformat before starting a project is that you know exactly that your working files are where you want them and cannot be fragmented (unless you delete one of them). When you start editing, then fragging will happen, but not to your captured files, as your editor master file expands. IMHO, it does make sense to defrag before rendering a new full project file without further compression from the capture files, so that this is in a single block. It is relatively unimportant doing this if you render to a more compressed format (e.g., MPEG), as the bitrate is much lower, in any case and a LITTLE fragging will not cause much harm.