Any opinions on RAID 0 being detrimental to capture?

[Will Hay]

Hi guys,
I'm gonna' do a little testing.
I've deleted everything from the drive and *just* captured a three minute clip with the MJPEG at quality 17, 18, 19 & 20 + huffyuv.
I'll then do a full format and do the same six tests.
Will post the results when done
Will

[Will Hay]

Assuming I create two partitions, A & B....
Thinking about it, my problem with a partition is this:
If I leave partition A for capture, on the basis that this will be a regulary occurrence I'll have to move the captures to partition B before encoding, or would it simply be easier to leave the captures on partition A and when I can encode do it to partition B?
My problem is I might capture more than one thing in an evening, and I don't have time to encode between them and then perform a quick format on the capture partition.
The more I think about it the more i feel I need a seperate drive for storage
Would it be fair to say another drive as slave to the RAID on the controller card would *not* be detrimental to capture?
You'll see from my system I don't have any on-board IDE's left!
Will

[Eddy]

Will,

You don't have to move captures to the other partition. When you do a capture on an empty disk, it just fills the disk starting from the beginning, the free space won't be fragmented. When you have two files, delete the first and then start capturing, eventually, the heads of the disks will have to move over the second file. So in that case it would be better to defrag before starting the capture.

Do you really need the high transfer rates of a stripe set? I think it could only be really necessary if you also do uncompressed captures. When you capture using MJPEG, the data rate can easily be handled by a single disk (even my old 5400 RPM disk can handle that). If not you could just configure the RAID controller to use them as 2 disks.

Adding a disk as slave should not have any impact on the stripe set of the other two disks when you don't access it during capture. Just make sure the 2 disks in the stripe set are not on the same IDE channel.

Eddy.

[Will Hay]

Hiya Eddy,
This is kinda my entire point of whether or not I actually need to configure my two drives as a RAID configuration
I needed the 160gb storage anyway so the small addition of a controller card wasn't a problem - I would have needed that anyway due to taking up all my onboard IDE contollers.
As you'll see from my earlier post a friend has suggested it *might* be detrimental (clearly he was right considering the way I'm using the device!) and it might be as easy simply to capture to a standard drive.
I would like the option to capture not just with mjpeg but also huffyuv. I appreciate this isn't uncompressed but seems to be a real drag for me - I've never got it right (aprt from 352 x 288 which isn't really suitable for dvd ouput!) so I'd like the opportunity to compare mjpeg to huffyuv.
Thanks for clearing up my 'you don't have to move captures to the other partition' problem. You're now in the same band as Brian who appreciates who appreciates how little I know!....but is kind enough not to reflect this in his replies!
I guess then that as long as I don't shift the captures around I'll be okay to continue to capture to the same drive/partition, and as long as I defrag (I imagine I'll perform a quick format each time rather than defrag) after I delete the captures/enocde, I'll be no worse off.
I've been playing around with the disk management tools in (WinXP) control panel, and have already formatted the drive.
I'm going to test on a clean full drive, and also a partition.
I played around with setting this up, I created a 100gb partition called V (*v*ideo, clever eh? ) and a 60gb partition called 'S' (*s*torage, wow! ) so imagine as I created 'V' first that would be the primary partition (and therefore the one I should capture to).
I'll do some three minutes captures to this partition, then wipe it and re-configure the drives to one big 'un (the 160GB-er).
Might be interesting to see if there's any difference, although I'm only capturing 3min clips, that should be enough to get a fair idea, right?
Will

[Will Hay]

Another concern I have with RAID is I don't think the Abit controller card I have is performing particulaly well. I've been getting intermittent errors and have been unable to access my data.
Normally a quick fiddle with the ide cables solves the problem (I can never see aything loose)
I'd hate to capture a tonne of stuff and then lose it.
I think, once I've tested the RAID, I'll remove the stripe aray and capture to a single drive, see what happens.
Will

[Eddy]

Brian,

What space might be lost? Have you ever seen that there is less free space when you delete all files including the ones in the recycle bin? It would seem strange to me, because all space used by files should be released. The only thing I can imagine is that extra space might have been allocated for the root or recycle bin directory when there have been a lot of files in it.

Using two (primary) partitions should not cause extra disk accesses when compared to a single parition. There is always a partition table which contains the size for the partition(s), even when there is only one. When you use an extended partition with logical drives, the system uses a linked list, which can cause extra lookups. But I think that the information about all partitions is cached by the OS. In DOS it certainly was, because you had to reboot after changing the partition table.

Recently I found some information that explains how NTFS works. There are 2 things that were quite different than what I expected and are useful to know when using an NTFS volume for capturing. The first is that the OS apparently maintains a pointer to the next available space (while running). That would mean that if you capture a file to an empty disk, delete it and capture again (without rebooting), the new file won't start at the start of the disk, but after the first file. That means that when the end of the partition is reached, the disk will have to seek to the start of the disk where there is still free space. Currently I don't have an empty NTFS drive to verify this. If this is the case, that could be a reason to format the drive instead of deleting all files, because formatting will probably reset the pointer. Defragmenting might reset the pointer as well.

The other thing is that it appears that when a disk is formatted, 12.5% of the free space as the start of the volume is reserved for administration purposes. When the disk fills up, some of the space is released and used by files. For capturing that means that the head will have to move to the start of the disk. This could explain what I have noticed myself, that after a certain amount of time, there are disk seems and some frames are dropped (when the disk is filling up). The advice was to leave 20% free space on NTFS drives for best performance.

When I can free a partition, I'll do some tests to check this out.

Eddy.

[Eddy]

Hi Will,

If a single harddisk can handle the captures in the format you want to use, and you don't need a single big volume, it would be better to use the disks individually. In that case, if one of them would fail, the other one would still be usable. And copying a file from one disk to the other is faster than copying a file that is on a RAID volume to a different directory on that same volume, because there is less seeking.

When you test something without using RAID on the Abit controller, you can check whether the problems are related to only one of the disks. In that case you can swap some things like cables, IDE channel on which the disk is etc to track down what the problem is. I'm not sure how big a 3 minute clip is in MJPEG. When testing it would be best to capture at least 1 GB of data, so you know it is not all cached in memory.

And we all know very little. Some people just know a bit more than others on certain subjects

Eddy.

[Brian Ellis]

Eddy

I haven't gone much into the theory of NTFS but what I've been saying is based on the practice. I have, as a video drive, 2 identical 60 Gb Maxtor drives configured as a RAID 0 array. This gives me 111.7 Gb of usable space, at the start, the rest (10 Gb estimated) being pinched by, as you say, admin purposes. I have 3.99 Gb (adjustable) reserved for the recycle bin. I assume the rest is used for "System Volume Information" which may actually not be fixed (I don't know).

If you go into the MS defragger, I always have a hefty lump of System files to the left (estimated at baout 10%) and a narrow one in the centre. I assume this is because of the RAID system. The files, as they're recorded, start off from the right of either green zone, so that the unused space is always at the end. I assume that each disk must have a master file table. I would guess that the files on the RH side start on disk 2 and those on the left on disk 1 (the majority) and they go alternately from there onwards. However, what is shown in blue (or red) is what is combined on the two disks, not what is individual.

The important observation is that I have found that if I delete all files on the disk (which is not partitioned) and empty the recycle bin when I've finished a project, instead of 111 Gb, I might have anything between about 100 Gb to 106 Gb free. I have not tried to analyse where the rest is nor even tried to defrag it. It may be in the system files. Nor do I know whether the Fast Indexing Service has any effect on the availability of space. As Microsoft claim that Quick Reformat only deletes, I assume it is the same as delete+empty bin in a single operation. What I do know is that I can recover the full 111 Gb after a full reformat.

Another problem with defragging is that the files are not put in the fastest part of the disk, as a solid block, but there often gaps in them, which will cause new files to fragment almost immediately. (I believe the Norton defragger is better in this respect.)

[Will Hay]

Not looking good
I've just formatted the drive, tried both PicVideo mjpeg (20) and Huffyuv and massive frame loss, the latter something like 25% lost.
I will do proper testing but think I'm wasting my time.
One thing, I was hoping I could hit pause break at bootup to ascertain what IRQ's had been asigned to which hardware - I have a real feeling my RAID is sharing with something else.
Any quick way to ascertain what IRQ is where from within Windows (XP)?
I've tried device manager but nothing's jumping out.
Thanks,
Will

[Will Hay]

Hi guys,
Sorry to flood the thread with constant cr*p but boy am I a happy lad!
I formatted the RAID and captured again, massive frame loss with huffyuv and mjpeg quality 20.
The huffyuv lost 26% of frames captured, not good, it now looks like a badly made 1950's Norwegian cartoon in slow motion!
I deleted the hardware from my system, removed the card and re-installed, hoping that the re-assigned IRQ would solve my problems - it didn't!
I then deleted the RAID and the set the drives up - independent of each other.
I've just captured three minutes of huffyuv and get this:
NOT ONE LOST FRAME!!!!!!!!
Am I dreaming?!??!
I'm gonna convert an old vhs movie of mine to dvd now (Hobson's Joice, John Mills, you know it Brian?) so I suppose a two hour movie is the ultimate test!
Thanks guys,
Will:

[Eddy]

Hi Will,

It sounds like your RAID controller has problems with one of the disks (or the connection to it). You can check by capturing to both drives and see whether one of them shows the same problem you were having before. When they are configured as a stripe set, the entire set will have the problem. So you can test capturing to both drives to see which one is the problem. Then you can try to swap the cables while keeping the disks on the same IDE channels. If the problem switches to the other disk, you know that it is the cable and you better replace it.

To see how NTFS works an what the consequences for capturing are, I've been doing some tests. The space that is sometimes "missing" when you delete all files when compared to a quick format is just permanently reserved for administrative purposes (the MFT to be exact). The space is reused later is probably continuous with the rest of the MFT. So it won't negatively affect performance. So it is not necessary to format in order to reclaim it. It will have to be allocated again when you create (a lot of) files. And the thing I read about the OS remembering where the next free block is, appears incorrect, at least voor Windows 2000 and XP. So it will start allocating at the start of the disk, and it is not necessary to reformat or reboot. But Quick Format does more than just delete all files. Otherwise it would not be possible to quick format from FAT to NTFS. It basically just rewrites all system files but does not check all sectors on the disk.

The allocated space in the middle of the drive is not related to RAID. It contains files needed by the filesystem and are placed there on purpose. As far as I know there is not much you can do about it. The idea is probably that the average time needed to seek there is minimal. On Windows XP, an NTFS filesystem is formatted a bit differently. The space that is allocated at the start of the disk on Windows 2000 is located more in the middle of the disk. This causes the free space to be more fragmented. Also, when the OS allocates space for a file, it starts in the middle of the disk. When it hits the other system stuff, it will continue allocating at the start of the disk. So for a capture drive, it would probably be better to format using Windows 2000 or NT 4.0 than with XP.

If you want to see some information about what is allocated by the filesystem, you can use one of the great utilities from Sysinternals: http://www.sysinternals.com/ntw2k/source/ntfsinfo.shtml

They also have a utility that can analyze fragmentation for individual files and defragment them.

Eddy.

[Brian Ellis]

Hobson's Choice

Yes, I saw it. I can't remember a thing about it, other than the title and an idea in the back of my mind that it was a comedy, but my guess is that it was between 50 and 55 years ago: am I right?

That was in the good old days of British film comedy à la Ealing Studios. The Alec Guinness ones stand out: The Man in The White Suit, The Lavender Hill Mob, The Ladykillers and, for me, the best comedy performance(s) ever by AG, Kind Hearts and Coronets. Other outstanding comedies of about that era include Genevieve (Kenneth More and Kay Kendall), Doctor in the House (Dirk Bogarde, James Robertson Justice) etc. An earlier one I saw, probably over 60 years ago, was the Ghost Train with Arthur Askey, an obnoxious little squirt.

[Will Hay]

Originally posted by Brian Ellis
Hobson's Choice
Yes, I saw it. I can't remember a thing about it, other than the title and an idea in the back of my mind that it was a comedy, but my guess is that it was between 50 and 55 years ago: am I right?

Yep, you are.
John Mills was a shoemaker in Lancashire, and set up his own business with the daughter of Hobson, aka Charles Laughton.
A great movie.
Will Hay.......32 years young, shouldn't I be watching death, kill, die, arnie movies?