Question for AVIsynth / VDub / TMPGENc Dual PIII CPU users

[droopy]

G'day yah all.


I've recently been very fortunate to get a very good deal on an old Tyan Thunderbolt with dual PIII-700 Xeon CPU's and 512Mb SDRAM. All looks great and is a HUGE improvement on the old PII-450. I've noticed something though for which I can't find an explanation and was wondering if some of you had seen similar and might even know a cause or solution. I use the following setup.

W2k + SP2 + Marvel G400 TV + PD 5.35eTV (Michael Carleers inf file trick) + VT 2.02 (To keep hardware MJPEG support). Added to that I use AVISynth 1.05 + HuffYUV 2.11 + Vdub 1.4.7 + TMPGEnc 2.01 with VFAPI. Boot HD is a 18Gb SCSI2 LVD disk and capture HD is a 40Gb EIDE HD.

I use next process to create MPEG2 files for SVCD.

1: Put captured HuffYUV in AVS script combined with 8 seconds black video file on the end. This last is done as I found that my DVD player doesn't show about last 5 to 6 seconds of a file)
2: If needed resize / add borders in the AVS script to get 16 to 9 or whichever needed.
3: Load AVS script in Vdub 1.4.7 and if needed change audio delay and add filters when appropriate.
4: Start Vdub's frameserver and open in TMPGEnc 2.01
5: Load SVCD script for TMPGEnc and change bitrates and other items needed. Additionally I activate all options under CPU in environmental settings including dual CPU usage and all "pipeline" options.

When I start this process all looks well when looking in the performance monitor of taskmanager. Both CPU's usage varies on average between 75% to 100% with occassional drops to 50%. Somewhere along the line the CPU usage drops to an incredibly low usage of about 0% to 15% on average. The HD's used in my system are working like mad which gives me the idea that a continues dataswap is going on with the pagefile?????????? When looking at readings in Physical Memory part they have dropped to an average of only about 2 to 10Mb?????????? Memory usage according to the taskmanager is never higher then about 275Mb though??????

Now for the curious part. If I leave out the AVS scripts from AVISynth and directly load the capture in Vdub, all works great and the file is decoded with no problems at all. At least the 2 45 minutes tests I did using this setup. Additionally if I use above process but disable the dual CPU usage all works great as well, again for the 2 45 minute tests I did. Using 1 CPU only though in a dual CPU board is like taking a Ferrari and go look for traffic jams to drive in ;-)

Since this is my first experience with dual CPU's and W2k, does anybody have any idea where this comes from???????????? Is AVISynth maybe very sloppy with memory?????? Am I doing something terribly stupid causing this?????? Is this a known problem?????? I've searched around the forum but couldn't find anything at all or similar on this.

Thanks in advance for any assistance to this cry for help / information.

Best regards, Leon

[droopy]

Just moving it up again!

Well, I'm just moving this up again in the hope that there is someone out there that might have an idea or can give me a hint. I've been fighting this problem now for about 3 weeks with all kinds of setting combinations but so far nothing.

So if anyone might have a clue. Please share it with me?

Thanks in advance, Leon

[droopy]

Here goes nothing in a last attempt

We'll, I can't believe that there is no one out there who at least has some idea why this problem occurs.

I've found a decent memory usage monitor in the mean time, much better then task manager for checking. I've narrowed it down now so far to be AVISynth for sure. It is EXTREMELY memory hungry appearently. It looks like it NEVER releases memory it's used properly. I've also found that it doesn't matter if I use the dual or single CPU option of TMPGenc in the environment settings window. The only difference it makes is that the problem occurs later. But it still occurs.

1:
If I make a simple AVS script where I combine a video capture and a sound file using the "Audiodub" command + a simple 704*396 resize. and load it in VirtualDub after which it's frameserved to TMPGEnc, the problem occurs.

2:
If I load that same AVS script in TMPGEnc directly, the problem occurs.

3:
If I separately load the video files and sound file in VirtualDub, not using the AVISynth, all is well. Memory usage does never go above above 124Mb and stays stable. While with AVISynth in time it went up to my maximum of 512Mb after which the pagefile swapping started.

I know you're gonna say now, why not stop using AVISynth alltogether. AVISynth though is extremely powerfull and has a lot of easy to obtain results which VirtualDub, TMPGEnc or any other editor DO NOT have. A simple thing is adding different borders on each side of a clip!!! A fadeout at the end of a video capture before an additional 8 second black clip is added!!! (For the people whose DVD players don't show the last few seconds of a (S)VCD.)

I would guess that 512Mb of memory should do for a job like this??? I've never experienced this kind of problems with me old PII-450. The only thing is that it took that machine 18 hours what this one should be able to do in about 3. At least when looking at some comparisons I've done with 20 minute sample files to be converted.

So please guys, if anyone knows a hint tip or other, let me know?

Regards, Leon

[Rob100]

Dumb question, but have you tried another version of AviSynth? An older version?

What about if you encode with something other than TMPEG? For example what if you load the AVS script into VirtualDub and encode as DivX - do you get the same problem then?

This might not be related, but I recently had some problems with AviSynth since "upgrading" a Win2k installation to DirectX 8. I re-installed to prove the point, but everything was fine with DX7, but as soon as I put DX8 on I got problems (unable to read source file and other crazy errors). Win98 and DX8 was no problem and I have not seen the problem in XP (DX8.1) either.

I have a dual PIII450 system here which currently has no HD, but I could set it up and try and replicate the problem if you want. It doesn't have AGP so I can't install my Marvel on it, but otherwise I can make it similar. You'll have to give me a few days though...

Rob.

[droopy]

Hi Rob100

Thanks very much for the reply. You got some good ideas in it of which I hadn't thought. So far I'd only been tinkering with settings in TMPGEnc, VirtualDub, AVISynth ans W2K.

I had already tried an older version of AVISynth, 1.0b3, with the same effect. By the way, there's no dumb questions, only dumb answers.

I'm sure gonna try you're suggestions though on the DirectX part as I indeed have installed 8.1. I'll also try the DivX suggestion just to see if there's a difference. I don't think it will be it though as a direct frameserve from the files through VirtualDub doesn't show the problem. I think though the DirectX suggestion is the most promising as I've heard of more problems going round related to it. Not just the 8.1 version, but DirectX in general.

Give me about 3 to 4 days and I'll let you know my results. I have to make a whole new installation as I've only made Ghost images from setups where DirectX 8.1 was already installed.

Thanks again for the suggestions, as I was already starting to think of giving up the dual CPU setup.

Best regards, Leon

[Rob100]

Let me know how you get on.

What really pis**s me off is when software installs stuff on your system without asking. When the installation I refered to went wonky on me it was after I installed Premiere 6.0 which without asking dump DX8 on my system - argh!

Best of luck anyway, Rob.

[droopy]

Well, I tried it and I failed miserably again. It still occurs and makes no difference.

I used a fully clean w2k installation now. The only additional thing installed was the PD 5.35 driver as VirtualDub requires at least 256 colors. Same problems re-appeared after about 15 minutes of encoding.

I even made it simpler now as I only used the videostream in combination with the "SegmentedAVISource" command, so no sound.

I've now also sent this question to Edwin v Eggelen, the current developer of AVISYnth, Premiere server plugin and some others. Sending it to Ben Rudiak is of no use as he has apparently disappeared in thin air. I hope he's alright though.

Well, I hope I find the solution. It would be a real shame to throw this dual PIII out of the window so to speak.

And if there's anyone else out there that migth have ANY IDEA OR CLUE. Do contact me PLEASE.

Regards, Leon

PS:
That's why I never choose for a default installation and always select custom installation where possible. If you do that with Premiere 6, it will ask you if you wish to upgrade DirectX and even warn you that it is not reversable.

[Rob100]

Leon,

I'm gonna try and replicate something similar on my dual PIII 450 box.

What resolution are you capturing at? Not that I think it is that relevant, but wanna make the test as accurate as possible...

Rob.

[teo]

i used een abit vp6 2*800 512mb whit een marvel G400 whit w2k sp2 direct x 8.0 en 8.1 pd 5.039.19 vt 2.04.05 en i have no probleem whit capaturing en whit avi en premiere 6.1can i capaturing

[IM_Riktar]

Have you considered just frameserving right into TMPGEnc with AVIsynth? I am just wondering why you would put a middle man in where one isn't needed.

For myself, I take my captures that are done in AVI_IO and after editing things in MSP 6.0 for content, transitions, effects, yadda-yadda-yadda, I save the capture files individually (IE: capture.00.avi becomes captureEDIT.00.avi) and just frameserve right into TMPGEnc for SVCD, VCD, XVCD, or whatever. My thoughts on this are the minimizing of system resources for each task.

And even with dual CPU's you can still use all the help you can get when it comes to rendering video.

Unless of course I am missing something in all this since my head is a bit tender at this point.

[droopy]

Hi Teo and IM_Riktar,

Thanks for the replies. They're much appreciated.

@Teo,

I have no problems with capturing, editing with this combination. The problems occur when I try to frameserve an AVS file through Vdub to TMPGEnc. All of a sudden a huge amount of memory is being used which I can't explain where it comes from or what's causing it. So basically Premiere works great for me. Also capturing, for which I always use AVI_IO, is wonderfull. It's just the encoding of AVS scripts through Vdub to TMPGEnc causing the headaches.

@IM_Riktar,

If you mean by "frameserving directly in TMPGEnc", directly loading the AVS script in TMPGEnc? Yes I tried this with the same results. Using a "middleman" has different reasons. One of them is lazzyness, but the more important one is ease of use. Most of the time I record programs from TV and just edit out the commercials. So transistions and other stuff are not needed for me. I simply put the files in a basic AVS script and load that one in Vdub. In the AVS script I also put things like different boarders on each side and other things which Vdub or other editors can not do. Well, you probably read my first and second messages. It's all there. The lazy part comes from the fact that I could load the Vdub filters that I use in the AVS script as well and use the "Trim" command from AVISynth to edit the capture. As said though, a direct load of the AVS script in TMPGEnc causes the same problems.

@ALL

It's so weard you know. When I directly load the captured files in Vdub and then frameserve to TMPGEnc, memory usage is stable at about 124Mb. As soon as AVISynth is added somewhere in the chain it all goes horribly wrong. Even for example when recompressing the files from an AVS script through Vdub shows the problem. That's the part I'm trying to solve. It would definitely be worth solving I think as AVISynth is extremely powerfull and has an awfull lot to offer. Maybe you guys could try the methods from my first and second message on your dual CPU boards and see what taskmanager shows you on memory usage and pagefile swapping? You will probably see and experience the same problems as I've found. I would be very curious on the results??? For me the problems started occurring after about 15 minutes of encoding on average. You will see an immediate continues decrease in available memory though as soon as you start the process.

Script I used is extremely simple and following. (Leave out the text"Line x:, its only for clarity)

Line 1:Video = SegmentedAvisource("f:\Capture_1.avi").AssumeFPS(2 5)
Line 2:Audio = WavSource("F:\Capture_1.wav")
Line 3:AudioDub(Video, Audio).FadeOut(50) + Avisource("d:\Black_704x368_8sec+audio.avi").Assum eFPS(25)

The black file added is because my DVD player doesn't show the last +/-6 seconds of an SVCD.

Regards, Leon

[droopy]

Hi yah again Rob100,

Did you get through to the new year allright? I don't know if you're also from Europe, but we've just gotten ourselves a new currency here. Seems to be OK so far. They're making a big fuzz about it on TV though. Lot's of big bellied politicians talking blah blah blah blah blah. It's just a means of paying though.

Anyway, thanks for your continued offer to help.

I tried 2 different ones. First was a normal TV capture at 704*576. Second I took from a DivX which was 704*368 which I recompressed using HuffYUV. I know 704*368 is not an even multiple of 16. Using these files though straight from Vdub gives no problems at all. I also tried this one with a resize in the AVS script to 704*384, no luck though. All my captures or editing is done with HuffYUV files. It takes little more space but it's worth it and helps speed in MPEG2 encoding.

I tried with and without sound but got similar results. Above message shows the one of the scripts I've tried. The only differences in results between all trials was that it sometimes took a little longer for the problems to occur. The more complex the AVS script, the sooner it occured. The black file I used was made using a properly sized BMP file from Premiere 6 with export movie command using HuffYUV as codec. Most captures I make are not longer then 45 minutes as I try to keep to the SVCD standard for best compatibility among different DVD players.

If you need more I'll be happy to provide the info. I'm just hoping the problem is not the memory management of the TYAN board itself. Am using the latest BIOS though + all default save settings of it.

Thanks in advance for your help.

Best regards, Leon

[Rob100]

Hi Leon,

Will reply properly later tonight or tomorrow (Saturday) morning, but I now have my dual PIII 450 system up and running and am currently carrying out some tests.

Rob.

[Rob100]

Leon,

Firstly... Yep I survived the new year. I'm in the UK, so no Euro here. Must be confusing for you... I wonder for how long you will keep converting things back to your old currency in your head?

Okay, my tests are going strong and I'll report my findings later. One thing that I must question is where do you resize to 480x576 for SVCD - I assume in VirtualDub?

Rob.

[Rob100]

Leon,

Okay I've done quite a few tests and have seen something similar. It takes about an hour on my system, but initially the overall CPU useage is about 70-100% with occasional dips to 50%. The suddenly it drops to almost nothing and barely peeks above 8%. I didn't notice the HD going mad and my memory useage did not increase. I left it doing this and went out of the room, but when I cam back CPU useage had resumed to normal.

I was using a slight different version of TMPEG to you and VD 1.4.8, but otherwise all was the same.

TBH though I wasn't impressed (on my system) with the gains made from using both processors. It still seemed painfully slow compared to my Thunderbird.

As you have enough ram and a fast HDD, it may be worth running two instances of TMPEG (as single CPU of course) and setting the affinity of CPU 0 to one copy and CPU 1 to the other. When I tried this, both CPU's were *always*maxed out at 100%, so I feel that overall you'd see an improvement encoding two projects at the same time using two copies of TMPEG than you would encoding two projects after one another using multi-threading support in TMPEG. I could be wrong, but you never know...

If there's any thing else I can do to help, then just ask.

Rob.