Nvidia is up to their old tricks

[TdB]

nVidia has done nothing, so far, in real games to do anything.

exactly, they spend resources in cheating, those resources could be spend on improving their drivers to benefit their costumers instead. If i had an nvidia card i would feel a bit cheated. A good service-oriented company should improve their drivers to benefit their costumers, and not to help marketing or PR. IMO

their costumers didn´t pay them money just so they could be victims in a benchmark-war.

Just imagine how great it would be if nvidia(and the others) didn´t care about benchmarks at all: we could probably have had the nv30 6 months earlier with a normal cooler, for example. sis would have made a card with good texture filtering as the default. maybe they could finally spend resources in figuring out how to get a sustained framerate.

[treilanin]

Heh.... okay bsdgeek, I see what you were trying to say. It just seemed like you were defending the cheating, when in fact you view it the same as us . Benchmarks are basically useless since they do not reflect anything remotely to real world performance.

[bsdgeek]

LOL, I reread my last post and it made no sense at all.

[DGhost]

Acctually... i really have to agree with Kyle over at HardOCP on this one...

just because it is having problems rendering certain scenes doesn't mean that it is an intentional problem.

3DMark game tests are running in perhaps the most ass backwards ways to create an excess of work to be done with no benefit. instead of writing code that is optimized and works well with driver optimizations (things such as clipping and culling). they intentionally force offscreen and hidden parts of the scene to be rendered, instead of letting the video card cull and clip them itself.

ie, they are trying to give a much better picture of what the card is capable of and not just what the drivers can make it do... however... this does not translate over into real world performance as it is very easy to come up with scenes that are more complex than the 3DMark 2003 ones and require so much less rendering work. nor does it translate well into good programming as pretty much everything is optimized for developers to take those short cuts (ie, you render from front to back and let the card handle all the object culling and clipping).


shit... it could very easily be a driver bug related to the way that Futuremark chose to render the scene... considering everything else that they have done up to this point i would not be suprised if they were doing something really far off the beaten path that resulted in this...

it would be far too easy for a driver to optimize out a part of the scene that is important when it is rendered in this manner. especially since it appears the application isn't really giving the video card any help with optimizing the scene...

plus, considering that ATI has been doing shit like this for a long time... people still insist that the Quake/Quack thing was a bug... if that was, why can't this?


and... considering the fact that 3DMark 2003 is written in so many horribly ass wrong ways and will not be representative of *any* games performance level, i think that anyone who uses this as a basis of their purchasing is an idiot.

[Technoid]

Originally posted by DGhost
Acctually... i really have to agree with Kyle over at HardOCP on this one...

just because it is having problems rendering certain scenes doesn't mean that it is an intentional problem.

a Bug that increeses perceptible performance???
ummm... yesssss

3DMark game tests are running in perhaps the most ass backwards ways to create an excess of work to be done with no benefit. instead of writing code that is optimized and works well with driver optimizations (things such as clipping and culling). they intentionally force offscreen and hidden parts of the scene to be rendered, instead of letting the video card cull and clip them itself.

Sure, and how often have'nt we seen ass backwards coded games?
Optimizing in games????
More like "alright, it don't limp on the p3@800 with a GF3, i think we are done with optimizing it"

ie, they are trying to give a much better picture of what the card is capable of and not just what the drivers can make it do... however... this does not translate over into real world performance as it is very easy to come up with scenes that are more complex than the 3DMark 2003 ones and require so much less rendering work. nor does it translate well into good programming as pretty much everything is optimized for developers to take those short cuts (ie, you render from front to back and let the card handle all the object culling and clipping).

Again, are you suggesting that they should have optimised their code to work good with certine cards?? or certine drivers???

shit... it could very easily be a driver bug related to the way that Futuremark chose to render the scene... considering everything else that they have done up to this point i would not be suprised if they were doing something really far off the beaten path that resulted in this...

it would be far too easy for a driver to optimize out a part of the scene that is important when it is rendered in this manner. especially since it appears the application isn't really giving the video card any help with optimizing the scene...

plus, considering that ATI has been doing shit like this for a long time... people still insist that the Quake/Quack thing was a bug... if that was, why can't this?

So if one cheats it means that everyone is allowed??
NO WAY!

and... considering the fact that 3DMark 2003 is written in so many horribly ass wrong ways and will not be representative of *any* games performance level, i think that anyone who uses this as a basis of their purchasing is an idiot.

sure sure
but many people who buys computer hw checks reviews, and reviewers often use 3dmark as a part in their benchmarking suits

[TdB]

and will not be representative of *any* games performance level,

it isn´t supposed to be representative of gaming performance at all, that was the main difference between 3dmark03 and their earlier products. It is true they coded that benchmark in a way no sane gamedeveloper would do at present time, they did that on purpose.

the point of 3dmark03 is not to tell which graphic-card is the best at current games, it is just supposed to bencmark the speed, and reveal the strenghts and weaknesses of the GPU. thus they choose to put an unrealistic strain on the gpu, to better illustrate the differences between the GPUs. to idea is that the strenghts and weaknesses 3dmark03 illustrates will show up in future games too(but maybe on a different level). we will prpobably know if 3dmark03 fullfilled that purpose, in a few years from now.

[DGhost]

not so much a bug that increases performance. say it is doing the intelligent culling and clipping like nvidia claims. the bug would be that it is clipping the scene incorrectly when the camera changes. depending on how nVidia is handling it, i could see a few ways that Futuremark could have written it that would make it incredably easy for a slightly overactive culling routine to misidentify what is being rendered and what isn't. it is all in how they chose to handle moving the camera and updating the scene. and since the game tests are designed to follow a static path (and most likely the ability to look around as it pleases) the camera handling code is probably not as integral to the render engine as say quake 3 or doom 3. it really just all depends.

the bug would not be that it increases performance. intelligent culling and clipping is not a bug. let me repeat that. intelligent culling and clipping is not a bug. that is an important part of any graphics card driver. the bug would be that it doesn't keep culling/clipping accurate to the screen when you move the camera.

if you don't think that intelligent culling isn't important, look at Morrowind. one of the reasons it performed so badly is that the water is *always* rendered. if you can cull or clip as much of the water off screen it will improve performance significantly.

honestly, ATI probably does the exact same thing in their drivers. the only difference is that their drivers don't show artifacts when you go off the set path...

yes, a lot of games have been coded backwards. although most are not nearly as backwards as Futuremark's rendering engine. whereas a lot of developers write games that are just inefficent and resource hogs, Futuremark is intentionally making scenes do a lot more work than required.

most games that are coded badly at least have a decent rendering engine. most simply are just inefficent... their models have more polys than they need to, they use multitexturing when they don't need it, they render (large) parts of the scene when it is offscreen, they get over zealous with their culling and wind up having huge visibility tests that slow it down, they pull a quake3 and render all objects and bullet impact marks at all the time, their engine uses a lot of processor time, etc etc...

most don't go out of their way to do the absolute most ass backwards things, though.

Not saying that cheating is alright. Just saying that ATI was caught in a far more compromising situation. looking at the situations, it is far easier to conclude that ATI was cheating (as you could get side-by-side comparisons both being done on the same video card and the same drivers - only difference was what it thought was running). yet, a large number of people both here and on other message boards (Rage3D, DriverHeaven) will go out of their way to convince others that it was a bug in the drivers, not an intentional thing. nothing more. and ATI certainly shouldn't be taking crap for it.

like i'm said earlier... i agree with Kyle on this one...

it's not so much that nVidia doesn't have a problem in their drivers... if it is intentional then yes, it is a huge problem... but... all we are seeing is a rendering issue and somehow infering that (without proof that it is specifically profiling for 3D Mark2003) nVidia is cheating...

i find it curious, especially after the Quake/Quack thing that they didn't try a similar test with 3DMark and see if the problems went away. or perhaps they did but the results were the same, so they forgot to mention it...

this very well could be a driver bug. like I said earlier, because of the nature of the way that Futuremark has chosen to do things, i could see a number of ways that such a bug could crop up very easily and very unintentionally.

unless someone can specifically prove that it is profiling 3DMark and because of that this is happening, you don't know if they are cheating or not.

I also have choice words for those reviewers who use 3DMark to base their conclusions about a video card on. it is designed horribly, way to easy to inflate scores, doesn't give any real indication of what level of performance a game will give, etc etc. it is a synthetic benchmark.

[DGhost]

Originally posted by TdB
it isn´t supposed to be representative of gaming performance at all, that was the main difference between 3dmark03 and their earlier products. It is true they coded that benchmark in a way no sane gamedeveloper would do at present time, they did that on purpose.

the point of 3dmark03 is not to tell which graphic-card is the best at current games, it is just supposed to bencmark the speed, and reveal the strenghts and weaknesses of the GPU. thus they choose to put an unrealistic strain on the gpu, to better illustrate the differences between the GPUs. to idea is that the strenghts and weaknesses 3dmark03 illustrates will show up in future games too(but maybe on a different level). we will prpobably know if 3dmark03 fullfilled that purpose, in a few years from now.

than, pray tell, how exactly do the game tests reveal the strengths and weaknesses of the GPU? and why should they be such a huge percentage of the overall score?

500 words or less.

[TdB]

hmm, I found this:

http://www.futuremark.com/companyinfo/Response_to_3DMark03_discussion.pdf

Use 3DMark03 for a comparable overall performance measurement of DirectX 9 compatible hardware

We continue to recommend to independent testers to complement their analysis by using published games in measuring performance. However, it has to be noted that those results are valid only for that game, whereas 3DMark03 can provide a forward-looking overall view of the performance and features of the hardware.

is a forward-looking tool that provides unique value to consumers in the form of impartial information to support their purchasing decisions today.

although it doesn´t directly address your question, it does say that they are aware that 3dmark03 can´t replace real game benchmarks, and that they are more focused in benchmarking hardware to get a generel impression of the hardware capabilities.

and for the gametests and what they each are supposed to test, is described here: http://www.futuremark.com/companyinf...whitepaper.pdf

I can´t find anything about why they only use the result from the 4 gametest.

[DedicatedFan3D]

It is very wise of HardOCP not to use 3DMark03.

Or any popular synthetic benchmark for that matter. I see that he makes a good point in that with these benchmarks, they can anticipate frame-by-frame everything in advance, and optimize the drivers to render those explicit frames in the fastest means possible. That means to render each triangle that is in place time after time the benchmark is run, to optimize for THAT PARTICULAR algorithm, and that particular position the bullets are at each possible microsecond, rather than try to make the engine render geometry or filter or sample faster.

I think that ATI cheats too. This entire discussion is not new, nVidia and ATI have been doing this since 3dMark 99. 3dfx did it too.

To be honest, I like nvnews.net's review of the 5900. What they did is turn on the timedemo in Unreal2 and play through the demo. This is how benchmarking SHOULD BE DONE.

By playing through a level, and going with the average fps acheived in that score, also publishing minimum framerate is critical. So far, who does this? No one.

Another idea, in Q3 benchmarks, play for 15 minutes in a level while recording, then benchmark the recording!!!! Publish the recording too so you can let people compare on their systems. That is the absolute of honest reviews. It also forces the GPU makers to optimize for game code rather than these pathetic synthetic benchmarks.

I think that there is no place for synthetic benchmarks at all whatsoever. No place at all, they do nothing for us anymore due to the inability to quantitatively judge the quality of a product due to the dishonesty in the industry.

[Kurt]

Originally posted by DedicatedFan3D
It is very wise of HardOCP not to use 3DMark03.

Or any popular synthetic benchmark for that matter. I see that he makes a good point in that with these benchmarks, they can anticipate frame-by-frame everything in advance, and optimize the drivers to render those explicit frames in the fastest means possible. That means to render each triangle that is in place time after time the benchmark is run, to optimize for THAT PARTICULAR algorithm, and that particular position the bullets are at each possible microsecond, rather than try to make the engine render geometry or filter or sample faster.

I think that ATI cheats too. This entire discussion is not new, nVidia and ATI have been doing this since 3dMark 99. 3dfx did it too.

To be honest, I like nvnews.net's review of the 5900. What they did is turn on the timedemo in Unreal2 and play through the demo. This is how benchmarking SHOULD BE DONE.

By playing through a level, and going with the average fps acheived in that score, also publishing minimum framerate is critical. So far, who does this? No one.

Another idea, in Q3 benchmarks, play for 15 minutes in a level while recording, then benchmark the recording!!!! Publish the recording too so you can let people compare on their systems. That is the absolute of honest reviews. It also forces the GPU makers to optimize for game code rather than these pathetic synthetic benchmarks.

I think that there is no place for synthetic benchmarks at all whatsoever. No place at all, they do nothing for us anymore due to the inability to quantitatively judge the quality of a product due to the dishonesty in the industry.

I agree 100%

But it doesn't stop at GPU. All other synthetic benchmarks (just check tomshardware for names...) are useless. The problem is that they make it too easy/convenient for testers. Reviews would take ages in everyone had to test a set of common apps, games, etc.

Instead they use, Sandra, Sysmark, 3dmark, etc.

[Indiana]

DGhost, are you sure, you understood the purpose of the massive use of pixel- and vertexshaders in 3DMark03? They did this deliberately to make the bench as gfx-card dependent as possible.
Now statements like yours or nVidias (in fact you sound like a nVidia sales-person) is like saying to a racing circuit:
Hey, that's probably "the most ass backwards ways" to drive from point a to point b. Look, after analyzing the track I found out that you don't need to drive around all the curves. I can drive a straight line and it's faster...

If nVidia would produce CPUs and someone would make a bench meant for measuring the computing-power, that did something like 2^10*2+2048, then nVidia probably would say: "What an ass backwards way to do this" and release a special driver that would instead do a
PRINT "4096"
just in order to be faster...

[DedicatedFan3D]

Which is kind of what ATI does in its drivers for 3dm03.

Synthetic benchmarks have ruined it for consumers. Because the only thing the cards are truly being optimized for is benchmarking....

[DGhost]

If you wanna stress the GPU, do it with on-scene geometry that cannot be culled or clipped. use dynamic camera and object paths/placement so that it makes it harder to hardcode optimizations. use more realistic render engines so that graphics card manufacturers cannot optimize for them so easily. make the amount of work done in each benchmark consistant across all hardware, ie don't have PS1.1, PS1.4 and PS2.0 render paths in one test. the difference between how much work is required when using PS1.1 and PS1.4 is phenominal.

in the track example, it doesn't make sense to make certain cars go to another town between ever other lap around the track. it doesn't make sense to make drivers drive the whole way in reverse either.

if it were GFX dependant, i would not have such a huge issue with it. but, they are doing so many things that make so little sense that it honestly becomes a matter of how good the drivers are, and how much you can optimize for the certain cases which arise in the benchmark...

its kinda the difference between using hand tuned ASM for a CPU benchmark and using Windows API calls. if you are using the API calls you can very easily wind up with different levels of performance just because of how optimized the OS is for the certain processor. and an update to the OS could change performance drastically.

oh well, synthetic benchmarks suck ass. they really, really do.

Synthetic benchmarks have ruined it for consumers. Because the only thing the cards are truly being optimized for is benchmarking....

bingo.

[DedicatedFan3D]

I wonder what would happen if this post were put in the futuremark forums?