Cheating vs. Optimizations

[Helevitia]

I had a few thoughts and wanted to share them with you and see what you all thought. I am no good at debating but I like to hear your thoughts.

I'm sure most of you have read all of the ATI and Nvidia scandals as of late. My questions are:

1. If ATI's method of 'optimizing' involves moving around the order in which instructions get executed but the IQ stays the same, is that considered cheating? At first I was thinking, no way in hell that could be cheating. After thinking about it some more and reading some more, it appears ATI, is in fact not using the standard shader?(I don't know what it's called) that 3DMark requires for benchmarking purposes.

2. If ATI's method above is considered 'acceptable' optimizations, then are they using this 'optimization' in any other games or just 3DMark? Also, since this is a DX9 benchmark and there are no DX9 games, then I guess the question is useless until a DX9 game comes out.

Just thinking out loud.

Dave

[az]

"optimizing" for a benchmark is OK, as long as the benchmark is run the way it's supposed to be - this is AFAIK not the case with the ATI or nVidia drivers, so it's cheating. A benchmark should either be run under the same conditions on different hardware (3dmark is of this sort), or should be open for optimization to everybody (a la SPEC).

Drivers get optimized for popular games as well, this is not cheating (unless of course, a quack situation appears).

AZ

[Indiana]

If you're reaching the EXACT SAME result with a simple shuffle of a few shader instructions (this is what ATI is doing), this is generally considered legitime optimization.
But if you're analysing the camera angles and just don't draw anything that you know that it isn't visible (in the views the demo shows) together with forcing a lower rendering precision and other tricks (nVidia), this is clearly cheating.

Specifically for a benchmark I can definitely understand Futuremarks standpoint to dislike ANY optimization in order to be able to provide a defined and comparable workload to all gfx-cards. What ATI is doing still is like if one of the athletes at a triathlon said: Hey, I'm faster at this course when I'm swimming first, then biking and then running, so I'll do it in that order....

John Carmack and Tim Sweeney both have released statements (without naming any companies, of course), that partly support ATIs driver optimization because it delivers the exact same result, but condemn nVidias driver tricks.

To quote John Carmack:

"Rewriting shaders behind an application's back in a way that changes the output under non-controlled circumstances is absolutely, positively wrong and indefensible."

This is what nVidia is admittedly doing.

"Rewriting a shader so that it does exactly the same thing, but in a more efficient way, is generally acceptable compiler optimization, but there is a range of defensibility from completely generic instruction scheduling that helps almost everyone, to exact shader comparisons that only help one specific application. Full shader comparisons are morally grungy, but not deeply evil."

This is ATIs approach.

I don't have Tim Sweeneys statement here, but can look it up, if you're interested - but it basically says the same thing.

[Indiana]

Originally posted by az

Drivers get optimized for popular games as well, this is not cheating (unless of course, a quack situation appears).
AZ

BTW, nVidias drivers show those sometimes massive rendering errors and lower precision not only in 3DMark03, but also in the earlier 3DMark2001 and in most everything that uses DX9 / PixelShader2.0.

[TdB]

most developers will most likely view the driver/GPU as a "black box", and just code against the general API. This "black box" shouldn´t rely on application detection, it should view the application as a "black box" too, and just offer the supported api-calls.

I think ATIs optimization in this case should be considered cheating, because in this specific case ATIs optimization required application detection, and is not an improvement of the "black box" and is therefore a benchmark-cheat, since 3dmarks purpose is to benchmark the "black box" and not application specific driver tweaks.
for games application detection could be acceptable(but stil a crude hack) because in a game the interesting output will be how it looks, and if it looks right everybody is happy.
in this case, the way it looks isn´t the interesting output, the interesting output is: how long does it take the "black box" to execute the following shader in this specific order.
And ATI is invalidating that output.

when you use a benchmark, like 3dmark, you are not interested in how good it looks, you are interested in how fast it is.

besides I think the idea behind the benchmark, is to tell how fast a GPU is in a scenario, where there hasn´t been GPU-specific optimizations thrown in the mix(the black box scenario).

If ATIs optimization is indeed application independent, then it is optimizing the the way the gpu works in a scenario without GPU-specific optimizations. And i think that is okay, so optimizations that improves this "black box"(so the "black box" is cabable in reordering general shader-instructions on the fly) is fine be me.

[Kurt]

IMO you do whatever you want as long as the end result is:

1/ What the programmer wanted it to be

2/ Faster than the "black box" way


Means to an end. What ppl are obsessed with right now are the means and not the end....which is backward thinking...

If you can't tell the difference between the normal and the optimized way, then there's no reason to criticize the process.

If quality suffers then you know the method is wrong.