DX10 Benchmark Fiasco editorial

[Ryan]

http://www.pcper.com/article.php?aid=408

Hey guys, I think you'll be interested in seeing this new editorial I posted this afternoon.

Leave me your comments/thoughts/questions in here!

[zirik]

dont really care much about these 'exclusive tech demos'. everyone does it. and everyone cheats... err, i mean optimizes their drivers to boost performance. half life 2 was 'optimized' for ati cards. 3dmark 2006 was 'optimized' for ati. but they were not considered a fiasco.

[_dangtx_]

i praise u for going against the flow, once again

[Poci]

LOL doesnt this kinda stuff happen every time a new DX version is realeased?
I think I remember this happening with DX 8 and DX 9

[AMDAthlonguy]

It would seem AMD Graphics group made a small mistake("Juarez") unintentionally and nvidia took offense and intentionally made a mistake with "The Lost Planet".

[Ryan]

http://digg.com/hardware/DirectX_10_...d_of_the_gamer

DIGG too if you can!

[Fluff n Stuff]

I don't care if one manufacturer gets a demo 'optimized' for them over the competition... as long as there is transparency. It is the job of the reviewer to find out as much as possible about a piece of hardware or software when presenting their review. A review that just follows the instructions given by the marketing department is not a review, it is a puff-piece... at best. At worst the reviewer KNOWS about the inequity, but does not pass it along in the review. There are many names for sites who do that kind of review, in most cases the common name is 'forgotten'.

I admire Ryan for pointing all of this out, but it would be nice to have this discussion (or at least a link to it) as a caveat early in a review. Not everybody who finds a review on Google or Yahoo will be 'up to speed' on these issues, and if reviews are going to be useful they should include a bit of background with their benchmarks.

If the benchmarks are too messed up to give a good result, then they are not benchmarks.

Makes me wish that Van Smith's Comprehensive Open Source Benchmark Initiative had gotten more traction... best I can tell COSBI withered on the vine back in 2005 'nStuff.

[Ryan]

It would seem AMD Graphics group made a small mistake("Juarez") unintentionally and nvidia took offense and intentionally made a mistake with "The Lost Planet".

Let me first say that I don't think anyone made a mistake. AMD definitely knew about the issues in "Juarez" as they would not have recommended it to press without seeing results for the competing cards first.

As for NVIDIA, Lost Planet was in development for some time and I think they just rushed the demo release a bit with DX10 to make a show of it, yeah.

[Ryan]

I don't care if one manufacturer gets a demo 'optimized' for them over the competition... as long as there is transparency. It is the job of the reviewer to find out as much as possible about a piece of hardware or software when presenting their review. A review that just follows the instructions given by the marketing department is not a review, it is a puff-piece... at best. At worst the reviewer KNOWS about the inequity, but does not pass it along in the review. There are many names for sites who do that kind of review, in most cases the common name is 'forgotten'.

I admire Ryan for pointing all of this out, but it would be nice to have this discussion (or at least a link to it) as a caveat early in a review. Not everybody who finds a review on Google or Yahoo will be 'up to speed' on these issues, and if reviews are going to be useful they should include a bit of background with their benchmarks.

If the benchmarks are too messed up to give a good result, then they are not benchmarks.

Makes me wish that Van Smith's Comprehensive Open Source Benchmark Initiative had gotten more traction... best I can tell COSBI withered on the vine back in 2005 'nStuff.

Excellent points! This problem exists in a smaller degree on a per-game basis for all titles currently for sale -- some times driver fixes and patches can make one side perform better than the other. But in THAT case, we are reporting on what the actual use experience is at that time; we can't do any more than that really.

For reviewers like myself, knowing the intricate details of each software application is a DAUNTNG task, but one I try to get as much information on before including it in our testing suite. With GPUs, the "real world" aspect though says "hey, I buy the game off the shelf, plug it in and what do I get?"

[nub]

Excellent article. But this stuff is expected really.

[elvin]

Ryan that's why i love it when you just buy the game on store and test it out and give us results.Keeps it honest and lets us know why we should or not get it.DX10 in my view will be in a baby stage for a while and any game with dx10 i think will have lots of patches and drivers to get it working right that's a sure thing.

[molestedbyasus]

Feeling a load on the server this morning?

http://hardware.slashdot.org/article...7/05/23/201243