Nvidia's Kal-El Demonstration Marred By Benchmark Confusion

When Nvidia announced its next-generation Tegra product at the Mobile World Conference, it pulled out all the stops in an effort to impress. The stats themselves were impressive—the chip packs a twelve-core GeForce GPU in with a quad-core ARM CPU—but NV opted to hammer home the point by showing benchmark results.

According to Nvidia, Kal-El turned in a score of 11,352 in the embedded processor benchmark Coremark while a T7200 Core 2 Duo (65nm, 2GHz dual-core, 4MB L2, 667MHz FSB) returned a score of just 10,136. If you want to see Nvidia's original video, you can do so here. We weren't completely sold on the results because, as we explained:
The program [Coremark], published by the Embedded Microprocessor Benchmark Consortium (EEMBC), is only designed to test the core functions of a processor. According to CoreMark.org: "It is encouraging to see the industry, as well as academia, adapting to a new standard so quickly, but let us not forget – CoreMark only targets core operations. EEMBC’s full-featured application benchmarks are much better suited for testing a processor’s capability in a real application. Furthermore, processors are becoming increasingly complex and one core-based benchmark is insufficient for a comprehensive analysis.
It turns out we were right. The credit for this discovery goes to ilsistemista.net, who caught the very different fine print under Kal-El vs. the T7200.

It turns out that the version of Coremark that Kal-El ran was heavily optimized and used a relatively new version of GCC, while the T7200 flavor was compiled using an old version of GCC with minimal optimization. The next step Ilsistemista took was to test what happens when Coremark is compiled and run on the T7200 using the same optimizations that were used for Kal-El. As you'll see, there's a bit of a difference.

When the scales are evenly balanced the T7200 turns in results nearly 50 percent faster than NV's published numbers.

What Was The Point?

The strangest thing about Nvidia's move is that there's no reason for it. Compare Kal-El to Tegra 2, and you'll note that Kal-El is no less than 94 percent faster than its predecessor. It wouldn't surprise us if Kal-El's graphics engine really is 5x faster than T2's, at least in some metrics. Certainly no one else is seriously talking about a mobile platform that can push 2560x1600 dropping by the end of the year. There's no reason to think Kal-El will be inherently limited to larger devices, either—NV should be able to fit the part into smaller power envelopes by disabling CPU/GPU cores. If a quad-core / 12-pipe configuration works well for a tablet, a dual-core / 6-pipe configuration should work just fine for a smartphone.

Second, there's no single test that's emerged on a consumer level as "the" benchmark for smartphones. Various browser-centric timed tests come closest, but browsers themselves are a significant confounding variable. An awful lot of testing boils down to whether or not Phone A "feels" faster than Phone B. Unfortunately, research has shown that human beings are pretty bad at actually judging such improvements and will often think that a newer / supposedly faster device actually is faster when timed tests prove it isn't.

Third, and perhaps most importantly, this sort of strategy inevitably raises questions about why NV felt it had to skew the numbers to make Kal-El look good. Kal-El, to our way of thinking, still looks pretty darn good on its own. If it delivers as promised, it'll be a huge leap from Tegra 2. As for its theoretical performance against a five year old Core 2 Duo that would never fit inside a tablet, we don't see a reason to care. 
coolice 3 years ago

This article essentially confirmed what what most of the members commented in last weeks article when nvidia first demoed this "next gen "kal-el" mobile processor"... goes to show that companies like to play us a fool, well at least the general public!

realneil 3 years ago

[quote user="coolice"]companies like to play us a fool[/quote]

And they never seem to learn that they usually get caught with their pants down when they try this kind of BS.

Case in point: Sony and their infamous root-kit embedded onto millions of their Music CD's.

The fact is that there's a whole lot of very smart, connected people out there that are looking hard at what these companies are doing. Getting away with 'slight-of-hand' shenanigans is not an option these days.

Here's a 1956 Volvo that looks almost new.

dodgers2213 3 years ago

Hit the nail on the head. So unnecessary to compare your latest and greatest by inflating numbers when you can easily run on the platform of "its way way better than our last product". C'mon  now green!

schmich 3 years ago

This is so pathetic on Nvidia's part. I don't mind too much descriptions that their products are the best but lying about performance? So sad.

AKwyn 3 years ago

Eh. This happens mostly every day. I don't blame them, they wanted to make their processor more powerful then it was.

While it is unnecessary and laughable that they would do this, they haven't lost me as a customer just yet. They have to do something much worse in order to lose me.

CDeeter 3 years ago

Wow they just lost a lot of respect in the tech community. Did they do this just to take a pot shot at Intel? Or were they trying to say that they can bring desktop class power to SOC devices? Either way they blew it.

SammyHayabuza 3 years ago

Post a Comment
or Register to comment