Why browser speed benchmarks are meaningless

Not everyone has a PhD. Not everyone should be running experiments. In this article, you learn the harsh yet simple truth why browser speed benchmark results, mainly based on ideal Javascript tests and a few other useless ideas, are meaningless in real-life browser usage.

You will hate me, but that's how it is ...

http://www.dedoimedo.com/computers/browser-speed.html


Cheers,
Mrk

 

Congrats. In my opinion one of your best articles. I like the car analogies (rush hour).

BTW. Do you still really need Golf GTI? Normally aspirated petrol 1.6 just as fast in rush hour and if there are speed limits also on wide open highway.

 

Here he comes with another sensationalist headline, hawking his website.

If he has something to say, why not post it here at Wilders rather than pulling hits to his website?

If many other websites did this, Wilders eventually would become just another *List of Lists* rather than a forum for discussing security.

 

Great article, and a highly factual one at that I might add.

I don't see where he's benefiting financially off it, trying to sell a product or advertise, although there is a donation suggestion, but that's not forced upon anyone. The site offers lots of computer-related help articles and tips. Seems harmless to me.

 

Bellgamin, feel free to report ...

Ocky, I sure do ... I have a highway only to work

Cheers,
Mrk

 

Mrk,

I agree with wat0114 that your article is interesting and your site is quite good. That isn't my point.

And it has nothing to do with "reporting" -- there are no rules involved that I know of. It's simply a matter of questioning the principles involved. Namely -- if other sites began doing as you are doing, Wilders could eventually become a catalog of websites instead of a forum. I just wonder: how hard would it be for you to cut & paste your articles to Wilders instead of periodically calling attention to your site?

(By the way, I have 12 sites of my own & am quite familiar with hit counts, unique visitor counts, and so forth.)

 

It would be fairly difficult.

Second, Wilders traffic to dedoimedo accounts to 1,000 page views a month, whereas I have some 500,000+, so it's 0.2% overall, if that. Besides, how many forums members have tech sites with original content and frequent updates? Maybe their stuff is quite noteworthy? Where do you get your info? Where do you read your news?

And I don't see myself ruining Wilders with my posts and links in the last 4.5 years.

Mrk

 

If he posted articles here someone would complain of that. I don't feel there is anything wrong with discussing articles here. If it were not his own site there likely wouldn't be any complaining.

I liked the article by the way.

I get tired of all of these "version x.0 is 400% faster than the other browser you use" sensationalistic browser reviews. I use the browser I like based on the features provided. I don't care if something is a millisecond faster at running the javascript I have most likely blocked anyway.

 

I like a browser that opens fast and loads as fast as I think my connection should load it. Many can do it, sometimes tweaks help. Firefox has always felt slower in every aspect. Currently for me, Chrome feels the fastest, by about 25% on my human eye meter.

Sul.

 

Good info.
But personal experience talks
And i feel Firefox is slower than Chrome in some aspects (Don't even talk about IE
But IE9 is a whole different thing, very fast and very good.
But i still prefer Chrome layout and love it's constant development.

 

All browser behave same on my pc except ie8 ofcourse

 

Browser benchmarks are not meaningless. With all this hype about speed benchamrks every developer tries to be the best and the results benefit everyone. Even if they are meaningless like in your article at the end all the optimizations for those benchmarks benefit end users.
Without all this benchmark "hype" and speed race we would still use ultra slow browsers like years before. Chrome started al this hype and now we have a fast Opera fast IE9 and Chrome allways pushing to the limit. Thats good.

 

What's exactly a "slow" browser? Can you define that?
Mrk

 

A slow browser is a browser thats slow to launch, slow at loading heavy JavaScript pages, slow at loading multiple websites at once with Opera beeing the best at loading 50 or whatever webpages at once. Also browsers are trying to imitate the minimal design of chrome. And whys that? Its clear - simplicity wins. Less bloat=Faster using=More time browsing=Less distraction from browsing. Its funny u ask me whats a slow browser. You write an article about meaningless of speed benchmarks and ask me whats a slow browser. What about giving an opinion on my arguments i wrote before?

 

Slow to launch - under what circumstances?

What scheduler, what disk, what disk cache, what processor, what load, what windows manager, what theme, what background processes, any anti-virus software, do you understand how many parameters there are. Less bloat - visually - is subjective, so again, no baseline.

Your argument starts with the definition of slow - which is incorrect, hence everything else is irrelevant.

Mrk

 

I dont need to answer all that crap because its a discussion with no ending. Specification of one pc to other pc spec and this pc and that pc blabla. If u dont notice that browsers are constantly getting faster and improving on all aspects im sorry for u. Oh wait i dont really care. Btw i also wrote

but its clear u dont have arguments and try to nail it down to subjective speed talking. Im not going to enter that.

 

I find all browsers the same speed for loading pages. the differences is the features and layout of the browser.

Internet explorer 8 feels slow on all the computers ive tryed it on. its slow to load and new tabs take awhile to load.
I use opera as my web browser because it is as fast with one tab as it is with 30 or more. Opera has all the features I need without any crap. firefox felt to bare and internet explorer has to much crap. I also find chromium is to bare.
competition helps make all browsers better.

 

Nice articles

 

Benchmarks are a basic requirement of any scientific endeavour. To argue that they are meaningless is a bit extreme. To that extent, the title of this thread is as misleading as the extrapolation of benchmark results to "real life" situations, whatever they are.

 

I, unfortunately, am stuck with dial-up and I find the true test is when I frequent my usual haunts, Wilders being one of them, and my online banking. Lately I have been finding Internet Explorer faster then Firefox and Opera so I have been using IE a lot lately.
In the past I was using IE for updating windows only.

 

No. A proper test plan and a control are the basic of a scentific endeavor. In this regard, you don't have either. You have guesses and half-assed conclusions made by fanboys.

Mrk

 

Turn off HW acceleration in Firefox and everything becomes so so slow right? No it does not and so we can conclude that HW acceleration, asking GPU to do something, is hot air, a hoax! Context is what matters and what article skips to some degree since it assumes nothing really changes. If web content in few years and outside developer circles will be filled with html5 goodies or web apps (example http://creativefan.com/20-shockingly-cool-html5-canvas-applications/ ) of course every tweak and performance increase matters. If relevant today, to you and your browsing patterns is another matter. Also there are other machines that beefy desktops to run a browser on. Performance is king and numbers cant be overlooked.

To understand test value or importance of course you must be a developer so taking screen shots and go all 3Dmarks (they naturally have their own browser benchmark) is stupidity at full force. But any topic that is considered an oldie but goldie, ("which is best AV?" everyone knows that one) give hits and is accepted as infotainment. Especially in tech world you can almost predict what give headlines. Same old.

I think there are still 100s of bugs to kill in Firefox 4 (as of today) but I can easily tell it will be way faster and better than Firefox 3.6.x - do not need benchmarks. And actually I have turned off HW acceleration since it make last few days builds crash like mad! Makes some new "tests" look like absolute crap compared to IE9 though, oh no!!!

About speed. Do a clean test with no extensions. Then add for example Adblock. Do same test. Now numbers looks worse right? (they did 2 years ago when I tried). Add-ons which actively play a role in how browser works will not exactly improve performance. And yet Adblock will in fact often speed things up in the real world because there are less slow ad-servers to wait for, even rendering though that is almost theory. So much for benchmarks

 

Human's consider less than 1 second responsive, 2-3 seconds average, more than 3 secs is slow, more than 10 secs is lost interest. This usability law exists from 1983. Internet useability guru's like Steve Krug (for developers a must read "Don't make me think") says it still holds for internet today.

So a cold start of a browser should be within 3 secs, page navigation on same site within a sec, url's from other websites should display within 3 secs. Depending on the target market and infrastructure commenly available within a country or region of those target markets, you can answer this question yourself.

But your point is problably that this (target market and infrastructure) is context dependant. On the other hand this does not mean you can not scope applictions and websites on it (as stated when you develop a website/application you have a target market in mind, the target market has an average mean of infrastructure, this is your guidance).

For commercial developers: benchmarks do matter (as the distribution of infrastructure bandwith and pixel/screen size of the most commenly bought monitors/screen displays), only the part of the web site vistors still using IE6 are more of a head ache to them. So I would ask all wilders nerds, please contact all your friends and ask them to update to the latest browser version of their choice.

So Mrk you are partly right. There are other more context related thresholds for the web experience. Since your aim is to think and reflect on hypes I like your 'stingy' article

Regards Kees

 

I see it both ways (as in it's both meaningless and meaningful, depending on the context) even though I don't gobble myself up with browser benchmarks. At least, not yet...a simple read when it's reported or mentioned in blogs/forums I go to is all I do. It doesn't affect how I choose my browser.

Statistics and figures are meaningful for those who can analyze it and understand the details. That is what rolarocka is saying or pointing out as he knows what those details mean. Without comprehension of the material in hand, the data can be said to be 'meaningless' for that person. And that person applies to most browser users. All that matters to most end-users who aren't "geeks" or who don't frequent forums like these is subjective speed of the browser. And that differs for each individual, depending on other variables as stated by Mrkvonic.

Am I not right to look at it that way?