Web Browser Wednesday
We’ve been asked for quite awhile to provide a performance comparison of the different mainstream browsers out there, and so today we are going to show you the stats from several different areas that users generally find to be the most important. Things like memory usage, page load time, and JavaScript performance will all be covered below.
One thing that you need to remember with these tests is that the results are all relative to each other. Each browser is running on the exact same machine so that the comparisons are accurate. What we heard the last time we did tests like this is “such and such browser performed a lot better/worse for me.” We appreciate hearing what your results are, but for the sake of accuracy they can’t really be compared to what we get.
Notes:
- All of these tests are performed on the same Windows Vista SP1 machine, and is wired into a network to minimize the effects of wireless disturbances.
- To test both IE7 and IE8 Beta I ran all of the tests in IE 7, installed IE 8, and then reran all the tests again.
- All browsers started with a clean profile and no add-ons/extensions installed.
- Caches were cleared before each test was run.
- Only one browser was open at a time and no other applications (other than standard Vista services) were running.
- Internet Explorer 8 was always used in the native rendering mode (a.k.a. standards compliant mode).
–JavaScript Tests–
We’ve previously run Apple’s SunSpider JavaScript tests, but there was apparently some controversy of using that because people felt that it could be a little biased. We wanted to pick a test that used tools you’ll find in sites you visit everyday. That’s why we went with the MooTools SlickSpeed test which checks the browser against different JavaScript libraries: Dojo 1.0.2, JQuery 1.2.3, MooTools 1.2B2, and Prototype 1.6.0.2.
Many sites use those libraries, and even we use JQuery for things such as the AJAX commenting. By putting the browsers back-to-back with the SlickSpeed test, we’ll be able to find out exactly which ones will give us the better JavaScript performance (this is what really matters on a day to day basis).
Pretty much none of the browsers were able to complete all of the tests error-free, and so we’re focusing purely on the speed. We ran each test three times, totaled the runtime (measured in milliseconds) for all four libraries, and then averaged the results. In the parenthesis you’ll see the results of each test we ran before averaging them together (the overall smaller number is better):
- Safari 3.1: 447.33ms (407,536,399)
- Opera 9.5.9841 Beta: 502.00ms (523,456,527)
- Firefox 3 Beta 4: 909.00ms (921,904,902)
- Opera 9.26: 1036.33ms (992,1034,1083)
- Firefox 2.0.0.12: 1507.67ms (1523,1472,1528)
- Internet Explorer 7: 5944.33ms (5965,5998,5870)
- Internet Explorer 8 Beta: 6690ms (6245,7206,6619)
It looks like the new Safari 3.1 takes the crown on this set of tests!
–Page Load Times–
I was trying to figure out what the best method would be to measure page load times in all the browsers. Sure some of the browsers report how long it takes for a site to load, but we wanted a universal way that would work across all of the browsers. We figured sitting here with a stop watch just wouldn’t cut it.
After some searching around I came across the Numion Stopwatch, which is a great tool for measuring how long it takes for a site to load. It’s entirely encased in a website so that there is nothing to install, and it uses JavaScript to notify you exactly how long it takes a page to load. From what I can tell it does a rather superb job!
We ran the page load test three times on two different sites so that we could really see what the results were like. We used the Official Google Blog and the Yahoo Search Blog for our benchmarks, and there are very good reasons that we chose those sites. Both of those serve up nearly the exact same content every time you load the site. If I chose a site such as ours we would run into the issue of different ads being served in the different browsers.
For each test the browser started with a cleared cache, and the three results were averaged together to get a single overall load time (measured in seconds). In the parenthesis you’ll see the results of each test we ran before averaging them together (the overall smaller number is better):
Note: I literally went and deleted each browser’s cache after each refresh just to remove any concern that a Control/Shift refresh was not deleting the site’s cache correctly.
The Google Blog:
- Opera 9.5.9841 Beta: 2.498s (2.129,2.606,2.760)
- Safari 3.1: 2.798s (2.619,2.963,2.811)
- Firefox 3 Beta 4: 3.009s (3.167,3.347,2.513)
- Opera 9.26: 3.360s (3.606,3.215,3.260)
- Internet Explorer 7: 4.235s (4.402,3.800,4.504)
- Firefox 2.0.0.12: 4.485s (4.852,4.258,4.346)
- Internet Explorer 8 Beta: 4.602s (4.409,4.238,5.158)
The Yahoo Search Blog:
- Safari 3.1: 1.411s (1.547,1.312,1.375)
- Opera 9.5.9841 Beta: 1.599s (1.578,1.625,1.593)
- Opera 9.26: 1.677s (1.547,1.625,1.860)
- Firefox 2.0.0.12: 1.771s (1.797,1.844,1.672)
- Firefox 3 Beta 4: 2.055s (2.430,2.143,1.591)
- Internet Explorer 7: 2.594s (2.563,2.219,3.000)
- Internet Explorer 8 Beta: 3.365s (2.875,3.750,3.470)
It looks like Safari 3.1 and Opera 9.5 both do really well in these tests, and I would consider it a tie between the two.
–Memory Usage–
This is probably one of the areas that interests most of you. Memory usage has become a big concern these days as we’ve seen some browsers (*cough* Firefox *cough*) use up insane amounts of our computer’s resources. So we took each of the browsers seen in the previous tests, developed a list of sites to open in each, and went at it.
We’re not going to list out all of the sites that we decided to visit, but they are all sites that are extremely popular. Places like MySpace, YouTube, CNN, and others were all included as we chugged through our four different memory usage readings:
- Started the browser, and took a memory usage reading.
- Loaded 10 predetermined sites in tabs, and took a memory usage reading after all the sites finished loading.
- Loaded 15 more predetermined sites in tabs (totaling 25 sites), and took a memory usage reading after all the sites finished loading.
- Let the browser sit for 10 minutes with the 25 tabs open, and then took a memory usage reading.
And now for the results! The table below lists the different browsers and the result from each test mentioned above. The best browser from each test is highlighted in green, and the worst is highlighted in red.
| Startup | 10 Sites | 25 Sites | 25 Sites After 10 Minutes | |
| Firefox 2.0.0.12 | 14.9MB | 110.8MB | 151.6MB | 172.8MB |
| Firefox 3 Beta 4 | 21.3MB | 68.9MB | 118.2MB | 124.7MB |
| Opera 9.26 | 12.6MB | 71.9MB | 127.6MB | 133.1MB |
| Opera 9.5.9841 Beta | 15.8MB | 98.3MB | 184.4MB | 186.5MB |
| Internet Explorer 7 | 6.3MB | 134.1MB | 248.3MB | 249.7MB |
| Internet Explorer 8 Beta | 5.6MB | 141.6MB | 244.2MB | 248.7MB |
| Safari 3.1 | 25.2MB | 97.1MB | 191.6MB | 210.4MB |
For this round it’s clear that Firefox 3 Beta 4 walks away as the clear winner. Mozilla has obviously put some work into making Firefox 3 a more memory efficient browser than it previously was, and this is proof of that.
I can’t say that I was surprised that some version of Internet Explorer almost always did the worst, but I was quite taken back that Internet Explorer 8 shows little improvement over version 7. Apparently that is not the focus of Microsoft right now.
–Overview–
It took us about 6 hours to compile all of the results that you see above, and I would say that doing this on your own is not really for the faint of heart. Things like not being able to run IE7 and IE8 side-by-side is really a time killer, but we wanted to be sure to include both versions in our results. And surprisingly the only browser that crashed on us was Internet Explorer 8 Beta when trying to open the 25 tabs.
We’d love to hear any comments you may have regarding our results, but remember that what you’ll see on your computer will likely differ from what we see. The important thing to take out of the stats is how each of them relatively rank up against each other.
This is just one of the hundreds of CyberNotes we have done. You can find more of them by visiting our CyberNotes category, or by subscribing to our CyberNotes feed. We also have a full feed available if you want to receive all of our articles in your reader!
Tags: CyberNotes, Firefox, Freeware, Software, Beta, Browser, Comparison, Firefox 3, Internet Explorer, Opera, Performance, Safari
Related Posts:
- Your Week in Review with Sunday’s Summary
- Our 5,000 Published Posts Milestone
- Firefox 3 Nearing Completion
- CyberNotes: Our Favorite Wednesday CyberNotes Articles
- Need A Laugh? PS3 & Wii Feature Comparison
Thanks, Ryan. A very well rendered comparison. Precise, objective. A bunch of guys are going to appreciate this work, me included.
Very nice article. Thank you, Ryan, for your efforts!
Nice stats. I appreciate CyberNotes when you’re working this way.
Opera has from far always been the best browser imho, and all in all, this is confirmed here.
that must have took a lot of your time. thanks
Ryan:
Thank you very much for a such a professional job. This is precisely why I read your site everyday, several times a day. Your dedication to your readers is way above average.
Congratulations!
Omar.-
awesome job,greatly appreciated!
Ditto that
Just forgot to say thanks for the time you spent compiling all these tests datas. Cybernotes are raising up again in quality since some days. Please keep on working this high quality articles.
What can we say but thanks?
Indeed
Thanks for this comparison. The more the better….it should be interested to see how IE8 final performs. And in a more practical scenario, Firefox 2 and 3 with the most common and popular extensions.
For improving startup time, I use Firefox Preloader ([sourceforge.net]) due to which it starts almost instantaneously.
P.S. Except for the startup time, I’ve always found even FF2 to be quite fast, stable and consumer moderate amount of memory. I find IE7 to be the worst. And I’ve several FF extensions installed (Auto Shutdown, Bookmark Previews, Calculator, Copy All URLs, Customize Google, Dictonary Search, Download Helper, DragToScroll, Favicon Picker 2, Fetch Text URL, FFClickOnce, Firebug, gDocsBar, Google Gears, Greasemonkey, MediaPlayer Connectivity, MIME Edit, Mozilla Archive Format, Navigational Sounds, Open in RegEdit, PDF Download, Piclens, Pop Alt Attribute, Remember the Milk for Gmail, ReminderFox, Searchbar Autosizer, Snap Links, Speak it, Tab Control, Tab Effect, Tab Scope AND ToolBar Buttons).
corrections* interested=interesting, consumer=consuming
Its is good to see visible improvements in Firefox 3 over FF2. I couldnt care about IE as i never use it, but Safari is clearly a worthy competitor.
Very nice, Safari and Opera are doing well.
Wow, those IE8 figures sure are nasty, I hope it’s because of its beta state, if not, I think Microsoft is doing something really wrong. Also, it strikes me the higher memory usage of Opera 9.5 compared to 9.2, again, I hope it’s because of the beta phase.
Also, what program did you use to measure memory usage? The Windows’ task manager, or something like Process Explorer?
Aside that, a nice comparison, much appreciated
Glad you guys appreciated the stats. We’ve been asked to compile results like this for months, but we wanted to wait until a lot of the browsers were on the same playing field. And by that I mean have a released version of the browser in addition to a Beta version. Once IE8 was released we thought it would be the perfect opportunity to do this.
A little over 6 hours.
Any chance of WinXP stats, since most people are still on this OS?
Numion results are VERY inaccurate. especially with flash, js, dynamic content.
FX3b4 does win one or 2 out of 4 library tests.Its quite unfair to mix 4 libraries together since they are not used with same frequency.
Finally, I don’t think slickspeed is particularly better than sunspider, slickspeed are testing more about theoretical stuff, sunspider does more about day to day practical usage.
I don’t think we’re going to rerun all of these tests on XP because for the most part the browsers are the same for XP. It wouldn’t really be worth spending another 6 hours testing the browsers when the results would still be pretty similar.
Since we had already done the Sunspider tests we didn’t want to do them again, while it may seem unfair to mix the test results from the libraries I believe that it makes sense. Many Web 2.0 sites use a mixture of those libraries, but regardless it would be difficult to put together a set of test results that everyone is content with.
Did you know it’s possible to have IE7 in stand alone mode? If not check out
[weblogs.asp.net]
Maybe this will with some of the testing so you don’t have to go back and forth so much.
well, fair enough
PS, I just did the test with safari 3.1 and firefox beta5 RC1, it comes out 715ms vs 834ms. and each browser varies alot when test multiple times.
This is a really comprehensive test and one that I’ve been looking around for but could never find a good one.
Thanks for putting the effort into this little project – I’m definitely going to take on board what these results say – IE8 is getting uninstalled (am sick of it crashing, even with less than 5 tabs open) and am going to take a proper look at Firefox 3 Beta, it’s looking good from those memory stats – I typically end up with around 100 tabs open at times!
Awesome article. Thanks for taking the time to write it. Could you update the results when Firefox 3, and the next version of Opera are released?
Thanx Ryan, not really too surprised with FF3b4 doing well, been noticing it here on my XP
Thanks for putting the time in to do this work. It confirms that Opera is the best browwser for me, although the FF results would probably be dinged pretty good if all of the popular extensions were added on.
Huh, didn’t think about that. But I wonder whether that would affect the performance? That’s one reason why I didn’t want to run portable versions of any of the browsers as well.
They do vary quite a bit, and that’s why I ran each browser three times and averaged the results. Generally, however, each of the results were rather close to each other.
Glad it helped you! I normally have a significant number of tabs open as well, and that’s why I really wanted to test out the memory usage.
I’ll definitely have to think about that, but I’m sure this is an article that I’ll keep updating. I’ll create a new post on the site when I do decide to update it.
I’m curious why you thought the Sunspider benchmark is biased. I’ve seen stats from Apple, Mozilla, and Microsoft using it for javascript performance. (Apple developed it, Mozilla has done a ton of blogs on their speed on it, Microsoft used it at their presentation at Mix, dunno about Opera).
I actually think that the Slickspeed is more biased because it has more variables than just pure javascript speed – like whether certain selectors are implemented natively in the browser or have to be run through the library (ie. nth selectors). Really I think that it has more to do with comparing the speeds of the library rather than browser.
Although I certainly do agree with you that there is a surprising lack of tools to measure pageload. I mean, that’s about the most important performance there is, right?
Impressive and very useful article! Thanks a lot
Do you know for a fact they will be similar, or is that an assumption? If it’s an assumption, then I know a saying about that
Also, you write “Pretty much none of the browsers were able to complete all of the tests error-free, and so we’re focusing purely on the speed”. Isn’t this also known as “If you don’t care what the result is, i can do it really fast”? Could you somehow indicate how accurate each browser was (how many errors were thrown)?
Anyway good job on this. Always nice to read a reasonably well executed comparison article. But as a final point I must berate you for using 3D graphs. They don’t look pretty and they make it harder to read and compare values. The only people who use such 3D graphs are marketing people and 1st year science students (and the students soon learn they’ll get laughed out the room if they do such a thing again
)
Actually I think it is much better to test on Vista rather than XP, if only for the memory tests. XP’s way of counting howmuch memory is being used pretty much sucks, and they are certainly not good enough for intensive testing like this. Vista’s way is actually IMO the best of any platform out there (though I don’t really know about Linux).
It’s not me, it’s some of our readers. The last time we used the Sunspider test we received several comments and emails stating that we shouldn’t have used a test developed by one of the development companies that create a browser.
Before commenting on that I did some quick tests on an XP machine that I have access to, and the results were proportionally the same. Meaning the ranks did really seem to differ.
Trying to find a JavaScript test that all of the browsers successfully complete is not an easy task. Even the Sunspider test has quite a few troubles with Opera.
The point of the graphs is not to try and sit there to interpret the values. If we would have not provided the detailed data with each test then we would have done a 2D graph, but all the graphs are there to represent is how large the differences are between some of the browsers. Again, it’s tough to please everyone on our comparisons which is one reason I think most sites refrain from doing them.
That’s true, and we’ve done an article on how Vista and XP report the memory usage differently.
Great article, thanks for taking the time to do this.
I have a question about your methodology. Did you do all tests of a given browser consecutively, or switch browsers after each trial?
For example, for the page-load time tests, did you do
Firefox 2 trial 1, IE8 trial 1, Safari trial 1, etc then
Firefox 2 trial 2, IE8 trial 2, Safari trial 2, etc then
Firefox 2 trial 3, etc
or did you do
Firefox 2 trials 1-3, IE8 trials 1-3, Safari trials 1-3, etc?
If you did it the second way, the trials could be skewed by longer-term fluctuations in your network speed, overall internet conditions, or test web site trafic over the six hour period the tests were performed. For instance: If some huge news event occurred between the Safari and IE7 runs, cnn.com and other news sites could be significantly slower for the latter. I know it would be more trouble to switch browsers after every trial, but it would be a more solid result.
The tests were all run consecutively for each of the browsers. Even if we didn’t run them consecutively and CNN got hit with a surge of traffic all of a sudden that would essentially render results prior to the “surge” inaccurate compared to the others.
The only way that you can really overcome this problem is to do the testing on a local network where you can control the fluctuations in traffic. But if we would have done that then people would have said that this isn’t a “real world” test.
Looks like FF3 is going to be a hit to me!
IE wasting 6 seconds to load some pages ? wow!
I have IE6 and XP and the blog pages loads in 1-2 seconds, i will not use IE7 or IE8..
6 secs, oh my god…
What about updating these results to today?
Guess they’ve changed.
We do plan on doing a follow up using the recent releases, but we would like to wait until IE7 Beta 2 is released in August.
@Ryan
Nice idea. Maybe you meant IE8 BetaX? Anyway good idea to compare all last updates/releases.
Impatient to see results.
I meant IE8 Beta 2. It’s supposed to be released in August, and I figure that will be one of the last released before it reaches the final form. So it’s a good point to do some more tests.
Definitivly, it is.
I’m impatient about these last results.
I’m sure there will be changes.
Hehe… this was nice to see Opera still setting the mark.
ROFL Wish more M$ pople would come read this, they might just maybe move to another browser. Hehe and then I don’t have to hack away at my webpages.
The Java thingy has various roots, but look at how it is implemented. Compression is the main culprit as Opera embeds the Java runtime the other browsers don’t thus stopping the expansion of the lib clientside. It just helps with security, everything stays in the browser.
Ryan, Thank you very much and apprciate you work. It is quite precise to me and … is it time to add chrome?
This article took a long time to put together, and unfortunately I don’t have the time right now to update it. Hopefully in the next few months I’ll be able to update it to include all of the latest browsers.
This may be a life saver, because I have been going mad with FireFox eating memory and utterly hate the Safari skin.
I will now be trying Leopard and Chrome ?
Please keep me updated with the latest version of this, and I suspect you would gain many followers on Twitter with all such performance comparisons.
Any chance you can come up with the ultimate one-stop shop PC performance diagnostic that would say “here’s where the problem is”, and” here’s what you need to do” ? It would save us poor non-techies running about (a) trying to find fixes and (b) turning into a techie for a day (or a week depending on skill set) to apply fixes piecemeal.
Unfortunately, I’ve had nothing but bad mem experiences w/FF (ver. 3 as of now)…usually taking up 550 mb of mem or more. Constantly crashing. I have maybe 2 extension installed. Sorry FF. Was a fan for a long time but I can’t take this shit anymore. Enough is enough.