Safari and Firefox are the two largest browsers in the Mac market. They’ve also both recently released new beta versions. This is the first in a short series on the similarities and differences you can expect in the final versions of Firefox 3 and Safari 3.
It’s important to note, I went with the latest nightly builds of each browser (as of March 5th, 2008)–dubbed Webkit (r30790 ) and Minefield (3.0b5pre). The advantage to this is the browsers are on a level playing field–but each may have potentially buggy or incomplete features.
The Fastest Browser Wins
Safari is advertised as the fastest browser in the world, but FireFox 3 is rumored to be catching up quickly.
The best tool available for benchmarking Javascript is WebKit’s SunSpider. According to SunSpider, Safari 3 is 1.32x faster than Firefox 3.
I performed the next test on CSS to make sure the results were representative. Unfortunately this benchmark also included some Javascript so the tests weren’t completely independent (test available here, but tuned down to not break the server).
The CSS benchmark tests the CSS background-color attribute on 60,000 divs. I ran each test 5 times. The results show Safari renders CSS 4.5x faster than Firefox.
Still not satisfied these results were completely representative, I wanted to try another test. The SunSpider test was created by Apple, so there was a chance for a bias. I wanted to run the browsers through one more test that was independent–I decided on Quirksmode Javascript benchmark.
The Quirksmode test is a benchmark for generating large amounts of data using the following Javascript methods: DOM 1, DOM 2, Table, innerHTML 1 and innerHTML 2. I ran each test 10 times and aggregated the results. The winner? You guessed it, Safari was 1.6x faster than Firefox.
The Fastest Browser is: Safari
From these results it’s clear Safari is much quicker rendering Javascript and CSS. Benchmarking other parts of the browser (SVG, Sockets, etc…) is more difficult and outside the scope of this article. I tried to give Firefox as fair a chance as possible–but the results always came up in Safari’s favor.
(This image looked so ridiculous I decided to keep it.)
Round 1 Winner: Safari
Don’t fret Firefox, there’s a few more rounds coming–and I have a feeling you’ll make up for your speed with your many other features.
Stay tuned for the next couple rounds where we outline the features in Safari 3 and Firefox 3.
Other rounds in Safari 3 vs Firefox 3 series:
Loading ...
Michael Zhao
Travis Jeffery
Brad Jasper
Travis, Yes I read up on that. Developers are pretty upset.
The methods exist, they're just not documented. Firefox has already put many of these new API calls into the latest nightly builds.
As far as the ethics go? Apple probably should have disclosed them, but I personally don't think it's as big a deal as everyone is making it out to be. March 6th, 2008 at 10:19 am
cris
1. most sunspider test of most recent nightly shows firefox is 1.1xslower than webkit, not 1.3. I would also mention that in the past 30 days, webkit has no improvement, while firefox speed increase 200%. No to mention there are still more javascript optimization no out yet, including PGO, Im quite sure eventually firefox 3 will be at least no slower than safari 3.1.
2. as a speed test, Im surprised you didn't test reading from history at all? you do realize there are a good chunk of browsing is back/forward operation?
3. the CSS test was created 4 years ago, I think you should use more modern test if you can find one, since the CSS standard has gong through 3 version changes.
4. methods exist, just not documented? thats exactly M$ style, Im surprised anybody would brush aside this unfair tactic. March 6th, 2008 at 11:20 am
cris
Xel
cris
Brad Jasper
Let's address some of the things you talk about here:
This article is Mac only, so this is irrelevant. I would bet Firefox 3 is faster than Safari 3 on windows--but as we're a Mac site, we don't focus on this.
I ran the test 3 times last night and got the same results.
Safari: http://nightly.webkit.org/
Firefox: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/
I did say this in my opening paragraph. I stated Firefox has made huge leaps recently in performance:
While it's nice that Firefox has gotten "faster," it's still not as fast as Safari.
Speed has always been the #1 concern for Safari, outlined here: http://webkit.org/projects/performance/index.html
I'm glad you're confident the final version of Firefox 3 will be as fast as Safari 3, but that's pure speculation. Let's look at the results. Right now, Safari has the upper hand.
Caching is not my main concern with benchmarks. My main concern is the render time. Safari renders Javascript and CSS faster--it's entirely possible Firefox can render cached pages quicker, but there's no results to support that.
Absolutely. I could have tested more advanced CSS features, but I thought it would be fair to test one of the most basic CSS features that both browsers have supported for years. Background color is used by most Web sites so it's the best benchmark in my opinion. I could have used newer CSS3 specs, but how does that translate into real world performance?
I agree it wasn't a "nice" or "fair" thing to do. But Apple is a business and as a business must make money to make these gadgets we all love.
The biggest marketing edge for Safari is it's the fastest browser in the world. If Apple didn't do anything to protect that claim it would loose its biggest advantage.
Again, I agree with you. Open source should be open--and Apple has deemed Webkit to be open. They should have disclosed, they didn't. That being said, Apple has the right to protect its business in any way it can.
If you really want to get into Apple business tactics lets talk about iTunes lock-in. Unfortunately that is not the topic of this entry, so we'll have to debate that elsewhere.
Regardless, I appreciate your comments Cris. It lends to a better discussion on the browser wars. My goal here is to be fair. Firefox has many advantages over Safari, but right now--speed is not one of them. March 6th, 2008 at 12:11 pm
cris
LOL
ok, let me say something to make it right
1. please follow the threads at macrumors.com and mozillazine.org. there are specific topics about the javascript changing in firefox 3. I would guess you guys aren't used to the idea of firefox' nightly, it actually has significant changes every day! Javascript speed just bumped another 5% two days ago.. PGO promises another 10~20%.
2. browser speed is a complicated issue, both javascript and CSS are tiny parts of the web, all these tests have to repeat same script 100000+ times in order to measure the statitical ignificant differences, there is very little real world impact for users.
3. on the other hand, page loading, DNS resolving, history reading, etc. have much more profound impact on users' day to day usage, and you guys pretty much tested nothing on these issues.
4. I would advise you wait for the final test, and redo the speed test when both firefox 3 and safari 3.1 out. So many changes, your test data are already 2 days too old. March 6th, 2008 at 12:16 pm
cris
macrumors thread http://forums.macrumors.com/showthread.php?t=426038
mozillazine thread http://forums.mozillazine.org/viewtopic.php?t=630259 March 6th, 2008 at 12:21 pm
Brad Jasper
But unfortunately we must choose a version and stick with it. I opted for the latest nightly versions of each release (because of the recent speed increases in Firefox).
I'm glad we agree browser speed is a complicated issue :).
And I agree Javascript and CSS are a small part of that.
We're talking milliseconds here, which for the most part users won't recognize.
Page render speed is loosely related with the tests above, but not completely. Render time is often what makes a page "feel" fast, so it was the focus of these benchmarks.
Caching may have been another good test to run--which I will revisit when we test the final versions.
DNS resolving is not a browser issue, it's a socket layer issue and fundamentally an operating system issue.
I considered all of the different benchmarks I could possibly use, and as I stated--most are outside the scope of the article.
The reason I added the Quirksmode benchmark is because I realized I needed something beyond pure Javascript and CSS.
While the Quirksmode involves Javascript, it's more about processing large amounts of data.
If you have any other methods of benchmarking Cris, I'd love to hear them. March 6th, 2008 at 1:27 pm
Brad Jasper
They're testing the newest version of Firefox with an older (but still recent) version of Safari.
We're not comparing apples to apples in this scenario.
In order for the tests to be relevant, you should test either:
Latest Safari Beta vs Latest Firefox Beta
OR
Latest Safari Nightly Build vs Latest Firefox Nightly Build
Mis-matching them is simply inaccurate. In this case I could test the latest Safari build with the most recent Firefox beta and claim Safari was 2x faster than Firefox.
Those results would be inaccurate and misleading.
I realize you're not making that claim, but the initial posts in both of those links are. March 6th, 2008 at 1:34 pm
Trackback
[...] el mundo OS X, Safari y Firefox dominan. MacTips comienza con la nueva versión beta de los dos navegadores, y testean su velocidad en varias [...]
March 6th, 2008 at 5:35 pmOmer Zach
cris
Did you NOT read the post by the macrumors moderater's post? he was using latest webkit nightly and firefox nightly. Maybe the OP's posts are based on safari 3.0.4. But moderator's posts are NOT. and His data shows 1.10X faster
AND. may I suggest you download a new firefox nightly today? since there are new improvements overnight, and my test shows latest webkit (30868) is only 1.06X faster now. 4800ms vs. 4500ms March 7th, 2008 at 9:59 am
cris
firefox 20080307 http://webkit.org/perf/sunspider-0.9/sunspider-results.html?{%223d-cube%22:[244,242,237,244,244],%223d-morph%22:[187,182,184,180,185],%223d-raytrace%22:[185,191,185,191,189],%22access-binary-trees%22:[72,69,72,67,72],%22access-fannkuch%22:[328,327,327,327,326],%22access-nbody%22:[255,250,276,254,252],%22access-nsieve%22:[97,98,99,98,99],%22bitops-3bit-bits-in-byte%22:[78,79,77,76,78],%22bitops-bits-in-byte%22:[119,120,121,118,121],%22bitops-bitwise-and%22:[122,125,126,122,124],%22bitops-nsieve-bits%22:[176,176,175,174,175],%22controlflow-recursive%22:[52,54,52,52,54],%22crypto-aes%22:[113,111,115,112,113],%22crypto-md5%22:[87,87,84,87,86],%22crypto-sha1%22:[86,86,93,89,91],%22date-format-tofte%22:[339,340,333,336,338],%22date-format-xparb%22:[223,225,220,225,225],%22math-cordic%22:[225,221,227,226,223],%22math-partial-sums%22:[195,210,199,197,203],%22math-spectral-norm%22:[112,112,112,111,112],%22regexp-dna%22:[397,392,393,400,393],%22string-base64%22:[127,122,126,122,122],%22string-fasta%22:[301,297,299,314,295],%22string-tagcloud%22:[225,230,225,231,231],%22string-unpack-code%22:[295,291,294,291,292],%22string-validate-input%22:[147,149,148,148,152]}
webkit 30868
http://webkit.org/perf/sunspider-0.9/sunspider-results.html?%7B%223d-cube%22:%5B190,182,183,183,183%5D,%223d-morph%22:%5B196,198,198,197,197%5D,%223d-raytrace%22:%5B163,170,166,165,166%5D,%22access-binary-trees%22:%5B91,94,94,93,93%5D,%22access-fannkuch%22:%5B318,321,317,317,318%5D,%22access-nbody%22:%5B194,196,197,202,195%5D,%22access-nsieve%22:%5B87,89,87,87,86%5D,%22bitops-3bit-bits-in-byte%22:%5B88,90,89,89,88%5D,%22bitops-bits-in-byte%22:%5B132,132,131,131,132%5D,%22bitops-bitwise-and%22:%5B243,249,240,240,242%5D,%22bitops-nsieve-bits%22:%5B153,152,153,153,152%5D,%22controlflow-recursive%22:%5B114,117,114,114,114%5D,%22crypto-aes%22:%5B107,110,108,108,107%5D,%22crypto-md5%22:%5B104,105,104,105,103%5D,%22crypto-sha1%22:%5B103,104,107,103,103%5D,%22date-format-tofte%22:%5B193,188,193,190,189%5D,%22date-format-xparb%22:%5B216,218,216,218,217%5D,%22math-cordic%22:%5B230,230,228,231,232%5D,%22math-partial-sums%22:%5B258,259,259,257,257%5D,%22math-spectral-norm%22:%5B114,115,116,116,115%5D,%22regexp-dna%22:%5B285,287,286,285,286%5D,%22string-base64%22:%5B136,135,133,134,135%5D,%22string-fasta%22:%5B245,249,245,247,248%5D,%22string-tagcloud%22:%5B192,191,188,190,190%5D,%22string-unpack-code%22:%5B180,181,181,182,181%5D,%22string-validate-input%22:%5B164,160,161,159,160%5D%7D March 7th, 2008 at 10:01 am
Brad Jasper
They may have fixed their tests in later posts, but the initial posts are off base. The fact they were mis-matching versions shows me their bias.
Also, testing nightly versions every day is of no interest to me. We could test every build and compare them with differing results.
I will test the final version of each release, but for this article I picked the most recent builds (at the time) and I'm sticking with these results for now. March 7th, 2008 at 10:20 am
Brad Jasper
I think I've done a fair job thus far comparing the two browsers.
Let's agree that I'll test the final release of each version (with more and better benchmarks) and let's move on. March 7th, 2008 at 10:24 am
cris
Trackback
[...] Safari 3 vs Firefox 3 - Round 1: Speed Test - MacTips Firefox és Safari sebességtesztek. [...]
March 12th, 2008 at 1:02 amIsaac
cris
Today's nightly, firefox 20080317 is already 1.02x faster than webkit today's nightly 31090.
Firefox is changing compiling environment of its mac version from Tiger (xcode 2) to leopard (xcode 3). My estimation from test build is that It will speed up firefox 3% on Tiger, and 8% on leopard.
As of now, Firefox 3 nightly is fastest on any platform in javascript performance.
PS. Safari 3 is quite different from safari 3.1, we all know safari 3.0.4 are 3x slower than 3.1 and firefox 3 nightly. March 17th, 2008 at 8:53 am
Brad Jasper
I will have new results when a new version of Firefox is released. March 18th, 2008 at 6:47 pm
Emilio
Ashu
I actually conducted some real-world tests (not necessarily the ones shown here) between the Firefox release and the latest Safari, and I'd have to say I was pleasantly surprised at how fast the new Firefox really is. You can read about it here:
http://blog.ashuc.com/2008/06/17/firefox-30-as-good-as-safari-311/ June 18th, 2008 at 6:53 am
Jonas Grimfelt