Last week we posted a demo of our WebGL port of the Fieldrunners particle system. What I didn’t mention in that post was that we were using the Google Analytics event tracking system to collect performance data every time someone ran the demo! We ended up collecting over 14,000 samples from 837 unique users, about 16 samples per user.
The big conditional block triggers roughly once a second, updating the FPS counter you see on screen in the demo. Each time this happens, we sum up the number of samples we’ve taken, and have a running total of the overall FPS. Then every 8 seconds we record the moving average of this 8-sample window to Google Analytics using the _gaq.push() function. We reset the moving average and start again. The eagle-eyed will notice that while the code runs every 8 seconds, it only starts doing so on the 7th second the demo is running. This is because I didn’t want to collect a 1-sample moving average as soon as the demo began.
The Google Analytics event tracking system is flexible enough that I’m able to pass it a category (‘FRWebGL’ in this case), the name of the tower selected, and the overall format of the towers in a concatenated string (‘r4c4x64y64’ would be a 4x4 array of towers with an x/y offset of 64). What’s great about using Google Analytics is that each of these data points gets tied to a user session, which in turn is tied to things like browser version, OS, location, etc. For example, I can open up Google Analytics and see this (image edited to condense the content into a smaller screenshot):
You can see from this that over ALL 478 unique users who contributed flame tower data, Safari performed the best out of any browser. (“Unique events” in this case is equivalent to unique users.) However, you can also see that our sample size for Safari is very small at just 14 users, which is why I’m excluding Safari from the rest of the data in this post. The data we collect also includes any demo configuration, including people who generated 1 tower, and people who generated 2,000+ towers. Fortunately, Google Analytics lets us drill down even further to get more useful data. You can also export the data to a CSV using the Google Analytics Data Export API (although I just used this tool and did a copy-and-paste into a spreadsheet).
There’s nothing inherently super interesting about this data except to say that average performance is more than acceptable across both Chrome and Firefox. (As mentioned above, Safari was left off because we didn’t have a big enough sample size.) Remember that these are WebGL-enabled browsers, so we’re talking Firefox 4-8 and Chrome 12-15. We do see a dip in Lightning Tower performance on Firefox, but we haven’t exactly tracked down the cause of that.
The overall data on operating system normally wouldn’t be super interesting (we care more about browser than OS), but Chrome OS is kind of neat. The reason why we’re seeing 29.3 FPS on Chrome OS, while other operating systems are at 51+ FPS, is because the Chrome browser on a Chrome OS system has its frame rate capped at 30 FPS.
Also of note: when the data refers to “Chrome” as a browser elsewhere, it’s stripped of Chrome OS data due to this artificial FPS cap.
Breaking down the general performance over browser versions on Firefox shows a slight increase in performance as version increases, but I suspect that’s more due to the fact that leet programmers with unstable builds of Firefox installed probably have better computers than the masses with Firefox 4.
Chrome 12 performance is better overall than Firefox 4 (about 17% faster), but Firefox 5 catches up to it. We see Chrome’s performance drop by 3 FPS on Chrome 15, but our sample size for Chrome 15 is pretty small, so I wouldn’t draw any conclusions from it.
Lastly, I’ve included a box plot to give a sense of distribution. This chart shows the quartile data for Firefox and Chrome running the 4x4 Flame Tower test with a 64 pixel x/y offset. You can see that while we did have some browsers score as low as 8.5 FPS on Firefox, 75% of all Firefox users showed a framerate of 45 FPS or higher. The lowest we saw on Chrome was an acceptable 28 FPS, with 75% getting above 47 FPS. You see that Chrome doesn’t go higher than 60 FPS – this is because Chrome’s rendering is capped at 60 FPS (the fact that we see 61 FPS is just an artifact of measuring frame rate).
These “in the wild” tests confirm what we’ve been seeing in-house at Bocoup: Fieldrunners is performing extremely well under WebGL!
Of course, to get really robust numbers we’d need to know CPU, GPU, RAM, and other basic stats that we have no good way of passively collecting from a web browser. But considering that our overall performance across all tests (even the ones with thousands of towers) was 51.07 frames per second, we’re bullish on WebGL for games.