I’ve been working on this Benchmark based on Iain Lobb’s BunnyMark. Being a bit confused sometimes about what things speed things up or slow things down, I didn’t want to guess anymore, so I grabbed Iain’s code base (cause I’m lazy, and didn’t want to start from scratch), and added some tests for things I suspect are slowing things down (or speeding things up). I think this will also help shed some light on why some folks see a huge gain in AIR 2.7 CPU mode, while others do not.
Some caveats – this only tests instances of flash.display.Bitmap on the display list, at the size they are, moving the way they move. It’s on my list to add Blitting (I have some initial work on that done, thanks to Iain, but I need to add the rotation, and alpha settings to it), and I’d like to add a vector test, and maybe some extra sized Bitmaps (I’ve heard that makes a difference).
Enough! Here are some results – quality had no effect on GPU mode, so I included only one line:
Note: some are reporting they see a difference in GPU mode, but I still don’t. Update: It appears some users are confusing “Mobile Performance Tester” with BunnyMark, which explains the discrepancy. BunnyMark is not currently in any App Store, which is one key distinguishing feature. 😉
BunnyMark Results – 500 Bunnies | |||||||||
---|---|---|---|---|---|---|---|---|---|
Alpha | ✓ | ✓ | ✓ | ||||||
Rotation | ✓ | ✓ | ✓ | ✓ | |||||
CaB | ✓ | ✓ | ✓ | ✓ | ✓ | ||||
CaBM | ✓ | ✓ | |||||||
iPhone 3GS – GPU | |||||||||
FPS | 24 | 18 | 17 | 22 | 13 | 13 | 19 | 1 | 19 |
iPhone 3GS – CPU | |||||||||
FPS-L | 28 | 21 | 19 | 9 | 19 | 5 | 7 | 5 | 5 |
FPS-M | 28 | 21 | 19 | 4 | 18 | 3 | 7 | 5 | 3 |
FPS-H | 28 | 21 | 19 | 3 | 18 | 2 | 7 | 5 | 2 |
FPS-B | 28 | 21 | 19 | 3 | 18 | 2 | 7 | 5 | 2 |
iPhone 4 (Retina) – GPU | |||||||||
FPS | 25 | 21 | 20 | 25 | 13 | 13 | 16 | 0.5 | 16 |
iPhone 4 (Retina) – CPU | |||||||||
FPS-L | 32 | 23 | 20 | 10 | 21 | 6 | 8 | 6 | 7 |
FPS-M | 32 | 23 | 20 | 5 | 20 | 3 | 8 | 6 | 3 |
FPS-H | 32 | 23 | 19 | 4 | 20 | 2 | 8 | 6 | 2 |
FPS-B | 32 | 23 | 19 | 4 | 20 | 2 | 8 | 6 | 2 |
Notes about the Benchmark:
- In general, the CPU mode seems pretty consistent with the way you’d expect things to work on the desktop – the same optimizations you’d apply for the browser plugin, you’d also apply to mobile for CPU mode.
- Rotation in this benchmark is not continuous – the Bunny graphics are only rotated at the edge of the stage, which is why cacheAsBitmap works to speed those up. If they were constantly updated, it would likely be much more expensive on CPU mode (probably more like rotation without CaB).
- Alpha is continuous – the alpha value of each Bunny is based on the y position and is updated every frame. I would like to add a mode similar to the rotation, so see what effect CaB has on alpha transparent objects that don’t constantly change.
- iPhone 4 and 3GS numbers aren’t directly comparable for practical purposes. The Bitmaps on the screen on 3GS take up much more real estate, since the 3GS screen res has 1/4 as many pixels as the iPhone 4. In a normal app, we’d probably resize things to look comparable between the two devices. I’ll try to add a mode that makes this more comparable (because I suspect we’ll find that 3GS can keep up with iPhone 4 with similar looking content).
- Touching the screen seems to cost about 4 fps across the board.
- I think there may be an issue with returning to rotation = 0 costing some performance in GPU mode. Still have to test that.
- I’m definitely getting some variance on default speeds – basically, before any settings are messed with on some runs I get the faster numbers (the baseline numbers in the tables above). Other times it runs at default settings a couple of FPS slower (on start, or after resetting the switches). With any of the settings, everything is consistent across multiple runs.
It’d be nice to have more benchmarks for more devices, but I only have the above devices available. This should run just fine on Android, Blackberry Playbook, and iPads. If anyone wants to contribute a set of benchmarks, hit the comments. Here is the source. One of these days I’ll make another post, and try to draw some conclusions, maybe wrap the bullet points into a narrative, and edit some of this, but the tables are there, and the source code, and that’s the important stuff.
In the midst of playing with this benchmark, I found (or was pointed at) some great resources. Here are some of them:
- Understanding GPU Rendering in Adobe AIR for Mobile – Ton of good info here.
- Mobile Performance Tester — Now Live in App Stores – This is very similar to what I’ve done, and has some of the same kinds of tests. It shows some different results though, which I’m a bit confused about. I suspect it has something to do with using larger images, which have more alpha transparency (smoother edges). I intend to look at it more closely to see what’s up in there.
- Original BunnyMark by Iain Lobb
- Source for BunnyMark for AIR Mobile
Here is the Benchmark to see it in action: