I’ve been working on this Benchmark based on Iain Lobb’s BunnyMark. Being a bit confused sometimes about what things speed things up or slow things down, I didn’t want to guess anymore, so I grabbed Iain’s code base (cause I’m lazy, and didn’t want to start from scratch), and added some tests for things I suspect are slowing things down (or speeding things up). I think this will also help shed some light on why some folks see a huge gain in AIR 2.7 CPU mode, while others do not.
Some caveats – this only tests instances of flash.display.Bitmap on the display list, at the size they are, moving the way they move. It’s on my list to add Blitting (I have some initial work on that done, thanks to Iain, but I need to add the rotation, and alpha settings to it), and I’d like to add a vector test, and maybe some extra sized Bitmaps (I’ve heard that makes a difference).
Enough! Here are some results – quality had no effect on GPU mode, so I included only one line:
Note: some are reporting they see a difference in GPU mode, but I still don’t. Update: It appears some users are confusing “Mobile Performance Tester” with BunnyMark, which explains the discrepancy. BunnyMark is not currently in any App Store, which is one key distinguishing feature. 😉
| BunnyMark Results – 500 Bunnies | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Alpha | ✓ | ✓ | ✓ | ||||||
| Rotation | ✓ | ✓ | ✓ | ✓ | |||||
| CaB | ✓ | ✓ | ✓ | ✓ | ✓ | ||||
| CaBM | ✓ | ✓ | |||||||
| iPhone 3GS – GPU | |||||||||
| FPS | 24 | 18 | 17 | 22 | 13 | 13 | 19 | 1 | 19 |
| iPhone 3GS – CPU | |||||||||
| FPS-L | 28 | 21 | 19 | 9 | 19 | 5 | 7 | 5 | 5 |
| FPS-M | 28 | 21 | 19 | 4 | 18 | 3 | 7 | 5 | 3 |
| FPS-H | 28 | 21 | 19 | 3 | 18 | 2 | 7 | 5 | 2 |
| FPS-B | 28 | 21 | 19 | 3 | 18 | 2 | 7 | 5 | 2 |
| iPhone 4 (Retina) – GPU | |||||||||
| FPS | 25 | 21 | 20 | 25 | 13 | 13 | 16 | 0.5 | 16 |
| iPhone 4 (Retina) – CPU | |||||||||
| FPS-L | 32 | 23 | 20 | 10 | 21 | 6 | 8 | 6 | 7 |
| FPS-M | 32 | 23 | 20 | 5 | 20 | 3 | 8 | 6 | 3 |
| FPS-H | 32 | 23 | 19 | 4 | 20 | 2 | 8 | 6 | 2 |
| FPS-B | 32 | 23 | 19 | 4 | 20 | 2 | 8 | 6 | 2 |
Notes about the Benchmark:
- In general, the CPU mode seems pretty consistent with the way you’d expect things to work on the desktop – the same optimizations you’d apply for the browser plugin, you’d also apply to mobile for CPU mode.
- Rotation in this benchmark is not continuous – the Bunny graphics are only rotated at the edge of the stage, which is why cacheAsBitmap works to speed those up. If they were constantly updated, it would likely be much more expensive on CPU mode (probably more like rotation without CaB).
- Alpha is continuous – the alpha value of each Bunny is based on the y position and is updated every frame. I would like to add a mode similar to the rotation, so see what effect CaB has on alpha transparent objects that don’t constantly change.
- iPhone 4 and 3GS numbers aren’t directly comparable for practical purposes. The Bitmaps on the screen on 3GS take up much more real estate, since the 3GS screen res has 1/4 as many pixels as the iPhone 4. In a normal app, we’d probably resize things to look comparable between the two devices. I’ll try to add a mode that makes this more comparable (because I suspect we’ll find that 3GS can keep up with iPhone 4 with similar looking content).
- Touching the screen seems to cost about 4 fps across the board.
- I think there may be an issue with returning to rotation = 0 costing some performance in GPU mode. Still have to test that.
- I’m definitely getting some variance on default speeds – basically, before any settings are messed with on some runs I get the faster numbers (the baseline numbers in the tables above). Other times it runs at default settings a couple of FPS slower (on start, or after resetting the switches). With any of the settings, everything is consistent across multiple runs.
It’d be nice to have more benchmarks for more devices, but I only have the above devices available. This should run just fine on Android, Blackberry Playbook, and iPads. If anyone wants to contribute a set of benchmarks, hit the comments. Here is the source. One of these days I’ll make another post, and try to draw some conclusions, maybe wrap the bullet points into a narrative, and edit some of this, but the tables are there, and the source code, and that’s the important stuff.
In the midst of playing with this benchmark, I found (or was pointed at) some great resources. Here are some of them:
- Understanding GPU Rendering in Adobe AIR for Mobile – Ton of good info here.
- Mobile Performance Tester — Now Live in App Stores – This is very similar to what I’ve done, and has some of the same kinds of tests. It shows some different results though, which I’m a bit confused about. I suspect it has something to do with using larger images, which have more alpha transparency (smoother edges). I intend to look at it more closely to see what’s up in there.
- Original BunnyMark by Iain Lobb
- Source for BunnyMark for AIR Mobile
Here is the Benchmark to see it in action:
Great work! Looking through your results it seems that cacheAsBitmap and cacheAsBitmapMatrix both seem to make the FPS slower?
Thanks! It depends on what you are doing whether CaB and CaBM make things slower. CaBM does seem to make things slower most of the time (in this test ALL the time). I’ve got an idea that it might help with constant rotation (this test only rotates when the bunnies hit the edge).
CaB helps when translating ONLY x/y position on CPU mode. You can see the benefit in CPU mode when you turn on rotation and CaB. Without CaB the CPU seems to rotate each Bitmap every frame, where if you use CaB, it just uses the cached rotated image, until it’s invalidated by further rotation. You do take a huge performance hit when rotation is on (you can even see that on the desktop), though the hit is much smaller with CaB.
In GPU mode – yeah, CaB always makes things slower for Bitmap instances. Even CaBM doesn’t seem to actually have any benefit. I’ll add a constant rotation mode soon, that’ll probably show further performance degradation with rotated Bitmaps (in CPU mode), and should test whether CaBM has any benefit at all (so far it hasn’t, not for Bitmap instances in either mode).
Very excellent work, useful information, helpful links, etc.