Jump to content

[TOPIC: topicViewTemplate]
[GLOBAL: userSmallPhoto]
Photo

Corona SDK Performance and Bunnymark challenge
Started by lemon07r Aug 01 2018 03:27 PM

- - - - -
24 replies to this topic
bunnymark benchmark performance challenge defold corona sdk
[TOPIC CONTROLS]
[/TOPIC CONTROLS]
[modOptionsDropdown]
[/modOptionsDropdown]
[reputationFilter]
[TOPIC: post.html]
#1

lemon07r

[GLOBAL: userInfoPane.html]
lemon07r
  • Observer

  • 6 posts
  • Corona SDK

Hello everyone. I'm a fan of Lua based frameworks/engines and have used a bunch, but not yet Corona SDK, which recently caught my interest. A few months ago I ran some comparisons of my favorite engines/frameworks through bunnymark to see how they all performed in this type of test. 

 

The defold tests are bottle necked by the max bunny limit so I decided to run the tests from my phone targeting the fps I got from the defold particlefx test because it is fixed at 15000 bunnies. I tested on my LG V20 with the chrome browser, everything kept consistent as possible and retested for the most accurate numbers.
 
Defold particlefx 22fps @ 15000
Defold go.animate() 23fps @ 10000
Phaser 22 fps @ 3100
Gideros (particles) 23 fps @ 7500
Pixi.js (4.7.3) 23fps @ 10500
Openfl 8.0.0 22fps @ 39000
Kha 21fps @ 50000
 
Then I tested two of my favorite Lua based engines natively on android:
Defold particlefx 31fps @ 15000
Defold go.animate() 31fps @ 12000
Gideros (particles) 31 fps @ 15000
 
I want to add Corona SDK to my bunnymark comparison on android, but did not see any bunnymark tests made for Corona SDK. I have no experience with Corona so I was wondering if anyone would be up to port or make their own Corona SDK bunnymark to rep for their game engine. Would love to see how it holds against Defold and Gideros since they are both free Lua engine alternatives. I think it's a great way to explore engine strengths and limitations.
 
Here are the source projects for those of you interested in how bunnymark was programmed in it's perspective engines/frameworks:

 



[TOPIC: post.html]
#2

lemon07r

[GLOBAL: userInfoPane.html]
lemon07r
  • Observer

  • 6 posts
  • Corona SDK

Bumping.

 

Is the community not all that active compared to other engines?



[TOPIC: post.html]
#3

davebollinger

[GLOBAL: userInfoPane.html]
davebollinger
  • Corona Geek

  • 1,297 posts
  • Enterprise

my guess is just that few here care how fast bunnies can be drawn. (but if you care, then why not port it yourself?)


  • roaminggamer likes this

[TOPIC: post.html]
#4

roaminggamer

[GLOBAL: userInfoPane.html]
roaminggamer
  • Corona Geek

  • 7,489 posts
  • Corona SDK

Agreed. I personally don't have the spare time to port a benchmark. I'm sure others are in the same boat. Just too busy for this.

[TOPIC: post.html]
#5

lemon07r

[GLOBAL: userInfoPane.html]
lemon07r
  • Observer

  • 6 posts
  • Corona SDK

my guess is just that few here care how fast bunnies can be drawn. (but if you care, then why not port it yourself?)

 

Agreed. I personally don't have the spare time to port a benchmark. I'm sure others are in the same boat. Just too busy for this.

 

Is that the general mindset of the Corona community? A little sad to see for me cause when I did this for other engines there was a lot more interest and activity, even in kha the main developer answered questions, defold had a very active thread with well over 50 replies, gideros had multiple people interested, and so forth. Im sure it would only take a half hour to a few hours to port if I was motivated to do so but no point if nobody is interested or if the community isnt super active.

 

Nothing personal, I understand that some people are busy or have other important priorities. Was just disappointed not to find individuals who might be interested like I did in other communities because that's one of my favorite parts about learning and trying other game engines.

 

If I do end up getting around to it I can leave my results here and how I felt about Corona coming from other engines, but I likely wont do the engine full justice like someone experienced could and might end up representing the engine in a poor way if my port isn't made optimally. 

 

One of the features Corona is advertised for literally on the main page is "Speed and performance

From bleeding-fast OpenGL graphics to the lightweight scripting power of Lua, Corona is optimized for performance at every core level. With Corona, your apps are automatically compiled at build time, streamlined for performance and stability." So I was surprised that there werent really any benchmarks or anything to actually test the validity of these claims.

Edited by lemon07r, 05 August 2018 - 07:25 PM.


[TOPIC: post.html]
#6

roaminggamer

[GLOBAL: userInfoPane.html]
roaminggamer
  • Corona Geek

  • 7,489 posts
  • Corona SDK

I don't understand the attitude in your posts. It doesn't endear me to your cause.
 

Is the community not all that active compared to other engines?


Is that the general mindset of the Corona community? A little sad ...


I'm probably misreading your posts. It is very hard to get tone right.

The people here (like myself and others) who answer questions for new folks are busy trying to make a living, so we are careful with our time.

Converting a benchmark is interesting, but time consuming. I personally can't justify the time to do it at this point.

Thanks for raising the topic, and I'd love to see it done, but I for one can't help. Still there are a lot of smart folks here, so if you're willing to wait a little you may be surprised and see someone post a conversion.

Edited by roaminggamer, 05 August 2018 - 07:54 PM.

  • lemon07r likes this

[TOPIC: post.html]
#7

roaminggamer

[GLOBAL: userInfoPane.html]
roaminggamer
  • Corona Geek

  • 7,489 posts
  • Corona SDK

One more thing. I don't see an original language-/ engine-agnostic design spec for this benchmark.

Without a spec I'd have to reverse engineer one of those tests to implement this, all the time risking an apples versus oranges comparison due to my misunderstanding of the port-from source.

This is a lot of work.

Interesting, but man time consuming. :(

PS - In 2013 I wrote a series of benchmarks for Corona. I'm checking to see if I still have the code and will post a link here if I do.

While is isn't your Bunnymark, you can still run it.

Edited by roaminggamer, 06 August 2018 - 07:51 AM.

  • lemon07r likes this

[TOPIC: post.html]
#8

roaminggamer

[GLOBAL: userInfoPane.html]
roaminggamer
  • Corona Geek

  • 7,489 posts
  • Corona SDK

Found it: https://github.com/roaminggamer/RG_FreeStuff/blob/master/AskEd/2018/08/cbench.zip

This is some seriously old code. It is still valid, but it did need a little love to convert it to modern SSK and to Graphics 2.0 from Graphics 1.0.

Here is an HTML5 build (not sure how well this will run in HTML5):

https://roaminggamer.github.io/RGDocs/pages/html5_tests/tests/cbench/

Edited by roaminggamer, 05 August 2018 - 08:48 PM.


[TOPIC: post.html]
#9

roaminggamer

[GLOBAL: userInfoPane.html]
roaminggamer
  • Corona Geek

  • 7,489 posts
  • Corona SDK

Well.. it runs but the scroller is hosed up so you can't see the end of the results.

[TOPIC: post.html]
#10

Michael Flad

[GLOBAL: userInfoPane.html]
Michael Flad
  • Contributor

  • 213 posts
  • Corona SDK

I really like Corona and, besides one project, all of my current inDev projects, are done using Corona as it has a lots of strengths. The unpleasant but honest truth ist, extreme performance is just not one of those - and I tried really hard before I added a second framework just for one project.

 

While I haven't written any bunnymark and I don't know the details but any serious implementation would need to be done *in* the engine itself, using f.i. particle systems (as some of the above listed implementions seem to use) which may run fully on the C++ side or even on the GPU is just nonsense and any such implementation is just plain useless. Haxe based engines like Kha of course are fine because the generated C++ is actually built from the same language you also use for the rest of your game/project.

 

Anyway - the benchmark seems really simple and would be a tiny project but implemented in the typical Corona way I predict you're in for a disappointing result.



[TOPIC: post.html]
#11

Melix19

[GLOBAL: userInfoPane.html]
Melix19
  • Observer

  • 26 posts
  • Corona SDK

Hello,
I tried to port this bunnymark test to Corona, just for the curiosity to see how it performs compared to the other engines. You can find it here: https://github.com/Melix19/corona-bunnymark

Let me know the results!

 

(If someone that has more experience with Corona than me finds out that the code can be improved, fell free to send a PR)


  • lemon07r likes this

[TOPIC: post.html]
#12

Melix19

[GLOBAL: userInfoPane.html]
Melix19
  • Observer

  • 26 posts
  • Corona SDK

I have also builded the project to html5. Here you can find it out: https://melix19.github.io


  • lemon07r likes this

[TOPIC: post.html]
#13

nick_sherman

[GLOBAL: userInfoPane.html]
nick_sherman
  • Corona Geek

  • 1,693 posts
  • Corona SDK

Looks good, but the fps counter seems to be wrong after 1000 bunnies. It's still running fairly smoothly but it's reporting 3-4 fps.
  • lemon07r likes this

[TOPIC: post.html]
#14

Melix19

[GLOBAL: userInfoPane.html]
Melix19
  • Observer

  • 26 posts
  • Corona SDK

There is not a real fps counter on Corona so i tried different codes to port it to the project. This is the best that i found.


  • lemon07r likes this

[TOPIC: post.html]
#15

roaminggamer

[GLOBAL: userInfoPane.html]
roaminggamer
  • Corona Geek

  • 7,489 posts
  • Corona SDK

@Melix19,

 

1.  You're awesome.  Thanks for porting.

 

2. I didn't agree with your implementation of the FPS counter so used my own and added a memory meter too.

 

3. I also made an optimized mover more like the code here: https://github.com/pixijs/bunny-mark/blob/master/src/Bunny.js

 

You can see my version of your port here: https://github.com/roaminggamer/RG_FreeStuff/tree/master/AskEd/2018/08/bunnymark

Direct downloadhttps://github.com/roaminggamer/RG_FreeStuff/raw/master/AskEd/2018/08/bunnymark.zip

 

Folders:

 

PS - Your mover was elegant by the way.  Very nice.

 

 

PPS - I posted an HTML build of 'modified' here: https://roaminggamer.github.io/RGDocs/pages/html5_tests/tests/bunnymark1/index.html


Edited by roaminggamer, 06 August 2018 - 01:15 PM.

  • lemon07r likes this

[TOPIC: post.html]
#16

roaminggamer

[GLOBAL: userInfoPane.html]
roaminggamer
  • Corona Geek

  • 7,489 posts
  • Corona SDK

I'd say the performance of the HTML5 build-target (still maturing) is sub-optimal, but on other build-targets the performance is pretty good.

 

I suspect if you knock the FPS to 30 in config.lua you will actually see a frame rate increase at higher bunny counts. 

 

This is because you are halving the calls to all calculations.


  • lemon07r likes this

[TOPIC: post.html]
#17

davebollinger

[GLOBAL: userInfoPane.html]
davebollinger
  • Corona Geek

  • 1,297 posts
  • Enterprise

keep in mind that html5 is still in beta and barely 4 months old



[TOPIC: post.html]
#18

lemon07r

[GLOBAL: userInfoPane.html]
lemon07r
  • Observer

  • 6 posts
  • Corona SDK

Some really cool stuff. I built an apk using the latest public release of Corona and @roaminggamer's optimized version of bunnymark, it got 18-23 fps at 1000 bunnies. I rebuilt the Defold version of the benchmark using their latest version as well to see if anything had changed and for consistency, tested on the same phone right after roaminggamer's version of bunnymark I still get 12k bunnies at 31 fps using their recommended method of go.animate(), particlesfx could hit up to 15k but wouldnt be a very real approach to drawing sprites. This was tested on my LG V20.

 

Some side observations, building the android apk was pretty hassle free like it is for defold, I had issues getting gideros to build me one some time ago, and my other observsation is apk size, I believe gideros had the largest one but I forget how big, defold's bunnymark which comes with the options to choose from 5 different methods of drawing bunnies was only 2.88mb, corona was in the middle at 6.86mb. However from what I understand Corona has a lot more higher level constructs and such, meanwhile defold works a little closer to the metal with less overhead so I think this was to be expected. Benchmarks only tell one side of a story, so there is no better engine here, just engines that are better at different things. 

 

Still this seems a little slow, I think there could be some possible tweaks made to get more bunnies out of corona.



[TOPIC: post.html]
#19

davebollinger

[GLOBAL: userInfoPane.html]
davebollinger
  • Corona Geek

  • 1,297 posts
  • Enterprise

image sheet/sprite might help.. test appears to be a batching nightmare


  • lemon07r likes this

[TOPIC: post.html]
#20

Michael Flad

[GLOBAL: userInfoPane.html]
Michael Flad
  • Contributor

  • 213 posts
  • Corona SDK

Yeah, batching is for sure the most required optimization here and I'm sure Defold does using a texture atlas, so at the moment the comparison is not remotely fair.

 

I'll give this a try later today and try some additional optimizations, but there's only so much you can do with such simple code. With regards to the general code of Defold I'd say they're in general not at a lower level as Corona and the update code for the bunnies isn't even that optimized (could use an additional local here or there) but then, it might just not have an impact in such a small piece of code calling into engine APIs that many times.

 

The probably biggest Defold performance advantage is it's usage of LuaJIT - it's just a so much faster VM even with a disabled jit compiler (as that's not allowed on iOS and does have some issues wrt the memory manager on Android).


  • lemon07r likes this

[TOPIC: post.html]
#21

lemon07r

[GLOBAL: userInfoPane.html]
lemon07r
  • Observer

  • 6 posts
  • Corona SDK

image sheet/sprite might help.. test appears to be a batching nightmare

 

 

Yeah, batching is for sure the most required optimization here and I'm sure Defold does using a texture atlas, so at the moment the comparison is not remotely fair.

 

I'll give this a try later today and try some additional optimizations, but there's only so much you can do with such simple code. With regards to the general code of Defold I'd say they're in general not at a lower level as Corona and the update code for the bunnies isn't even that optimized (could use an additional local here or there) but then, it might just not have an impact in such a small piece of code calling into engine APIs that many times.

 

The probably biggest Defold performance advantage is it's usage of LuaJIT - it's just a so much faster VM even with a disabled jit compiler (as that's not allowed on iOS and does have some issues wrt the memory manager on Android).

 

Defold does indeed use atlases to cut down on size and better performance. It also does most of the work for you in regards to batching. Does Corona not use LuaJIT? It does offer a huge performance advantage over just regular 5.x Lua.



[TOPIC: post.html]
#22

Michael Flad

[GLOBAL: userInfoPane.html]
Michael Flad
  • Contributor

  • 213 posts
  • Corona SDK

Corona does not use LuaJIT at the moment - also even though it is an exceptional piece of software (even more so given it's kind of the work of a single dev) but I can also understand that it's not the simplest decision as it's kind of stuck around Lua 5.1 (+ some features of 5.2).

 

But then Corona does not use a newer version of Lua anyway *and* it seems that more recent Lua versions are still slower (there was some discussion over at the Lua mailinglist) than 5.1 anyway so, besides native integer support and bitops there's not that much to be gained from newer versions IMO.

 

You can and should add your vote for a feature you like to see implemetned (or even request new ones) for Corona at the feedback site.

http://feedback.coronalabs.com/forums/188732-corona-feature-requests-feedback/suggestions/6655792-coronasdk-with-luajit



[TOPIC: post.html]
#23

davebollinger

[GLOBAL: userInfoPane.html]
davebollinger
  • Corona Geek

  • 1,297 posts
  • Enterprise

re lua:  just benchmark it, fe replace newImage() with {x=123,y=0}.  my guess would be ~100X performance, if so then lua isn't the limiting factor, move on.



[TOPIC: post.html]
#24

Hyunjoong Kim

[GLOBAL: userInfoPane.html]
Hyunjoong Kim
  • Enthusiast

  • 95 posts
  • Corona SDK

iPhone 5c (2013  released) : 25 fps

Samsung Galaxy A7 (2016 released) : 3 fps

 

lol

 

p.s: A7 is the better hardware specification.



[TOPIC: post.html]
#25

Michael Flad

[GLOBAL: userInfoPane.html]
Michael Flad
  • Contributor

  • 213 posts
  • Corona SDK

iPhone 5c (2013  released) : 25 fps

Samsung Galaxy A7 (2016 released) : 3 fps

 

lol

 

p.s: A7 is the better hardware specification.

 

Without any details, there's now way to guess what's going wrong.




[topic_controls]
[/topic_controls]

Also tagged with one or more of these keywords: bunnymark, benchmark, performance, challenge, defold, corona sdk