Comments on: Game Scripting Languages

By: Jay Sherby » Blog Archive » Scripting Language Benchmarks Game

Jay Sherby » Blog Archive » Scripting Language Benchmarks Game — Fri, 16 Dec 2011 04:11:41 +0000

[...] Some other languages I considered but ultimately rejected include Tcl, Pawn, Scheme, and AngelScript. Tcl and Scheme are a bit too old and stagnant for my tastes. Pawn and AngelScript both have negligible use in the real world, making support a considerable challenge. Also, Pawn and AngelScript are so similar to C and C++ respectively that it makes the languages pretty boring. I will note for the record, though, that Pawn has shown excellent results in others’ benchmarks. [...]

By: Lewis

Lewis — Mon, 23 May 2011 18:14:23 +0000

I’m planning on repeating the tests soon, and I’ll release the full test harness when that happens.

I’m staying pretty busy right now, though, so it may be a couple of months out, unfortunately.

By: dotsquid

dotsquid — Mon, 23 May 2011 09:13:32 +0000

Hi. Thanks for the article.
However I’d like to join Deryck’s request to repeat the tests.

By: Deryck

Deryck — Sat, 14 May 2011 12:20:43 +0000

Interesting results – I was considering using AngelScript for a game engine, mainly due to the ease of adding native functions to the scripting engine, but judging from these tests, Pawn looks like a great alternative in terms of raw performance.

Judging by Pekuja, though, it appears things have changed since when you posted this (2009?). Just wondering whether you could repeat the same tests with your test harness, or whether you could post your test harness up so we can test it ourselves under similar conditions?

By: Alberto

Alberto — Sat, 09 Apr 2011 20:46:58 +0000

Ok, I noticed that indeed MinGW creates executables that are 2 times the size compared to Visual Studio when using C++ but not when using C.
So I played a bit with my makefiles and I brought the exe from 400Kb to 290Kb… (BTW static libs are still the same size)
First of all I found that I was compiling with C++ exception handling enabled, that squirrel doesn’t use. Apparently hits performance and size very hard in GCC (Add -fno-exceptions in the GCC parameter list and TADA! good performance gain).
Running a “strip sq.exe” (like lua) reduces size even more. I tried running ‘strip’ on my libraries crashes strip.exe(at least the version I have).

So I apologize if I couldn’t believe your results, it seems my GCC makefiles are way too simple and by default GCC leaves symbols both in static libraries and executables and they have to be stripped explicitly(my bad).

ciao
Alberto

By: Pekuja

Pekuja — Sun, 27 Mar 2011 10:33:02 +0000

I replicated some of the tests using latest versions of AngelScript, GameMonkey and Squirrel, as well as ChaiScript. I didn’t do a compile to bytecode first though, but I think the effect should be mostly negligible. I’ve posted my results on pastie.org as plain text for now: http://pastie.org/1721408

I actually got some pretty different results on some of the tests. AngelScript did a lot better in my tests. Squirrel also did better, and GameMonkey did a bit worse in some tests. ChaiScript was the slowest of the bunch overall, but did surprisingly well in comparison when passing large strings to native code.

By: Pekuja

Pekuja — Sat, 26 Mar 2011 23:33:02 +0000

I found this blog post extremely interesting. I think it might be the only such benchmark around. Now, I was hoping to try and replicate the test with the latest versions, but you seem to have omitted the embedding C code from the package. Do you still have it?

By: Lewis

Lewis — Wed, 02 Mar 2011 06:50:10 +0000

Just a follow up: I re-compiled Lua and Squirrel again, to double-check my previous results.

The results from today are:
Lua Library: 197 KB
Lua Executable: 197 KB
Squirrel Library: 397KB (or 462 KB including sqstdlib)
Squirrel Executable: 377 KB

I compiled both Lua and Squirrel with the included Makefiles. It appears that both used optimization level 2 (-O2). My compiler is MinGW GCC 4.5.0 and the tested versions are Lua 5.1.4 and Squirrel 2.2.2.

I also reran my “fib” and “prime” benchmarks, and got results inline with my previous results.

If I have some free-time later in the week, I’ll run some tests with Visual Studio. If I get different results, I’ll let you know.

By: Lewis

Lewis — Tue, 01 Mar 2011 21:09:59 +0000

Thanks for your feedback, Alberto.

The libraries were compiled with GCC. I used the supplied Makefiles to compile both Lua and Squirrel. I will double check my comparisons between the two, and post back later.

I don’t believe that looking a static library’s size is “nonsense.” It is the total compiled size of all the code. If you link it to an executable, and stripped out any uncalled functions, then obviously the size should be smaller. In any case, I’m displaying the information at face value, so readers should interpret it as they see fit.

If you think my performance tests are off “big times,” then please, by all means, download the code I used, and run it (at least run “fib” and “prime”). Report back the run-times for Lua vs Squirrel, and the tools/configurations you tested with. I’d be happy to hear your results.

I’m probably going to update this benchmark soon (it’s a year and a half old), and publish all my scripted benchmarking code. Tell you what, I’ll even compile some executables to compare library size with.

I’m sorry if feel that this benchmark showed Squirrel in a bad light, but I am trying to be as unbiased as possible.

By: Alberto

Alberto — Tue, 01 Mar 2011 17:59:56 +0000

Your numbers on squirrel are quite off. On windows and VC++ 2010 squirrel is like +/- 2Kb compared to Lua.
Plus looking at a .lib file to determine the code footprint is non sense, you should look at linked executables expecially when comparing C++ and C libraries. And make sure you don’t link the CRT statically.
Your preformance tests also seem rather off(big times). You should probably specify the tools/configurations you tested with.
(BTW I’m Squirrel’s author :) )

By: Lewis

Lewis — Tue, 07 Sep 2010 17:42:48 +0000

For this post I used Gnuplot. For some other projects I’ve used Ploticus. They both work quite well.

The trick is to have everything scripted. I just hit go, all the tests are run, and the graphs are drawn.

By: xorgman

xorgman — Tue, 07 Sep 2010 14:53:25 +0000

What did you use to create the charts and graph?

By: Lewis

Lewis — Thu, 04 Feb 2010 04:02:37 +0000

I recently discovered The Jim Interpreter. It's a very small TCL implementation. If you like TCL (I do), it's definitely worth checking out.

By: benoit jacquier

benoit jacquier — Fri, 23 Oct 2009 11:29:27 +0000

Thanks for the article!
However i did a small perf test with a fibonnaci function ( recursive ), and my result gives gamemonkey 2x faster than Lua
Here with fibonnaci number: 43
GameMonkey( 1.25 ): 151 sec
Lua ( 5.1.4 ) : 317 sec

Core 2 Duo / VS 2008

By: Joe Andresen

Joe Andresen — Thu, 27 Aug 2009 17:08:56 +0000

That is a very nice article. I use Lua for my stuff and have found the community to make the experience a lot better. A lot of my code/modules i use (lua pickle, lua interface, NumericLua, Event System) are from peoples public/free projects. You might also want to add Decoda as the debugger for Lua. It is literally like the Visual Studio debugger, and as long as your application has .pdb files it will work with embedded lua VMs. Here is a Video of how it works.