Showing posts with label kahlua. Show all posts
Showing posts with label kahlua. Show all posts

Friday, June 18, 2010

Kahlua on Android

Yes, that's right, Kahlua supports Android (or Android supports Kahlua, depending on how you see it). And it worked surprisingly easy. All in all, I spent two hours installing the Android SDK, reading the Android documentation and creating an example program that used Kahlua. I even managed to create a very simple interpreter. And it didn't just load the Kahlua core, it also supports the J2SE extensions. It was almost out of the box, and I'll expand on that later.

This was a great success, and I attribute it to the following reasons:
  • The Android SDK is really polished. It was easy to install, easy to get started, and the documentation was pretty good!
  • Kahlua (core and J2SE) is compatible with the same subset of Java 5 that Android supports.
As far as I can tell, the classes in Android are fully compatible with Suns Java classes; at least I haven't found any bugs in Kahlua on Android yet.

The only problem I had was that I got a NoSuchMethodError when invoking Arrays.copyOfRange. It turns out that this method was introduced in Java 6, which explains why Android didn't support it. I simply converted the code into using the old and trusted System.arrayCopy and the code worked!

The Android interpreter even supported the reflection- and annotation-based exposing of Java methods, which I demonstrated by exposing a method that called Thread.sleep. (This was also useful for testing that the UI components in Android worked correctly while executing a script.)

The Android Kahlua interpreter is checked in to the git repository for Kahlua2, if anyone is interested in seeing it. It is basically a single Java file in 148 lines of code.

Friday, June 4, 2010

More details of the exposer

As I said before, there is more to the exposer than I wrote before.

Lua is, as you already know, a language that supports multiple return values. Java however does not, and that makes integration slightly less straightforward.

It is not obvious how to do it, but it is possible to expose Java methods and have them return multiple values.

Exposing multiple return values
The workaround is to create your java method like this:
@LuaMethod
public void getUnitLocation(ReturnValues rv, String unitName) {
Unit unit = service.getUnit(unitName);
rv.push(unit.getX());
rv.push(unit.getY());
}
You can then use the function like this:
local x, y = getUnitLocation("theUnit")
This looks a lot like how it's done using a raw JavaFunction, you might exclaim! This is not by coincidence. In fact, ReturnValues is just a thin layer on top of the original LuaCallFrame.

In order for this to work, the exposer mechanism performs special handling if the first parameter of the method is of the type ReturnValues.

Varargs
If the java method is a vararg method, i.e. defined like this:
public String concat(String... strings) {
return Arrays.toString(strings);
}
then Lua scripts can call it with concat("a", "b", "c") and get "[a, b, c]" back. This may seem obvious and it should be, but I'll mention it anyway.

Error handling
What if you do something wrong, such as calling getUnitLocation(), thus not passing in the name? The exposer will detect an incorrect number of arguments and throw an error message at you.

This however, is ok: getUnitLocation("theName", "something else"). The exposer will simply disregard the extra parameter. You may expect it to similarly throw an error, but this is in fact in line with how Lua generally handles too many arguments.

If you supply the incorrect type, you'll also get an error (unless the type can be converted to the Java type of course). If you supply nil, it'll go through into the method - so you have to add the null checks yourself.

Method overloading
I mentioned method overloading earlier, and that the exposer can handle it to some extent. I'll clarify that here.

Assume you have the following class:
public class Example {
public void myMethod() {
}
public void myMethod(Object a, Object b, Object c) {
}
public void myMethod(String a, String b) {
}
public void myMethod(int a, int b) {
}
}
If all these methods are exposed, the exposer will detect that it's already been exposed, and will convert the generated JavaFunction to a dispatching JavaFunction.
This dispatcher will inspect the arguments passed in the call to try to determine which method to call. The actual algorithm for selecting is very naive, but most likely good enough for almost all cases. In any case, you shouldn't use overloading too much - that's bound to confuse your users.

The dispatcher first sorts the list of matching methods by decreasing parameter count. Varargs count as zero. In our case, this means the following ordering:
public void myMethod(Object a, Object b, Object c);
public void myMethod(String a, String b);
public void myMethod(int a, int b);
public void myMethod();
The middle two have the same number of parameters, and thus can occur in any order.
When getting a call, it simply goes through the list in order and tries to call it. If it can't be called, it simply tries the next.
A call like myMethod("a", "b") will fail the first method, since it doesn't have enough arguments, but it will match the second.
myMethod("a", 123) won't match 1, 2 or 3, but it could be reduced to the 4th.
myMethod(nil, nil, nil) will match the first (since nil matches all types).

Methods and functions
What is a method? What is a function?

I'd like to define method as a function that's associated with an object. In Java, all instance methods falls under that definition of method. Static methods in Java can be considered functions, since they are not associated with specific objects.

Real methods don't really exist in Lua - all functions are plain functions. However, you can simulate methods easily. There is a simple syntactic sugar in Lua for this:

Sugar for a method definition:
function obj:method(args...)
end
is equivalent to:
function obj.method(self, args...)
end
That means that in the first case, it simply injects a new local variabel "self" assigned to the first argument.

Similarly, there is sugar for a method call:
obj:method(args...)
is equivalent to:
obj.method(obj, args...)
which of course is a regular function call.

This also how the exposer differentiates between methods and functions. If it's annoted with @LuaMethod(global = true) or is static, it's a function, and arguments are passed straight through from Lua to Java.

If global = false (which is the default), the first argument is consumed by the exposer and used to find the actual object to invoke the method on, and it's not passed along to the method.

Thursday, June 3, 2010

Differences between Kahlua2 and Lua 5.1

There are some noticable differences between Lua 5.1 and Kahlua2, and it may be good to know what they are.

Tables
Tables in Kahlua2 are implemented using a backing Hashtable or ConcurrentHashMap and as a result, implementing "next" properly would be inefficient and instead of implementing "next", it was removed. The function "pairs" still works, so use that instead. One common idiom using next in Lua 5.1 was checking if a table is empty:
function isempty(t) return next(t) == nil end


An alternative implementation of that in Kahlua2 would be:
function isempty(t) return pairs(t)() == nil end

but it would be inefficient, since it would create an iterator for each call to isempty.

Instead, you can use the method table.isempty(t) in Kahlua2.

Another side effect of using Java maps to implement tables, is that key equality is slightly different between Lua 5.1 and Kahlua2. In Lua 5.1, keys are equal in a table if they refer to the same object while in Kahlua2 they are equal if their hashcodes are equal and their equals()-methods return true. For strings, it works just as in Lua 5.1, but for numbers it's slightly different. Java treats -0 and +0 as different keys while Lua treats them as the same.

tostring
When converting things to strings in Kahlua2 (for example when calling tostring(obj)), it will not just return "userdata" if it doesn't have a better guess, it will instead use the toString() method for the object.

Weak tables
Kahlua2 doesn't support the __mode = "[k][v]" metatable magic to create weak tables. If you need weak tables, consider exposing a new method for creating a KahluaTable backed by a WeakHashMap or something similar.

Kahlua2 doesn't support the __gc feature for metatables, since there is no good way in Java to get notified when an object is about to get garbage collected. If you need this functionality, consider putting the object in a SoftReference to manually check when it's collected.

Libraries
Kahlua2 does not include any io or debug library. Some parts of the debug library may be added in the future, but the io library is better left added as a custom extension if necessary. The same thing goes for the os library, which is only partially implemented.

xpcall
Kahlua2 doesn't implement xpcall yet, but it might come in the future.

random and randomseed
random and randomseed in Lua uses a mutable global state, which is undesirable in Kahlua2.
Instead, use the following:
local myrandom = newrandom()
myrandom:random() -- real number in the range 0 to 1
myrandom:random(max) -- integer number in the range 1 to max
myrandom:random(min, max) -- integer number in the range min to max
myrandom:seed(obj) -- seeds the random instance with the hashcode of the object.

Wednesday, June 2, 2010

Kahlua - the J2SE goodies

In this post, Lua will generally refer to the language specification, whereas Kahlua will refer to this actual implementation so don't be confused when I talk about Lua scripts running under Kahlua.

The goodies
Kahlua has a couple of useful extensions that makes your life as a developer easier when integrating Kahlua into your applications. Some of them depend on each other, so I'll go through them in dependency order to make the reading more digestible.

Converting objects
Since Lua only has a few built in types (String, Double, Boolean, KahluaTable) and Java has many many more types, it's sometimes useful to automatically convert types when going back and forth between Lua and Java.

For instance, if Java were to send an Integer object to Kahlua, that object would just be an opaque reference, without any operations. If we instead convert it to a Double, Kahlua can work with it without any problems. This is the rationale for introducing the first J2SE goodie - the KahluaConverterManager.

This is a class that can help you convert types back and forth between Lua and Kahlua. By itself, it can't convert anything at all, but you can register actual converters with it. For convenience, a couple of generic converters are bundled with Kahlua and you can also write custom converters if you need to.

The converter manager has two conversion methods: fromLuaToJava and fromJavaToLua.
Converting from Lua to Java takes an object and a Java class.
The converter manager then uses the type of the object and the Java class to determine which converter best matches the conversion. If nothing matches it returns null instead.

Converting from Java to Lua is different - it doesn't have a target type, since Lua is dynamically typed. Instead the converter just looks at the Java type to choose converter. If no suitable converter is found, it returns the object as it is.

Here are the already existing converters:

The number converter
Registering the number converter allows you to automatically convert Lua numbers (which are implemented as Double) to the numeric types in Java (int, long, char, byte, short, float, double).

It also works the other way around, converting the numeric types in Java to Double.

The table converter
This one also converts both ways. KahluaTable instances can be converted both to List and Map, and Lists and Maps can be converted back to KahluaTable. It also converts maps recursively, so be careful to avoid converting cyclic structures.

The enum converter
You can also convert enums to their string representations (enum.name()) and back, which means that you can
call Java methods that takes an enum just by passing in a string that matches one of the enums.

Calling Lua from Java
LuaCaller.protectedCall is a utility method that simplifies the usage of thread.pcall. Instead of returning a raw array where the first return value always is true or false, it returns an abstraction that hides the details: the LuaReturn class.

LuaReturn has the methods isSuccess(), getErrorMessage(), getStacktrace(), getJavaException() which makes it easier to use than extracting result[0] (and 1, 2, 3, respectively).

LuaCaller.protectedCall additionally applies the installed converters to the inputs before the call. It doesn't apply it to the return values, because it doesn't know which types it wants.

Calling Java from Lua
Be prepared - here comes the most advanced, the most useful and simply best feature of the J2SE goodies.

The LuaJavaClassExposer is a class that eliminates the need to manually define JavaFunction implementations for things that you want to expose to Lua.

Imaging writing a simple game, which is scriptable with Kahlua, and you want to be able to query a game units health from the Lua script.
Without the exposer, you'd have to write something like this:

public class GetUnitHealth implements JavaFunction {
public int call(LuaCallFrame callFrame, int nArguments) {
if (nArguments < 1) {
throw new IllegalArgumentException("Expected at least one argument");
}
Object arg = callFrame.get(0);
if (arg == null || !(arg instanceof String)) {
throw new IllegalArgumentException("Expected a string argument but got " + arg);
}
String s = (String) arg;
Unit unit = unitService.getUnit(s);
callFrame.push(unit.getHealth());
return 1;
}
}

Using the exposer instead, you can write something like this:

public class UnitFunctions {
@LuaMethod
public int getUnitHealth(String unitName) {
Unit unit = unitService.getUnit(unitName);
return unit.getHealth();
}
}

It removes a lot of the boilerplate right? The exposer takes care of all the conversions and argument matchings for us!

Instead, you can write an ordinary method in any class, and let the exposer expose that class. One JavaFunction per method will be created, and the method will be available in Lua. The generated JavaFunction will automatically use the installed converters which makes it really easy to integrate the Java and Lua code.

There are basically two different styles available when exposing:

Exposing like java
The first style is exposing things to work almost the same as in Java.
You simply call exposer.exposeLikeJava(yourClass, staticBase).
After that, a Lua script may call obj:yourMethod(arg1, arg2, ...) if obj is of type yourClass and yourMethod is a method in yourClass.

You may be wondering what staticBase means (and if not, you're not paying enough attention). A method are only available for the object if it's a regular java method.
Static methods and constructors can't be used this way. Instead, Kahlua uses the staticBase argument, which is a KahluaTable, to fill up with methods that don't belong to specific objects. If you expose the class java.util.Date, then the constructors will be available as:
staticBase.java.util.Date.new
and static methods such as parse are available as:
staticBase.java.util.Date.parse

If the class name is unique enough, it is also available directly from the staticBase root, such as staticBase.Date

These static methods and functions are what I call functions instead of methods.
Instead of invoking them with obj:method, do it like this:

local unixTimeStamp = 123
local dateObject = staticBase.java.util.Date.new(unixTimeStamp)
local dateObject = staticBase.Date.new(unixTimeStamp)
local NewDate = staticBase.Date.new
local dateObject = NewDate(unixTimeStamp)

Any of those would work.

You can also use exposer.exposeLikeJavaRecursively(yourClass) which works the sam but additionally it also exposes all other classes that yourClass interacts with in any way. This can be useful if you want to avoid manually setting up a lot of exposing, but don't use it when dealing with untrusted code.

Annotation based exposing
For untrusted code, or simply when you need more custom configuration, consider using the annotation based exposing instead.

Use one of these methods to expose a class or objects by using annotations:
exposer.exposeGlobalFunctions(object) and exposer.exposeClass(class),

Exposing global function takes an actual Java objects and scans it for method annotations. All methods which are marked as global will be exported to Lua as functions, implicitly bound to the object. This means that you should not use the obj:method() notation of invoking them, just use method().

Exposing a class instead creates Lua methods for objects of the class, and also exposes static methods as function, if they're marked as global.

The available annotations are: @LuaMethod and @LuaConstructor.

@LuaMethod has two optional parameters. "name" gives you the option to expose
the method with a different name. If this is left out, it uses the method name. The second parameter is "global" which can be true or false. True means that the method will be exposed as a function, and false means that it will be an object method. False is the default.

@LuaConstructor similarly takes a "name" parameter.
Note that it doesn't take a "global" , since constructors are never bound to an object; they are always implicitly global.

Overloading
Java supports method overloading based on parameter types, but since functions are first class objects in Lua, it can't support overloading. However, the Kahlua exposer can detect overloaded Java methods and create a limited form av overloading by creating a function that can call any of the overloaded Java methods.
Kahlua then uses the parameter types to try to invoke the method that best matches the supplied arguments.

Here are some concrete usage examples:

public class Vector {
private final double x, y, z;
@LuaConstructor(name = "newvector")
public Vector(double x, double y, double z) {
this.x = x;
this.y = y;
this.z = z;
}

@LuaMethod(name = "length")
public double length() {
return Math.sqrt(x*x + y*y + z*z);
}
}


If this class were exposed by exposer.exposeClass(Vector.class), then Lua scripts could use it like this:

local v = newvector(3, 4, 0)
local len = v:length()
assert(len == 5)

The other method is mostly useful for service-type features:

public class NetworkService {
private final Pinger pinger;

@LuaMethod(name = "ping", global = true)
public int ping(String host) {
return pinger.ping(host);
}
}
exposer.exposeGlobalFunctions(new NetworkService());

Used as:

local pingTime = ping("localhost")
That's it for this time! I could probably go on more in depth with the advanced features in the exposer, but this should already be enough for you to digest.

Time for you get started with embedding Kahlua in your Java application, and let me know how it works out! Feedback is always appreciated.

Friday, May 28, 2010

Getting started with Kahlua2

This post contains some words and acronyms that are good to know about.
  • J2SE - Java 2 Standard Edition - is the Java version that is available on desktops. Kahlua core depends on J2SE 1.4 or higher, while the J2SE extensions require J2SE 1.5 or higher.
  • CLDC - Connected Limited Device Configuration - is a slimmed down version of Java, with many features removed. Kahlua requires CLDC 1.1 or higher to work.
  • Midlet - Application for mobile devices using CLDC (and MIDP, but let's not go too deep).

Kahlua2 is built with a modular design.
This means you can configure your system the way you want it but it also means that there are some basic things that need to be set up to get started.

The platform
First of all, you need a Platform.

The core of Kahlua2 has some dependencies that are platform specific. For instance, Kahlua2 needs to be able to create tables, and the table implementations are different for J2SE and CLDC 1.1. They are backed by ConcurrentHashMap and Hashtable respectively.
For that reason, Platform has the method:
KahluaTable newTable();


Fortunately for you, getting hold of a platform is easy!
Just call J2SEPlatform.getInstance() or CLDC11Platform.getInstance().
These platform implementations are immutable, so don't worry about sharing the platform with others.

Typically you want to use CLDC11Platform for writing midlets and J2SEPlatform for everything else. If for some reason, neither of these platforms suit your purposes, you can simply create your own.

An environment
The global scope in Kahlua (as well as in Lua) is just a regular table. The compiler makes sure to translate any variable that isn't declared as a local to a lookup in the environment table.

If the environment is just a regular table, can we simply create one by calling platform.newTable()? Yes! That works perfectly well.

However, this environment will be empty, so you can't find the standard library functions in it, and most of the scripts you want to run probably needs a couple
of standard functions to do anything interesting.

An alternative way of creating an environment is to call platform.newEnvironment(); This creates a new table, but also populates it with useful stuff. J2SEPlatform for instance loads the environment with BaseLib, MathLib, RandomLib, StringLib, CoroutineLib, OsLib, TableLib and the compiler.

If you're curious, take a look inside J2SEPlatform, it should be easy to follow.

Compiling scripts
Having an environment is not very exciting in itself, you probably also want to run some scripts! Kahlua has two ways of creating closures (or functions) for these scripts. The easiest and best way is to use one of the static methods in LuaCompiler; use loadstring if you have the input as a string, and loadis if you have an inputstream or reader. All closures needs environments, so you need to pass in your environment as the last parameter (or another environment if you prefer).

Note that the compiler will throw a KahluaException (which is a RuntimeException) if the script is malformed. You should be prepared to handle it.

The other option is to load the bytecode directly. This is mostly useful on devices where you dont want to bundle the full compiler, such as limited mobile devices. The solution is to compile the scripts offline, either with the standard luac bundled with Lua, or with Kahluas own Java class called LuaC. The resulting binary file can then be loaded with KahluaUtil.loadByteCodeFromResource.

The thread
With a platform, an environment and some closures, you're almost set to start running your scripts, but there is one more thing needed - a thread to run them in.
KahluaThread t = new KahluaThread(platform, env)

is all you need to create a thread.

Kahlua is concurrent in the way that you can create multiple threads that reference the same environment, but don't try to run scripts on the same KahluaThread from different Java threads! Things would blow up in mysterious and unexpected ways.

The most basic way of invoking your closure is by using thread.call(closure, args...). However, if the closure invocation should throw an error, a RuntimeException will propagate out to the Java code. A safer option is to use thread.pcall(closure, args...) which traps all Lua errors and returns the error information as part of the return values.

For J2SE, there are few more ways of invoking closures, but more on that in a later post.

Java functions
Most applications that use an embedded script engine have a need for the scripts to access domain specific functions. If I were to write a game engine, then my AI scripts might need a way to execute orders. So the opposite of Java invoking Lua scripts is needed; we need to be able to call Java methods from Lua.

There are several ways of doing this in Kahlua, but they all boil down to this core functionality:

interface JavaFunction {
int call(LuaCallFrame callFrame, int nArguments);
}


Kahlua can call any object that implements this interface. This is how all of the standard library in Kahlua is implemented. This is also the only way to do it in CLDC 1.1, due to lack of a reflection API.

The parameters to call is a callframe, which contain the arguments passed to the function, and the number of arguments. The return value of call should be the number of return values, since Lua supports multiple return values. The return values themselves need to be pushed on the Kahlua stack, and there are convenience methods for that in the LuaCallFrame object.

Here is a simple example a JavaFunction implementation intended to be used like this:

local x, y = GetUnitLocation("The unit name")


public class GetUnitLocation implements JavaFunction {
public int call(LuaCallFrame callFrame, int nArguments) {
if (nArguments < 1) {
throw new IllegalArgumentException("Expected at least one argument");
}
Object arg = callFrame.get(0);
if (arg == null || !(arg instanceof String)) {
throw new IllegalArgumentException("Expected a string argument but got " + arg);
}
String s = (String) arg;
Unit unit = unitService.getUnit(s);
callFrame.push(unit.getX());
callFrame.push(unit.getY());
return 2;
}
}

Due to the dynamic nature of Kahlua, there needs to be a lot of boilerplate error checking, but the rest of the implementation is straight forward.

Exposing things
Just having a JavaFunction implementation is not enough - it needs to reach the lua scripts in order to be useful. This can be done in several ways. The first, and most obvious, is to put it in the environment:

environment.rawset("GetUnitLocation", new GetUnitLocation());


The lua script can then call it by just doing:
GetUnitLocation("the unit name")


The lua script can also copy it somewhere, redefine it - "GetUnitLocation" is just a key in a table, and can be manipulated like any other object, so you can do things like this:
fun = GetUnitLocation
fun("the unit name")

or
local oldfun = GetUnitLocation
function GetUnitLocation(s)
print("calling GetUnitLocation here")
return oldfun(s)
end

Conclusion
That's all you need to know to get started with Kahlua. It may seem like there was a lot of steps involved, but each step is really very simple.

For mobile platform targets, there isn't much more to add but for J2SE users there are additional goodies available, which I'll go over some time in the near future.

Friday, May 21, 2010

The history of Kahlua

In 2007, there was really only one alternative if you wanted to run Lua on Java.
LuaJava implements a native binding from Java to the real Lua,
which requires you to bundle a platform specific library with your Java application.

This meant that the integration with Java wasn't as easy to use as it could have been - the memory spaces were completely separated, just like the memory space of regular Lua is separated from C.
It also meant that Lua on J2ME devices was complete out of the questions - you're not allowed to bundle native libraries in J2ME applications, and even if you did,
they would probably be much too large.

So I thought it would be a fun idea to implement a pure Java implementation of Lua to see how similar it would be. I figured that it wouldn't be too much work, since several data types in Lua would be easily mapped by existing classes in Java:
  • strings, numbers and booleans in Lua could be implemented by the String class, Double class and Boolean class respectively in Java. These types are immutable in both languages and have the same behaviour, which
    makes it a good fit.
  • the special value nil was trivially mapped to null in Java.
  • tables can be implemented as a plain Java class.
  • java functions (a Java object implementing a specific interface).

Regular Lua implements sophisticated memory management with garbage collection. This is completely avoided in Java by just using the same memory space and taking advantage of the built in garbage collector in the JVM.

Error handling in Lua were adapted similarly, by simply mapping Lua errors to Java RuntimeException.

Kahlua was originally written with the goal of implementing the basic virtual machine for running Lua bytecode. There were no plans for having a compiler or supporting advanced features such as coroutines. It was aimed at both J2ME and J2SE, and thus only supported the common subset between the two - CLDC 1.1.
Since the project goals were fairly modest, and the actual virtual machine can be written fairly simple and straightforward, it didn't take long until Kahlua could load bytecode compiled from the regular Lua and run it with a limited implementation of the base library.

After the success of having the toy project actually work, I set out to implement more of the advanced features of Lua - Upvalues, coroutines, weak metatables and the complete string library. Getting all this to work took some time, but it ended up quite compatible with Lua.

Soon after that, I started working on a hobby J2SE game project, and thought it would be fun to start using Kahlua as a scripting language there. Kahlua worked nicely there, but it relied on external compilation so we had to either precompile scripts with the commandline luac or make a JNI binding to the lua library. We started with the precompiling, but once we realized that we needed hot reloads,
doing the second approach was unavoidable.

Within the next couple of years after the initial release of Kahlua, several other pure Lua implementations emerged.
First after Kahlua, I think, was Mochalua which was also aimed at J2ME and had more core functionality for directly interacting with MIDP instead of just CLDC, which unfortunately made it harder to use in J2SE. Mochalua was more of a direct port of the original C code than a reimplementation but just like Kahlua, it lacked a compiler.

Not long after Mochalua, LuaJ appeared, which also was based on a port of Lua.
LuaJ had however also implemented/ported a compiler and had a coroutine implementation based on Java threads. This was a quite sophisticated project, with a very modular design. It had a clever solution for supporting both J2ME and J2SE based on a platform interface that both J2ME and J2SE implemented, which I ended up getting inspiration from later on.

It also turned out that LuaJ's compiler could, with a little bit of refactoring, be adapted to fit in with Kahlua, so I forked it and put it into Kahlua and seemingly over night, Kahlua had its own compiler and the JNI solution could be dropped. Thanks LuaJ!

A while after this project, yet another Lua project in Java appeared. JILL, or Java Implementation of the Lua Language wasn't really a new project - Nokia had funded it and it had just recently been open sourced. I haven't really looked into it much, but it seemed to use a similar approach as Kahlua, adapting the implementation to Java instead of doing a straight port.

Now, back to the game project!

We were writing our entire UI system in Lua, with gui and game logic methods implemented in Java and made available to Kahlua. Creating the raw JavaFunction's needed for Kahlua required a lot of low level operations, such as verifying correct number of arguments and the types of the arguments as well as pushing return values, so we decided to implement an easier mechanism for exposing methods.

This ended up as a contrib-library in Kahlua that supported exposing Java classes annotated with @LuaMethod. It handled all of the problems above, which made our lives much easier. In time, this library grew to also include conversion between Lua and Java types. Since Lua only used Double as a numeric type, we needed conversions from the other numeric types in Java. Lists and Maps in Java were converted to KahluaTable.

This was pretty much the last real development for Kahlua, but I started getting other ideas for the further development of Kahlua.

The birth of Kahlua2
I started getting a lot of ideas for how to modernize Kahlua which meant departing from many of the original design decisions, and introducing big incompatibilities with the original Kahlua. So, Kahlua2 was born, this time on github instead of Google Code. The reason for this was unrelated to the actual development of Kahlua - I just wanted to learn git, and investigate if its workflow and features had any real life benefits over Subversion.

I've identified three main categories of design changes.

Modularity
Kahlua didn't differentiate the versions for J2SE and J2ME, which meant that J2SE had to suffer a bit (not much, but a bit). The most significant effect was that most of the math operations were implemented in software, which is both slower and less precise than the built in operations in J2SE.

The solution in Kahlua2 was to introduce a platform interface implemented for both J2SE and J2ME, and creating two separate math libraries, similar to the solution used in LuaJ.

Compatibility
Keeping compatibility with Lua was one of the original goals of Kahlua, and it reached pretty far. Tables in Kahlua had the same behaviour as tables in Lua.
This was done with a complete low level implementation of tables and special handling of doubles (0 and -0 are the same key in lua tables, but in Java they are distinct). The low level implementation was required to achieve a reasonably fast implementation of next().

I meant for Kahlua to be primarily a way of extending Java with a good script language and thus compatibility with Java is more important than compatibility with Lua. Also, keeping the code simple and efficient should have a high priority.
The handcoded table implementation in Kahlua is likely not as a good as the built in maps in Java, so Kahlua2 simply drops compatibility on table keys, removes the next function, and removes support for weak tables. I hope no one will miss these features too much.

Concurrency
Kahlua, and all other Lua implementations I knew of were inheritly single threaded.
Each lua state could only be run by one thread at a time, and states couldn't really share data.
With the loosening of compatibility, Kahlua2 can instead implement tables as a simple facade to either Hashtable (for J2ME) or ConcurrentHashMap (for J2SE),
which means that Kahlua2 suddenly becomes concurrent.

All native datatypes in Kahlua (and Lua) are immutable except for tables, upvalues and the stack itself. The stack concurrency is solved by not letting multiple Java threads use the same KahluaThread at the same time. Upvalues are solved by simply making the set and get-methods be synchronized.

Unlike Kahlua, Kahlua2 separates the script environment and thread. Multiple KahluaThreads can share the same environment, which is the key to concurrency.
The threads are not to be confused with Coroutines, which still work just like in Lua.
Many KahluaThreads can share Coroutines, but only one thread can run a coroutine at a time.

Present day
You can currently find Kahlua2 on github and it's still evolving, although the core ideas for Kahlua2 described above are already implemented. The changes at the point in time mostly consist of bug fixes, refactoring of the compiler (it's still mostly a C port, which fits badly with Java).

That's it for now!
Stay tuned for The History of Kahlua part 2, expected somewhere around 2013 perhaps.

Relevant links:
LuaJava
Mochalua
LuaJ
JILL

Kahlua
Kahlua2

The hobby game, with working title Spaced

Tuesday, May 18, 2010

What's up with String.format

I noticed a while back that Javas own String.format() is slower than I expected it to be. I compared it to the handbuilt version in Kahlua which is built with a naive approach using only Java 1.4 features.

To follow up on this, I wrote a benchmark to see if I can really be sure that it is in fact slower. Writing benchmarks is always tricky and is easy to get wrong. Here's my approach of writing a test.


public class StringFormatTest {
static interface Runner {
String run(String format, Double x, Double y);
String name();
}

static class JavaRunner implements Runner {

@Override
public String run(String format, Double x, Double y) {
return String.format(format, x, y);
}

@Override
public String name() {
return "Java String.format";
}
}

static class LuaRunner implements Runner {
private final KahluaThread thread;
private final LuaClosure closure;

public LuaRunner(KahluaThread thread, LuaClosure closure) {
this.thread = thread;
this.closure = closure;
}

@Override
public String run(String format, Double x, Double y) {
return (String) thread.call(closure, format, x, y);
}

@Override
public String name() {
return "Lua string.format";
}
}

@Test
public void testFormat() throws IOException {
String format = "Hello %3.2f world %13.2f";
Double x = 123.0;
Double y = 456.0;


Platform platform = new J2SEPlatform();
KahluaTable env = platform.newEnvironment();
KahluaThread thread = new KahluaThread(platform, env);
LuaClosure closure1 = LuaCompiler.loadstring("" +
"local stringformat = string.format;" +
"return function(format, x, y)" +
"return stringformat(format, x, y)" +
"end", null, env);
LuaClosure closure = (LuaClosure) thread.call(closure1, null, null, null);

Runner luaRunner = new LuaRunner(thread, closure);
Runner javaRunner = new JavaRunner();

List<Runner> list = new ArrayList<Runner>();
for (int i = 0; i < 20; i++) {
list.add(luaRunner);
list.add(javaRunner);
}
Collections.shuffle(list);

for (Runner runner : list) {
int count = 0;
long t1 = System.currentTimeMillis();
long t2;

while (true) {
t2 = System.currentTimeMillis();
if (t2 - t1 > 1000) {
break;
}
runner.run(format, x, y);
count++;
}
double performance = (double) count / (t2 - t1);
System.out.println(String.format(
"%30s %10.2f invocations/ms",
runner.name(),
performance));
}
}
}



I've done my best to make this test be fair. The obvious first thing to do is to run each test many times, to ensure that cache misses, JIT optimizations, warmups, et.c. are all taken into account. The second obvious thing is to shuffle the ordering of tests to avoid introducing accumulating interference.
For instance, if I were to run the tests as A, B, A, B, ... then any garbage being produced at the end of A would punish B when it ran its garbage collector.

Having run this test suite for a bit, I consistently get these results:

Lua string.format 232.71 invocations/ms
Java String.format 170.07 invocations/ms
Lua string.format 231.61 invocations/ms
Java String.format 170.34 invocations/ms
Lua string.format 232.73 invocations/ms
Java String.format 170.10 invocations/ms
Java String.format 170.11 invocations/ms
Java String.format 171.23 invocations/ms


Why is the Lua version faster? It should be much slower considering:
  1. The Lua version has to go into its interpreter loop as additional overhead.
  2. The Lua implementation of string.format is a naive and short implementation, and not built by the elite development team of Sun.
Perhaps the difference is that the default Java implementation handles a lot of additional use cases such as locale, but that still doesn't really explain it.

Does anyone have any ideas why Javas String.format is slow?