Tuesday, May 18, 2010

What's up with String.format

I noticed a while back that Javas own String.format() is slower than I expected it to be. I compared it to the handbuilt version in Kahlua which is built with a naive approach using only Java 1.4 features.

To follow up on this, I wrote a benchmark to see if I can really be sure that it is in fact slower. Writing benchmarks is always tricky and is easy to get wrong. Here's my approach of writing a test.


public class StringFormatTest {
static interface Runner {
String run(String format, Double x, Double y);
String name();
}

static class JavaRunner implements Runner {

@Override
public String run(String format, Double x, Double y) {
return String.format(format, x, y);
}

@Override
public String name() {
return "Java String.format";
}
}

static class LuaRunner implements Runner {
private final KahluaThread thread;
private final LuaClosure closure;

public LuaRunner(KahluaThread thread, LuaClosure closure) {
this.thread = thread;
this.closure = closure;
}

@Override
public String run(String format, Double x, Double y) {
return (String) thread.call(closure, format, x, y);
}

@Override
public String name() {
return "Lua string.format";
}
}

@Test
public void testFormat() throws IOException {
String format = "Hello %3.2f world %13.2f";
Double x = 123.0;
Double y = 456.0;


Platform platform = new J2SEPlatform();
KahluaTable env = platform.newEnvironment();
KahluaThread thread = new KahluaThread(platform, env);
LuaClosure closure1 = LuaCompiler.loadstring("" +
"local stringformat = string.format;" +
"return function(format, x, y)" +
"return stringformat(format, x, y)" +
"end", null, env);
LuaClosure closure = (LuaClosure) thread.call(closure1, null, null, null);

Runner luaRunner = new LuaRunner(thread, closure);
Runner javaRunner = new JavaRunner();

List<Runner> list = new ArrayList<Runner>();
for (int i = 0; i < 20; i++) {
list.add(luaRunner);
list.add(javaRunner);
}
Collections.shuffle(list);

for (Runner runner : list) {
int count = 0;
long t1 = System.currentTimeMillis();
long t2;

while (true) {
t2 = System.currentTimeMillis();
if (t2 - t1 > 1000) {
break;
}
runner.run(format, x, y);
count++;
}
double performance = (double) count / (t2 - t1);
System.out.println(String.format(
"%30s %10.2f invocations/ms",
runner.name(),
performance));
}
}
}



I've done my best to make this test be fair. The obvious first thing to do is to run each test many times, to ensure that cache misses, JIT optimizations, warmups, et.c. are all taken into account. The second obvious thing is to shuffle the ordering of tests to avoid introducing accumulating interference.
For instance, if I were to run the tests as A, B, A, B, ... then any garbage being produced at the end of A would punish B when it ran its garbage collector.

Having run this test suite for a bit, I consistently get these results:

Lua string.format 232.71 invocations/ms
Java String.format 170.07 invocations/ms
Lua string.format 231.61 invocations/ms
Java String.format 170.34 invocations/ms
Lua string.format 232.73 invocations/ms
Java String.format 170.10 invocations/ms
Java String.format 170.11 invocations/ms
Java String.format 171.23 invocations/ms


Why is the Lua version faster? It should be much slower considering:
  1. The Lua version has to go into its interpreter loop as additional overhead.
  2. The Lua implementation of string.format is a naive and short implementation, and not built by the elite development team of Sun.
Perhaps the difference is that the default Java implementation handles a lot of additional use cases such as locale, but that still doesn't really explain it.

Does anyone have any ideas why Javas String.format is slow?

1 comment:

MrCoder said...

I think this one works pretty well for micro benchmarks, while still being ridiculously easy to use (with deviation and other statistics): http://www.ellipticgroup.com/html/benchmarkingArticle.html

Post a Comment