Sunday, February 21, 2010

Mockito vs Mockachino

I wrote down some differences between Mockito and Mockachino, ordered by how important they are for actual mock usage.

Note that I am quite biased since I actually wrote Mockachino, but I'm also a great fan of Mockito and I still prefer Mockito over all other java mocking frameworks. I have tried to be as objective as possible, but if I have missed some important aspect to compare, please let me know, and I'll add it.

The basic syntax
The syntax is definitely one of the most important aspect of mocking, especially when comparing frameworks that are similar in most other regards.
Mockito and Mockachino have slight differences in syntax, with the biggest difference being when stubbing a mock method:

// Mockito
when(mock.theMethod(eq(value1), eq(value2), same(value3))
.thenReturn(theReturnValue);
// Mockachino
stubReturn(theReturnValue).on(mock).theMetod(value1, value2, same(value3));

Mockito has a better flow here: you first type what you want to match on and then what you want to return. Mockito also has a type safe return value, since the compiler will enforce that theReturnValue is of the same type as the return type of mock.theMethod().

There are a couple of disadvantages of this approach:
  1. It actually needs to invoke the mock first, and then undo the side effect it produces. This is actually impossible to do if the mock is actually a spy. What Mockito actually does here is remove the invocation from the mocks call history so it won't be a false match in any later verification.
  2. The mock invocation might actually throw an exception, which will kill the test. The workaround for this is to use Mockitos alternative syntax, which is more similar to Mockachinos. More in this problem later.
  3. It's not possible to use mocks inlined here, since that will confuse the stubbing mechanism, which basically just remembers the latest invoked mock when adding the stubbing.
Mockachino's approach may seem a bit backwards at first but it's generally not much more difficult to grasp than the difference between these sentences:
  1. When you call my by the phone number 08-123 456, I will send a fax reply back.
  2. I will send a fax reply back when you call me by the phone number 08-123 456.
Mockachino is a bit more consistent in its syntax here. Every kind of interaction with the mock (stubbing, adding observers, verifying behaviour) has the same syntax:

// Stubbing
stubReturn(theReturnValue)
.on(mock).theMetod(value1, value2, same(value3));
// Observing
observeWith(observer)
.on(mock).theMetod(value1, value2, same(value3));
// Verifying
verifyOnce()
.on(mock).theMetod(value1, value2, same(value3));
As you can see, they all end with
on(mock).theMetod(value1, value2, same(value3));
which is quite consistent. It also keeps the mock object close to the method call.
Compare it with Mockito, which breaks up this pattern by having additional parameters between the mock and the method call when verifying:
verify(mock, times(1)).theMetod(value1, value2, same(value3));

Clean stacktrace
Both Mockito and Mockachino clean up stacktraces upon errors, which means that the stacktrace will exclude all Mockito/Mockachino stack elements

Mixing argument matchers and plain arguments
When stubbing or verifying a method in Mockito, you may choose to use argument matchers instead of values, which gives you great flexibility. However, you can not mix them. The solution is simple: just add the eq()-matchers for everything that you don't want to match explicitly.

Mockachino has a slightly more friendly approach. It allows mixing plain values with matchers, as long as the plain values are not the default values for their type (i.e. null, false, 0, '\0').

In order verification
Mockito has a couple of bugs (or design choices) when trying to verify things in order. Here is one:

@Test
public void testInOrder() {
List mock = Mockito.mock(List.class);
mock.add("Foo");
mock.add("Bar");
mock.add("Foo");

InOrder order = Mockito.inOrder(mock);
order.verify(mock, Mockito.atLeast(1)).add("Foo");
order.verify(mock, Mockito.atLeast(1)).add("Bar");
order.verify(mock, Mockito.atLeast(1)).add("Foo");
}

This fails, even though it's clear that it shouldn't.
Compare with the Mockachino example below. This works as expected.

@Test
public void testMockachinoInOrder() {
List mock = Mockachino.mock(List.class);
mock.add("Foo");
mock.add("Bar");
mock.add("Foo");

InOrder order = Mockachino.verifyOrder();
order.verifyAtLeast(1).on(mock).add("Foo");
order.verifyAtLeast(1).on(mock).add("Bar");
order.verifyAtLeast(1).on(mock).add("Foo");
}
Singleton ordering per mock object
In Mockito, one mock object can only have one ordering-object. This means that the following does not work:

@Test
public void testMockito() {
List mock = Mockito.mock(List.class);

mock.add("Foo");

InOrder order1 = Mockito.inOrder(mock);
order1.verify(mock).add("Foo");

InOrder order2 = Mockito.inOrder(mock);
order2.verify(mock).add("Foo");
}

In Mockachino, each ordering object is different and holds its own verification state. This works as expected:

@Test
public void testMockachino() {
List mock = Mockachino.mock(List.class);

mock.add("Foo");

OrderingContext order1 = Mockachino.newOrdering();
order1.verify().on(mock).add("Foo");

OrderingContext order2 = Mockachino.newOrdering();
order2.verify().on(mock).add("Foo");
}

Multithreading
Mockito fails with incorrect error messages when trying to use the same mock in multiple threads.
With Mockachino, multithreading is not a problem. You can share mock contexts and even mocks between threads

equals, hashCode and toString
These three methods can be tricky, since they are base methods used a lot by Java standard libraries (Sets, Maps, Lists, et.c.). Mockachino supports stubbing of all these methods, Mockito only supports stubbing toString. Stubbing these methods can be very useful at times, such as using mocks as keys in maps where a correct equals and hashCode are essential.

When it comes to verifying, all these are possible to verify with Mockachino, but it's not always a good idea, since these methods can be called almost anywhere. Also, the correctness of your code should probably not depend on specific calls to these methods.

API that never lies

Mockito uses a generic VerificationMode parameter to the verify methods, but all verification modes do not work for InOrder verification. This is very unclear in the API.
Mockachino instead hides its unsupported modes in the ordering API.

Support for inline mocking
Since Mockito does a lot of dark magic to enable its terse syntax, it's quite frail when you interrupt its expected flow of constructs. Inline mocking is a perfect example of this.
I have on several occasions had the need to let a mock return another mock.

This, and similar constructs fails in Mockito:

@Test
public void testInlineMockito() {
List mock = Mockito.mock(List.class);
Mockito.when(mock.get(0)).thenReturn(Mockito.mock(List.class));
}

The workaround is at least simple:

@Test
public void testInlineMockitoWorkaround() {
List mock = Mockito.mock(List.class);
List secondMock = Mockito.mock(List.class);
Mockito.when(mock.get(0)).thenReturn(secondMock);
}

No workarounds are necessary with Mockachino since it does not rely on ordering of calls.
Thus, the equivalent in Mockachino works fine:

@Test
public void testInlineMockachino() {
List mock = Mockachino.mock(List.class);
Mockachino.stubReturn(Mockachino.mock(List.class)).on(mock).get(0);
}
Calling mock methods when verifying
Another interesting case is this. Due to Mockitos design choice of letting calls to verify and stub apply to the most recently invoked mock method, the following case breaks, since verify will actually try to verify mock.size() instead of mock.add()

@Test
public void testMockito() {
List mock = Mockito.mock(List.class);
mock.add(0);
Mockito.verify(mock).add(mock.size());
}

It works fine in Mockachino, due to a different design choice:

@Test
public void testMockachino() {
List mock = Mockachino.mock(List.class);
mock.add(0);
Mockachino.verifyOnce().on(mock).add(mock.size());
}

Deep mocks
Update 2010-03-13 - Both the latest Mockito and Mockachino have deep mock handlers that behave in the same way.

Time ranges Mockachino supports verifying that interactions with a mock happened in a specific time range.
See additional-features-of-mockachino for a more detailed explanation.
Mockito does not include this feature.

7 comments:

Szczepan Faber said...

Hey,

Here's my feedback:

1. You mention many features of Mockito so the comparison is not likely to be up-to-date. Example: deep mocks.

2. I don't get the multithreading comparison - Mockito also allows to share mocks. If you found a bug could you submit it?

3. Good idea: Mockachino instead hides its unsupported modes in the ordering API

4. Several things you mention I consider as disadvantages of Mockachino but that's ok - we can have our own biased opinions :) If I ever have time I will replay back.

Kristofer said...

1. I thought I was fairly up to date, but I can rerun some test code for deep mocks and be better at describing exactly what I meant.

2. Sure I can add the test case I used for multithreading. That's probably a good idea in any case. Basically I found that Mockito works fine for just using mocks multithreaded, but fails if you also try to verify/stub multithreaded.

4. I definitely see your point here. I think there's a tradeoff for most if not all features. Most specifically the when().thenReturn() vs doReturn().when().

It would be fun to get some more feedback from a real mock framework expert :)

dm3 said...

I love the fact that you have managed to distill the inconveniences that have been bugging me for months with Mockito. Hate you for fragmenting my choice in mocking frameworks :)

Mokachino approach to stubbing will probably make default thenThrows stubs possible (problem described http://dm3.github.com/2010/09/18/repeated-stubbings-with-mockito.html). Would love to see this approach implemented as an alternative in Mockito.

Kristofer said...

Mockito has actually supported the doThrow(...).when(mock).method(...) syntax for a very long time. And since I wrote this blog, I've also added the when(mock.method(...)).thenReturn(...) syntax to Mockachino. (It was a very small patch, so I thought it was worth it)

So, this specific thing shouldn't affect your choice of framework so much.
That said, I always welcome people to try it out, I am usually very quick to respond to bug reports and feature requests. :)

dm3 said...

Yeah, my bad. This makes the differences between the frameworks only interesting to *really* sophisticated mockers :)

Also, you have a typo in "Mockachino supports stubbing of all these methods, Mockachino only supports stubbing toString."

Kristofer said...

Thanks. Fixed!

om3ga said...

Thanks for the informative post on Mockito. If you are willing to discuss more on unit testing with mockit and unit testing for other languages

can check out Mockito Mock forum which is exclusively for unit testing for

different languages. It is a upcoming forum and needs support to make it strong so we can have forum where people can share their knowledge on

Testing.

Post a Comment