When mocks fail

A mocking framework has three fundamental features:

1. Makes it easy to stub behaviour to make your test code work.
2. Makes it easy to verify that some behaviour has taken place.
3. Makes it easy to debug a broken test.

Today I'll focus on the third part.

When a test fails, it either means that your code is wrong or your test is wrong.
Either way, you want to find the cause of the problem as quickly as possible.

To assist with this, mocking frameworks try to give you as much relevant information as possible when a verification fails.

EasyMock usually says something like:

@Test
public void testFailEasymock() {
 List mock = createNiceMock(List.class);
 expect(mock.add("Hello")).andReturn(true);
 replay(mock);
 mock.remove("World");
 mock.remove("Hello");
 mock.add(null);
 mock.add("World");
 verify(mock);
}

java.lang.AssertionError:
Expectation failure on verify:
  add("Hello"): expected: 1, actual: 0

which tells you that add was expected, but never called.

Mockito is a bit nicer and shows what was actually called instead:

@Test
public void testFailMockito() {
 List mock = mock(List.class);
 mock.remove("World");
 mock.remove("Hello");
 mock.add(null);
 mock.add("World");
 verify(mock, atLeastOnce()).add("Hello");
}

Argument(s) are different! Wanted:
list.add("Hello");
-> at ordering.MixedOrderTest.testFailMockito(MixedOrderTest.java:95)
Actual invocation has different arguments:
list.add(null);
-> at ordering.MixedOrderTest.testFailMockito(MixedOrderTest.java:93)

Expected :list.add("Hello");
Actual   :list.add(null);

Which is more helpful than EasyMock.

However, Mockito only suggests the first invocation of list.add as the actual invocation instead of the best matching.

Mockachino goes a step further and suggests the three best matching calls, sorted by how well it was matched:

@Test
public void testFailMockachino() {
 List mock = mock(List.class);
 mock.remove("World");
 mock.remove("Hello");
 mock.add(null);
 mock.add("World");
 verifyAtLeast(1).on(mock).add("Hello");
}

se.mockachino.exceptions.VerificationError: Expected at least 1 call but got no calls
Method pattern:
add("Hello")

Near matches:
add("World")
    at se.mockachino.order.InOrderTest.testFailMockito(InOrderTest.java:116)
add(null)
    at se.mockachino.order.InOrderTest.testFailMockito(InOrderTest.java:115)
remove("Hello")
    at se.mockachino.order.InOrderTest.testFailMockito(InOrderTest.java:114)
... skipping 1 calls

    at se.mockachino.order.InOrderTest.testFailMockito(InOrderTest.java:117)

As you can see, both Mockito and Mockachino show where the verification call was made,
and where the actual matches were made. This is really helpful if you run the tests in an IDE, just click on the lines to jump to the corresponding line in the code.

It's clear that EasyMock shows too little information here, but how much is too much?

Is the simple approach of Mockito to just show the first partially matching method call sufficient? Yes, it probably does cover most of the cases but I think it would be improved without adding any noise if it tried to do some smarter matching.

I think it's equally likely that a mock called the wrong method but with the correct parameters compared to calling the correct method but with the wrong parameters. I think the approach of showing multiple suggestions is a good one.

In conclusion I think that both Mockito and Mockachino do a good job of reporting failed verifications, and that Mockachino may have an advantage in debugging some use cases where the mockito suggestion is the wrong one.