20 May 2007

Testability vs. Design (again)

I'm sure TypeMock is a fine piece of software, but I just cannot agree with their notion that API design and testability are in conflict. Their new case study includes the statement:

The key benefit we get from TypeMock is having the ability to fully unit-test the code without impacting the API design. [...] For us, the API is part of the deliverable. We need to make it fairly easy to consume and can't have the architecture of the solution overshadow the usability of the API.

In the other corner we have Michael Feathers who has been running a tutorial called API Design as if Unit Testing mattered. Michael is one of the most interesting people I know on the Agile circuit, although he doesn't make a fuss about it. To quote him:

Have you ever tried to write a unit test for a class that uses an external API? An API that is not mockable? Testing seems to be the last thing on our minds when we design APIs. It isn't that we don't test our APIs, we just don't make it easy to test code that uses our APIs.

His point is that, for library designers, even writing tests for your API is not enough to produce something that's really useable, you have to try writing tests for the code that will use your API. Or, as Michael Puleio wrote,

If your API is hard to unit test, how in the heck are your users going to be able to use it?


dorko said...

It's important to have good programming but at the same time if nobody can use it then what's the point? That's why we incorporate design AND usability together...

Steve Freeman said...

Agreed, but the point is that usability and testability should not be in conflict.

Someone correct me, but I can't think of another technical discipline where testability is regarded as an "extra".

Scott said...

Agreed - unit testing is a form of "use" and in most cases the tests shouldn't look that different from typical API usage so in my experience, testability has a strong positive correlation with usability.

That said, a good mock framework is handy when you are dealing with API authors who think that way. ;-)

Nat Pryce said...

I guess the crux of the issue is in the words "easy to consume". What does that mean? Easy to learn? Or easy to adapt to new, unanticipated situations.

For example, the java.io API is seen by beginners as very complicated, compared to the IO APIs of C or Python. But it's use of the Decorator and Chain of Responsibility patterns makes it easy to apply in different situations and adapt different underlying transports to the java.io stream model.

TDD with mock objects drives an OO design towards one like the java.io API, where design focuses on discovering common patterns of communication between objects and the end-to-end system behaviour is defined by composing objects instead of writing algorithmic code. That makes code more malleable by experienced programmers but, arguably, makes it harder for newcomers (to the codebase or OO programming itself) to learn.

Colin Jack said...

I don't write API's at the minute so maybe take this with a pinch of salt.

If you want to test at a really fine grained level using interaction testing (or even just stubbing) then I would imagine that the complexity of the API could rise. You'd end up exposing enacapsulated details without necessarily improving the overall API.

Could be missing something here though.

Steve Freeman said...

Not sure how to respond to your comment. I don't think that's been my experience.

One thing, though. Designing a published API has different constraints from designing an internal API. There's a lot more to think about, and some of what we do might not be as relevant.

Colin Jack said...

Yeah, its been a long time since I did any work on public API's so I could be well off.

I know in the .NET world a lot of people want API designers to expose more than they do and give us more capability to hook in and replace parts where appropriate.

Thats fine but I'm just not 100% sure that testing is always going to make you expose the most useful parts in the simplest manner, leaving you with the risk that your API is very flexible and so on but harder to use than your users are really going to want.

Don't do a lot of reading about API design these days though so I could be well off.