August 10, 2007

Why Testing code should be Laissez-faire

I've been working on the HFA Library some more, and got to the point now were I really want to start testing it deeply. Since HFA is in Haskell, the weapon of choice for testing pure code is QuickCheck. The interesting bit about QC is that test data is fundamental random. This is great for two reasons.

1) It forces you to actually test for _general_ properties of objects/functions, rather than specific properties of certain instances of those structures, like a unit test does
2) By Generating Random Data, you don't have to worry about being too nice or too mean to your program, as with unit testing.

I'll state it now, for the record, Unit Testing is great. I wouldn't test my Java (or soon enough, Scala, ^_^) programs with (mostly) anything else. However, when you have referential transparency on your side, QuickCheck is a superior system.

The real brilliance of QuickCheck though, is that it is extensible enough that you can define new "checkable" items, that is, instead of having to use the standard checkable types when checking a function, you can define how QuickCheck can generate random data for a particular type by making it an instance of the Arbitrary class, which is defined by QuickCheck. This means that, as long as you can define a couple of methods for your datatype, it is damn near trivial to have QC generate examples of your datatype and test them quickly.

Why is this good? Well, consider you're writing unit tests for your code. You've been intimately involved with this mangled peice of imperatively worded text for days and weeks. You know every inch of it, and you have in your mind the next new feature you want to add. It is not uncommon (at least for me) to begin writing and toying with the code in your mind, figuring out where potential bugs might be. As a human being, you are typically not looking to make things harder for yourself than needbe. So maybe, when you're writing those unit-tests that will guide your programming to ensure that the code is written correctly, you -- maybe subconciously, maybe not -- decide to make those few unit tests a little bit easier on the part of the code that you think is more fragile. I'm guilty of this, certainly, and I'm sure if you're honest with yourself and you've developed a good bit of code with the test-first methodology (which I like only mildly less than the test-shortly-after-you-realize-you're-supposed-to-write-tests-first methodology), that you would find that you've done it too. QuickCheck fixes that wagon pretty quickly, you don't get to reason about how hard some tests will be on the code, QuickCheck enforces a "Hands Off" or "Laissez-faire" (I'm french, and I like history, sue me.) form of testing. You don't get to decide what is tested, just what the result should be, which is how it _should_ be done. I _shouldn't_ be thinking about what data I want to test, I shouldn't have to write all the test-data, ideally, I should only have to say, "minimizing a DFA twice is equivalent to minimizing a DFA once" or "if the regular expression foo is generated by DFA f, and the expression foo' is generated by the minimized version of f, then foo == foo' for all f::DFA." I guess the point is, the computer is unbiased, it won't be nice to the code, it won't be mean to it, it'll be fair. Is that to say that coders will be biased towards or against their code? Yes, it is, we spend alot of time with these projects, we develop a vested interest in seeing them work, finding out that you did something wrong can be discouraging. Granted, small things may not devastate you, like using the wrong function name, or misplacing a variable. But if you're unit test catches a major flaw in the structure of a program you designed, that represents alot of work that just got blown to peices. At the very least, if not for your pride, testing systems like QC are good for your productivity, they allow you to test a (potentially) huge number of arbitrary cases every time you run your program. Heck, you could even have QC running live against your code in the background, telling you in real time what tests your failing, what cases gave you failures, etc. All of that is Automatic and Vital data, its 24/7/365 testing of your code, for free. Slowly building assurance that your code is, in fact, doing the right thing on all inputs.

By the way, Yes, I know many, many people have written about QuickCheck before, and about how wonderful it is, but I think it's always worth saying again. Good Ideas deserve to be talked about, QuickCheck is a _very_ good idea.

6 comments:

Unknown said...

I see you're talking about soon-to-be-coding in Scala, and to look forward to unit test it.

Well, you'll probably be overjoiced to know that you can also Check your scala.

It's by no mean as stable and robust as QC, but you'll probably find ScalaCheck (http://code.google.com/p/scalacheck/) interesting ;)

Unknown said...

I think QuickCheck would deal with another sort of testing problem that you don't mention.

I have a different problem from you when I'm testing, I don't find myself unconsciously willing the code to pass. I've gotten quite used to testing/debugging taking twice as long as the original coding, and I will sit down and put together a suite of tests to test everything I can think of.

My weakness is that I take short cuts with the test data. I'll run a whole bunch of different tests with the same data because it's more straight forward to do it that way than create a whole lot of different data for each test. I've been bitten in the past because of this ("why didn't I think to try the test with blah data?"). The randomized data used in the tests would get around this problem. Great idea.

kowey said...

You're French? Are you in #haskell-fr? And on the francophone mailing list?

Joe Fredette said...

@kowey

I'm french in name only, I don't speak it (more than a few sentence hither and thither). My grandfolks are French Canadian, so I'm a good 3/4's french (my mom's only part french.)

So, no, I'm not on haskell-fr and on the francophone mailing list. Though someday, after I have found time to learn french fluently, I hopefully will be.

@masklinn

Nifty!

@simon

Ah- thats a good point, I've done that a few times too.

7894789452789359958734809534890543089 said...

Hey, good ideas DO deserve to be talked again. And again is relative, I am new to Haskell, I am learning it by myself, and had not came across QC until now. I am starting to document my road with Haskell in my blog, and I certainly will add a post or two when I start to use QC (I don't have yet code to test, I learn by doing baby steps).
Also good ideas need to be repeated to spread the word!

Anonymous said...

This is great info to know.