Wednesday, January 27, 2010

Unit Testing - Part I

Introduction




Unit testing matters because (a) your code will be more reliable, especially when your code is updated over time and (b) you will reduce programming hours over time. It is tempting to solve the problem and hack a solution together as quickly as possible, doing ad-hoc testing as you go. Taking the time to unit test may add approximately 15% of initial coding time.

A common scenario is that everything is crystal clear when the initial coding is done. However, we live in a dynamic world and things change. So when adding a feature months later, all of a sudden it becomes a lot harder to verify if everything is working properly. You may spend a full day on a silly bug that could have easily been pointed out by unit tests. This is frustrating and not sustainable over time. The investment in Unit testing will earn back those 15% of extra effort you put in initially, quickly.





How it works

Let's consider a simple black box model. A function has input parameters and as output it has the returned result. In addition, an exception can occur, in which case no output is returned. This is diagrammed below.


 

RawDev make the setup of unit tests as easy as possible by using a fluid interface so that you can typically fit a test on one line of code. The time it takes to do unit tests now solely depends on how deep you are going to test. Typically you would like to produce fewer tests that cover as many as possible scenarios. By nature certain functions require more testing based on how critical they are. Tests are just like code, you have to maintain them over time.

The fluid methods to setup your unit test are:
setTitle(string $title); #optional, but useful, the title is displayed when a test fails
setInput(mixed $param, [$param 2 ...]); #sets the input parameters
setOutput(mixed param); #sets the expected returned result
setExpectedException(string $type); # sets the expected RException type

function setInput($input[, $...]) [fluid]

function setOutput($output) [fluid]

function setExpectedException($expecteException) [fluid]

function setTitle($title) [fluid]


The final test execution method is:

bool function test()

Example


Consider a simple function that returns the sum of multiple floats or integers based on two or more input parameters. The function sum is displayed below:

<?
function sum() {
  $params = func_get_args();

  if (count($params) < 2) throw new RException('math_too_few_params', 'Two or more parameters please.');

  $sum = 0;
  foreach($params as $param) $sum += $param;

  return $sum;
}
?>

My test strategy is to quickly test 0, 1, 2 and 3 params and see if the right output is produced. See the RawDev code below. In addition, for demo purposes I include a test for the invalid output, an expected exception that didn't happen and an exception that wasn't trapped.




RFunctionUnitTest::construct('sum')->setTitle('Zero Params')->setInput()->setExpectedException('math_too_few_params')->test();
RFunctionUnitTest::construct('sum')->setTitle('One Params')->setInput(1)->setExpectedException('math_too_few_params')->test();
RFunctionUnitTest::construct('sum')->setTitle('Two Params')->setInput(1, 2)->setOutput(3)->test();
RFunctionUnitTest::construct('sum')->setTitle('Three Params')->setInput(1, 2, 3)->setOutput(6)->test();
RFunctionUnitTest::construct('sum')->setTitle('Invalid Output')->setInput(1, 2)->setOutput(4)->test();
RFunctionUnitTest::construct('sum')->setTitle('Expected Exception Incorrect')->setInput(1, 2)->setExpectedException('math_too_few_params')->test();
RFunctionUnitTest::construct('sum')->setTitle('Untrapped Exception')->setInput(1)->setOutput(1)->test();
Output:

....X?E

X: Invalid Output              : Value is [3] but should be [4].
?: Expected Exception Incorrect: Exception [math_too_few_params] was expected.
E: Untrapped Exception         : Two or more parameters please.

Conclusion

RawDev makes it easy to define the expected output (result/exception) of a function based on the input. It also has an easy way of labeling a test and displaying the results of many tests. What is not yet incorporated is (a) the output of non-scalar variables such as hashes and objects (not such a big deal) and testing of object functions that change the state of an object. These topics will be added and discussed in the near future.

Links

Exceptions
Fluid functions
Function Test API Doc

6 comments:

  1. Because PHP allows functions to accept references as arguments, write to global variables, and perform I/O, your black box analogy is incorrect.

    How do you test that what was sent to the browser contains X? How do you check to ensure your function modified the global state correctly, or even more importantly, ensure that the state is cleaned up before the next test is run?

    ReplyDelete
  2. Thanks Andrew. This is the first iteration and as I state in my conclusion a function can also impact object state. Thanks for pointing out that it can also impact global state as well as the state of other objects and global state if you will. This functionality will be added but I envision that it is not needed most of the time.

    At this point I envision that object State can be checked with a function such as $this->setState(array) the array would have the values that you expect changed.

    There will also be a setOutputCallback through which you can check whatever you want in complex scenarios, but obviously this is a pain in the ... because you have to create a function, maintain it etc.

    Prepare and cleanup methods should be available in the future as well.

    RawDev does not seek to have the academically most correct and most-featured unit testing. The functionality for unit testing will (a) adhere to the RawDev Principles such as described in: http://blog.bokenkamp.com/2010/01/about-rawdev.html and (b) in the first phase of this project it will be optimized to (b1) building the RawDev framework with tests to get to the first goal of a comprehensive data independent model and (b2) unit testing of applications that use the RawDev framework.

    ReplyDelete
  3. > RawDev does not seek to have the academically most correct and most-featured unit testing. ...

    I'm not sure I follow what "setOutputCallback" would do, and how it would check "whatever you want." And, though I agree that maintaining a function is a pain in the ..., so is maintaining unit tests, and I don't see how a "fluid" interface, or any of the things you mention makes that easier.

    You might think about striving for automated testing for simple functions like that of `sum' using the principles of quickcheck (http://www.haskell.org/haskellwiki/Introduction_to_QuickCheck) and instead spend time to simplify setup and teardown of HTTP requests, responses and assertions about them, as an example.

    For instance:

    RHttpUnitTest::constructRequest('/within/the/box/')->setTitle('Where Ya At Has It')->setInput(array('POST' => array('x' => 10, 'y' => 15)))->assertResponseBodyContains('YES')->test().

    Though, as you can see here, the "fluidness" sort of makes this impossible to read, and you'll have to litter your code with array declarations and things anyway... but I digress.

    ReplyDelete
  4. >I'm not sure I follow what "setOutputCallback" would do, and how it would check "whatever you want." And, though I agree that maintaining a function is a pain in ...

    setOutputCallback would get a handle to the Test and
    can check global variables as well as internal class variables as well as any other class variable that's somehow linked... This is useful for larger organizations with their own test teams ...

    RawDev makes it easier to do function unit tests because you don't actually have to trap errors, compare complex objects or learn 200 assert functions ...

    I will report on the practicality and my recommended best practices as I will be eating my own dog food in the upcoming months.

    Once we will get to UI testing I like your suggestion.

    As far as the easy to read factor that may be personal style. Alternatives, dependent on the situation, are setting variables on different lines, or splitting it up into 2 or 3 lines, or just using one line per function in which case you prefer not to use the return param of a fluid function ...

    As far as automated test I think you just get what you pay for. They can test certain standard behaviors but unless you can describe the behavior of the function they cannot actually predict what to test. Although I should really read your suggested article before I open my big mouth on this ;-)

    ReplyDelete
  5. > setOutputCallback would get a handle to the Test and
    can check global variables as well as internal class variables as well as any other class variable that's somehow linked... This is useful for larger organizations with their own test teams

    Why do they need an instance of the Test to get access to global variables? Are you expecting to execute all of this code in a sandbox of some sort?

    > RawDev makes it easier to do function unit tests because you don't actually have to trap errors, compare complex objects or learn 200 assert functions ...

    And instead, you have to learn 200 methods in the fluid interface.

    So, how do you expect to provide a robust testing framework with only something as powerful as assertTrue (as your example shows)? Are you expecting that instead of providing methods such as assertContains($haystack, $needle), you'll make programmers do something such as assertTrue(strpos($haystack, $needle) !== FALSE))? Doesn't that seem problematic to you? You're allowing testing to become more error prone than it already is.

    And how do you not have to trap errors, and what does that mean exactly? Are you catching all exceptions that happen and reporting them as failures or something? Are you allowing division by 0? I mean...

    ReplyDelete
  6. G.

    A lot of the answers to your questions are in the RawDev sample and source code.

    There are only 5-6 fluid functions for Function Unit Testing. And I don't see why we would need more at this point.

    In concept I much prefer assert assertTrue(strpos($haystack, $needle) ) because it is more MODULAR so that not every php function needs to become an assert function. With that being said you are raising a point that the current implementation does not support assertContains since this needs a handle to the actual output result. I need to think about this more.

    The testing results will look like:
    ....EX?....

    .=OK
    E=Exception
    X=Failure
    ?=Invalid Expected Exception

    All the failed tests will in addition show detail in terms of the title of the test and the specific error message.

    It's all in the sample RAWDEV_HOME.'/Samples/Core/Exception.php.

    ReplyDelete