A minimalist unit test framework for JavaScript

When I started doing some serious JavaScript programming for a project based on Mozilla's XULRunner, I figured it would be easy to find a suitable unit testing frameworks -- something similar to JUnit, only for JavaScript. After some searching, I found a number of frameworks. Every one, however, had too many features.  Most relied on browser features of some sort that XULRunner just doesn't have. No doubt if I were doing web-site development, I'd be only too thankful. But I'm not and I needed something.

As a result, I created a very very simple framework that uses only standard ECMA script. The framework is called "xulUnit", at the moment, but it should be emphasized that it really has nothing to do with XUL except for being compatible with XULRunner.

An example


Once the framework is loaded, one adds a set of test objects. The function xulUnit.addTest accomplishes this. This function takes two arguments. One is the name of the test and the second is the test object. Below we add a test object that tests the basic functionality of the framework.
This is file testXulUnitItself.js This test object contains four test methods. (Test method names must start with the letters "test".) If we run the framework at this point, we get as output.
Now this is not pretty, but then it's not meant to be. You can see that the four test methods are run, bracketed by setUp and tearDown. Each test has one of three results.

• Pass. If it returned normally.
• Fail. If some assertion fails.
• Error. If an exception is thrown.

A summary of how many tests had each result is printed at the end.

A complete example

Now let's see a complete example of using xulUnit.

This is file xulUnit.html

• Line 5 loads the framework. (The file xulUnit.js can be found at the end of this article.)
• Line 7 loads the test file we saw earlier. 
• Lines 10-14 redefine xulUnit's xulUnit.printlnfunction. All output from xulUnit uses this one function. The default definition uses JavaScript's dump function, which doesn't work well in my browser. This redefinition sends output to the html document. Another useful redefinition is
• Finally line 20 runs the tests.

The API

The API is summarized as follows

• xulUnit.addTest( testName : string, testObject : Object ) -- add a test object.
• xulUnit.runTestsNow() --  run all tests that have been added since the last time it was called.
• xulUnit.assertTrue( q : boolean, message : string ) -- fail unless q is true. (Second argument is optional.)
• xulUnit.assertFalse( q : boolean, message : string ) -- fail unless q is false. (Second argument is optional.)
• xulUnit.assertNull( q : boolean, message : string ) -- fail unless q is null. (Second argument is optional.)
• xulUnit.assertNotNull( q : boolean, message : string ) -- fail unless q is not null. (Second argument is optional.)
• xulUnit.assertSame( expected, actual, message : string ) -- fail unless expected===actual. (Third argument is optional.)
• xulUnit.assertEquals( expected, actual, message : string ) -- fail unless expected==actual. (Third argument is optional.)

Known problems.

• Stack traces only work in Mozilla based implementations of JavaScript.
  

Here is the framework.
This is file xulUnit.js

Testing concurrent code doesn't

Here is an example that I used in my concurrent programming course this winter.

I'd originally intended to illustrate how easy it is to create race conditions. The example I ended up with illustrates a more important point: how futile it can be to try to discover race conditions through testing.

Here is the code in Java

public class Race {
 volatile int x = 0 ;
 
 public static void main(String[] args) throws InterruptedException {
  (new Race()).go() ;
 }
 
 public void go() throws InterruptedException  {
  final int ITERATIONS = 1000 ;
  Thread p = new Thread() {
   public void run() {
    for( int i = 0 ; i <ITERATIONS ; i++ ) {
     ++x ; } } } ;
  
  Thread q = new Thread() {
   public void run() {
    for( int i = 0 ; i <  ITERATIONS ; i++ ) {
     --x ; } } } ;

  
  System.out.println( "The initial value of x is: " + x ) ;
  
  // The two new threads start to run.
  
  p.start() ;  q.start();
  
  // Now both new threads are running or done.
  // The main thread waits until both are done.
  
  p.join() ;  q.join();
  System.out.println( "After "+ITERATIONS+" increments and "+ITERATIONS+" decrements" ) ;
  System.out.println( "the final value of x is: " + x ) ;
 }
}

This was intended to show the race between incrementing and decrementing. There are one thousand increments and one thousand decrements of x. You might expect that the final value of would be the same as its initial value, which is 0. But increment and decrement are not indivisible actions. Each increment consists of a load of x, an add, and a store of x; decrements are similar. If an increment and a decrement both load before either stores, the first action to store will have its result overwritten by the second. Given a thousand chances, it seemed likely that at least one increment or decrement would be lost. Unless the number of lost decrements and the number of lost increments just happen to be equal, we should see a nonzero final value for x.

I gave it a try:

The initial value of x is: 0
After 1000 increments and 1000 decrements
the final value of x is: 0

What gives? Perhaps not enough chances for the race to manifest. I tried ten thousand increments and decrements:

The initial value of x is: 0
After 10000 increments and 10000 decrements
the final value of x is: 0

OK, how about one million:

The initial value of x is: 0
After 1000000 increments and 1000000 decrements
the final value of x is: 0

At this point, I was starting to wonder if my understanding of Java was correct or whether my virtual machine had translated the conceptual load-increment-store sequence to a single --atomic-- instruction.

Ten million:

The initial value of x is: 0
After 10000000 increments and 10000000 decrements
the final value of x is: 4073375

Finally the result error manifested itself.

In retrospect it was obvious why my testing strategy was so ineffective. The start up time for the q thread was so great that the p thread was finished before the q thread even started.

The moral. When concurrency is in play: if even obvious errors that we know are there aren't easily revealed by testing, what chance can we have against subtle errors we don't know are there?