dotCover – Inaccurate or Misunderstood?

I’ve recently been using dotCover, a tool recently introduced by JetBrains to get real-time coverage of my application code. Coverage is something valuable to developers that use Unit Testing to get an idea of how well their tests cover their application. Overall I find the integration with Resharper and within the IDE a very valuable feature, but I’ve come to realize that you can’t use dotCover for precision coverage, and even then it leaves some interesting problems to be solved.

An interesting question to pose first: what is covered? My definition of covered is a test that exercises all branches of logic within a block of code. For instance, given this code:

   1:      public static Color ColorBasedOnValue(int value)

   2:      {

   3:          return value > 10

   4:                  ? Colors.Red

   5:                  : Colors.White;

   6:      }

This is pretty straight forward. If the value is greater than 10, return Red, otherwise, return White. There are two, possibly three tests to write for this.

1. The method returns Red when greater than 10.
2. The method returns White when less than 10.
3. The method returns White when exactly 10. This might be excessive to some people, but I’ve been burnt a few times by > vs. >=.

A coverage tool can easily figure out if parts 1 and 2 have been exercised. 3 is more of guarding against mistakes than anything – a good TDD’er might write #3 to ensure they use the greater-than-equal-to rather than greater-than.

So theoretically, a coverage tool should never say 100% covered unless case #1 and #2 are satisfied. Indeed, this is the case with testing tools like NCover.

Imagine my surprise when dotCover reported 100% test coverage despite case #1 only being tested.

   1:  public static class Colorize

   2:  {

   3:      public static Color ColorBasedOnValue(int value)

   4:      {

   5:          return value > 10

   6:                  ? Colors.Red

   7:                  : Colors.White;

   8:      }

   9:  }

  10:   

  11:  [TestFixture]

  12:  public class Tester

  13:  {

  14:      [Test]

  15:      public void ShouldReturnRedIfGreaterThan10()

  16:      {

  17:          var value = 20;

  18:          var result = Colorize.ColorBasedOnValue(value);

  19:          Assert.AreEqual(Colors.Red, result);

  20:      }

  21:  }

dotCover alone told me that the Colorize class was 100% tested.

More baffling was that the coverage highlighter highlighted code that wasn’t exercised:

This seemed totally wrong compared to everything I’ve ever known! This lead me to discover something about dotCover: It is a statement coverage tool. A ternary operator (The ?: operator) is a single statement, or at least that’s how dotCover is treating it. If however, we change our ternary operator to an if else block with two separate statements, dotCover gives the results I would expect:

And the coverage highlighter behaved as expected:

This made a little sense to me, but non-the-less it makes me question the coverage of my application now. dotCover failed to cover a very simple condition which affected the exit condition of the method. This lead me to another question: are there any other cases where it thinks code has been exercised that really hasn’t?

This has lead me to think that dotCover is still a useful tool, but only for a generalization. I would still like a coverage tool that can ensure every exit condition of a method is tested.

I’d like to see some improvement on this in the later versions of dotCover. If this issue clears up in a later release of dotCover, I will write a new post about it.