This is going to sound lame, but stars are pretty good. I dread making my Miniposts plugin available in the wordpress plugin listing because my users will suddenly have a mechanism to tell the public what they think of my software. The bugs that I’ve been allowing to linger, because they’re outside my use cases will suddenly be important.

Forget coverage, forget visualization, forget everything complex. A fair system that allows users to honestly comment on a piece of software is probably more than enough to tell if users think a piece of software works well. At the end of the day, it’s the users perspective that matters most.

You’re right, the key thing is nailing down the right metrics.

It’s true that code coverage (for example) doesn’t necessarily imply any particular level of correctness or stability. But, it does give an indication of how much effort is put into testing.

There are lots of metrics and qualifications in the real world that don’t guarantee anything, but in the absence of any other information, they can still be helpful to ensure quality. For example, if I go to a restaurant which has a Michelin 2-star rating, I might still get food poisoning; but relative to some of the restaurants near my house the chances are probably lower.

You’ll never get everyone to agree – but I’d really find something along these lines useful.

Some of the problems come from what makes code attractive. You gave some samples – but some of those areas are heavily debated across the industry. Something like Test Coverage is hard to visualize in a complex way. I can currently use a tool like NCover and see a visualization of my code in the test coverage sphere, and I find that useful . . . but it has to be considered carefully. I can have nearly 100% coverage – but still not have true stability, and worse, the code could be extremely fragile or just not meet requirements at all.

I know that you’re not suggesting using a single metric to form a visualization. But, I think that greatest challenge would be to find the right combination so that the visualization really means something. If we can’t trust the visualization it becomes more of a fun thing to have, but not actually useful for really fixing software.

But, every day I use source code analysis tools, test coverage tools, profilers and unit testing. There are also tools that will discover the level of testability that your source code offers. If these sorts of things could be combined together to display a picture, it would be pretty darn cool.