The Art of testing Code
The Twitterverse, various blogs, and some news portals published discussions about a bug in libgcrypt. The code contained a loop which could read past the end of a buffer. The error condition was found by using a test suite. Given the C code base of libgcrypt cases like this can often be found by using the static code analysing features of modern compilers. If you read the ticket concerning the particular overrun bug, then you will notice that it contains more than just the error description. The reason for emotional discussion around bugs are the many ways to find them.
Modern compilers contain a lot of helpful tools to audit your code. Even if the compiler lacks auditing/testing features, you can resort to other tools such as Valgrind (which turned 20 years of age last year). Even without adding secure coding practices to your project, using all available features of your toolchain should be part of your software development process. You don’t have to do this from the start, but at some point your test cycles have to take advantage of these features. Even other run-time environments such as interpreters or other code generators have ways to help you.
Throwing analyser options at your code is not enough. You also need data to feed to your code. Few applications can do without input or output, both always involve some kind of data. If you have reference data sets, then you can use them. With luck your bug reports contain some useful data for checking what should not happen (again). Structured data often passes the basic entry checks. This is why fuzzing is used to create a wide array of test data for checking how your code will handle exceptional states. Randomly creating test data won’t give you complete coverage of your code paths. You will get to test more code paths than with using manually created test sets, but the idea is to test for situations you don’t know yet. The error conditions and exceptions are the things you want to look for.
With all this in mind, let’s go back to the typical bug report. Creating a sound testing infrastructure requires a lot of work. This is especially true for portable code. Running an executable on a single computer is fine, but what about code that supports multiple platforms? Code doesn’t always look the same. Of course the multitude of supported cases in the pre-Heartbleed are of OpenSSL is something your development process should take care of eventually – before the testing begins. Even if your run your continuous integration pipelines and throw data fuzzing at your code, you will still miss cases and states your code can be in. You will probably have to add more code for selectively introducing errors to see what happens next in a specific step of your data processing.
Secure coding always means to add more code. Not every project has the resources to do so. Small teams can do less. Source must be available, too. Free Software is available, but few projects feature a test framework for hardening their code. Keep this in mind when reporting bugs as a security researcher. Be constructive, don’t judge.