The most important metric for comparing fuzzing approaches is the amount of flaws the method finds. A simple approach, like random fuzzer, requires virtually no investments, but they can only find 10% of the vulnerabilities hiding in the software. A mutation-based fuzzer can find around 50% of the flaws. Once again the tool investments are minimal, but cost of using the tools and integrating them into the development processes are a lot greater. A fully model-based fuzzer can find as much as 80-90% of the flaws, but it can be the most expensive method to build and maintain. The choice of the tool is often based on integration capabilities and challenges, not coverage. If the protocol is standards-based, a model based fuzzer is often the right choice. But especially with emerging technologies and agile development processes, the specifications needed to create model-based tests are not always available. There might be no consensus on the protocol specification or the specification changes rapidly, or in some special cases the specifications are proprietary and not available to testers. In such cases traffic captures can be used to create mutation-based fuzzers. When the amount of interfaces is vast, random fuzzing might be a budgetary choice to get at least some fuzz test coverage before more advanced capabilities are introduced to the development process.
Microsoft Security Program Manager in the SDL team, Bryan Sullivan: “It is important to fuzz your parsing code periodically, but you are probably not going to find so many potential vulnerabilities doing so that you need to fuzz it every sprint.”
Interestingly, with some type of fuzzing you do not need to be that picky. In many environments that I have visited, fuzzing is automated to the build process, i.e. automatically when the code builds, the fuzz process starts in the background. And why not? It is not like you need any person to monitor a fuzz process especially if you do not expect to find anything. Fuzzing actually fits very nicely to any programmers’ automated unit test process, especially if you are coding protocol stacks or simple applications on top of industry standard protocols such as HTTP, SOAP and SIP.]]>
The BSIMM study by Cigital creates a new mile stone in the domain of market studies in fuzzing. It might not even attempt to describe how common the use of fuzzing is. The sample of companies also do not really indicate anything about the rest of the users of fuzzers. And the interview process itself might have not really given much emphasis on fuzzing, as all the authors really come from the static analysis mindset. But surprisingly enough, all top product security teams were found to be doing fuzzing already!
Another major milestone is studying the use of fuzzing is the inclusion of fuzzing related questions to the Forrester questionnaire, completed by thousands of CIO/CSO/CISO people annually.
I personally look forward to hearing what Cigital and Forrester have to say about the use of fuzzing. If you are interested, please give us a shout here: Fuzzing 101]]>
The authors (Gary McGraw, Brian Chess, and Sammy Migues) interviewed leading product security teams in the industry, and collected the findings. The most important discovery (or maybe the biggest surprise to the authors?) was:
0. Fuzz testing is widespread.
What kind of “last bullet” is that on a top ten list?! Let us explain. Way back in 1997 in the book Software Fault Injection, Jeff Voas and McGraw wrote about many kinds of testing that can be imposed on software. We wondered whether security was a special case for software testing. One classic way to probe software “reliability” is to send noise to a program and see what happens, i.e., fuzzing. Somehow the security community has morphed this technique into a widely applied way to look for software security problems. Wow. Who would have guessed that reliability trumps security?”
The importance of finding real and certainly critical issues in software has finally been noted as the highest priority by all leading security organizations! But we knew that, because we have been helping them in the process.]]>
For example Jayaram & Mathur from Purdue are explaining interesting measurements of using statecharts as the basis of generating message sequences for complex protocols such as TLS. Sounds pretty similar to fuzzing, at least to me, although the research at this phase is nowhere in the same domain. Today most block-based fuzzers (although some of them call themselves model-based) use extremely limited message sequence coverage, with the worst of them only take a capture of traffic, and then mutate that. The drawback with this is that you will only do message structure fuzzing, the most basic form of fuzzing.
Then if you look at the work of e.g. Gotlieband and Petit from INRIA, you can get a glimpse of what the QA people are looking at in the area of test generation. Any individual field in the protocol message can (potentially) automatically generate its own set of data based on a very basic assumptions, and therefore optimize those to finally do some intelligent permutations of multi-anomaly fuzzing. Long gone are those static libraries of anomalies (again very few real fuzzers use them today). The result is less test cases, and better test coverage.
It is interesting to see where fuzzing will go in the future, and how companies with QA background, and companies with security background will either end up in the same direction, or very different direction.]]>
The best “Why Me” comments are also under selection process. Here is a sample of some of them (from current winners, who unfortunately will not get a chance to get a second copy):
Congratulations to all winners!]]>
Update: My ITworld blog]]>