Automated Code Audit’s First Customer
Last month our Cyber Reasoning System (CRS) -developed for DARPA’s Cyber Grand Challenge– audited a much larger amount of code in less time, in greater detail, and at a lower cost than a human could.
Our CRS audited zlib for the Mozilla Secure Open Source (SOS) Fund. To our knowledge, this is the first instance of a paid, automated security audit by a CRS.
This represents a shift in the way that software security audits can be performed. It’s a tremendous step toward securing the Internet’s core infrastructure.
Choice where there once was none
Every year, public, private, and not-for-profit organizations spend tens of thousands of dollars on code audits.
Over a typical two-week engagement, security professional charge a tidy fee to perform an audit. Their effectiveness will be limited by the sheer volume of the code, the documentation and organization of the code, and the inherent limitations of humans — getting tired, dreaming of vacations, etc.
You can only analyze complex C code effectively for so many hours a day.
Furthermore, a human assessor might have great experience in some subset of possible flaws or the C language, but complete or nearly complete knowledge is hard to come by. We’re talking about expertise acquired over 15 years or more. That level of knowledge isn’t affordable for non-profits, nor is it common in 1-2 week assessments.
It makes more sense for a piece of software to conduct the audit instead. Software doesn’t get tired. It can audit old, obfuscated code as easily as modern, well-commented code. And software can automatically re-audit code after every update to make sure fixes are correct and don’t introduce new errors.
Mozilla’s SOS
In August, as a part of their Secure Open Source (SOS) Fund, Mozilla engaged us to perform a security assessment of zlib, an open source compression library. Zlib is used in virtually every software package that requires compression or decompression. More than one piece of software you are using to read this very text bundles zlib.
It has a relatively small code base, but in that small size hides a lot of complexity. First, the code that runs on the machine may not exactly match the source, due to compiler optimizations. Some bugs may only occur occasionally due to use of undefined behavior. Others may only be triggered under extremely exceptional conditions. In a well-inspected code base such as zlib, the only bugs left might be too subtle for a human to find during a typical engagement.
To identify any especially subtle bugs from a human-powered audit, Mozilla would have had to spend many thousands of dollars more. But they’re a non-profit, and they have an array of other projects to audit and improve.
Great coverage at a great price
Automation made the engagement affordable for Mozilla, and viable for us. They paid 20% of what we normally have to charge for this kind of work.
Our automated assessment paired the Trail of Bits CRS with TrustInSoft’s verification software to identify memory corruption vulnerabilities, create inputs that stress varying program paths, and to identify code that may lead to bugs in the future.
- Read the report that we delivered to Mozilla. (It goes into great detail about how our CRS -paired with TrustinSoft‘s verification program- found more vulnerabilities for a fraction of the cost of a human-powered audit.)
- Read Mozilla’s release about the report.
For non-profits working to secure core infrastructure of the Internet, this is a wonderful opportunity to get a detailed assessment with great coverage for a fraction of the traditional cost.