On 2014-06-04, 20:22, Andy Isaacson wrote:
On Wed, Jun 04, 2014 at 08:50:14AM -0400, Tom Ritter wrote:
On 4 June 2014 01:54, Stephan Neuhaus <stephan.neuhaus@tik.ee.ethz.ch> wrote:
If you fail the audit, it's your duty as a professional auditor to provide evidence that there is something actually wrong with the software. It's OK to single out some pieces of code for closer inspection because of code smells, but if you try your darnedest to find something wrong with it and can't, then either the code is OK or you're not good enough an auditor. In either case, you can flag the code, you can recommend rewriting it according to what you think is better style, but you can't in good conscience fail the audit.
Stephan,
I strongly disagree. There are implementations that are Just Too Complicated and are Impossible To Audit. Such implementation choices *do*, empirically, provide cover for bugs; and as we as a society build more and more software into the fabric of our life-critical systems it's imperative that "the implementor liked this complexity and refuses to change it" gives way to the larger goals at stake. The auditor absolutely must have leeway to say "no you don't get to write your own string processing, you are going to use the standard ones."
I think that we are mostly in agreement, except perhaps in wording. We both agree that auditors rarely "pass/fail" software in a binary fashion. And as I wrote, the auditor absolutely has the leeway to recommend rewriting. But my gripe was with the "automatic fail" in the original post, to which I said that this was "going too far". If you do go that far (i.e., don't just recommend changes, but "fail" the audit), your verdict must be founded on evidence. For example, if it were actually true that complexity, "empirically, provides cover for bugs", that would be a perfectly good argument in favour of failing an audit. It's just that I've worked for a few years in precisely this field and all the studies I saw simply failed to show the necessary correlations. (The best study I know, by Yonghee Shin and Laurie Williams, shows rho <= 0.3, and that on the vulnerability-infested Mozilla JavaScript engine. See http://collaboration.csc.ncsu.edu/laurie/Papers/p47-shin.pdf) This shows, I think, that auditors must be extra careful not to confuse folklore with evidence. You can say "this code is too complex for me to audit", and you can add "this should give you food for thought and you should consider rewriting it in a simpler style", but *as the auditor* you cannot say "I fail the code because I can't audit it" unless auditability was a design requirement. (For the *owners* of the code, their options are of course much greater, but we were talking about this from the auditor's perspective, and the OP talked about an "automatic fail" if the code turned out to have certain smells. If a smell isn't backed up by evidence, it's just a personal prejudice or folklore. Which, incidentally, would be excellent new terms to replace "Best Practice" in many cases.) Again, please note that I agree with you that auditability and simplicity, using braces even for single-line if-statements, library functions rather than self-made string libraries, and all these other things, ought to be design requirements, especially for security-critical software, because they make auditing easier. It's just that if it wasn't, then you can fault the design requirements (though that may be outside your remit as auditor), but you can't "automatically fail" the implementation. Fun, Stephan