On Wed, Jun 04, 2014 at 08:50:14AM -0400, Tom Ritter wrote:
On 4 June 2014 01:54, Stephan Neuhaus <stephan.neuhaus@tik.ee.ethz.ch> wrote:
If you fail the audit, it's your duty as a professional auditor to provide evidence that there is something actually wrong with the software. It's OK to single out some pieces of code for closer inspection because of code smells, but if you try your darnedest to find something wrong with it and can't, then either the code is OK or you're not good enough an auditor. In either case, you can flag the code, you can recommend rewriting it according to what you think is better style, but you can't in good conscience fail the audit.
Stephan, I strongly disagree. There are implementations that are Just Too Complicated and are Impossible To Audit. Such implementation choices *do*, empirically, provide cover for bugs; and as we as a society build more and more software into the fabric of our life-critical systems it's imperative that "the implementor liked this complexity and refuses to change it" gives way to the larger goals at stake. The auditor absolutely must have leeway to say "no you don't get to write your own string processing, you are going to use the standard ones." This kind of feedback is precisely what happens in the higher quality audits that are becoming standard practice for security-critical software.
Perhaps this is getting too far into nits and wording, but I audit software for my day job (iSEC Partners). I'm not speaking for my employer. But, with very few exceptions (we have a compliance arm for example), one does not 'Pass' or 'Fail' one of our audits. (Perhaps they might be better termed as 'security assessments' then, like we call them internally, but we're speaking in common english, and people tend to use them synonymously.)
As a satisifed iSec customer (at a previous job), I have a bit of insight here. iSec is a leader in this space and definitely leads by example. Across the industry, the average quality of discourse in the source auditing business is pretty good in my experience; only the bottom-skimming truly awful auditors reduce their customer-facing feedback to just a binary pass/fail. However, inevitably, in the societal analysis of software quality for practical purposes, reductive reasoning happens. (This is not a bad thing, it's absolutely necessary -- we humans don't have the cognitive capacity to hold a complete decision tree in our head while doing this reasoning.) Thus statements like "you should use $OSS_CRYPTO_PACKAGE, it has passed its audits" end up playing a role in the discourse. We as domain experts have an obligation to ensure that our contribution is given appropriate weight in the debate and decisions -- in both directions. For example if an auditor sees their results being mis-interpreted in customer marketing material or media coverage, the auditor has a moral obligation to correct that and insist that the mischaracterization stop. (And yes, I believe that this moral obligation would override an NDA between the customer and the auditor; the contract should be structured to recognize this fact.) -andy