On 4 June 2014 01:54, Stephan Neuhaus <stephan.neuhaus@tik.ee.ethz.ch> wrote:
If you fail the audit, it's your duty as a professional auditor to
provide evidence that there is something actually wrong with the
software.  It's OK to single out some pieces of code for closer
inspection because of code smells, but if you try your darnedest to find
something wrong with it and can't, then either the code is OK or you're
not good enough an auditor.  In either case, you can flag the code, you
can recommend rewriting it according to what you think is better style,
but you can't in good conscience fail the audit.

Perhaps this is getting too far into nits and wording, but I audit software for my day job (iSEC Partners).  I'm not speaking for my employer. But, with very few exceptions (we have a compliance arm for example), one does not 'Pass' or 'Fail' one of our audits.  (Perhaps they might be better termed as 'security assessments' then, like we call them internally, but we're speaking in common english, and people tend to use them synonymously.)

Our customers are (mostly) on board with that too.  They never ask us if they 'passed' or failed' - I'm certain some of them look at a report where we failed to 'steal the crown jewels' as a successful audit - but the expectation we set with them, and they sign on with, is not one of 'Pass/Fail'. And engagements where they want a statement saying they're secure, we turn down - we're not in the business of rubber stamps*.

Our goal is to review software, identify bugs, and provide recommendations to fix that issue and prevent it from occurring again. AND, in addition to the specific bugs, provide general recommendations for the team to make their application and environment more secure - provide defense in depth. Maybe I didn't find a bug that let me do X, but if there's a layer of defense you can put in that would stop someone who did, and you're missing that layer, I would recommend it.

Examples: I audited an application that had no Mass Assignment bugs - but no defenses against it either. Blacklists preventing XSS instead of whitelist approaches, and like Andy said, homebrew C-code parsing JSON. We 'flag'-ed all of that, and told them they should rewrite, rearchitect, or add layered defenses - even if we couldn't find bugs or bypasses.

So the notion of 'Passing' or 'Failing' an audit is pretty foreign to me.  Perhaps people mean a different type of work (compliance?) than the one I do.

-tom

* The closest we get is one where we say 'We tested X as of [Date] for Y amount of time for the following classes of vulnerabilities, reported them, retested them Z months later, and confirmed they were fixed.'  As we do this very rarely, very selectively, for clients we've dealt with before.