"a skilled backdoor-writer can defeat skilled auditors"?

[Tom Ritter]

On 4 June 2014 01:54, Stephan Neuhaus <stephan.neuhaus at tik.ee.ethz.ch>
wrote:

> If you fail the audit, it's your duty as a professional auditor to
>  provide evidence that there is something actually wrong with the
> software.  It's OK to single out some pieces of code for closer
> inspection because of code smells, but if you try your darnedest to find
> something wrong with it and can't, then either the code is OK or you're
> not good enough an auditor.  In either case, you can flag the code, you
> can recommend rewriting it according to what you think is better style,
> but you can't in good conscience fail the audit.
>

Perhaps this is getting too far into nits and wording, but I audit software
for my day job (iSEC Partners).  I'm not speaking for my employer. But,
with very few exceptions (we have a compliance arm for example), one does
not 'Pass' or 'Fail' one of our audits.  (Perhaps they might be better
termed as 'security assessments' then, like we call them internally, but
we're speaking in common english, and people tend to use them synonymously.)

Our customers are (mostly) on board with that too.  They never ask us if
they 'passed' or failed' - I'm certain some of them look at a report where
we failed to 'steal the crown jewels' as a successful audit - but the
expectation we set with them, and they sign on with, is not one of
'Pass/Fail'. And engagements where they want a statement saying they're
secure, we turn down - we're not in the business of rubber stamps*.

Our goal is to review software, identify bugs, and provide recommendations
to fix that issue and prevent it from occurring again. AND, in addition to
the specific bugs, provide general recommendations for the team to make
their application and environment more secure - provide defense in depth.
Maybe I didn't find a bug that let me do X, but if there's a layer of
defense you can put in that would stop someone who did, and you're missing
that layer, I would recommend it.

Examples: I audited an application that had no Mass Assignment bugs - but
no defenses against it either. Blacklists preventing XSS instead of
whitelist approaches, and like Andy said, homebrew C-code parsing JSON. We
'flag'-ed all of that, and told them they should rewrite, rearchitect, or
add layered defenses - even if we couldn't find bugs or bypasses.

So the notion of 'Passing' or 'Failing' an audit is pretty foreign to me.
 Perhaps people mean a different type of work (compliance?) than the one I
do.

-tom

* The closest we get is one where we say 'We tested X as of [Date] for Y
amount of time for the following classes of vulnerabilities, reported them,
retested them Z months later, and confirmed they were fixed.'  As we do
this very rarely, very selectively, for clients we've dealt with before.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/html
Size: 3509 bytes
Desc: not available
URL: <https://lists.cpunks.org/pipermail/cypherpunks/attachments/20140604/3e41e3c9/attachment-0001.txt>