----- Original Message ----- From: "Anonymous" <nobody@remailer.privacy.at>
Ross Anderson's paper at http://www.ftp.cl.cam.ac.uk/ftp/users/rja14/toulouse.pdf has been mostly discussed for what it says about the TCPA. But the first part of the paper is equally interesting.
Ross Andseron's approximate statements: Closed Source:
"the system's failure rate has just dropped by a factor of L, just as we would expect."
Open Source: bugs remain equally easy to find. Anonymous's Statements:
For most programs, source code will be of no benefit to external testers, because they don't know how to program.
Therefore the rate at which (external) testers find bugs does not vary by a factor of L between the open and closed source methodologies, as assumed in the model. In fact the rates will be approximately equal.
The result is that once a product has gone into beta testing and then into field installations, the rate of finding bugs by authorized testers will be low, decreased by a factor of L, regardless of open or closed source.
I disagree, actually I agree and disagree with both, due in part to the magnitudes involved. It is certainly true that once Beta testing (or some semblance of it) begins there will be users that cannot make use of source code, but what Anonymous fails to realize is that there will be beta testers that can make use of the source code. Additionally there are certain tendencies in the open and closed source communities that Anonymous and Anderson have not addressed in their models. The most important tendencies are that in closed source beta testing is generally handed off to a separate division and the original author does little if any testing, and in open source the authors have a much stronger connection with the testing, with the authors' duty extending through the entire testing cycle. These tendencies lead to two very different positions than generally realized. First, closed source testing, beginning in the late Alpha testing stage, is generally done without any assistance from source code, by _anyone_, this significantly hampers the testing. This has led to observed situations where QA engineers sign off on products that don't even function, let alone have close to 0 bugs. With the software engineers believing that because the code was signed off, it must be bug-free. This is a rather substantial problem. To address this problem one must actually correct the number of testers for the ones that are effectively doing nothing. So while L is the extra difficulty in finding bugs without source code, it is magnified by something approximating (testers)/(testers not doing anything). It's worth noting that (testers) > (testers not doing anything) causing the result K = L*(testers)/(testers not doing anything), to tend towards infinite values. In open source we have very much the opposite situation. The authors are involved in all stages of testing, giving another value. This value is used to adjust L as before, but the quantities involved are substantially different. It must be observed, as was done by Anonymous, that there are testers that have no concept what source code is, and certainly no idea how to read it, call these harassers. In addition though there are also testers who read source code, and even the authors themselves are doing testing, call these coders. So in this case K = L*(harassers)/(harassers+coders). Where it's worth noting that K will now tend towards 0. It is also very much the case that different projects have different quantities of testers. In fact as the number of beta testers grows, the MTBD(iscovery) of a bug must not increase, and will almost certainly decrease. In this case each project must be treated separately, since obviously WindowsXP will have more people testing it (thanks to bug reporting features) than QFighter3 (http://sourceforge.net/projects/qfighter3/ the lest active development on sourceforge). This certainly leads to problems in comparison. It is also worth noting that it is likely that actual difficulty in locating bugs is probably related to the maximum of (K/testers) and the (testers root of K). Meaning that WindowsXP is likely to have a higher ratio of bugs uncovered in a given time period T than QFighter3. However due to the complexity of the comparisons, QFighter3 is likely to have fewer bugs than WindowsXP, simply because WindowsXP is several orders of magnitude more complex. So while the belief that source code makes bug hunting easier on everyone, is certainly not purely the case (Anonymous's observation), it is also not the case that the tasks are equivalent (Anonymous's claim), with the multiplier in closed source approaching infinite, and open source towards 0. Additionally the quantity of testers appears to have more of an impact on bug-finding than the discussion of open or closed source. However as always complexity plays an enormous role in the number of bugs available to find, anybody with a few days programming experience can write a bug free Hello World program, but it takes significantly more effort to write something the complexity of Windows or Linux bug-free. That is where open source receives the biggest boost, where Microsoft has a limited number of software engineers, and the Linux development process which has fewer testers but many more developers. This will reduce the number of bugs simply due to the effort put into the process by developers. Which is better? That depends entirely on your situation. Joe