Issue 5 - Testing is one of the Least Effective Defect Removal Techniques

Jeff De Luca's picture

De Luca on FDD Newsletter Issue 5

Testing is one of the Least Effective Defect Removal Techniques


Inspections are one of the most effective defect removal techniques. Testing is one of the least effective defect removal techniques. We know this from measurements; it isn't speculation. And yet, what do most organizations do? They test. What do most organizations not do? They don't inspect. Why is this?

Over the years I've observed several reasons for this. One reason is the fear people have of their work being inspected. Thus setting the inspection culture correctly is a critical success factor. Another reason is related to pre-object oriented systems where you get a request like "here's a 500 page COBOL program, we're going to do an inspection on it in 3 days." It's just too big. Thus ensuring inspections are performed on the right artifacts, at the right level of abstractions, and the artifacts are at an appropriate granularity are all critical success factors.

In well designed object oriented systems, functions are provided by the collaboration of classes and the methods they call on each other. We model this using the UML sequence diagram and the granularity and abstraction of FDD features especially helps here.

In most cases, a sequence diagram will only have 3-5 classes in it and the typical shape is that the first two calls are simply delegating down, and then there's a bit of work in the right hand classes and possibly some kind of accumulation or checking done back near the sender. Thus, the UML sequence diagram for an FDD feature, and then the code in each class to implement that feature, is a very small piece of work (tiny compared to monolithic programming). Furthermore, every FDD feature directly represents a client-valued function. These aspects mean that FDD delivers the right artifacts at the right abstractions and of the right granularity.

Inspections are one of the most effective defect removal techniques. Inspections complement other QA techniques and they are an important milestone for meaningful progress and risk reduction.

Some facts about inspections:

  • 55% of maintenance changes in error before inspections. After 2%. Freedman, Weinberg, 1982.
  • Before inspections 4.5 errors per 100 lines of code. After 0.82 errors per 100 lines of code. Freedman, Weinberg, 1982.
  • Aetna Insurance found 82% of errors using inspections and decreased resources by 25%. Fagan, 1976.
  • AT&T 14% increase in productivity and 90% decrease in defects. Fowler 1986.
  • Inspections find different kinds of errors than testing does. Myers 1978. Basili, Selby, Hutchens 1986.
  • Combination of design and code inspections removes 60-90% of all defects. Fagan 1976.
  • Reducing the number of defects improves development time.
  • If you know your work will be inspected, you are more careful with it (programmers write their code differently if they know someone else is going to read it).
 

Requirements
Defects

Design
Defects

Code
Defects

Document
Defects

Performance
Defects

Reviews - Inspections

Fair

Excellent

Excellent

Good

Fair

Prototypes

Good

Fair

Fair

N/A

Good

Testing (all forms)

Poor

Poor

Good

Fair

Excellent

Correctness Proofs

Poor

Poor

Good

Fair

Poor

Table 1 Defect Removal Methods. Capers Jones, 1996.

This is more than just defect removal. Inspections are a fantastic culture and team-building activity and they actively complement the feature team and Chief Programmer-oriented structures in Feature Driven Development. FDD applies inspections to both the design (in Design By Feature) and the code (in Build By Feature).

Inspections make it so much easier to propagate team culture, syntactic and semantic standards, naming standards, process, and so on. Programmers learn to improve their work (design and code) through participating in inspections.

Inspections, like many things, need to be introduced in a way that removes fear from a programmer. Programmers generally don't like to be told they did something wrong, or could have done it better. Setting the inspection culture is key so that these fears are removed and everyone sees them as a great learning tool as well as a quality tool. You should not micro-manage and use inspections as some kind of punitive measurement tool for your programmers.

However, to only list these two reasons thus far is to suggest that despite the evidence - the measured effects of inspections - people don't inspect due to the fear programmers have of their work being inspected and due to the size of the design or code to inspect. This is misleading as most people aren't aware of the evidence in the first place. As is the case for so many of the problems in I.T., the solutions have been identified long ago.

Inspections are also a time to market (or time to deployment) tool. That is, less defects means faster projects. However, a common knee-jerk reaction is that "inspections are taking too much time, so to speed up, we are going to stop inspecting." This is a false economy. It might appear that you are speeding up, but in fact you aren't. All you are doing is trading the appearance of different kinds of work - and in doing so, the total project time is being lengthened.

By not inspecting, and therefore doing only coding, what people are doing all the time is coding. Coding is incorrectly connected to "getting it done" and therefore by doing this all the time, things must be going faster. It is an appearance and it is an illusion in total project terms.

Inspections, however, are not coding. People are in a room - without keyboards, and screens, and compilers, and editors. The inspection itself takes time. Time that would otherwise be spent coding.

The issue here is that of connecting the act of coding with real work and fastest progress. There is no mystery here. This has been measured many times before. More code does not equate to fastest progress; correct and standards-conforming code does. Inspections speed you up, not slow you down. Understanding this common mental model can help you "get over it," help those around you "get over it," and allow you to see through the misleading appearance of "always coding."

Feature Driven Development directly facilitates the implementation of known best practices such as design and code inspections by delivering the right artifacts, at the right level of abstractions, and of the right granularity, and also by directly implementing team structures that map to these artifacts and that promote the mentoring and team building aspects of inspections.


© Nebulon Pty. Ltd.

With thanks to the review team: Gavin Baker, Phil Bradley, David Bye.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Is old data reliable?

The data you provided showing that inspection is superior to testing is old, very old. The most recent is from 1986.

Is data from this far back reliable, given the changes in software engineering in the past two decades?

Is there any data from the past few years covering inspection versus testing?

Wiegers and Jones from the last 3 years

Try Karl Wiegers, Peer Reviews in Software: A Practical Guide, Addison Wesley 2002 and Capers Jones, Software Assessment, Benchmarks and Best Practices, Addison Wesley 2000

There is some newer Jones material in IT Measurement, edited by Ed Yourdon, Addison Wesley, 2002

Meanwhile, what advances in software engineering in the last 20 years? As someone whose been here at least that long I am wondering what your on about - faster compilers, more sophisticated IDEs? minor incremental improvements in my view - like fuel injection for cars - the engine is still a basic internal combustion block. Just because bench checking your punch cards before processing is no longer de rigeur doesn't invalidate inspections versus testing data ratios, it simply reduces the time periods involved. We don't fire people for having 5 failed compiles these days, but we do catigate them for 5 broken builds in a sequence. How different is that!

How different is it today - really? How much of what has changed has been noise, of fad or just plain technology? The big changes were object-orientation, Java and runtime engines like it, and recently aspect-orientation. The biggest noise was Use Cases and everything that came with them. How different is RUP to IDD methods that preceeded it? Much of what is written in my Roger Pressman and Ian Somerville books on Software Engineering from the late eighties still appears today in the latest versions of those books.

I firmly believe that the stars are aligning for a paradigm shift in software engineering - a genuine discontinuous innovation - but I also firmly believe that it didn't happen in the last 20 years. What we've seen during that period was continuous innovation, and incrmental improvement - at best - and the results are almost certain not to have affected the metrics by much. "Much" meaning a significant amount or an order of magnitude which sufficient that it would change how you would behave in response to the numbers.

Regards,

David

David J. Anderson
author of "Agile Management for Software Engineering"
http://www.agilemanagement.net/

Changes in the past 20 years

The changes in the past 20 years that I had in mind are mostly the size of programs. Present day programs routinely involve hundreds of thousands of line of code, and a line of todays Java code may be worth several lines of C/C++. We also depend more heavily on third party libraries now.

The increase in size makes inspections more time consuming, probably more so than testing.

Two more changes worth noting.

Many applications today are driven by large, complex schemas that probably didn't exist two decades ago. Inspections (and testing) have to account for the range of permutations of possible underlying data.

Two decades ago it seems like most programmers came from engineering, science, or math backgrounds whereas today there are more computer science and MIS grads doing design and code. The culture is different now. Scientists are used to having their work scrutinized by peers whereas CS/MIS types probably are not so used to it. (Scientists are notoriously bad code writers, however. Take a look at the Numerical Recipes books.) My familiarity with CS grads is that they got used to cranking out code in a hurry that met some set of tests but wasn't ever reviewed.

Mostly I was curious as to why the dates given were from so far back.

Jeff De Luca's picture

No special reason

I'm confused. You assert a line of Java is worth several of C/C++ and imply the use of Java. This must mean that programs are smaller today (have less lines of code) - not bigger (more lines of code). Also, you assert that we depend more heavily on third party libraries now. This also must mean programs are smaller today - not bigger (again, by lines of code).

However, there are more problems I have with this, not the least of which is that program size as defined by lines of code is not a good metric even within the same language, let alone trying to compare the size of program A in language 1 (by lines of code) with program B in language 2 (by lines of code).

Furthermore, common designs and implementations today such as object oriented make for much smaller units of inspection - FAR smaller than of monolithic programming years ago. FDD's features are about as small as it can get for something meaningful to inspect.

I don't understand your comment about complex schemas today that didn't exist previously.

Finally, there was no science or particular reason as to why I used the dates the did. I've been doing this for 20+ years and what I am primarily doing via this site and the newsletter now is writing about it. Inspections are not a new technique and I already had (from years ago) a good set of cited, measured studies over the first 20 years or so of inspections usage. I didn't see any need to go search for more recent data. That's why I used the citations I did.

In part (but it was a minor consideration) I did bias towards the very fact that it is not a new technique and to quote Victor Basili (from memory) "inspections are well tested experimentally." I see the problem as more a lack of awareness than a failure of implementation.

Jeff

Larger and more compact

"I'm confused. You assert a line of Java is worth several of C/C++ and imply the use of Java. This must mean that programs are smaller today (have less lines of code) - not bigger (more lines of code). Also, you assert that we depend more heavily on third party libraries now. This also must mean programs are smaller today - not bigger (again, by lines of code)."

My point is that programs are both larger (in terms of lines of code) and more compact in that one line of code does what several used to do. (Maybe "information density" would be a better term than "compact".) Being both larger and more compact means greater functionality.

Inspections mean reading the code, hence today's larger programs will take longer to inspect.

The point about database size and complexity has to do with the difficulty testing an application whose behavior depends on the data underlying it. In my current application there are still lots of untested areas of functionality because the underlying data used in testing is incomplete and does not represent nearly the range of variability that real data will have. (I don't have control over the schema even for testing.) Inspections have limited usefulness here unless we can correctly anticipate all paths of execution for all ranges of data.

Your point that inspections and testing are both needed is well taken. I wasn't trying to start an argument.

Jeff De Luca's picture

The unit of inspection is smaller

My point is that programs are both larger (in terms of lines of code) and more compact in that one line of code does what several used to do. (Maybe "information density" would be a better term than "compact".) Being both larger and more compact means greater functionality.

Ok, now I get what you mean. You mean that even though one line of code today does more than before, today's programs have more lines of code then before.

Hmmm, I don't know about that - but my comments regarding lines of code still stand and they (IMHO) invalidate the comparison you're trying to make. To make the case, I think you'd have to use something like function points and look for studies in relation to FPs and FP to language density over time, to prove this. Capers Jones did publish a FP-Language table, but this backfiring approach has serious issues. What might be more interesting is to simply look at app sizes today by FP compared to before. But even then, the leap from FPs to language to lines of code is problematic (and you need this for the "more lines of code to inspect" assertion to prove true).

Irrespective of your size assertion, the unit of inspection with more modern approaches, such as OO and iterative/incremental, is far less than before. This is the key point regarding size and inspections.

The point about database size and complexity has to do with the difficulty testing an application whose behavior depends on the data underlying it... Inspections have limited usefulness here unless we can correctly anticipate all paths of execution for all ranges of data.

But this is precisely one of things you should be doing in an inspection. Simpler forms of this, known as value tracking, are even implemented in some analysis tools (such as lints, advanced editors, compiler front-ends). Value tracking, boundary conditions and parameter validation are indeed checking behaviour against values (I prefer this term to data).

You didn't start an argument - we're not having one. I didn't understand what you meant in your previous comment.

Jeff

szego's picture

Boundary Conditions

Simpler forms of this, known as value tracking, are even implemented in some analysis tools...

I can tell you from the point of view of a CP, who runs inspections all the time, we *do* check values as part of the inspection process. It's not left to analysis tools.

I'd say that not dealing with boundary conditions are some of the most prevalent (but often trivial) issues. The classic example in a language like Java is checking for nulls, usually as parameters and values returned from other methods.

I'm sure we all know lots of examples, my point was that it is done as part of the inspection.

The focus is different to just looking at paths of execution. If you focus purely on that you're more worried about how it can work, not how it can break. To spot boundary conditions for data values you're almost better off "switching off" and focusing on the low-level mechanics of the code. Once you spot these errors however they often do raise questions about paths of execution, in particular how to deal with unexpected situations.

Paul.

Jeff De Luca's picture

SPR on LOC and Backfiring

Yep, SPR have a nice half-page summary of this problem [spr.com].

Getting back to "it's different now"

I just can't buy the argument that larger code bases (and I don't believe they are - Jeff knows this only too well from his IBM mainframe days) and more functionality per square inch of listing, do represent a significant change.

Significant change comes from technology shifts which cause discontinuous innovation - and as a result leave practitioners using the old technology in the dust.

For example, how steel changed engineering. How carbon fibre changed it again. How the assembly line changed vehicle production. And so on.

If for example, MDA takes off - then we are talking about a paradigm shift. Then we can throw away the data on LOC and inspections and start again. If we stop building software as a craft of code writers and start doing genuine re-use of components or services in a value chain, similar to how other products like cars or PCs are built, then we can throw away the old books and the old data. The bottom line is that we build software today, in a similar way to how they built cars in 1911 - by craftsmen in small shops and every system is different.

My observation is that code is written today pretty much the way it was 20 years ago and that was pretty much the way it was 20 years before. All that changed was the input mechanism, the speed of the compilers, and the decomposition methods - procedural, functional, object-oriented.

Until we see a quantum leap (an order of magnitude) change in re-use and the ability to build a value chain of independent development - a production method for software - then it remains a craft based system. While it remains a craft, the human element remains a constant. The human condition doesn't change in such a short time frame. Hence, I will believe the old data to be true until I see proof otherwise.

David

David J. Anderson
author of "Agile Management for Software Engineering"
http://www.agilemanagement.net/

Stars aligning for a paradigm shift...

Several comments on the thread (from various places)

INEFFECTIVENESS OF TESTING vs EFFECTIVENESS OF INSPECTIONS
I have a suspicion this may be both highly situational and a somewhat invalid conclusion, even based on the original data.
I view the primary aspect of software development to be discovery of (previously unknown) knowledge, rather than cataloguing of what is known. All validation and verification is a knowledge diff--a comparison between two knowledge sources, representations, and repositories. For instance, one could plausibly argue that if the people attending an inspection do not know something about the system which is needed to be known, they can hardly be expected to identify that such knowledge is missing from a work product. What saves us (and inspections as a quality device, as several commentators have asserted) is the human dynamic whereby the exposure to your point of view allows me to extrapolate and interpolate what I know to things I don't know and between us deduce the presence of things we weren't originally aware of.

My experience has been:

(a) Testing tends to find different kinds of defects. For instance, executable testing is one of the few ways we have of uncovering complex platform-dependent situations.

(b) The effectiveness of testing is closely related to the setup (and inspection) of the test cases and expected results. Someone (Pressman? Myers?) stated that the act of creating a good test is more effective at detecting defects than running it. So the two mechanisms are quite closely coupled.

(c) Testing is (IMHO) the only aspect of software development that truly acknowledges that the real job is to find out things we don't know. And particularly the things we don't know we don't know (what I call "Second Order Ignorance"). It seems to me that all other aspects of development embody a tacit assumption that our job is to translate what we know (the application of our "Zeroth Order Ignorance"). In no other part of software development is the exposure of ignorance truly acceptable. Can you, for instance, imagine a manager saying "Bob, you've done a great job on the design of this system--look at all the things that don't work!"

(d) We aren't restricted to just one technique anyway. Who says (I mean apart from Michael Fagan) that we should globally replace testing effort with inspection effort? We can have both.

THE PARADIGM SHIFT
The term "Paradigm Shift" was popularized by Thomas Kuhn, but it's a well known and easily provable aspect of cognition. I wholeheartedly agree with Mr. Anderson that one is coming in software. The problem with such revolutions is always (a) it's really hard to see what it is because it isn't here yet (b) it's even harder to see what will happen as a result of it because of (a).

A true paradigm shift is a different way of thinking, and that is really really hard for people, for anyone. We can only think the way we can think, and we reflexively apply our proven existing thought processes to determine the viability of new thought processes, or even to try to use the new thought processes. This often ends in a stalemate condition. Using thought process A I can't see why thought process B is better (or even understand it). Because I don't have thought process B available to me I cannot see the limitations of thought process A.

For what it's worth (approx $0.02), using my limited thought process A, I think the coming change will involve two things: minds and software (gasp! What a revelation!):

Minds: We will socially and organizationally realize that software development is only a cognitive process and that, to create development methodologies we must look, not to the "outside world" of real life entities and their relationships, but to the operational processes of the (collective) human mind.

Software: We will start to use software to create software (a heck of a concept, I know). Presently, very little software is used in the creation of software: mostly brain-dead CASE tools, word processors, and domain-non-specific compilers. This is an area where testing scores over inspections; done well, testing can be considered to be a computer assisted inspection. But I'm thinking more in terms of, say, simulation.

The Agile movement in general has made some inroads in a couple of these areas, most specifically in the sociology of development, which has been astonishingly lacking in almost every development trend until very recently. We are so wedded to our technologies and methodologies, that we scarcely recognize that they are only conventional ways of thinking and that we can select the conventions and change the ways, even though it's hard to do. But hey, it's the way we think.

Phil Armour
Corvus International Inc

New thread in response to this post

I created a new thread in response to this post.

Jeff De Luca's picture

That's Astonishing

I have a suspicion this may be both highly situational and a somewhat invalid conclusion, even based on the original data.

That's an astonishing statement Phil. There's a LOT of actual data on this. Have you read of some of these studies? They are not lightweight. Capers Jones (to name just one person) collected data on something like 5,000 to 8,000 actual projects (I don't recall the exact number right now). Try reading, if you haven't already, the Assessment and Control of Software Risks, Applied Software Measurement and Patterns of Software Systems Failure and Success which document extensively Jones findings.

It is also well known and measured that the cost of fixing defects escalates as we move closer to deployment. Boehm in Software Engineering Economics showed this in 1981 from an analysis of 63 projects.

Now, I'll quote Gilb who wrote "Inspection can be used long before executable code is available to run tests. Inspection can be applied much earlier than dynamic testing, but can also be applied earlier than test design activities."

Can you connect these dots?

Now, I'll quote Jones

"In terms of the removal methods themselves, detailed code inspections are the most efficient form of defect removal yet measured, whereas most testing steps are less than 35 percent efficient. That is, most forms of testing find less than one bug out of every three bugs that actually exist. Once you have measured a reasonably large sample of projects (more than 50), you will be able to use the data you've collected to make very accurate quality and reliability estimates."

"Some of the implications of Table 5.5 deserve discussion. First, given the low efficiencies of testing, it is obviously impossible to achieve high levels of cumulative efficiency without up-front activities such as prototyping, reviews, or inspections. To be blunt, companies that only perform testing will never go much beyond 75 percent in cumulative defect removal efficiency; that is, they will deliver at least one out of every four bugs to clients."

"Second, testing ranges from moderate to useless in its ability to come to grips with front-end defects such as requirements and design. Indeed, it is surprising that testing can find requirements defects at all. Therefore, it is imperative to utilize the high-efficiency pretest activities in order to control these major sources of system problems."

I say and quote all this to at least establish what should be a very simple and intuitive case - even without having read the volumes of text on this subject. But I do find it astonishing that you seem to dismiss all this as highly situational and an invalid conclusion based only on your suspicion. You're saying that the actual measurements and text of Jones, Gilb, Weinberg, Fagan, McConnell, Boehm - to name just a few "lightweights" is wrong.

Noone I know says we should globally replace testing effort with inspecting effort. It isn't one or the other. I am speculating, but I'd be very surprised if Fagan said this.

I said, and I quote on Mon, 11/10/2003 - 18:28: I don't believe inspections vs testing is the most useful comparison to be making. It reads as "one or the other" and I strongly believe the answer is "both."

Jeff

Ineffectiveness of Inspections

Ok, the "invalid" statement WAS too strong. I did not intend to diss inspections at all. I have read much and practiced more at inspections and I am a strong believer in them. Really.

My point, which I belabored and probably lost, was the one Jeff makes at the end of the last post and made before: we can have and need both--inspections and testing are different--and that is a good thing. Like any tool, inspections are effective only within their application scopet. There are certain types of (say) platform dependent behavior or (say again) concurrent processing race conditions that would be quite excruciating to try to expose through an inspection, but probably would be discovered somewhat easier through a good test. Equally, there are validation content errors which cannot be reasonably caught by a test, unless you already know there is a defect present, in which case, why test? For these an inspection format whereby knowledgeable people look at the thing and pass judgment on it is the only way.

With testing, we always have the issue of how do we know the test is "good" unless the test itself is either (a) inspected or (b) tested? With option (b), of course we would have to determine a "goodness of test" test, which would have to be inspected or tested. So it looks like we are back to people looking at it to assess its viability. So we always end up back at inspections, I guess. So we have to do inspections and we have to do them well.

I was being facetious regarding the replacement of testing, though I have personally heard people say this very thing. The specific phrase used was "Defect-Free Software Through Inspections". Defect-less, perhaps, but not *free*. The main thrust of my post was that software is primarily a knowledge *discovery* process, and that as such inspections must be limited in their capability (which does NOT make them less valuable, merely more limited), since it is difficult for me to spot a deficiency in an artifact if I have the same deficiency in my understanding. I have participated in many inspections where we did the very best we could given what we knew at the time of the inspection. But we still missed a lot of defects which were exposed later thtough some other means. However, we were only able to recognize that they were defects because of the greater knowledge we had accumulated, sometimes just because it was later in time and we had learned more. The inspections were "good". They were certainly the best we could do at the time, and we would have done a lot worse a job, if we had not inspected. But they were not, and could not be, "perfect".

In some ways, the most effective "discovery" action, for certain kinds of defects, especially extrinsic defects, is to ship the system. But saying that dumping a defective system on the poor user is a guaranteed way of exposing defects, does not mean I think we should do that.

Jeff De Luca's picture

Not that class of defect

There are certain types of (say) platform dependent behavior or (say again) concurrent processing race conditions that would be quite excruciating to try to expose through an inspection, but probably would be discovered somewhat easier through a good test.

Actually, it's well documented and understood that testing is quite ineffective for a race condition. Here's just one citation.

The difficulty of finding data races and deadlocks is well known. Detecting such errors with testing is hard since they often depend on intricate sequences of low-probability events. This makes them sensitive to timing dependencies, workloads, the presence or absence of print statements, compiler options, or slight differences in memory models. This sensitivity increases the risk that errors will elude in-house regression tests yet make grand entrances when software is released to thousands of users. Further, even if a test case happens to trigger an error it can be difficult to tell that it has happened. Data races in particular are hard to observe since often they quietly violate data structure invariants rather than cause immediate crashes. The effects of these violations only manifest millions of cycles after the error occurred, making it hard to trace back to the root cause.

Dawson Engler and Ken Ashcraft Computer Systems Laboratory Stanford University Stanford, CA 94305, U.S.A. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SOSP 03, October 19 22, 2003, Bolton Landing, New York, USA. Copyright 2003 ACM 1-58113-757-5/03/0010 ...$5.00.

My point in my newsletter was to enlighten people to what we know works and to some of the typical presumptions we have about what we think works but is actually less effective.

Jeff

Jeff De Luca's picture

Use Both

Actually, the most recent I cited was 1996.

It's a fair enough question. I've not tried to track studies more recent than these as they, and my own substantial direct experiences, are more than enough for me. However, there are plenty of people still very active in this area. Magazines such as IEEE Software often have articles on inspections (usually under the banner of quality assurance - there were two in the May/June 2003 issue). You would find many articles and many more references within them. (IEEE has a digital library you can search). FWIW - I just did a search on "inspections" and the first 20 hits (by date) were all from 2003.

I don't believe inspections vs testing is the most useful comparison to be making. It reads as "one or the other" and I strongly believe the answer is "both."

Jeff

The problem is NOT in the techniques

The problems in the low productivity and low quality of software development, are not caused by using which technique. Actually, we have too many techniques to mislead the project development teams. They all are valid in theory, but few of them are valid for a real project development. Projects have been failing for so many years. If we are still insisting that Project Development Methodology is not part of the Project Management Methodology, projects will be continuously failing, just like the low productivity and low quality of Microsoft's products and Sun's J2EE deliverables.

Earlier in this thread, phila

Earlier in this thread, philarmour said:
"(b) The effectiveness of testing is closely related to the setup (and inspection) of the test cases and expected results. Someone (Pressman? Myers?) stated that the act of creating a good test is more effective at detecting defects than running it. So the two mechanisms are quite closely coupled."

And later, Jeff da Luca quotes Jones as "Indeed, it is surprising that testing can find requirements defects at all"

I would argue that test design should be done early in the process of software design.

When trying to decide requirements (or features), I find it useful to ask how can we test if this requirement is met.
Or: How can this feature be tested?

If the test is not at least conceptually clear, then the requirement (or feature, or class) should be treated with caution.

An approach that helps when implementing a class is design unit tests early, even before the class coded.

Not only does this help put automated tests into the build process, but it also helps pick up sloppy design quite early on.

I once heard someone say that inspecting documents was even more effective than inspecting code. I believe they had actually measured this in a large corp, but I did not get a reference. It makes sense as documents are easier to read than code, and impossible to test.

Document faults can be more expensive and harder to fix once they have been used to make a faulty system, so the savings can be significant if fixed up front. Document inspections also force more people to actually read the documents, and help authors to pay more attention to the quality, all of which has a number of benefits.

Although we value working code, we still need documents.

Jeff De Luca's picture

He said

Earlier in this thread, philarmour said:
b) The effectiveness of testing is closely related to the setup (and inspection) of the test cases and expected results. Someone (Pressman? Myers?) stated that the act of creating a good test is more effective at detecting defects than running it. So the two mechanisms are quite closely coupled."

And later, Jeff da Luca quotes Jones as "Indeed, it is surprising that testing can find requirements defects at all"

I would argue that test design should be done early in the process of software design.

They're not opposing statements (necessarily). They're actually saying different things. Jones never said that test design should not be done early. He never said it should be either.