Testing

One of the hardest tasks for any IT department is that of
testing. This is, in my experience, the area where we have the
greatest failures (next to estimating the cost and length of a
project) and where we have incredibly room for improvement.

I have been in the business for a long, long time, and the lack
of testing never fails to amaze me. There have been times when
I've received "finished" programs from developers which didn't
even run! Obviously the code had never been tested, at least not
in any meaningful way.

Before any testing can begin (and obviously this should also be
done before coding starts) you must have a thorough analysis and
design. You see, a program or system must be tested against the
specification and a set of standards. It cannot be done
arbitrarily or randomly.

Your specification explains what your systems are trying to
accomplish. The specification might say something like "a
standard URL will be accepted in the address field". Your
standards would state that all buffers must be checked for
overrun conditions, URLs in a valid format, and so on. The
standards apply to ALL testing, while the specifications apply
to the specific program or system.

A very critical fact (which seems to be completely unknown to
Microsoft) is the marketing department is not in charge of
testing. To be done correctly, testing actually requires
top-notch people who have been specially trained and who are
highly motivated to do their jobs well.

You also cannot make a hundred thousand copies of a product and
send it out to tens of thousands of beta testers without a clear
set of goals, expert supervision and constant management and
expect to get anything meaningful back. Beta testing is vital to
a project, but it does not and cannot replace professional
testing staff. Another fact which seems to be invisible to
Microsoft is the purpose of beta testing is to test, not to
market a product. Marketing is an essential part of a product
plan, but it has absolutely no place in the testing plan.

What are some of the common testing mistakes?

Testing to prove a program or system works - I know you want
your programs to work, but the purpose of testing is simply to
test, not to prove you are the best programmer on the planet.
Testing needs to hit a program hard, right between the eyes.
Your job as a tester is to ensure that the program meets the
specifications, and that any deficiencies are found and properly
recorded.

Trying to prove a program does not work - Again, the purpose of
testing is to test, not to prove anything. You should always
have a well defined testing plan and follow that plan.

Using testing to prototype a product - Prototyping is an
extremely useful part of the analysis and design phases of a
project. The purpose is to give your users and customers a way
to see what something will look and feel like before
implementing a project. Once design is done prototypes should be
thrown away and not used again.

Using testing to design performance - Performance goals must be
understood before a project leaves the design phase. By the time
a project is implemented (much less tested) you should
completely know how it will perform (minus the possibility of
bad programming, which is a different problem which testing is
designed to uncover). Testing will, however, validate that the
product does perform as indicated in the specifications.

Testing without a test plan - I don't know how many programmers
I've seen that just wade right in and start testing. Come on
people, how can you test something if you don't have a plan?
What are you trying to prove?

Testing without a specification - Remember, the purpose of
testing is to prove that a system or program meets the
specifications. That's all. It's very difficult to do that
without a specification right in front of you. Of course, this
assumes that you have a specification to begin with ...

Asking the developers to test their own programs - This is one
of the biggest mistakes (next to writing any code without a very
good specification) that you can make. How can a programmer test
his or her own code? First of all, programmers make lousy
testers - testing is a field all to itself and programmers are
almost never trained well in this area. Second, the developers
of a system have a conflict of interest - they want their
software to work. Testers need to approach with a more open mind.

Testing without a goal - If you don't have a goal in mind for
your testing, you don't know when you are done. What are you
trying to accomplish? Absolutely no bugs of any kind (not very
practicable)? The best goal is 100% compliance with the
specifications. This does pit the burden on the analysis and
design team - but that is exactly where the responsibility lies.

Expecting an unsupervised beta testing group to do anything
meaningful - Beta testers need well defined goals, constant
supervision and strong leadership to be successful. Without
these things beta testing is simply a numbers game which does
nothing useful at all.

Testing is not the appropriate time to make design decisions -
Something that I commonly see from Microsoft and other large
companies is they create a product and send it out to their beta
testers for feedback. Guys, come on. Beta testing is not the
place for this. Design decisions need to be made well before a
product is sent out for testing. You want to find out if your
users will like a feature? Create a prototype, send it to a
statistically valid sample audience and ask them for their
opinions. Clearly define it to the audience as a prototype and
survey them for their opinions afterwards. Don't send out a
poorly defined "beta test" to a hundred thousand people and try
and get their opinions on features. The only thing you are going
to accomplish is to get yourself slammed in the media. You also
find yourself making design changes in a product at the wrong
stage of the product life cycle. Design changes need to be made
during the analysis and design phases of the project, not AFTER
implementation.

So how is good testing done?

Analysis and design must be done first - No matter how large the
project, you will be much more likely to succeed if you do these
two steps thoroughly and completely before implementation and
testing. Many years ago I had a boss, name of Gary, who didn't
understand this simple rule. He asked me to start implementing a
warehousing system for a client without writing a specification
(over my objections). His concept of design was to spend an hour
or two asking the customer what was needed, then to start
coding, then to show the customer, make changes, code some more,
show the customer, make changes and so on until the customer
said "it's fine". Needless to say, the project took far longer
than necessary and did not do a great job of meeting the
customers needs. It was also very buggy and required an immense
amount of support during the first couple of years of it's life
cycle.

The only phase where the marketing department should be involved
is analysis - A well trained analyst understands that the
marketing department is a customer and must be included in the
analysis phase of the project. This is the only time (until the
product is through testing) that marketing should have any
input. If you don't follow this rule you will wind up with a
product which changes direction during testing, and thus
invalidates your test.

Understand that a specification is a contract - The goal is to
implement something that meets the specification. This is the
only way that I know of to produce a software project that (a)
gets finished at all and (b) meets the customers expectations.
Of course, this assumes that your analysis and design is top
notch. What does this really mean? It simply means that changes
to the design are only allowed during the analysis and design
phase. Period. If your customer changes anything at all after
the analysis and design, you must reanalyze, redesign and
renegotiate - always and without fail.

Let's say you are the contractor who has been hired to create a
new warehouse system. You do your analysis and design and it is
approved by the customer. You now have a contract and it is
important that your customer understands this from the
beginning. Okay, you begin the project and your customer decides
he wants to add bar coding. This seems pretty simple so you say
"sure". Wrong thing to do. You should say either "let's finish
the project as designed then add things" or "okay, we will need
to stop, see how that effects the project (at the customers
cost), then we will submit a cost and new delivery date".

Maintain standards - Testing measures the implementation against
the specifications and standards. Standards should be made known
to the customer as part of the entire package. These might
include things like all fields will be validated in specific
ways, all buffers will not overflow, screens will have a certain
look and so on.

Remember the purpose of testing - Testing should prove the
implementation meets everything included in the specifications
and standards. Testing does NOT mean the product is measured
against customer expectations (that's a marketing function which
should have been nailed down during the analysis and design
phase). You see, the specification MUST meet the customer
expectations before implementation beings. Then the final
product WILL meet customer expectations as the specification is
the expectation.

Richard Lowe Jr, the webmaster