Archive-name: software-eng/testing-faq
Posting-Frequency: semi-monthly
Last-modified: 1996/09/24
Version: $Revision: 1.13 $
URL: http://www.rstcorp.com/c.s.t.faq.html


                      comp.software.testing FAQ
	              and information resource	

This is the Frequently Asked Questions list for comp.software.testing.
This FAQ is posted twice a month.  If your site is set up properly, you
should always be able to find the latest version in news.answers or
comp.answers.

You'll also find the FAQ at the various rtfm ftp sites.  See the FAQ 
"Introduction to the *.answers newsgroups" in news.announce.newusers 
for more information.  One such location is:
<ftp://rtfm.mit.edu/pub/usenet/news.answers/software-eng/testing-faq>.

The FAQ is available on the web at http://www.rstcorp.com/c.s.t.faq.html.
For a list of some of the web sites which carry all FAQs in the rtfm
archives, see the "Introduction to the *.answers newsgroups" at
<ftp://rtfm.mit.edu/pub/usenet/news.answers/news-answers/introduction>.

The FAQ is still under construction (and like most FAQs, it always will
be).  I'm maintaining this FAQ to collect a consensus from the
comp.software.testing community, so it depends on feedback for it to
take shape.  Please send me corrections and additions and other
comments.

Herein, unless otherwise stated, "I" refers to Danny Faught.  

------------------------------

Subject: 1. Table of Contents

    1. Table of Contents
    2. What is this newsgroup about?
    3. What other FAQs are relevant?
    4. What other resources are available?
    5. World Wide Web resources
    6. Netiquette 
    7. I'm looking for a test tool...
    8. Beta testing
    9. How do I find information about testing object-oriented programs?
   10. What is black box/white box testing?  (under construction)
   11. Conferences
   12. Periodicals
   13. Books
   14. Bibliographic resources
   15. Organizations
   16. Contributors

------------------------------

Subject: 2. What is this newsgroup about?

If you're new to Usenet, please read through the FAQs in
news.announce.newusers and hang out in news.newusers.questions for a
while before you consider posting.

The original charter for comp.software.testing can be found at
<http://tsunami.jpl.nasa.gov/TEL/docs/cst-charter.html>.  Here's an
excerpt:

    CONTENTS:

    For the purpose of this discussion, we will assume that a system is
    comprised of [sic] hardware, software, people, and procedures. The
    proposed group should be chartered to include discussions
    characterized by the following list of subjects, compiled from
    preliminary discussions:

        system test automation 
        system test planning 
        system test optimization (e.g. minimize regression testing) 
        test tools 
	test metrics 
        measurement technology 
        nature of testing under various development models 
	    (e.g. object-oriented, real time, etc...) 
        testing in a rapid prototyping environment (i.e. sans spec) 
	relationship of various classes/types of tests to requirements, ... 
	conference and symposium announcements 
	the definition of "software testing" and its relationship to 
	    SQA and debugging. 
	requirements traceability 
	risk-based testing 
	the most useful form of specifications and functional 
	    requirements from the tester's point of view. 
	testing techniques, e.g., structured testing using control 
	    flowgraphs and basis path testing, equivalence class 
	    partitioning, boundary value analysis, cause-effect graphing, 
	    path predicate testing, data flow testing, program slices, 
	    data object state and usage analysis, data flow anomaly 
	    analysis, and sensitivity analysis. 
	 test coverage criteria, e.g., statement (C1), path, branch (C2), 
	    module (S0), and call-pair (S1) coverage criteria. 

    All levels of testing for both hardware and software should be
    considered appropriate subject matter. While it is likely that the
    group will take on a software flavor early on, we should try to
    generate interest and input from people with hardware and system
    testing experience and perspective. It is especially important that
    we foster discussion of *system* level testing issues, since this
    is the weakest area of our collective knowledge.

------------------------------

Subject: 3. What other FAQs are relevant?

Brian Marick maintains the following FAQs and posts them periodically:
    Testing Contractors and Consultants
    Testing Courses
    Testing Tool Suppliers

Please consult the appropriate FAQ before before asking questions
about the information already contained therein.  The Testing Tool 
Suppliers FAQ also contains a nice list of the available software test
tools.

These FAQs are also available at <http://www.stlabs.com/marick/root.htm> 
and <ftp://cs.uiuc.edu/pub/testing/faqs/>.

Also, since software testing is a subdiscipline of software engineering, 
you will probably find the FAQs posted to comp.software-eng useful.  
They can be found at <ftp://rtfm.mit.edu/pub/usenet/comp.software-eng/> or 
<http://www.qucis.queensu.ca/Software-Engineering/>

------------------------------

Subject: 4. What other resources are available?

A searchable archive of every message ever posted to c.s.t. continues to 
operate at 

    <http://tel.jpl.nasa.gov/bin/browse>

(but some articles seem to be missing)

Articles are also available a month at a time in "mbox" format from:

    <ftp://tel.jpl.nasa.gov/pub/cst-archive>

There's a mailing list gateway for comp.software.testing.  To
subscribe, send mail to majordomo@tel.jpl.nasa.gov with "subscribe cst"
in the body of the message.  To receive the articles periodically in
digest format, use "subscribe cst-digest" instead.  Send 
"unsubscribe cst <optional: address>" to unsubscribe.  Contact
Majordomo-Owner@tel.jpl.nasa.gov if you have questions.

Statistics about comp.software.testing are available at
<http://sunsite.unc.edu/usenet-i/groups-html/comp.software.testing.html>.

Danny Faught and Mark Wiley maintain a mailing list for people who like
to talk about the implementation details of OS testing and test tools.
Mail swtest-discuss-info@convex.com with "subscribe swtest-discuss" in
the body of the message to subscribe.  Send "unsubscribe swtest-discuss"
to unsubscribe.

There is an ISO 9000 mailing list.  Mail to Listserv@VM1.nodak.edu with 
"subscribe iso9000 (your name)" in the body of the message, or 
"Signoff iso9000" to unsubscribe.

There is also a list devoted specifically to ISO 9000-3.  To join,
send mail to majordomo@quality.org with "subscribe iso9000-3" in the
body of the message.  Send "unsubscribe iso9000-3" to leave the list.
There's also an iso9000-3-digest list to get the same information in 
digest format.

There is a mailing list for Microsoft Visual Test.  To join, mail to
mt_info-request@eskimo.com with "subscribe" in the subject line.  See
also <http://www.stlabs.com/mst.htm#List_serv>.  You can also read the
microsoft.public.visualtest newsgroup from the msnews.microsoft.com
news server.

Segue Software sponsors the QAPUSER mailing list for QA Partner.  
To join, send "subscribe QAPUSER <your_name>" in the body of a mail
message to listproc@segue.com.  Send "unsubscribe QAPUSER" to
unsubscribe.  The list uses a digest format.

To join the discussion list about ADL (Assertion Definition Language
system), send your name, affiliation, and e-mail address to
xopubadl@xopen.org.

SQA Suite users have a mailing list.  Send mail to
sqa-suite-users-request@maillist.dundee.net with 
"SUBSCRIBE sqa-suite-users" in the body of the message.  Send 
"UNSUBSCRIBE sqa-suite-users" to unsubscribe.  For help, send a "HELP"
message to the address above, or contact postmaster@sugaree.dundee.net.

------------------------------

Subject: 5. World Wide Web resources

Here are a few web sites that each try to connect you to much of the
software testing information that is available on the web.

    The RST Reliability Hotlist 
	<http://www.rstcorp.com/rst-web-top.html> 

    STORM (Software Testing Online Resources/MTSU) 
	<http://www.mtsu.edu/~storm>

------------------------------

Subject: 6. Netiquette 

Did I mention that you should read the FAQs in news.announce.newusers if
you haven't done so lately?  Also, you can reduce the chances of
embarrassing yourself by reading comp.software.testing for a while
before posting.  At the very least, scan all the articles in the group
currently on your server to make sure your question isn't already been
asked and answered.

If you post an article and later decide you shouldn't have posted it,
please cancel it.  It is much better to correct the problem than to
send *another* post to apologize for the first one.  Situations where
this would be appropriate include posting empty articles, multiple
copies of the same article, and any goof where you'd like to correct
something you posted shortly after it was sent.  Look in your
newsreader documentation for help with cancelling an article; some also
have a "supercede" feature which makes it easy to correct a posting.

While the cancel feature has been common for quite some time, some
newer newsreaders unfortunately do not offer it.  If this is the case,
complain to the author, and consider using a more reasonable
newsreader.  Here's a manual method to cancel an article - start to
follow up to the errant posting.  Change the subject to 
"cmsg cancel <message-id>", where message-id is taken from the
References: header or the attribution.  The body of the message is not
important.  Post it, and the news system should interpret it as a
cancel request rather than a real posting.  You can only do this for
an article you posted yourself.

There are classes of postings that are of interest to some readers, but
others don't welcome them at all.  The easiest compromise is to clearly
identify certain types of postings in the subject line.  Decent
newsreaders allow you to automatically delete articles with subject
line patterns that you specify.

Job postings must have a subject line that starts with "JOB:", exactly 
as shown, for example:
"Subject: JOB: need an experienced foo tester"

Job postings that are cross-posted to the various job-related
newsgroups should not be posted to comp.software.tesing, since the
subject conventions for those groups do not allow them to be easily
identified as job postings, and the misc.jobs FAQ specifies that job
postings should not be cross-posted between the job-related newsgroups
and non job-related newsgroups.

Other advertisements and commercial product announcements should 
have a subject that starts with "AD:".  For example: 
"Subject: AD: foo tester 1.0 released"

People have also expressed concern about postings that are totally
off-topic.  These postings are typically "spam" postings that go to
most of the thousands of newsgroups that exist.  There is very little
that a group can do proactively to prevent them, besides recruiting a
moderator (and I don't think that would be appropriate for
comp.software.testing).  See news.admin.net-abuse.announce for more
information.

Please don't post test messages to comp.software.testing.  This group
is for discussing software testing, not for testing your news
software.  If you want several automatic confirmations that your post
worked, post a test message to misc.test.  There are other test groups
at various levels of the news hierarchy that all end in ".test".  It's
best to try a local one first.

------------------------------

Subject: 7. I'm looking for a test tool...

Did you read the Testing Tool Suppliers FAQ (see "What other FAQs are
relevant?" above)  It has a list of many of the available tools, plus a
list of the commercial tools listings.  While every question about test
tools isn't answered there, it's a good start, and it'll at least allow
you to ask a more directed question if you do post a question about
test tools.

------------------------------

Subject: 8. Beta testing

Discussions about managing a beta test program are within the scope of
this newsgroup.  However, companies looking for beta testers, and
aspiring beta testers looking for something to test, should go to
comp.sources.testers instead.  A possible exception is test tool
vendors looking for beta testers.

If you are looking for a career in software testing,
comp.software.testing will welcome you.  Just don't call it "beta
testing".  Beta testing is typically conducted by end users of a
software product who are not paid a salary for their efforts.

------------------------------

Subject: 9. How do I find information about testing object-oriented programs?

Information provided by Brian Marick:

Books that include some discussion of testing OO programs:
    Jacobson, Object-Oriented Software Engineering
    Marick, The Craft of Software Testing
    McGregor, Object-Oriented Software Development

There are a vast number of articles about testing OO programs.  I
suggest starting with the September 1994 Communications of the ACM,
which is devoted to this topic.  Chase references.   The February 1996
issue of Object has a piece on system testing with use cases. 

Conferences like STAR and Quality Week and Pacific Northwest Software
Quality Conference seem to always have papers on this topic.  I
suggest going to a conference and buttonholing people.

Web sites with discussion:
    <http://www.clemson.edu/~johnmc/johnmc.html> (this link is not
	working as of 6/23/96)
    <http://www.cs.washington.edu/homes/gmurphy/testSTApp.html>
    <http://www.stlabs.com/marick/root.htm>
    <http://www.toa.com>

Three courses on testing object-oriented code (one by Robert Binder,
one by Ed Berard, and one by Reliable Software Technologies) are
described in the Testing Courses FAQ.  Also, a new addition is an
offering from Software Quality Engineering.

Bob Binder says:

My approach to oo testing is called FREE (Flattened Regular
Expressions) -- see <http://www.rbsc.com/pages/FREE.html>

You might also find the oo testing biblography of interest.
    <http://www.rbsc.com/pages/ootbib.html>

------------------------------

Subject: 10. What is black box/white box testing?

Black-box and white-box are test design methods.  Black-box test design
treats the system as a "black-box", so it doesn't explicitly use
knowledge of the internal structure.  Black-box test design is usually
described as focusing on testing functional requirements.  Synonyms for
black-box include:  behavioral, functional, opaque-box, and
closed-box.  White-box test design allows one to peek inside the "box",
and it focuses specifically on using internal knowledge of the software
to guide the selection of test data.  Synonyms for white-box include:
structural, glass-box and clear-box.

While black-box and white-box are terms that are still in
popular use, many people prefer the terms "behavioral" and
"structural".  Behavioral test design is slightly different from
black-box test design because the use of internal knowledge isn't
strictly forbidden, but it's still discouraged.  In practice, it
hasn't proven useful to use a single test design method.  One has
to use a mixture of different methods so that they aren't hindered 
by the limitations of a particular one.  Some call this "gray-box"
or "translucent-box" test design, but others wish we'd stop talking 
about boxes altogether.

It is important to understand that these methods are used during the
test design phase, and their influence is hard to see in the tests
once they're implemented.  Note that any level of testing (unit
testing, system testing, etc.) can use any test design methods.  Unit
testing is usually associated with structural test design, but this
is because testers usually don't have well-defined requirements at
the unit level to validate.

Definitions of these terms can be found in these references:

Beizer, _Black Box Testing_, p. 8.
Beizer, _Software Testing Techniques_, pp. 10-11.
Daich, et al., _STSC Software Test Technologies Report_, pp. 198, 206.
Mosley, _The Handbook of MIS Application Software Testing_, pp. 68-76.
Myers, _The Art of Software Testing_, pp. 8-11.
Wilson, _Unix Test Tools and Benchmarks_, 298, 306

------------------------------

Subject: 11. Conferences 

Comments in this section are by Boris Beizer.

All of these conferences have published proceedings.  Generally,
materials presented at these conferences and in their proceedings lead
the publication in regular journals (above) by about 2-3 years.

1.  ISSTA (Annual), theory, academic, leading edge practitioners (see 3b
    in the periodicals section.)

2.  IEEE Conference on Software Engineering.  Spring, world-wide.
    Technical.  Primary source after ISSTA for leading edge results.

3.  Quality Week.  Annual, San Francisco.  Biggest Conference on
    Testing and QA.  Typically 700+.  Many vendors.  Good balance
    between technical/theoretical and practitioners.  Very broad base.
    Workshops.  End of May.  Software Research Institute:  625 Third
    Street, San Francisco, CA 94107-1997:  qw@soft.com,
    http://www.soft.com/QualWeek/index.html

4.  Pacific Northwest Quality Conference.  Annual, Portland Oregon, End 
    September.  Definitely not a regional conference, despite the
    name.  Very broad from technical/research to practitioners.  A
    shade more academic and leading edge participation than quality
    week.  Terri  Moore, Conference Coordinator.  Pacific Agenda, 2425
    NW Overton Str., (PO BOX 10142) Portland, OR 97210.
    terri_moore@aortel.org
    http://www.teleport.com/~pnsqc/

5.  International Conference on Testing Computer Software.  Washington
    DC, Begin June.  13th annual coming up.  Less technical and more
    practitioner/management than Quality Week.  Smaller, 400+ .  June
    10-13.  Workshops.  USPDI:  uspdi@clark.net

6.  STAR.  Software Quality Engineering.  Also EuroSTAR in Europe.
    May, various locations.  About the same target as the International
    Conference (5) above.  Comparable level and interest.  Workshops.
    SQE 1-800-423-8378.

7.  QAI International Conference.  More of a tutorial/workshop than a
    conference.  Newbie orientation.  Fall, Orlando.  Quality Assurance
    Institute, 407-363-1111.

8.  IEEE Compsac, world-wide, various locations. Fall.  Typical IEEE
    technical conference with a substantial number of papers of direct
    interest to testing and QA.  Not as many as IEEE Software
    Engineering, though.

There are about a dozen other annual conferences with a substantial
testing content: Korea, Japan, Germany, to name a few.  In addition,
there are about two-dozen "conferences", privately sponsored by various
individuals and organizations.  These are not real conferences in that
there is no formal review process.  Speakers are invited by the
conference organizer.  Often, a heavy tutorial content.  These are
usually small ( under 100 participants, very few vendors).  They range,
depending on the organizer, from superb to fair-value, to outright
rip-off.  Sorry.  I won't play 20 questions on this one.

(end of Beizer's comments)

Also of note:

    The International Conference on Software Quality, sponsored by the
    ASQC.  http://wwwsel.iit.nrc.ca/6ICSQ/

    International Symposium on Software Reliability Engineering.
    Contact: Ram Chillarege, ramchill@watson.ibm.com, 914-784-7375.
    http://www.research.ibm.com/softeng/cf/issre96.html

------------------------------

Subject: 12. Periodicals

Comments in this section are by Boris Beizer

1.  IEEE Transactions on Software Engineering.  (monthly)
    The most prestigious journal for testing stuff.  Volume 1 number 1
    published the landmark article by Goodenough and Gerhart.  Almost
    every issue since has had papers on testing and quality
    assurance.  I have all the issues back to V1 #1.  Much of that
    stuff is still relevant.

2.  ACM TOSEM (Transactions on Software Engineering Methodology)-Quarterly.
    Relatively new journal  (1992).  Has prestigious editorial
    board.  Somewhat more theoretical than IEEE SE.  In all ways,
    equivalent.

3a.  ACM SIGSOFT Software Engineering Notes.  (Special Interest group on
     Software Engineering)  (Monthly)
     Goes back to May 1976.  This is the place where people publish
     stuff to stake out claims.  It is not as strictly refereed as IEEE
     SE or TOSEM, but some overview is provide to keep out the obvious
     crap.  Important because there is a two or three year waiting list
     for publication in IEEE SE or ACM TOSEM.  People who know this
     field read ACM SIGSOFT for advanced information on what will
     be published "officially" in IEEE SE and ACM TOSEM.  You read an
     article and contact the author directly for more information.

3b.  Annual ISSTA conference proceedings.  ISSTA is the annual theoretical/
     Academic conference devoted to testing.  This is the most
     prestigious place to publish new results in testing theory and
     software reliability theory.

4.  IEEE Software.  (Six times a year). 
    Rarely the latest stuff.  More like surveys and overviews once a
    subfield has become established.  Refereed, generally high
    standards.  Mostly overviews, but occasionally new stuff.

    Additional comments from Scott Killops:

	"You might mention in your blurb about 'IEEE Software' the
	regular "Quality Time" feature.  This is wide ranging and
	generally quite useful.  I've often thought that a collection
	of all of the 'Quality Time' articles published to date would
	make a worthwhile book."

5.  Software Testing, Verification, and Reliability.  Quarterly.
    International journal published by John Wiley and Sons.
    Prestigious international editorial board.  High quality original
    articles and excellent surveys.  Comparable in scope to IEEE SE
    and ACM TOSEM, but sharply focused on testing, verification, and
    reliability.  ISSN 0960-0833.

	http://www.csc.liv.ac.uk/users/mrw/stvr.main.html
	http://www.wiley.com/
	info@qm.jwiley.com
	John Wiley & Sons, Inc.
	Professional, Reference and Trade Group
	605 Third Avenue
	New York, NY  10158-0012

6.  ACM Computing Surveys.  (Quarterly).
    Not specific to testing and QA, but contains the most prestigious
    survey articles in the field.  Typically, only when a subfield is
    well established.  The authors are usually authorities.  Articles
    are long and comprehensive.  When an ACM Survey on a topic
    appears, it usually means that the field has matured to the point
    where it is possible to write meaningful books.

7.  Communications of the ACM.  (Monthly)
    Survey articles and overviews.  Sometimes (rarely) original stuff.
    More academic and foundational/theory oriented than IEEE Software,
    but generally same level.

8.  American Programmer.  Monthly, Cutter Information Corp.  37 Broadway,
    Suite 1, Arlingon MA 02174-5552 (617-648-8702).
    Private journal with prestigious editorial board (Larry
    Constantine, Bill Curtiss Tom DeMarco, Capers Jones, Ed Yourdon.
    Non technical, philosophical and overviews.  Management
    orientation. Essentially a good newsletter.

9a. Software Maintenance News.  (6 times a year)  Software Maintenance News 
    Inc.  B10, Suite 237, 4546 El Camino Real, Los Altos CA, 94022.
    General on testing and quality assurance and not just
    maintenance.  Invited articles, usually in an overview/survey
    style with minimal technical prerequisites by authors who may have
    previously published more advanced versions of this material in
    IEEE SE or TOSEM.  Essentially a good newsletter.

9b. Software Management Technology Reference Guide (annual)
    Editor same as Software Maintenance News--Nicholas Zvegintzov.  A
    superb guide to tools, vendors, resources, edited by the unique
    Nicholas Zvegintzov in his inimitable style.  Really packed.
    Probably the  highest signal to noise ratio in the industry.
    73670.22227@compuserve.com

10. Software QA Magazine.   (Bi-monthly).  Subscriptions: Software
    Quality Engineering, sqeinfo@sqe.com, 800-423-8378, fax 904-268-0733.  
    Published by Ridgetop Publishing, Ltd., ridgetop@teleport.com, 
    phone/fax 503-829-6806, http://www.teleport.com/~ridgetop/
    QA, testing,   Practitioners, management.  Somewhat conservative.   
    Myers' The Art of Software Testing (1978) was reviewed in Winter 
    1994 issue.

	Formerly called Software QA Quarterly.

11. The Software Practitioner (news letter, 6 times a year).
    Computing Trends, 350 Dalkeith Avenue, Los Angeles Ca. 90049 
    310-440-9982.  Excellent newsletter.  Non technical, management, 
    overviews.

12. Crosstalk. Journal of Defense Software Engineering.  Software Technology
    Support Center, Ogden AFB UT 84056.  801-777-8057.  
    (6 times a year) Heavy DoD orientation.  Practitioner/Management/
    Philosophical.  Broad based on software engineering, including IV&V.
    http://www.stsc.hill.af.mil/www/crostalk.html

13. Other Journals.  I'm sure I've offended several editors.  There are
    Journals concerned with Testing, QA, and Software Engineering
    published in Japan, Korea, Germany, and Australia.   Other
    Journals, such as IEEE-Transaction Computers, ACM SIGPLAN, ACM
    SIGARCH, ACM SIGSIM, ACM SIGMETRICS, ACM OOPSLA,  all publish
    material of interest to testing, especially insofar as hardware
    testing results and theory apply to software testing; also 
    language issues vis-a-vis testing.

(end of Beizer's comments)

commercial newsletters-

    STL Report: A Newsletter about Quality Assurance for Market-Driven Software
    bi-monthly, subscriptions are free
    for subscriptions: Trina Primitivo (206) 682-5832, ext. 138, 
		       trinap@stlabs.com
    editorial concerns: Prue Cuper at (206) 682-5832, ext. 137, 
			pruec@stlabs.com

    TESTING TECHNIQUES NEWSLETTER
    Free.  TTN/Online published monthly, TTN hardcopy is biannual
    Software Research, Inc.
    (415) 957-1441, (800) 942-SOFT, FAX: (415) 957-0730
    ttn@soft.com

    The Outlook
    McCabe & Associates, 800-638-6316

------------------------------

Subject: 13. Books

The book list has been growing quite a bit lately, to the point that
it's hard to decide which ones to start with.  Based on votes from
comp.software.testing readers in answer to the question "What books
should every software tester read?", the following books were
suggested most often, with the number of votes in parentheses:

    Testing Computer Software, Kaner et al. (11)
    The Art of Software Testing, Myers (10)
    Software System Testing and Quality Assurance, Beizer (8)
    The Craft of Software Testing, Marick (6)
    SafeWare, Leveson (4)
    Software Testing Techniques, Beizer (4)

The votes will be maintained and the list will be periodically
adjusted as necessary.  To add or change your vote, send email to the
FAQ maintainer.

Below is a book list originally based on one posted by Scott Killops
quite a while ago.  It has been proposed that we add the following
information for each (but we haven't found volunteers yet to provide
this information):

-Audience (i.e. beginner, intermediate, expert level needed for comprehension)
-Book topics (i.e. black box, white box, sub-system, oo, etc)

See <http://sqe.com/single/single.htm> for Software Quality
Engineering's SingleSource guide, which has a list of recommended books
and useful summaries.  See the misc.technical.books FAQ
<ftp://rtfm.mit.edu/pub/usenet/news.answers/books/technical> for
contact information for technical book publishers.
Comp.software.testing readers have recommended Reiter's Scientific and
Professional Books <http://www.reiters.com> and the Computer Literacy
Book Shop <http://www.clbooks.com/>.

Arnold, Thomas R.  NuMega's Practical Guide to Software Testing with 
Visual Test 4.0.  Indianapolis, IN: IDG Books Worldwide, 1996.  
ISBN 0-7645-8000-0.

Beizer, Boris.  Black Box Testing.  New York: John Wiley & Sons, 1995,
ISBN 0-471-120904-4.

Beizer, Boris.  Software System Testing and Quality Assurance.
New York: Van Nostrand Reinhold, 1984, ISBN 0-442-21306-9.

Beizer, Boris.  Software Testing Techniques, Second Edition.
New York: Van Nostrand Reinhold, 1990, ISBN 0-442-20672-0.

Bryan, William L. and Stanley G. Siegel.  Software Product Assurance:
Techniques for Reducing Software Risk.  New York: Elsevier, 1988.
ISBN 044401120X.

Daich, Gregory, Gordon Price, Bryce Ragland, and Mark Dawood.  STSC
Software Test Technologies Report.  USAF Software Technology Support
Center, 1994.

DeMarco, Tom.  Controlling Software Projects: management, measurement
& estimation.  New York, NY: Yourdon Press, 1982.  ISBN 0917072324.

DeMillo, R.A., W. M. McCracken, R. J. Martin, and J. F. Passafiume, 
Software Testing and Evaluation.  Menlo Park, CA: Benjamin/Cummings,
1987.  ISBN 0-8053-2535-2.

Deutsch, M.S.  Software Verification and Validation - Realistic
Project Approaches.  Englewood Cliffs, NJ: Prentice Hall, 1982.
ISBN 0-13-822072-7.

Dunn, Robert H.  Software quality: concepts and plans.  Englewood
Cliffs, NJ:  Prentice Hall, 1990.  ISBN 0-13-820283-4.

Friedman, Michael and Jeffrey M. Voas.  "Software Assessment:
Reliability, Safety, Testability".  New York:  Wiley Press, 1995.
ISBN 0-471-01009-X.  <http://www.rstcorp.com/book.html>

Hetzel, Bill.  The Complete Guide to Software Testing, Second Edition
Wellesley, MA: QED Information Sciences, 1988.  ISBN 0-89435-242-3.

Jorgensen, Paul.  Software testing: a craftsman's approach.  Boca
Raton: CRC Press, 1995.  ISBN 084937345X.
<http://www.crcpress.com/PRODS/7345.HTM>

Ince, Darrel.  An introduction to software quality assurance and its
implementation.  London, New York:  McGraw Hill, 1994.  ISBN
0-07-707924-8.

Jacobson, Ivar, etc. al.  Object-oriented software engineering: a use 
case driven approach.  New York:ACM Press, Wokingham, Eng.; Reading,
Mass.: Addison-Wesley, 1992.  ISBN 0201544350.

Gause, Donald C. and Gerald M. Weinberg.  Exploring Requirements: 
quality before design.  New York, NY: Dorset House Pub., 1989.  ISBN
0932633137.

Grady, Robert B. and Deborah L. Caswell.  Software Metrics:
establishing a company-wide program.  Englewood Cliffs, N.J.:
Prentice-Hall, 1987.  ISBN 0138218447.

Jain, Raj.  The Art of Computer Systems Performance Analysis:
techniques for experimental design, measurement, simulation, and
modeling.  New York: Wiley, 1991.  IBSN 0471503363.

Kaner, Cem., Falk, J., Nguyen, H.Q.  Testing Computer Software, Second
Edition.  New York: Van Nostrand Reinhold, 1993.  ISBN 0-442-01361-2.

Kaplan, Craig, Ralph Clark, and Victor Tang.  Secrets of software
quality: 40 innovations from IBM.  New York: McGraw-Hill, 1995.  ISBN
0079117953.

Kit, Edward.  Software Testing in the Real World: Improving the
Process.  ACM Press/Addison-Wesley, 1995.  ISBN 0-201-87756-2.

Knight, Peter and James Fitzsimons.  The Legal Environment of
Computing.  Sydney; Reading, Mass.: Addison-Wesley Pub. Co., 1990.
ISBN 0201417014.

Leveson, Nancy G.  Safeware: System Safety and Computers.  Reading,
Mass.: Addison- Wesley, 1995.  ISBN 0-201-11972-2.
<http://www.cs.washington.edu/research/projects/safety/www/book.html>

Lyu, Michael R., ed.  Handbook of Software Reliability Engineering.
New York: McGraw Hill, 1996.  ISBN 0070394008.
<http://www.research.att.com:80/orgs/ssr/book/reliability>

Marick, Brian.  The Craft of Software Testing.  Englewood Cliffs,
NJ: Prentice Hall, 1995.  ISBN 0-13-177411-5.
<http://www.prenhall.com/013/177410/17741-0.html>

Marks, D.M.  Testing Very Big Systems.  New York: McGraw Hill,
1992.  ISBN 0-07-040433-X.

McConell, Steve.  Code Complete:  a practical handbook of software
construction.  Redmond, Washington: Microsoft Press, 1993.  ISBN
1-55-615484-4.  

McGregor, John D. and David A. Sykes.  Object-oriented software 
development: engineering software for reuse.  New York, N.Y.: Van
Nostrand Reinhold, 1992.  

Mosley, Daniel J.  The Handbook of MIS Application Software Testing:
methods, techniques, and tools for assuring quality through testing.
Englewood Cliffs, NJ: Prentice Hall, 1993.  ISBN 0-13-907007-9.
<http://www.prenhall.com/013/907006/90700-6.html>

Musa, John D.  Software reliability: measurement, prediction,
application.  New York: McGraw-Hill, 1990 professional ed., 1987.
ISBN 007044093-X (1987) 0070441197 (1990).

Myers, Glenford J.  The Art of Software Testing. New York: John Wiley
and Sons, 1979.  ISBN 0-471-04328-1.

Nielsen, Jakob.  Usability engineering.  Boston: Academic Press,
1993.  ISBN 0125184050.

Ould, M.A., Unwin, C., eds.  Testing in Software Development.
Cambridge: Cambridge University Press, 1986.  ISBN 0-521-33786-0.

Parrington, Normal and Marc Roper.  Understanding Software Testing.  Ellis
Horwood Limited, 1989.  ISBN 0-7458-0533-7 (Ellis Horwood Limited), ISBN
0-470-21462-7 (Halsted Press).

Paulk, Mark C., et al.  The capability maturity model: guidelines 
for improving the software process.  Reading, Mass.: Addison-Wesley 
Pub. Co., 1995.  ISBN 0201546647.

Perry, William E.  Effective Methods for Software Testing.  New York:
Wiley, 1995.  ISBN 0471060976.

Perry, William E.  A standard for testing application software.
Boston: Auerbach Publishers, 1992.  ISBN 0791309754.

Perry, William E.  A structured approach to systems testing.
Wellesley, Mass.: QED Information Sciences, 1988.  ISBN 0894352334.

Royer, Tom C.  Software Testing Management - Life on the Critical
Path.  Englewood Cliffs, NJ: Prentice Hall, 1993.  ISBN 0-13-532987-6.
<http://www.prenhall.com/013/532986/ptr/53298-6.html>

Rubin, Jeffrey.  Handbook of usability testing: how to plan, design,
and conduct effective tests.  New York: Wiley, 1994.

Wiener, Lauren.  Digital Woes: why we should not depend on software.
Reading, Mass.: Addison-Wesley Pub. Co., 1993.  ISBN 0201626098.

Wilson, Rodney.  UNIX Test Tools and Benchmarks.  Prentice Hall, 1995.
ISBN 0-13-125634-3.

------------------------------

Subject: 14. Bibliographic resources

Comments by Boris Beizer.

1.  ACM Reviews -- Montly.  the primary review journal.  Most good papers on
testing and quality assurance are eventually reviewed here.  Extensive
evaluations and criticisms by excellent reviewer pool.  
  
2. ACM Guide to Computing Literature.  Annual.  Does not contain reviews.
Most comprehensive published literature guide for authors, titles, and
subject.

3.  Key Abstracts in Software Engineering.  Published by INSPEC, but
available through IEEE membership.  Monthly.  About halfway between ACM
Reviews and ACM Guide to the literature.  Thorough, best articles.  However,
only author abstracts and no evaluation or criticism.  

------------------------------

Subject: 15. Organizations

I'm building a list of organizations that are related to software
quality, both local and global.  At least, I'll work on the list of
local organizations until it grows out of control - there seem to be
quite a few of them.

National/International
----------------------

American Society for Quality Control
    611 E. Wisconsin Ave.
    P.O. Box 3005
    Milwaukee, WI 53201-3005
    800-248-1946, (414) 272-8575
    <http://www.asqc.org>

    The ASQC administers the CQE (Certified Quality Engineer) designation.
    Also, the software division of the ASQC is developing the CSQE
    designation which is specific to software quality.
    
    The ASQC press has many titles that may be of interest to software
    testers.  Stolen from a conference announcement:

    > Founded in 1946, the American Society for Quality Control (ASQC)
    > provides a variety of professional, educational, and informational
    > programs reflecting the changing needs of business and industry.
    > Headquartered in Milwaukee, WI, ASQC has been the leading quality
    > improvement organization in the United States for almost 50 years.
    > The Society is composed of more than 135,000 individual members and
    > 1,000 sustaining members worldwide. ASQC's vision is to be the
    > world's recognized champion and leading authority on all issues
    > related to quality.  The mission of ASQC is to facilitate
    > continuous improvement and increased customer satisfaction by
    > identifying, communicating, and promoting the use of quality
    > principles, concepts, and technologies.
    >
    > ASQC's Software Division
    > 
    > ASQC's Software Division is comprised of [sic] more than 5,000 members
    > including software quality professionals and software engineers
    > interested in applying quality principles to the field of software
    > development.  The Division develops a software engineer
    > certification program, publishes a quarterly newsletter, works with
    > the Software Quality System Registration Committee on establishing
    > an ISO 9000 Software Registration Program in the United States,
    > interacts with other professional software organizations such as
    > the IEEE and the ACM (Association for Computing Machinery), and is
    > involved in many other activities.

National Quality Month
    October is National Quality Month (in the U.S.), as decreed by 
    Ronald Reagan in 1984.

    The ASQC sponsors the National Quality Forum during this month.
    Many local sections set up local events that receive a live
    broadcast from the Forum.

Quality Assurance Institute  
    7575 Dr. Phillips Blvd., Suite 350
    Orlando Florida  32819
    Phone: (407) 363-1111
    Fax:   (407) 363-1112

    The QAI publishes The Journal of the Quality Assurance Institute
    monthly.

    They provide CQA certification.  Steve Driscoll wrote:

    > The CQA is someone who has demonstrated a mastery of the Common
    > Body of Knowledge(CBOK) for ensuring the quality of systems (which
    > often include software).  While ISO-9000 is considered part of the
    > CBOK, it's actually independent of the CQA designation (there is a
    > certification program for ISO auditors).
    > ...  while my life is still pretty much the same as it was prior to
    > obtaining the CQA designation, I found that the reading and
    > studying to prepare for the exam extremely beneficial in my duties
    > as a Quality Analyst.  I recommend it to anyone who is interested
    > in producing quality software (which includes developers as well
    > as quality analysts).

Society for Software Quality
    PO Box 86958
    San Diego, CA 92138-6958
    619-297-1544

Software Process Improvement Network
    > The Software Process Improvement Network is comprised of
    > individuals who want to improve software engineering practice. The
    > individuals are organized into regional groups called "SPINS" that
    > meet and share their experiences initiating and sustaining software
    > process improvement programs. They meet annually at the SEPG
    > (Software Engineering Process Group) Conference, which is
    > co-sponsored by the SEI and a regional SPIN.  
    (from the web page)

    <http://www.sei.cmu.edu/participation/spins.html>


Local organizations
-------------------

Most of the organizations above sponsor local organizations.  Contact 
them to find local organizations near you.  In this section I'll list
local organizations whose national affiliation is unknown.


The Software Quality Institute At UT

    (stolen from a conference announcement)
    > The Software Quality Institute (SQI) at The University of Texas at
    > Austin is a multidisciplinary partnership between UT-Austin and the
    > software and information systems industries in Texas.  It is
    > recognized as a leading authority on and champion for software
    > quality.  SQI was founded in 1993 for the purpose of strengthening
    > organizations to compete more successfully in global markets
    > through sponsorship of seminars and conferences. An advisory group
    > of 24 industry and academic representatives guides the Institute.
    > Its popular programs include: a 13 week long Software Project
    > Management Certificate Program, one- and two-day seminars offered
    > to the public and as in-company contract courses, our "Software
    > Quality Matters" quarterly newsletter that addresses issues of
    > concern relative to quality issues, sponsorship of monthly forums
    > for discussion of software issues (including the Software Process
    > Improvement Network (A-SPIN), the Austin Software Executives' Group
    > (ASEG), and the Austin Forum for Object-Oriented Technology
    > (AFOOT), and on-line services which include a Worldwide Web
    > Home Page and topical newsgroups.

    The web page is <http://www.utexas.edu/coe/sqi/>.  The newsgroups
    seem to be reachable only at UT.  For information on newsletter
    subscriptions, contact info@sqi.utexas.edu.

Central Ohio Quality Assurance Association
    
Australian Software Quality Research Institute
Griffith University, Brisbane, Australia
<http://www-sqi.cit.gu.edu.au/>

Triangle Information Systems Quality Association (North Carolina)
contacts - Dennis Brandel 72652.1146@compuserve.com
	   David Wood  DAVDWOOD@VNET.IBM.COM

Triangle Quality Council (North Carolina)
    "A membership organization that fosters the understanding and
    implementation of quality management principles.   It is a
    non-profit organization affiliated with N.C. Quality Leadership 
    Foundation and Triangle Area Chambers of Commerce"  (paraphrased
    from Laura Rose)

    Mission:  To promote quality management principles and practices
    through networking, experience sharing and education - providing
    a complete and diversified quality management resource for the
    Greater Triangle Area.

    contact - Margaret O'Brien  obrien@ies.ncsu.edu

North Carolina Quality Leadership Foundation 
    (800) 207-5685

New England Software Quality Assurance Forum (NESQAF)
    Monthly meetings in Cambridge, MA.  
    contact - Peter J. Vasile (pvasile@sqp.com)

Software Testing Institute 
    Offers industry research, surveys, publications, online services
    and seminars to software development and testing professionals.  A
    list of software testing product vendors, including web site and
    e-mail links, is available from our web site:

    http://www.ondaweb.com/sti

    contact - Susan Archer, Director (sarcher@metronet.com)
    2639 Elm, No. 304, Dallas, TX  75226 
    (214) 741-3046  FAX (214) 741-3319

------------------------------

Subject: 16. Contributors

My thanks to the people who have contributed to the FAQ in various
ways.  Email addresses for some of those mentioned herein are listed 
below.

Boris Beizer - BBeizer@gnn.com
Bob Binder - rbinder@smtp.mcs.net
Steve Driscoll - sdriscol@oclc.org
Danny Faught - faught@convex.com 
Scott Killops - Scott_B_Killops@ccm.jf.intel.com
Brian Marick - marick@cs.uiuc.edu or marick@testing.com 
Laura Rose - laura@PACorp.com
