






























































                       June 23, 1992





                             2





           Using Weak Mutation Coverage with GCT


                        Brian Marick
                    Testing Foundations

            Documentation for version 1.3 of GCT
                    Document version 1.2





The _G_e_n_e_r_i_c _C_o_v_e_r_a_g_e _T_o_o_l (_G_C_T) _U_s_e_r'_s _M_a_n_u_a_l describes  the
most  common  uses  of GCT.  This manual explains how to use
GCT to measure weak mutation coverage, a "researchy"  cover-
age measure.





































                       June 23, 1992








_P_r_e_f_a_c_e



GCT is free software; you can redistribute it and/or  modify
it under the terms of the GNU General Public License as pub-
lished by the Free Software Foundation; either version 1, or
(at your option) any later version.

GCT is distributed in the hope that it will be  useful,  but
WITHOUT  ANY  WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A  PARTICULAR  PURPOSE.   See
the GNU General Public License for more details.



For more information about  GCT  or  the  other  services  I
offer, contact:

Brian Marick
Testing Foundations
809 Balboa
Champaign, Illinois  61820

(217) 351-7228
Email: marick@cs.uiuc.edu

You can join the GCT mailing list by sending  mail  to  gct-
request@ernie.cs.uiuc.edu.
















This document is Copyright 8c9 1992 by Brian Marick.  Portions
are  Copyright  8c9 1991 by Motorola, Inc.  Under an agreement
between Motorola, Inc., and Brian Marick,  Brian  Marick  is
granted rights to these portions.

Brian Marick hereby permits you to reproduce  this  document
for personal use.  You may not reproduce it for profit.





                       June 23, 1992





                             2                 Weak Mutation


                        Introduction


_1.  _O_p_e_r_a_t_o_r _M_u_t_a_t_i_o_n _C_o_v_e_r_a_g_e

You can consider weak mutation coverage a generalization  of
relational operator coverage (though, in fact, weak mutation
coverage came first).  Relational coverage hypothesizes that
the  program  contains a misuse of a relational operator and
requires tests that detect that misuse.   The  same  can  be
done with misuses of other kinds of operators, in which case
we call the approach _o_p_e_r_a_t_o_r _m_u_t_a_t_i_o_n _c_o_v_e_r_a_g_e.

For example, it may be that a line of code is

    if (A && B)

but should be

    if (A || B)

The coverage condition that distinguishes the  correct  pro-
gram  from  the buggy program is (A&&B)!=(A||B).  One of the
arguments must  be  true,  the  other  false.   Notice  that
multi-condition  coverage  will  require  A true and B false
(because of C's short-circuit evaluation  of  &&).  In  this
case,  multi-condition coverage also satisfies weak mutation
coverage.   That's  not  true  of  all  operators,   though.
Another potential fault is using

        A + B

instead of

        A - B

This is ruled out when A+B != A-B, or, simplifying, B!=0.

Such measures  have  been  described  in  [Howden82],  [DeM-
illo78], [Hamlet77], and many later papers.

_2.  _O_p_e_r_a_n_d _M_u_t_a_t_i_o_n _C_o_v_e_r_a_g_e

Another common fault is to use the wrong  variable  or  con-
stant.   For example, consider the assignment statement A=C,
which should have been A=B.  The fault will  never  cause  a
failure  if B always equals C, so we must write at least one
test in which the two variables have different values.

These measures are due to [Howden82] and  [DeMillo78].   For
commentary  on  the  cost  of  weak  mutation  coverage, see
[Marick91a].





                       June 23, 1992





Weak Mutation                3                  Introduction


_3.  _T_h_e _R_e_s_t _o_f _T_h_i_s _D_o_c_u_m_e_n_t

The next chapter describes control file options important to
users of weak mutation.  The chapter after that describes in
detail what weak mutation does and what the reporting  tools
say  about  it.  The final chapter will help you when troub-
leshooting.


















































                       June 23, 1992





Weak Mutation                4                  Control File


                      The Control File


_4.  _T_u_r_n_i_n_g _o_n _W_e_a_k _M_u_t_a_t_i_o_n

The two types of weak mutation coverage  may  be  turned  on
separately:

    (coverage operator operand)
    (coverage operator)
    (coverage operand)

These two types of coverage are not  compatible  with  race,
routine, or call coverage.  You may use weak mutation cover-
age together with any other type of  coverage.   To  get  as
much coverage as you can, use

  (coverage branch loop multi operator operand)

You do not need to call for relational coverage, since  it's
a subset of operator coverage.


_5.  _O_p_t_i_o_n_s

The following options may be set (or turned off) in the con-
trol file.  They won't make sense until you've read the fol-
lowing chapter.

macros  Default: off

     Under operand coverage, macros that expand  to  a  con-
     stant,   variable   name,  array  reference,  structure
     dereference,  or  pointer  dereference  are   partially
     instrumented.  The top level construct is instrumented,
     but not any components.  That is, given

     #define ACCESS(ARRAY, OFFSET, VAL)  ARRAY[OFFSET+abs(myfun(VAL))]

     ACCESS(a, o, v)

     condition counts for the array access will be kept, but
     not  for  the  computations within the square brackets.
     greport will make reports like

     "lc.c", line 252: operand ACCESS might be index.
     "lc.c", line 252: operand ACCESS might be constant.
     "lc.c", line 252: operand ACCESS never differs from the previous element.

     Operators that expand into operators are ignored.  That
     is, in

     #define ADD(A+B)
     ADD(a, b)+5



                       June 23, 1992





Weak Mutation                5                  Control File


     operator coverage will be applied  to  the  +,  operand
     coverage will be applied to the 5, and the expansion of
     ADD(a,b) will be ignored.


dereference     Default: on

     Pointers are a problem for operand coverage.   Consider
     a statement like

             A := *P

     We would like to rule out the case where P  might  have
     been  Q by requiring a test case in which *P!=*Q.  This
     is dangerous if Q is not a valid pointer.  GCT attempts
     to  avoid the dangers by (for example) initializing all
     uninitialized pointers to  zero  and  ruling  out  null
     pointers  without  dereferencing them.  Not all dangers
     can  be  avoided,  but  GCT's  effectiveness  would  be
     markedly  diminished  if  it never dereferenced suspect
     pointers.  Consequently, the default is to  dereference
     pointers  when needed.  If a pointer is invalid, a core
     dump may result.  In this case, you  might  modify  the
     program  (to use calloc instead of malloc when allocat-
     ing arrays of pointers, for example) or  turn  off  the
     _d_e_r_e_f_e_r_e_n_c_e option.

array-substitutions     Default: on

     This is like the _d_e_r_e_f_e_r_e_n_c_e option, but it  turns  off
     fewer substitutions.  In an expression like

             array[i]

     turning _a_r_r_a_y-_s_u_b_s_t_i_t_u_t_i_o_n_s off will eliminate a cover-
     age condition that would require

             array[i] != other_array[i]


constants       Default: on

     By default, operand coverage requires every variable to
     actually  vary,  to  be  non-constant.   In practice, a
     large number of variables in correct  programs  do  not
     vary.  (Typically, these are variables whose values are
     calculated only once.)  This results in a great  number
     of  messages from greport, most of which are impossible
     to eliminate.  Sifting through them is a waste of time.
     The constants option can be used to turn them off.







                       June 23, 1992





Weak Mutation                6         The Types of Coverage


                      The Types of Coverage

_6.  _C_h_a_n_g_e_s _t_o _M_u_l_t_i_c_o_n_d_i_t_i_o_n _C_o_v_e_r_a_g_e

When multicondition  coverage  is  selected  alongside  weak
mutation  coverage,  it applies to more operators.  The com-
plete set of operators is

8     __________________________________________________
           Tree type          Point of instrumentation
8     __________________________________________________
      and                    &&
8     __________________________________________________
      or                     ||
8     __________________________________________________
      bitwise and            &
8     __________________________________________________
      bitwise or             |
8     __________________________________________________
      bitwise exclusive or   ^
8     __________________________________________________
7    |7|7|7|7|7|7|7|7|







                          |7|7|7|7|7|7|7|7|







                                                      |7|7|7|7|7|7|7|7|









As before, both the left-hand and right-hand sides  must  be
both 0 and non-zero.

Similarly, the set of operators that yield "boolean"  values
has been expanded:


8             _________________________________
              <    <=   >    >=   ==   !=
              &    |    ^    &&   ||   ~    !
              &=   |=   ^=
8             _________________________________
7            |8|7|7|

9                                             |8|7|7|



9
These operators must yield both zero and non-zero when  used
as the argument to return or an assignment statement.


Further, these kinds of  assignment  statements  must  yield
both zero and non-zero values:

8                       ______________
                        &=   |=   ^=
8                       ______________
7                      |8|
8                                    |8|

9
These extensions seldom add much cost to testing, since  the
operators are not commonly used, and the coverage conditions
are often ruled out without your ever seeing  them.   Still,
it's  doubtful they add much benefit, and they may disappear
in a later release.  Your opinion is welcome.


_7.  _O_p_e_r_a_t_o_r _C_o_v_e_r_a_g_e

All  operator  coverage  is  expressed   in   terms   of   a
hypothesized  correct  operation  that  can't  be ruled out.
Suppose we have the following code:


9                       June 23, 1992





Weak Mutation                7         The Types of Coverage


    if (a + b < c + d )

To rule out the fault where the first + should have  been  a
-, B must not be 0, (since a+0=a-0).  For the same reason, D
must not be 0.  To rule out the fault where < might  be  <=,
there  must  be  a case where A+B==C+D.  If a fault is ruled
out, nothing regarding it will appear in the  report  unless
the -all option is given.

A report will look like:

"example1.c", line 9: operator + might be -.
"example1.c", line 9: operator < might be <=.
"example1.c", line 9: operator + (2) might be -.

If -all is used, the number of times the condition is satis-
fied is reported:

"lc.c", line 275: operator <= might be >=  [1]
"lc.c", line 275: operator <= might be <.  [3]


_7._1.  _T_h_e _P_o_i_n_t _o_f _I_n_s_t_r_u_m_e_n_t_a_t_i_o_n

In all cases, the point of instrumentation is  the  operator
itself.

_7._2.  _A_r_i_t_h_m_e_t_i_c _O_p_e_r_a_t_o_r_s


8          _______________________________________
                       expr1 + expr2
8          _______________________________________
           Alternate          Rule out with
8          _______________________________________
           -            expr2!=0
8          _______________________________________
           *            expr1+expr2!=expr1*expr2
8          _______________________________________
7         |7|7|7|7|7|




                     |8|7|7|7|


9                                                |7|7|7|7|7|








8               _____________________________
                       expr1 - expr2
8               _____________________________
                Alternate     Rule out with
8               _____________________________
                +            expr2!=0
8               _____________________________
7              |8|7|7|7|


9                          |7|7|

                                           |8|7|7|7|




9

8          _______________________________________
                       expr1 * expr2
8          _______________________________________
           Alternate          Rule out with
8          _______________________________________
           +            expr1+expr2!=expr1*expr2
8          _______________________________________
7         |8|7|7|7|


9                     |7|7|

                                                |8|7|7|7|




99


                       June 23, 1992





Weak Mutation                8         The Types of Coverage


8       _____________________________________________
                       expr1 / expr2
8       _____________________________________________
        Alternate             Rule out with
8       _____________________________________________
        %            expr1 / expr2 != expr1 % expr2
8       _____________________________________________
7      |8|7|7|7|


9                  |7|7|

                                                   |8|7|7|7|




9

8       _____________________________________________
                       expr1 % expr2
8       _____________________________________________
        Alternate             Rule out with
8       _____________________________________________
        /            expr1 / expr2 != expr1 % expr2
8       _____________________________________________
7      |8|7|7|7|


9                  |7|7|

                                                   |8|7|7|7|




9

8         _________________________________________
                          - expr1
8         _________________________________________
                Alternate           Rule out with
8         _________________________________________
          non-negated expression   expr1!=0
8         _________________________________________
7        |8|7|7|7|


9                                |7|7|

                                                 |8|7|7|7|




9


































                       June 23, 1992





Weak Mutation                9         The Types of Coverage


_7._3.  _B_o_o_l_e_a_n _O_p_e_r_a_t_o_r_s


8        ___________________________________________
                      expr1 && expr2
8        ___________________________________________
         Alternate            Rule out with
8        ___________________________________________
         ||           expr1&&expr2 != expr1||expr2
8        ___________________________________________
7       |8|7|7|7|


9                   |7|7|

                                                  |8|7|7|7|




9
Ruling out & is too hard to satisfy, so that coverage condi-
tion is omitted.


8      _______________________________________________
                      expr1 || expr2
8      _______________________________________________
       Alternate              Rule out with
8      _______________________________________________
       &&           expr1 || expr2 != expr1 && expr2
8      _______________________________________________
7     |8|7|7|7|


9                 |7|7|

                                                    |8|7|7|7|




9
Ruling out | is, again, too hard to satisfy.


8               _____________________________
                           !expr
8               _____________________________
                Alternate     Rule out with
8               _____________________________
                ~            !expr != ~expr
8               _____________________________
7              |8|7|7|7|


9                          |7|7|

                                           |8|7|7|7|




9
The only case that does _n_o_t rule out ! is a  word  with  all
ones.
























                       June 23, 1992





Weak Mutation                10        The Types of Coverage


_7._4.  _B_i_t_w_i_s_e _O_p_e_r_a_t_o_r_s


8       ______________________________________________
                       expr1 & expr2
8       ______________________________________________
        Alternate             Rule out with
8       ______________________________________________
        |            expr1!=expr2
8       ______________________________________________
        &&           !!(expr&expr2) != expr && expr2
8       ______________________________________________
7      |7|7|7|7|7|




                  |8|7|7|7|


9                                                    |7|7|7|7|7|







In  the  second  case,  notice  that  the  _t_r_u_t_h  _v_a_l_u_e   of
expr1&expr2  must  be distinguishable from expr1&&expr2. The
weaker condition (expr1&expr2 != expr1 &&  expr2)  would  be
satisfied by

        expr1 = 0x5, expr2 = 0x3

The differences between  the  original  expression  and  the
variant  are more likely to cause a noticeable effect if one
of them is forced to 0.


8     __________________________________________________
                       expr1 | expr2
8     __________________________________________________
      Alternate               Rule out with
8     __________________________________________________
      &            expr1!=expr2
8     __________________________________________________
      ||           (expr1 | expr2) != (expr1 || expr2)
8     __________________________________________________
7    |7|7|7|7|7|




                |8|7|7|7|


9                                                      |7|7|7|7|7|







Unlike the previous case, the truth value of the  left  hand
side  is  not  used  --  it always has the same value as the
right hand side.


8          _______________________________________
                       expr1 ^ expr2
8          _______________________________________
           Alternate          Rule out with
8          _______________________________________
           |            expr1|expr2!=expr1^expr2
8          _______________________________________
           &            expr1^expr2!=expr1&expr2
8          _______________________________________
7         |7|7|7|7|7|




                     |8|7|7|7|


9                                                |7|7|7|7|7|







The second case is used because &-for-^ is a likely typo  --
the two keys are next to each other on the keyboard.


8               _____________________________
                          ~ expr
8               _____________________________
                Alternate     Rule out with
8               _____________________________
                !            ~expr != !expr



                       June 23, 1992





Weak Mutation                11        The Types of Coverage


8               _____________________________
7              |
7
7                                           |


The only case that does _n_o_t rule out ! is a  word  with  all
ones.


8              ________________________________
               expr1 << expr2, expr1 >> expr2
8              ________________________________
                Alternate       Rule out with
8              ________________________________
               each other      expr1!=0
8              ________________________________
7             |8|7|7|7|


9                            |7|7|

                                             |8|7|7|7|




9








































9


                       June 23, 1992





Weak Mutation                12        The Types of Coverage


_7._5.  _A_s_s_i_g_n_m_e_n_t

Coverage conditions rule out alternate operators, as  usual.
They also force test cases in which the assignment statement
makes a change in the program state.  For example, if, in "A
=  B",  A  and  B  always  have  the same value, perhaps the
assignment statement is never needed.

In the descriptions of how to rule out alternates,  _l_v_a_l__n_e_w
refers  to  the post-assignment value of the variable on the
left-hand-side of the assignment statement.  _l_v_a_l__o_l_d refers
to its pre-assignment value.

8    ___________________________________________________
                        lval = expr
8    ___________________________________________________
      Alternate                Rule out with
8    ___________________________________________________
     never needed   lval_old!=lval_new
8    ___________________________________________________
     ==             (!!lval_new) != (lval_old == expr)
8    ___________________________________________________
7   |7|7|7|7|7|




                 |8|7|7|7|


9                                                      |7|7|7|7|7|






The double negation converts the  value  of  the  assignment
statement  into  a truth value, which must be different from
the truth value of the expression gotten when = is  replaced
with  ==.   This  is  because  this  fault is often found in
expressions like:

             if (a = b)                           /* Meant a == b */

and the comparison of truth values forces the  other  branch
to  be  taken,  making  it  more  likely  that the effect is
observable.

8  _______________________________________________________
                       lval += expr
8  _______________________________________________________
      Alternate                  Rule out with
8  _______________________________________________________
   -=, never needed   lval_old!=lval_new
8  _______________________________________________________
   *                  lval_new != lval_old*expr
8  _______________________________________________________
   ==                 (!!lval_new) != (lval_old == expr)
8  _______________________________________________________
7 |8|7|7|7|7|7|7|





9                   |7|7|7|7|7|




                                                        |8|7|7|7|7|7|7|







9
== is used because it's a likely typo.

8          _______________________________________
                       lval -= expr
8          _______________________________________
              Alternate          Rule out with
8          _______________________________________
           +=, never needed   lval_old!=lval_new
8          _______________________________________
7         |8|7|7|7|


9                           |7|7|

                                                |8|7|7|7|




9


9


                       June 23, 1992





Weak Mutation                13        The Types of Coverage


8_____________________________________________________________
                        lval *= expr
8_____________________________________________________________
          Alternate                     Rule out with
8_____________________________________________________________
 /=, never needed                lval_old!=lval_new
8_____________________________________________________________
 +=                              lval_new != lval_old + expr
8_____________________________________________________________
 &=lval_new != lval_old & expr
8_____________________________________________________________
7|8|7|7|7|7|7|7|





9                              |7|7|7|7|7|




                                                            |8|7|7|7|7|7|7|







9
&= is used because it's a likely typo.

8      ________________________________________________
                        lval /= expr
8      ________________________________________________
          Alternate              Rule out with
8      ________________________________________________
       *=, never needed   lval_old!=lval_new
8      ________________________________________________
       %=                 lval_new != lval_old % expr
8      ________________________________________________
7     |7|7|7|7|7|




                       |8|7|7|7|


9                                                     |7|7|7|7|7|








































9


                       June 23, 1992





Weak Mutation                14        The Types of Coverage


8        ____________________________________________
                        lval %= expr
8        ____________________________________________
          Alternate            Rule out with
8        ____________________________________________
         never needed   lval_old!=lval_new
8        ____________________________________________
         /=             lval_new != lval_old / expr
8        ____________________________________________
         ^=             lval_new != lval_old ^ expr
8        ____________________________________________
7       |8|7|7|7|7|7|7|





9                     |7|7|7|7|7|




                                                   |8|7|7|7|7|7|7|







9
^= is a likely typo.

8       ______________________________________________
                lval <<= expr, lval >>= expr
8       ______________________________________________
              Alternate              Rule out with
8       ______________________________________________
        the other, never needed   lval_old!=lval_new
8       ______________________________________________
7      |8|7|7|7|


9                               |7|7|

                                                    |8|7|7|7|




9
8        ____________________________________________
                        lval &= expr
8        ____________________________________________
          Alternate            Rule out with
8        ____________________________________________
         never needed   lval_old!=lval_new
8        ____________________________________________
         |=             lval_new != lval_old | expr
8        ____________________________________________
         *=             lval_new != lval_old * expr
8        ____________________________________________
         ^=             lval_new != lval_old ^ expr
8        ____________________________________________
7       |7|7|7|7|7|7|7|7|







                     |8|7|7|7|7|7|7|





9                                                   |7|7|7|7|7|7|7|7|










^= and *= are likely typos.

8        ____________________________________________
                        lval |= expr
8        ____________________________________________
          Alternate            Rule out with
8        ____________________________________________
         never needed   lval_old!=lval_new
8        ____________________________________________
         &=             lval_new != lval_old & expr
8        ____________________________________________
         +=             lval_new != lval_old + expr
8        ____________________________________________
7       |8|7|7|7|7|7|7|





9                     |7|7|7|7|7|




                                                   |8|7|7|7|7|7|7|







9
+= is a likely typo.

8____________________________________________________________
                        lval ^= expr
8____________________________________________________________
  Alternate                    Rule out with
8____________________________________________________________
 never needed   lval_old!=lval_new
8____________________________________________________________
 |=             lval_new != lval_old | expr
8____________________________________________________________
 %=             expr == 0 || (lval_new != lval_old % expr)
8____________________________________________________________
 &=             lval_new != lval_old & expr
8____________________________________________________________



8                       June 23, 1992


9


8Weak Mutation                15        The Types of Coverage


999|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|
77777777777777777777777777777777777777777777799             |9|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|
7777777777777777777777777777777777777777777777899                                                           |99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|99|
7777777777777777777777777777777777777777777
%= and &= are likely typos.

expr == 0 rules  out  the  %-substitution  because  such  an
alternate program would get a divide-by-zero error and hence
could not be correct.


8              ________________________________
               lval++, lval--, ++lval, --lval
8              ________________________________
                Alternate       Rule out with
8              ________________________________
7             |7|7|

                            |8|
8                                             |7|7|




There  is  no  instrumentation.   Any  evaluation  of  these
expressions  has  a  different value from any of the others;
hence, simple execution rules out those substitutions.

Similarly, any evaluation of --lval is distinguishable  from
-lval because the value of lval is different.
































9


                       June 23, 1992





Weak Mutation                16        The Types of Coverage


_7._6.  _F_l_o_w _o_f _C_o_n_t_r_o_l


8        ___________________________________________
                           while
8        ___________________________________________
         Alternate            Rule out with
8        ___________________________________________
         if            two or more loop executions
8        ___________________________________________
7       |8|7|7|7|


9                   |7|7|

                                                  |8|7|7|7|




9
In the current implementation, operand  coverage  for  while
statements  is  the same as loop coverage.  The greport mes-
sage is the same:

"example1.c", line 9: loop zero times: 1, one time: 0, many times: 0.

Executing the while test more than once rules out the possi-
bility that it should have been an if.

Unfortunately, there's no equivalent  condition  that  rules
out  the  more  common  fault where an if should have been a
while.


8               _____________________________
                   goto, continue, break
8               _____________________________
                Alternate     Rule out with
8               _____________________________
7              |7|7|

                          |8|
8                                           |7|7|




There is no instrumentation for goto; if the goto target  is
incorrect,  the error will likely be detected by a test that
executes the goto, which branch coverage requires.

Similarly, the error where a continue  should  have  been  a
break,  or  a  break  a continue, will likely be detected if
executed, especially if loop coverage is required.

_8.  _O_p_e_r_a_n_d _C_o_v_e_r_a_g_e

In operand coverage, values of variables  or  constants  are
compared   against   each  other.   The  following  are  all
operands:

        A
        1
        Array[1]
        Array[1+1]
        S.field
        S->field
        *ptr
        *(ptr+S.field)

Operand coverage is a  requirement  on  pairs  of  operands.
Suppose we have a program containing this code:


9                       June 23, 1992





Weak Mutation                17        The Types of Coverage


        function(A);

Operand coverage might require the following:

        A != B
        A != C
        A != D

This could be accomplished in one test in  which  the  vari-
ables had these values:

        A == 0, B == 1, C == 1, D == 1

or it might take three tests:

        A == 0, B == 1, C == 0, D == 0  => Rules out B (A!=B).
        A == 1, B == 0, C == 1, D == 0  => Rules out B (A!=B) and D (A!=D).
        A == 0, B == 0, C == 1, D == 0  => Rules out C (A!=C).

The idea of operand coverage is that if a particular operand
has the same value as another in every test, we have no evi-
dence that the  correct  program  shouldn't  use  the  other
operand.   (Of  course, if the operand has the same value in
every possible execution, it doesn't matter which we use.)

Output  is  in  terms  of  the  existing  operand  and   the
hypothesized substitution.  For example, given

if (A < F(5, A))

output might be

"example1.c", line 9: operand A might be B.
"example1.c", line 9: operand A might be C.
"example1.c", line 9: operand A might be constant.
"example1.c", line 9: operand 5 might be A.
"example1.c", line 9: operand A (2) might be B.

(In the third line, notice that there is no way to  indicate
which constant A might be.)

_8._1.  _C_o_m_p_a_t_i_b_i_l_i_t_y

A pair of operands can be compared only if  C  defines  ine-
quality between them.  For example, integers can be compared
to floats, but floats cannot be  compared  to  whole  struc-
tures.   (There  is  almost no circumstance in which a float
should have been a structure without generating  a  compiler
error or warning.)

This is not ideal.   For  example,  consider  the  following
code:

   func(s1, s2)



                       June 23, 1992





Weak Mutation                18        The Types of Coverage


   char *s1, *s2;

   if (func2(s1))
        return 5;
   ...

GCT will rule out the substitution of S2 for S1  if  S2!=S1.
However,  this  comparison  of  pointers is not optimal.  In
this context, S1 and S2  are  intended  to  be  strings.   A
better comparison would be to rule out the substitution if

        string_not_equal(s1, s2)

However, this cannot be done automatically, since GCT has no
way  of  knowing  that  these  character  pointers represent
null-terminated strings.

_8._2.  _S_e_l_e_c_t_i_n_g _A_l_t_e_r_n_a_t_e_s

There may still be an  infinite  number  of  type-compatible
operands.  How are alternates selected?  For example, should
A be compared to all of these possible operands?

        Array[0]
        Array[1]
        Array[2]
        ...
        Array[A]
        Array[B]
        Array[C]
        ...
        Array[A+B]
        ...

Clearly, this is not useful.  GCT uses two general rules  to
pick  alternate operands.  Each kind of operand (e.g., array
references, pointer  dereferences)  extends,  restricts,  or
modifies the set of alternates produced by those two rules.

constants
     For any operand, all constants of a compatible type are
     potential alternates. (For variables, this infinite set
     of comparisons can be implemented by requiring that the
     variable actually take on at least two values.)

local variables
     All variables of a compatible type, defined in the same
     function,  are  potential alternates.  Global variables
     are _n_o_t  alternates.   There  are  often  a  surprising
     number  of them (hidden in include files) and not test-
     ing for them doesn't (yet) seem to lead to any loss  of
     effectiveness.

     For the example above, A would be compared to B, C,  D,



                       June 23, 1992





Weak Mutation                19        The Types of Coverage


     but  not  to  any  array reference.  An array reference
     like Array[A] would be compared to A, B,  C,  D,  etc.,
     but not to any other array reference.  (Under this rule
     -- we'll see in the section after next how  comparisons
     between array references are generated.)

_8._3.  _W_e_a_k _S_u_f_f_i_c_i_e_n_c_y

It's not always enough just to check for  differing  values.
Consider, for example,

    if (var1 == var2)

Suppose var2 should be var3.  Suppose further that  var1  ==
1,  var2 == 2, and var3 == 3.  var2!=var3, so operand cover-
age is satisfied.  However, the if statement  has  the  same
value  in  both cases, so the fault might quickly "damp out"
and become unlikely to be  detected.   We  really  want  the
relational expression to take on a different value.

Consequently, if a variable  comparison  is  made  when  the
variable  in  question is the ENTIRE right-hand or left-hand
side of a relational expression, then not only must the com-
parison  reveal  a  difference, but the results of the rela-
tional operator must be different.

This  rule,  whose  technical  name  is   _w_e_a_k   _s_u_f_f_i_c_i_e_n_c_y
[Marick91b],  will  certainly  cause you confusion at first.
You will have an expression like

        f(a, b, c)
        {
           if (a < b)
           ...

and see this message from greport

"example1.c", line 9: operand B might be C.

and say, "That's impossible -- the test case was F(1, 2,  3)
and  2!=3",  having  forgotten  that, because of weak suffi-
ciency, you must instead satisfy this coverage condition:

        (a < b) != (a < c)

The early annoyance will pay off in better tests.


_8._4.  _W_e_a_k _S_u_f_f_i_c_i_e_n_c_y _f_o_r _C_o_m_p_o_u_n_d _O_p_e_r_a_n_d_s

Weak sufficiency also  applies  to  compound  operands  like
array  references, structure field references, and so forth.
The rule is similar: Whenever an operand is immediately con-
tained  in  a  larger operand, the entire containing operand



                       June 23, 1992





Weak Mutation                20        The Types of Coverage


must have a different value from the operand formed by  sub-
stituting  the  alternate  contained operand.  Some examples
will make this clear:

Array[A] must differ from Array[B].  It is not enough for  A
to differ from B.

Array[Array2[A]] must differ from Array[Array2[B]].

*PTR must differ from *PTR2.  It is not enough  for  PTR  to
differ from PTR2.

However, in the case of *(ptr+1), we  only  require  PTR  to
differ from PTR2.  We do not require *(PTR+1) to differ from
*(PTR2+1), because PTR is not immediately contained  in  the
dereference.   (The  increased  difficulty  of devising test
cases for such comparisons doesn't seem to be worthwhile.)

_8._5.  _R_e_f_i_n_e_m_e_n_t_s _f_o_r _P_a_r_t_i_c_u_l_a_r _K_i_n_d_s _o_f _O_p_e_r_a_n_d_s

_8._5._1.  _C_o_n_s_t_a_n_t_s

Constants follow the general local variable rule.  Thus, you
might expect to see messages like

"lc.c", line 153: operand 3 might be c.
"lc.c", line 252: operand '{' might be index.
"lc.c", line 919: operand "panic: %s" might be reason.

The constant rule would seem to  be  unneeded:   what's  the
point  of comparing a constant to another constant?  In gen-
eral, none.  But there is  a  point  when  weak  sufficiency
holds.  Consider

if (a < 5)

GCT will generate tests that require you to demonstrate that
no  other  constant  but  5 would do.  In this case, setting
A==5 rules out any constant greater than 5.   (If  5  should
have  been  6,  say, the if statement would branch the other
way.)  Setting A=4 rules out any constant less than 4.   (If
5  should  have  been 4, the if statement would again branch
the other way.)  Hence, 5 is likely to be correct.

In such cases, the message from greport will be "5 might  be
another constant".  Here are the various cases, and coverage
conditions that will eliminate them:

        A < 5
                A==4
                A==5
        A <= 5
                A==6
                A==5



                       June 23, 1992





Weak Mutation                21        The Types of Coverage


        A > 5
                A==6
                A==5
        A >= 5
                A==4
                A==5
        A != 5
                A==5
        A == 5
                A==5


_8._5._2.  _S_i_z_e_o_f

The _s_i_z_e_o_f operator is a  constant.  Constant  substitutions
are  done  for  _s_i_z_e_o_f  operators used in relational expres-
sions.  Variable substitution  tests  aren't  done  --  it's
unlikely  that  a sizeof expression should have been a vari-
able.

_8._5._3.  _A_r_r_a_y _R_e_f_e_r_e_n_c_e_s

The usual constant and variable substitution tests are  used
for array references.  Expect to see messages like

"lc.c", line 869: operand chars[...] might be ch.
"lc.c", line 869: operand chars[...] might be constant.


For an operand like Array[A] you will also see messages like

"lc.c", line 869: operand A might be B.

As discussed in section 4.6.4, this means that Array[A] must
differ  from  Array[B].   Notice,  however,  that GCT cannot
blindly insert such comparisons: suppose  B  had  the  value
-100,000,000?   They are made only if 0 <= B < A.  Expect to
be confused several times because you know that  B  is  dif-
ferent  from A, but you still get the greport message that A
might be B.  You've forgotten that B must both be  different
than A and also safe as an index.

GCT will also consider B ruled out if B<0,  though  this  is
less reliable.

For any array reference, GCT also requires a  test  case  in
which  Array[A]  differs  from  Array[A-1].  This encourages
non-homogeneous test arrays, which are much more  likely  to
find  faults.   Of  course,  such a test can only be made if
A>0.

GCT also requires test cases  where  Array[5]  differs  from
Array2[5]  (or  Array2  is  null).   These  test  cases  are
dangerous  if  Array2   is   uninitialized.    The   -array-



                       June 23, 1992





Weak Mutation                22        The Types of Coverage


substitutions or -dereference options turn them off.

_8._5._4.  _F_I_E_L_D _R_E_F_E_R_E_N_C_E_S

This discussion applies  to  both  references  of  the  form
A.field and also P->field.

The usual constant and variable substitution tests are used,
so you'll see messages like

"lc.c", line 476: operand name_tally->pure_code might be constant.
"lc.c", line 476: operand name_tally->pure_code might be B.

In addition, the tests must rule out substitutions of a dif-
ferent  field  in  the  same structure.  For example, if the
structure is defined as

        struct S
        {
           int field1;
           int field2;
        };

an operand like p->field1 will yield this message:

"lc.c", line 476: operand p->field1 might use field field2.

The normal variable substitution on P,  when  combined  with
weak  sufficiency,  will  require that you rule out possible
substitutions like

        q->field1

Failing to do so will yield a message like

"lc.c", line 476: operand p might be q.


Notice that

        p->field is compared with p->field2
and
        p->field is compared with q->field
but
        p->field is _n_o_t compared with q->field2.

Double typos of this sort are rare  and  are  likely  to  be
caught by tests adequate for single typos [Offutt92].

_8._5._5.  _D_e_r_e_f_e_r_e_n_c_e_s

Pointer dereferences use the standard constancy and variable
substitution tests.  Expect to see messages like:




                       June 23, 1992





Weak Mutation                23        The Types of Coverage


"lc.c", line 250: operand *p might be argc.
"lc.c", line 250: operand *p might be constant.

You will also see messages like

"lc.c", line 250: operand p might be q.

Because of weak sufficiency, you must construct  test  cases
such that *p != *q.

_8._6.  _M_i_s_c_e_l_l_a_n_e_o_u_s

There is no instrumentation  for  the  address-of  operator.
Any  likely  alternate operand for &var (such as &var2) will
almost always yield a different value; hence, merely execut-
ing the expression is enough of a coverage condition.

There is no instrumentation for casts.  Although  there  are
faults  associated  with  cases, I don't have enough of them
yet.





































                       June 23, 1992





Weak Mutation                24              Troubleshooting


                      Troubleshooting

See _G_C_T _T_r_o_u_b_l_e_s_h_o_o_t_i_n_g for help with general GCT problems.

If you have trouble  that  you  think  others  might  share,
please    mail    a    description   of   the   problem   to
marick@cs.uiuc.edu, so that this chapter might be improved.


_9.  _T_r_o_u_b_l_e _I_n_s_t_r_u_m_e_n_t_i_n_g

Compiler runs out of space
     You are most likely using operand coverage and an old C
     compiler.  You'll have to turn off operand coverage.

GCT panics.
     GCT will panic if given an expression like

          ((int) p) = 5;

     This is not legal C, but some code  contains  it.   The
     only solution is to change the source.

Warnings about comparisons.
     Suppose we have this code:

          (i < 10000000)

     Suppose c is a character.  We will  end  up  with  this
     test, because characters are comparison-compatible with
     integers:

          _G(i < 10000000 != c < 1000000)

     Because the constant is larger than any character, this
     will lead to warnings:

     Sun CC:
     "file.c", line 208: warning: constant 1007 is out of range of char comparison
     "file.c", line 208: warning: result of comparison is always true

     GCC:
     file.c: In function remove_intermediates:
     file.c:208: warning: comparison is always 1 due to limited range of data type

     It can also lead to warnings from the  assembler.   You
     can safely ignore all these warnings.

_1_0.  _T_r_o_u_b_l_e _H_a_n_d_l_i_n_g _G_r_e_p_o_r_t _O_u_t_p_u_t

Cannot figure out which complex operand is meant.
     If you see

     "example1.c", line 9: operand <...>[...] (2) might be C.



                       June 23, 1992





Weak Mutation                25              Troubleshooting


     this refers to the second complex  array  reference  on
     the  line  (even if they are not identical).  For exam-
     ple, in this line:


          p->array[5] + Array[p->array[4]];

     p->array[5] is printed as operand <...>[...].
     Array[p->array[4]] is printed as operand Array[...].
     p->array[4] is printed as operand <...>[...] (2).


Cannot rule out operand coverage when operand  is  in  rela-
     tional expression.
     Greport messages of the form

     "example1.c", line 9: operand A might be B.

     can be frustrating to rule out because of  weak  suffi-
     ciency.  (See the discussion earlier in this document.)
     If A is a part of  a  relational  operator  expression,
     such as

          A < 3

     It is not enough to provide a  test  case  where  A!=B.
     You must provide a test case where (A<3)!=(B<3).

Cannot rule out operand coverage when operand is in nested
     operand. (See also above case.)  Weak sufficiency  also
     applies  to  operands built up from structures, arrays,
     and pointer dereferences.  For example, in the  expres-
     sion

          function(A[3].foo)

     the "A might be B"  message  must  be  ruled  out  with
     A[3].foo!=B[3].foo.  Even worse, if the expression is

          A[3].foo < 5

     the needed coverage condition is

          (A[3].foo) < 5 != (B[3].foo < 5)


Cannot rule out operand coverage when operand is array
     index. In A[I], "I might be J" can only be ruled out if
     0<=J<I.  It would be dangerous to use other values of J
     to index into the array.   GCT  will  also  consider  J
     ruled  out  if it is less than zero.  That is, to elim-
     inate the greport message, you must satisfy this cover-
     age condition:




                       June 23, 1992





Weak Mutation                26              Troubleshooting


          j<0 || ( 0 <= j < i  && A[i]!=A[j])


Wrong variable seems to be named.

     In lines like

     "example1.c", line 9: operand A (2) might be B.

     the parenthetical number is NOT instances of A  on  the
     line,  but rather the number of instances that have had
     instrumentation generated for them.  These  are  almost
     always  the  same  number.   However,  if  a particular
     instance of A on a line has no associated coverage con-
     ditions,  the  numbering  will  be confusing.  The only
     common case of this is in assignment.  For example, in

     A = A + 1

     the first A has no coverage conditions.  greport  would
     therefore show:

     "example1.c", line 9: operand A might be B.
     "example1.c", line 9: operand A might be C.

     referring to the _s_e_c_o_n_d A on the line.

     Sorry about that.

Variable is always reported to be constant.

     Operand mutation requires variables to  actually  vary;
     they must take on two different values _i_n _t_h_e _s_a_m_e _e_x_e_-
     _c_u_t_i_o_n _o_f _t_h_e _p_r_o_g_r_a_m.  It is not enough for a variable
     to  have the value 5 when the program is run once, then
     value 3 when the program is run again.  This  can  make
     certain  variables impossible to vary, because they are
     set once in the  beginning  and  never  change.   argv,
     argc,  and  standard  I/O library buffers (stdin, etc.)
     exhibit this  problem.   You  can  use  the  -constants
     option to suppress constancy instrumentation entirely.

Instrumented program dumps core.

     Most likely, instrumentation caused dereferencing of an
     uninitialized (but non-zero) pointer or array.  See the
     discussion of the -dereference and -array-substitutions
     options   in   the   section   on   the  control  file.
     gcorefrom(1) can help you locate which  instrumentation
     caused the problem.

     It's also possible the program has a bug.

Cannot rule out = might be ==.



                       June 23, 1992





Weak Mutation                27              Troubleshooting


You must have (!!B) != (A==B).  The easiest thing to try for
is two non-equal, non-zero numbers.  (This also rules out "=
is never needed".) Having both A and B be 0 also works.

Watch out for the !!:  it means that A==2, B==2,  for  exam-
ple, will not rule out the condition.



















































                       June 23, 1992





Weak Mutation                28                 Bibliography


_I_I_I. _B_i_b_l_i_o_g_r_a_p_h_y


[DeMillo78]
     R.A. Demillo, R.J. Lipton, and F.G. Sayward, 'Hints  on
     test  data  selection: help for the practicing program-
     mer'. _C_o_m_p_u_t_e_r.  vol. 11, no.  4,  pp.   34-41,  April,
     1978.

[Hamlet77]
     R.G. Hamlet.  _I_E_E_E _T_r_a_n_s_a_c_t_i_o_n_s _o_n  _S_o_f_t_w_a_r_e  _E_n_g_i_n_e_e_r_-
     _i_n_g, vol. SE-3, No. 4, pp. 279-289, 1977.

[Howden82]
     W. E. Howden.  _I_E_E_E _T_r_a_n_s_a_c_t_i_o_n_s _o_n _S_o_f_t_w_a_r_e  _E_n_g_i_n_e_e_r_-
     _i_n_g, vol. SE-8, No. 4, pp. 371-379, July, 1982.

[Marick91a]
     Brian Marick. 'Experience with the Cost of  Test  Suite
     Coverage Measures'.  _P_a_c_i_f_i_c _N_o_r_t_h_w_e_s_t _S_o_f_t_w_a_r_e _Q_u_a_l_i_t_y
     _C_o_n_f_e_r_e_n_c_e,  October,  1991.   Also  available   as   a
     compressed          postscript          file         in
     cs.uiuc.edu:/pub/testing/experience.ps.Z.

[Marick91b]
     Brian Marick. 'The Weak Mutation Hypothesis'.  _P_r_o_c_e_e_d_-
     _i_n_g_s _o_f _t_h_e _A_C_M _S_I_G_S_O_F_T _S_y_m_p_o_s_i_u_m _o_n _T_e_s_t_i_n_g, _A_n_a_l_y_s_i_s,
     _a_n_d _V_e_r_i_f_i_c_a_t_i_o_n, October, 1991.

[Offutt92]
     A. Jefferson Offutt. 'Investigations  of  the  Software
     Testing Coupling Effect'.  _A_C_M _T_r_a_n_s_a_c_t_i_o_n_s _o_n _S_o_f_t_w_a_r_e
     _E_n_g_i_n_e_e_r_i_n_g _a_n_d _M_e_t_h_o_d_o_l_o_g_y, vol. 1,  No.  1,  January,
     1992.























                       June 23, 1992


