
Subject: [l/m 3/17/92] Measurement environments..(12/28) c.be FAQ
Date: 12 Apr 1996 12:25:06 GMT

12.Measurement environments...<This panel>
13.SLALOM
14
15.12 Ways to Fool the Masses with Benchmarks
16.SPEC
17.Benchmark invalidation methods
18
19.WPI Benchmark
20.Equivalence
21.TPC
22
23
24
25.Ridiculously short benchmarks
26.Other miscellaneous benchmarks
27
28.References
1.Introduction to FAQ chain and netiquette
2
3.PERFECT
4
5.Performance Metrics
6.Temporary scaffold of New FAQ material
7.Music to benchmark by
8.Benchmark types
9.Linpack
10.Network Performance
11.NIST source and .orgs

Benchmarking environments
Taxonomy described by: A, B, C, D, E
Examples:
.A Cray Y-MP or better with integrated hardware performance monitor
.B Cray-2, Convex C-1/C-2...
.C VAX-11/780, IBM PC, IBM 370-class
.D Apple II, Timex Sinclair
.E Incomplete hardware, chip, module, board-level

An A environment should have state of the art hardware and software
to aid benchmarking.  Hardware should be minimally or non-intrusive.
This category should change with time.

A B environment is a "good" environment to do benchmarking.
It has as a minimal, a near cycle-time clock, profiling software,
etc.  The environment at this level or higher may also be more hostile
for measurement.  Good performance is likely to be an issue, so the
compilers will have good optimizers.

A C environment is an "average" benchmarking environment.
This is just typical: a 60 or 100 Hz clock.  Some profiling software.

A D environment is a difficult environment to do benchmarking.
This environment may not have a clock or software to help the poor
benchmarker.  Timing by wrist watch or maybe oscilliscope.
Software? What software?

An E environment:
Another difficult environment to do benchmarking.  It might have software.
It might be an incomplete computer: a processing using of a larger
multiprocessor: it touches on the issue of composition.

If dealing with a simulator:
there maybe an F category for simulated timing.

                   ^ A  
                s / \ r                
               m /   \ c              
              h /     \ h            
             t /       \ i          
            i /         \ t        
           r /           \ e      
          o /             \ c    
         g /               \ t  
        l /                 \ u
       A /                   \ r
        <_____________________> e   
                Language
 
From billf@mccoy.dbaccess.com  Tue Jan 12 20:36:23 1993
From: billf@mccoy.dbaccess.com (Bill Frantz)
Message-Id: <9301122252.AA11791@mccoy.dbaccess.com>
Subject: Re: [l/m 3/17/92] Measurement environmentsII(12/28) c.be FAQ

Please note that the IBM-370 series including the XA and ESA systems
(but not the 360 series) have a Store Clock instruction which stores
a 64 bit clock value.  All of the models that I know of (with the
exception of the XT/370 and AT/370 cards for PCs) increment the clock
at a rate that is close to the rate at which the machine issues
instructions.  (The definition of the value is in units of 1/4096
microseconds, but the actual increment and frequency is implimentation
defined.)

If you have any further questions, please send mail to me at:
   frantz@netcom.com

Regards - Bill Frantz


