# Performance Tuning Tips for Digital Unix

Below is a set of newsgroup posts made by an engineer from DEC in response to queries about how to modify DEC's Digital Unix OS for more heavily loaded web sites. Copied with permission.

## Update

From: Jeffrey Mogul
Date: Fri, 28 Jun 96 16:07:56 MDT

1. The advice given in the README file regarding the "tcbhashsize" variable is incorrect. The largest value this should be set to is 1024. Setting it any higher will have the perverse result of disabling the hashing mechanism.
2. Patch ID OSF350-146 has been superseded by

   > Patch ID OSF350-195 for V3.2C
   > Patch ID OSF360-350195 for V3.2D

   Patch IDs for V3.2E and V3.2F should be available soon. There is no known reason why the Patch ID OSF360-350195 won't work on these releases, but such use is not officially supported by Digital. This patch kit will not be needed for V3.2G when it is released.

       From           mogul@pa.dec.com (Jeffrey Mogul)
       Organization   DEC Western Research
       Date           30 May 1996 00:50:25 GMT
       Newsgroups     comp.unix.osf.osf1
       Message-ID     <4oirch$bc8@usenet.pa.dec.com>
       Subject        Re: Web Site Performance
       References     1

       In article <skoogDs54BH.9pF@netcom.com> skoog@netcom.com (Jim Skoog) writes:
       >Where are the performance bottlenecks for Alpha AXP running the
       >Netscape Commerce Server 1.12 with high volume internet traffic?
       >We are evaluating network performance for a variety of Alpha AXP
       >runing DEC UNIX 3.2C, which run DEC's seal firewall and behind
       >that Alpha 1000 and 2100 webservers.

       Our experience (running such Web servers as altavista.digital.com
       and www.digital.com) is that there is one important kernel tuning
       knob to adjust in order to get good performance on V3.2C.  You
       need to patch the kernel global variable "somaxconn" (use dbx -k
       to do this) from its default value of 8 to something much larger.

       How much larger?  Well, no larger than 32767 (decimal).  And
       probably no less than about 2048, if you have a really high volume
       (millions of hits per day), like AltaVista does.

       This change allows the system to maintain more than 8 TCP
       connections in the SYN_RCVD state for the HTTP server.  (You
       can use "netstat -An |grep SYN_RCVD" to see how many such
       connections exist at any given instant).

       If you don't make this change, you might find that as the load gets
       high, some connection attempts take a very long time.  And if a lot
       of your clients disconnect from the Internet during the process of
       TCP connection establishment (this happens a lot with dialup
       users), these "embryonic" connections might tie up your somaxconn
       quota of SYN_RCVD-state connections.  Until the kernel times out
       these embryonic connections, no other connections will be accepted,
       and it will appear as if the server has died.

       The default value for somaxconn in Digital UNIX V4.0 will be quite
       a bit larger than it has been in previous versions (we inherited
       this default from 4.3BSD).

       Digital UNIX V4.0 includes some other performance-related changes
       that significantly improve its maximum HTTP connection rate.  However,
       we've been using V3.2C systems to front-end for altavista.digital.com
       with no obvious performance bottlenecks at the millions-of-hits-per-day
       level.

       We have some Webstone performance results available at
               http://www.digital.com/info/alphaserver/news/webff.html
       I'm not sure if these were done using V4.0 or an earlier version
       of Digital UNIX, although I suspect they were done using a test
       version of V4.0.

       -Jeff

       ----------------------------------------------------------------------------

       From           mogul@pa.dec.com (Jeffrey Mogul)
       Organization   DEC Western Research
       Date           31 May 1996 21:01:01 GMT
       Newsgroups     comp.unix.osf.osf1
       Message-ID     <4onmmd$mmd@usenet.pa.dec.com>
       Subject        Digital UNIX V3.2C Internet tuning patch info

       ----------------------------------------------------------------------------

       Something that probably few people are aware of is that Digital
       has a patch kit available for Digital UNIX V3.2C that may improve
       Internet performance, especially for busy web servers.

       This patch kit is one way to increase the value of somaxconn,
       which I discussed in a message here a day or two ago.

       I've included in this message the revised README file for this
       patch kit below.  Note that the original README file in the patch
       kit itself may be an earlier version; I'm told that the version
       below is the right one.

       Sorry, this patch kit is NOT available for other versions of Digital
       UNIX.  Most (but not quite all) of these changes also made it into V4.0,
       so the description of the various tuning parameters in this README
       file might be useful to people running V4.0 systems.

       This patch kit does not appear to be available (yet?) from
               http://www.service.digital.com/html/patch_service.html
       so I guess you'll have to call Digital's Customer Support to get it.

       -Jeff

       DESCRIPTION: Digital UNIX Network tuning patch

                    Patch ID: OSF350-146

                    SUPERSEDED PATCHES: OSF350-151, OSF350-158

               This set of files improves the performance of the network
               subsystem on a system being used as a web server.  There are
               additional tunable parameters included here, to be used
               cautiously by an informed system administrator.

       TUNING

               To tune the web server, the number of simultaneous socket
               connection requests are limited by:

               somaxconn               Sets the maximum number of pending requests
                                       allowed to wait on a listening socket.  The
                                       default value in Digital UNIX V3.2 is 8.
                                       This patch kit increases the default to 1024,
                                       which matches the value in Digital UNIX V4.0.

               sominconn               Sets the minimum number of pending connections
                                       allowed on a listening socket.  When a user
                                       process calls listen with a backlog less
                                       than sominconn, the backlog will be set to
                                       sominconn.  sominconn overrides somaxconn.
                                       The default value is 1.

               The effectiveness of tuning these parameters can be monitored by
               the sobacklog variables available in the kernel:

               sobacklog_hiwat         Tracks the maximum pending requests to any
                                       socket.  The initial value is 0.

               sobacklog_drops         Tracks the number of drops exceeding the
                                       socket set backlog limit.  The initial
                                       value is 0.

               somaxconn_drops         Tracks the number of drops exceeding the
                                       somaxconn limit.  When sominconn is larger
                                       than somaxconn, tracks the number of drops
                                       exceeding sominconn.  The initial value is 0.

               TCP timer parameters also affect performance. Tuning the following
               require some knowledge of the characteristics of the network.

               tcp_msl                 Sets the tcp maximum segment lifetime.
                                       This is the maximum lifetime in half
                                       seconds that a packet can be in transit
                                       on the network.  This value, when doubled,
                                       is the length of time a connection remains
                                       in the TIME_WAIT state after a incoming
                                       close request is processed.  The unit is
                                       specified in 1/2 seconds, the initial
                                       value is 60.

               tcp_rexmit_interval_min
                                       Sets the minimum TCP retransmit interval.
                                       For some WAN networks the default value may
                                       be too short, causing unnecessary duplicate
                                       packets to be sent.  The unit is specified
                                       in 1/2 seconds, the initial value is 1.

               tcp_keepinit            This is the amount of time a partially
                                       established connection will sit on the listen
                                       queue before timing out (e.g. if a client
                                       sends a SYN but never answers our SYN/ACK).
                                       Partially established connections tie up slots
                                       on the listen queue.  If the queue starts to
                                       fill with connections in SYN_RCVD state,
                                       tcp_keepinit can be decreased to make those
                                       partial connects time out sooner.  This should
                                       be used with caution, since there might be
                                       legitimate clients that are taking a while
                                       to respond to SYN/ACK.  The unit is specified
                                       in 1/2 seconds, the default value is 150
                                       (ie. 75 seconds).

               The hashlist size for the TCP inpcb lookup table is regulated by:

               tcbhashsize             The number of hash buckets used for the
                                       TCP connection table used in the kernel.
                                       The initial value is 32.  For best results,
                                       should be specified as a power of 2.  For
                                       busy Web servers, set this to 2048 or more.

               The hashlist size for the interface alias table is regulated by:

               inifaddr_hsize          The number of hash buckets used for the
                                       interface alias table used in the kernel.
                                       The initial value is 32.  For best results,
                                       should be specified as a power of 2.

               ipport_userreserved     The maximum number of concurrent non-reserved,
                                       dynamically allocated ports.  Default range
                                       is 1025-5000.  The maximum value is 65535.
                                       This limits the numer of times you can
                                       simultaneously telnet or ftp out to connect
                                       to other systems.

               tcpnodelack             Don't delay acknowledging TCP data; this
                                       can sometimes improve performance of locally
                                       run CAD packages.  Default is value is 0,
                                       the enabled value is 1.

                                  Digital UNIX version:

                                         V3.2C
       Feature                    V3.2C  patch  V4.0
       =======                    =====  =====  ====
       somaxconn                   X      X      X
       sominconn                   -      X      X
       sobacklog_hiwat             -      X      -
       sobacklog_drops             -      X      -
       somaxconn_drops             -      X      -
       tcpnodelack                 X      X      X
       tcp_keepidle                X      X      X
       tcp_keepintvl               X      X      X
       tcp_keepcnt                 -      X      X
       tcp_keepinit                -      X      X
       TCP keepalive per-socket    -      -      X
       tcp_msl                     -      X      -
       tcp_rexmit_interval_min     -      X      -
       TCP inpcb hashing           -      X      X
       tcbhashsize                 -      X      X
       interface alias hashing     -      X      X
       inifaddr_hsize              -      X      X
       ipport_userreserved         -      X      -
       sysconfig -q inet           -      -      X
       sysconfig -q socket         -      -      X