NWTP Note 0501 "About the encryption mechanism"

This note discribes the password encryption mechanisms as used by Novell
Netware 3.x/4.x. Passwords are encrypted by workstations
and servers. The password verification process is also based on encryption.

This note describes the entire process of password verification and
password change. It has 3 appendices:
1. A description of the ASM calls involved with encrypted passwords;
2. Sourcecode of the encryption routines and tables in Turbo Pascal;
3. Sourcecode of the encryption routines and tables in C.

 Initial state
 =============

 Our transaction model starts with a description of the initial state
 of the server.

 The following tables are stored at the server:

  -The EncryptionKeyTable, containing an 8 byte EncryptionKey for
     every workstation connection. This table can be queried by
     workstations using the GetEncryptionKey (INT 21h, AX=F217h, subf. 17h).
     Table entry [c] is renewed whenever connection c used a call using
     encrypted passwords.

  -The PasswordTable containing an 16 Byte encrypted password
     (socalled "Shuffled" Password) for every connection.

 A short description of the various encryption mechanisms involved:

  -Shuffling. ('S' for short)
     The encryption of the password (string of char) into 16 bytes (the
     shuffled password), using a number of static tables and the objectID
     of the object the password is associated with.

     In Mathematical terms:
      ShuffledPassWord=S(PasswordString) or Spw=S(pw)

  -Encryption ('E' for short)
     The main encryption process, encrypting the shuffled password (S(pw)
     for short) into 8 bytes, using the same static tables as the Shuffling
     functions and a dynamic encryption key requested from the server.

     In mathematical terms:
      EncryptedPassWord=E(EncryptionKey,ShuffledPassword) or Epw=E(Ekey,Spw)

  -EncryptDifference ('D' for short)
     Encrypts the 'difference' between the Shuffled old password and the
     Shuffled new password, using a static table. The encrypted difference
     is passed to the server. The computed 'difference' consist of
     16 bytes of data and 1 checksum byte.

     In mathematical terms:
      PasswordDiff=D(ShuffledOldPW,ShuffledNewPW) or pwDiff=D(SOpw,SNpw)

  -DecryptDifference ('Dinv' for short)
     Server decryption process. Decrypts a 'password-difference'
     encrypted block as suplied by a workstation to a shuffled version
     of the new password. The shuffled old password, as stored in the
     servers' EncryptionKeyTable is used in the decryption process.

     In mathematical terms:
      ShuffledNewPW=Dinv(ShuffledOldPW,PasswordDiff) or SNpw=Dinv(SOpw,pwDiff)
      Notes:-ShuffledOldPassword is taken by the server from its'
             EncryptionKeyTable, i.e. ShuffledOldPassword=EncryptionKeyTable[c]
             where c is the connection number of the object the password
             is associated with.
            -SNpw=Dinv(SOpw,D(SNpw,Opw)), hence the name "D inverse".

  -GenerateNewKey ('GNK' for short)
     The server process creating a new encryption key for a certain
     connection after the previous one has been used by that connection.

     In mathematical terms:
      EncryptionKeyTable[c]:=GenerateNewKey(EncryptionTable[c])


 Password Verification
 =====================

   Password verification procedure when the encrypted password calls
   (VerifyEncrBinderyObjectPassword and LoginEncrToFileServer)
   are used:

    Workstation                       Server
    ===========                       ======

    GetEncryptionKey ----------------> Return EncryptionKeyTable[c]

       EncrKey       <---------------------

    Epw:=E(EncrKey,S(pw))

    Verify/Login(Epw) ---------------> Epw'=E(EncryptionKeyTable[c],
                                              PasswordTable[c])

                                       Epw'=Epw ?
     completion code  <--------------------

                                       EncryptionKeyTable[c]=
                                              GNK(EncryptionKeyTable[c])

    Note: c  = Workstation connection number
          pw = Password (string, max. 128 characters)
          Epw= Encrypted Password (8 bytes)


   Password verification procedure when the calls using unencrypted
   passwords (VerifyBinderyObjectPassword and LoginToFileserver)
   are used in combination with a 3.x server:

    Workstation                       Server
    ===========                       ======


    Verify/Login(pw) ---------------->

                                       S(pw)=PasswordTable[c] ?
     completion code  <--------------------

                                       EncryptionKeyTable[c]=
                                              GNK(EncryptionKeyTable[c])

    Note: c  = Workstation connection number
          pw = Password (string, max. 128 characters)



 Changing Passwords
 ==================

   The process of changeing a password using encrypted passwords:

    Workstation                       Server
    ===========                       ======

    GetEncryptionKey ----------------> Return EncryptionKeyTable[c]

       EncrKey       <---------------------

    SOpw=S(oldPW)
    SNpw=S(newPW)

    EOpw=E(encrKey,SOpw)

    PWdif=D(SNpw,SOpw)

    ChangeEncrBinderyObjPW
     (EOpw,PWdiff)
                       --------------> Epw'=E(EncryptionKeyTable[c],
                                              PasswordTable[c])

                                       Epw'=Epw ?
     completion code  <--------------------

                                       SNpw'=Dinv(PasswordTable[c],
                                                  PWdiff)
                                       PasswordTable[c]=SNpw'

                                       EncryptionKeyTable[c]=
                                              GNK(EncryptionKeyTable[c])


     Note: c     = Workstation connection number
           OldPW = Old password (string, max. 128 characters)
           NewPW = New password (string, max. 128 characters)
           SNpw  = Shuffled new password  (12 bytes)
           SOpw  = Shuffled old password  (12 bytes)
           EOpw  = Encrypted old Password (8 bytes)


   THe process of chageing a password when using unencrypted passwords:

    Workstation                       Server
    ===========                       ======


    ChangeBinderyObjPW
     (OldPW,NewPW)
                       -------------->
                                       S(OldPW)=PasswordTable[c] ?
     completion code  <--------------------

                                       PasswordTable[c]=S(NewPW)

                                       EncryptionKeyTable[c]=
                                              GNK(EncryptionKeyTable[c])


     Note: c      = Workstation connection number
           OldPW  = Old password (string, max. 128 characters)
           NewPW  = New password (string, max. 128 characters)


Sources: NVPW.C by Itsme [Itsme@Hacktic.nl]
         LOGON.PAS by Barry Nance/Terje Mathesen, Byte, March 1993.