=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= = P/HUN Issue #3, Volume 2: Phile #3 of 11 = =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= VAX/VMS System Security ======================= Written for P/HUN Inc.,P/HUN Online Magazine -------------------------------------------- By Lawrence Xavier January, 1989 VAX/VMS may be the worlds best operating system. It certainly beats the pants off each and every IBM OS, and wins over Unix hands down. Native VAX/VMS security is rated higher (by the U.S. Government) than all IBM mainframe OSs, even after such security packages as RACF and Top Secret are added to them. VMS is not without its foibles and kludges, however. For one thing, enabling all the security features of VMS is guaranteed to crash the system! For another, many of VMS's security features are annoying to set up, encouraging lazy system managers to put off doing so indefinitely. VMS got a bad reputation when young hackers were able to routinely break into many systems by using default accounts and passwords such as username SYSTEM with password MANAGER. This has all changed with VMS 4.7: in the upgrade procedure the installer is required to change passwords on these accounts or eliminate them entirely. Let's go over some of the basic features of VMS security, then look at some common problems and loopholes. Once you know what the loopholes are you can take steps to close them on systems you manage and increase security. VMS Security Features ===================== Logging In: ----------- VAX/VMS systems have several types of protection that can be set up on logins. Logins can be restricted by time of day, day of the week, and by terminal ID. Logins can also be restricted by where they come from: Local, Remote, Dialup, etc. Local are logins on direct connect ports or DECservers. Remote are logins across DECnet. Dialup are logins across X.25 or on ports set with the DIALUP characteristic. Usually VMS will present a Username: prompt after it sees one or two characters (which are used by VMS to set the Baud rate, if AutoBaud is enabled). If a System Password has been set on the port, VMS will BEEP after the first , and will then seem to be dead. Only after the correct System Password has been entered will the Username: prompt be given. VMS gives no indication of whether a correct username has been entered: it always asks for a Password:. VMS passwords can be like any other passwords, or they may be generated nonsense words. The /GENERATE_PASSWORD qualifier may be placed on user accounts by the system manager, forcing them to select from lists of supposedly easy to remember but nonsensical passwords. The system manager may also enforce a minimum password length and can even impose dual passwords on accounts. If a Username with dual passwords is entered, the system will prompt for Password: twice in a row. Automatic Password expiration dates can be set, forcing users to change their passwords every so often: from once a day to once a year or never. After the Username and Password have been entered, the system will either log the user in, or will print the familiar message, User Authorization Failure and will hang up after a settable number of failures (the default is 3) if the port characteristics include DIALUP and HANGUP. Breakin Detection: ----------------- If a hacker were trying to get into the system he could just continue to dialup and try again. But VMS has some features to discourage this too. If breakin detection and evasion is enabled, VMS will start to get cagey. If the count of login failures from a specific source gets high enough, the system assumes a break-in is in progress. Only login failures caused by invalid Passwords are counted, NOT invalid usernames. And the attempts must be coming from one of these three sources: . A specific valid Username, and (if setup this way, A specific terminal. . A specific remote DECnet node and remote Username. . The Username of the creator of a detached process. By default, VMS allows five failed login attempts from any one source within the time period specified. But it's not as simple as that! Each time a failure occurs, time is added to the time period in which a certain number of failures can occur. To take an example from DEC: Assume the default values are in effect. LGI_BRK_LIM specifies no more than five login failures from one source. LGI_BRK_TMO is set for five minutes. Assume that an outsider starts sending user names and passwords to the system. When the first password fails, the clock starts to run and the user has four more tries in the next five minutes. When the second attempt fails about 30 seconds later, the user has three tries left that will be counted over the next 9.5 minutes. When the third attempt fails 30 seconds later, the login failure observation time has reached 22.5 minutes. As a result, the next login failure from that source within 22.4 minutes will trigger evasive action. The system tolerates an average rate of login failures that is the reciprocal of the parameter LGI_BRK_TMO... When breakin evasion is triggered, the system will give a: User Authorization Failure message even when a valid Username and Password are entered, giving no indication of what it is doing. Note that ONLY the Username(s) in question are treated this way: other Usernames can still log in from the same terminal even if terminal-specific breakin detection is enabled. The length of time VMS will hide in this way is controlled by the sysgen parameter LGI_HID_TIM. But VMS doesn't hide for exactly this time. Rather, it will hide for a length of time determined by the following equation: Evasion time = LGI_HID_TIM * (random number between 1 and 1.5) The parameter LGI_BRK_DISUSER can be set, and will tell VMS to permanently disable accounts rather than just hiding for a time. The system manager then has to re-enable them manually. This is a dangerous parameter to set, however, because malicious individuals could deliberately disable accounts then! If the SYSTEM account is disabled this way, it will only be allowed to login on the VAX system console. Security Alarms: ---------------- Although breakin attempts to different Usernames don't activate VMS Breakin detection, they can trigger Security Alarms. Security Alarms can also be triggered by different types of access to specific files or memory areas. Security Alarms cause messages to be displayed on the system console, on the terminals of any user enabled as Security Operator, and in the Operator Log file. As DEC says, Because security auditing affects system performance, enable security alarms only for the most important events. Damn right! If all security alarms are enabled the system will hang! It starts writing alarms about the fact it is writing alarms, ad infinitum.... Security alarms can be triggered on multiple login failures, on breakin, on successful login from given ports, on failed attempts to access files, on successful attempts to access files, etc. So even if you get privilege to override protection or to defeat it a security alarm may still be triggered. Security alarms typically might be enabled on the AUTHORIZE program, which adds and modifies user accounts, on SYSUAF.DAT, the authorization database, on RIGHTSLIST.DAT, the access rights database, etc. and on critical database files. But many sites don't bother with them because of their inconvenience. Accounting: ---------- Besides Security Alarms, Accounting can be enabled. Accounting can show successful logins, login failures, how much resources are consumed by processes, what programs are executed, etc. Not all sites enable accounting, and not all sites enable the same amount of it. Accounting records show login failures but only show the username that attempted to login if it is a valid username. File and Device Protection: ========================== UIC: ---- The primary access protection mechanism is the UIC. This consists of a Group and a User code, numerically represented as [nnn,nnn]. It is an Octal number. Since VMS 4.x versions the UIC can also be expressed as [name] or [name,name], but internally this is translated back to the old format. Users, processes, files, devices, memory sections, etc. all have UICs. Files, devices, memory sections, etc. can have access by System, Owner, Group and World, any combination of Read, Write, Execute, Delete for each category. System are the system accounts. Owner is the account(s) who's UIC is the same as that on the object (file, device, etc.). Group are accounts with the same first UIC number. World is everyone. So a process with UIC [23,7] could access an object with UIC [23,4] if that object allowed access by Group or World. The process could access an object with UIC [25,3] only if World access was allowed, and could access objects with UIC [23,7] if Owner, Group, or World was allowed. ACL: ---- Also, there's a protection mechanism called the ACL or Access Control List. This is in addition to, and can override UIC protection. With ACLs an Identifier is created, like MODEM for one or more modem ports. An ACL is created on the port(s) desired, and in the ACL are multiple Access Control Entries (ACEs). If one of them is: (Identifier=MODEM, Access=Read+Write) for example, user who has been Granted the identifier MODEM can access those ports. These access privileges, like UICs apply to processes in general. Granting and managing Identifiers is done in the AUTHORIZE program. Loopholes, Ways of Defeating Security... ======================================== Although VMS has great security it's often applied poorly. For one thing, protection is often not set up properly, programs are installed with too much privilege etc. (Programs can be installed so they have privilege when run even if the user running them has no privilege). Getting a $ prompt: ------------------- If a hacker logs into a VMS system and finds himself trapped within application programs the first thing he will want to do is to get out to the normal DCL command mode from where more things can be done. Hackers will try several things and you should check to make sure they can't try these tricks on your system. AllInOne: In AllInOne, DEC's popular Electronic Mail and menuing Office Automation system, typing $ (the dollar sign) will by default take the user to DCL level. Typing ADMIN will get the user into the AllInOne administrator menu. From there they can create accounts with $ access. AllInOne mail also has a feature where macros can be attached to mail and executed when the mail is read. If the hacker sends a message of this type to some user with privilege, the macro can go off in the background and create accounts, etc. for the hacker. This feature should be disabled. Other Captive Account tricks: Holding down and letting it repeat for a while will often cause accounts that are trapped in a command procedure but not marked as CAPTIVE in the UAF to exit from the command procedure to DCL. If an account has access to VAXMAIL (the MAIL command) it can often use MAIL's SPAWN command to spawn a process with DCL access. The TPU editor has a similar SPAWN command. If an account is not marked CAPTIVE the user can try to add /NOCOMMAND after the username, like the following: Username: fred/nocommand This will cause the command procedure to not be executed, leaving the hacker at a DCL $ prompt. There are many more too. For this reason you should mark all accounts that are supposed to be captive as CAPTIVE using the AUTHORIZE utility. When at the $ Prompt: --------------------- Since protection is often set incorrectly, hackers can take advantage and use this to bypass security. A couple of examples will serve to show that you must be diligent in setting the protections properly on systems you manage. If SYS$SYSTEM:AUTHORIZE.EXE is not protected, it can be run by non- privileged users. The hacker would then run AUTHORIZE and create a new SYSUAF.DAT file in his own directory (AUTHORIZE will do this by default if not run in the SYS$SYSTEM directory). The hacker would add a privileged username to the new SYSUAF.DAT, copy it back to SYS$SYSTEM:, log out, log in again as the new privileged user, and quickly delete the new SYSUAF.DAT so that other users don't get "Authorization Failure" messages. The hacker would then be able to add privileged accounts to SYSUAF.DAT at his leisure. Another clever idea would be for the hacker who has gained access to copy SYSUAF.DAT to another directory and then try to find out what passwords are in it. VMS uses a one-way encryption algorithm, but a gifted hacker will use the same algorithm to repeatedly encrypt different passwords until he finds ones that match. A copy of the VMS assembly language code to do this encryption can be found in the appendix, for your information. Again, setting the protection properly will keep this from happening to your system! Conclusion: =========== This has been a brief overview of VMS security. For more information, read your DEC manuals. A good place to start is the handy VMS System Manager's Manual, Order Number AA-LA00A-TE, which can be obtained from DEC Direct and should have come with your VMS update. The importance of proper security cannot be over emphasized, but if you overdo it performance will suffer. Experiment on your system to find a good balance. Don't ignore security or you may regret it rather intensely. Appendix -- VMS assembly code for encrypting passwords: ======================================================= .TITLE HPWD - hash user password .IDENT 'V02-002' ; Hash PassWorD: ; Hash a password irreversibly. This is one way encryption with ; no decryption possible. ; This code was obtained by disassembling the AUTHORIZE program. ; See the VMS microfiche for the fully commented code: ; e _lib$code:_lib$code+68 ; Input Parameters: ; PWDDSC - Address of password descriptor ; ENCRYPT - Encryption algorithm index (byte) ; SALT - Random number (word) ; USRDSC - Address of username descriptor ; Output Parameters: ; OUTDSC - Address of encrypted output descriptor OUTDSC=4 PWDDSC=OUTDSC+4 ENCRYPT=PWDDSC+4 SALT=ENCRYPT+4 USRDSC=SALT+4 .PSECT _LIB$CODE RD,NOWRT,PIC,SHR,BYTE,EXE ; AUTODIN-II polynomial table used by CRC algorithm AUTODIN: .LONG ^X000000000,^X01DB71064,^X03B6E20C8,^X026D930AC,^X076DC4190 .LONG ^X06B6B51F4,^X04DB26158,^X05005713C,^X0EDB88320,^X0F00F9344 .LONG ^X0D6D6A3E8,^X0CB61B38C,^X09B64C2B0,^X086D3D2D4,^X0A00AE278 .LONG ^X0BDBDF21C ; Purdy polynomial coefficients. Prime, but don't need to be Purdy_Poly: c: .LONG -83,-1 .LONG -179,-1 .LONG -257,-1 .LONG -323,-1 .LONG -363,-1 .ENTRY LGI$HPWD,^M MOVAQ @outdsc(AP),R4 MOVAQ @4(R4),R4 TSTB encrypt(AP) BGTRU 10$ MNEGL #1,R0 MOVAQ @pwddsc(AP),R1 CRC autodin,R0,(R1),@4(R1) CLRL R1 MOVQ R0,(R4) BRB 20$ 10$: CLRQ (R4) MOVAQ @pwddsc(AP),R3 BSBB COLLAPSE_R2 ADDW2 salt(AP),3(R4) MOVAQ @usrdsc(AP),R3 BSBB COLLAPSE_R2 PUSHAQ (R4) CALLS #1,PURDY 20$: MOVL #1,R0 RET COLLAPSE_R2: MOVZWL (R3),R0 BEQL 20$ MOVAL @4(R3),R2 PUSHR #^M MOVL R0,R1 5$: CMPB (R2)+,#32 BNEQ 7$ DECL R1 7$: SOBGTR R0,5$ MOVL R1,R0 POPR #^M 10$: BICL3 #-8,R0,R1 ADDB2 (R2)+,(R4)[R1] SOBGTR R0,10$ 20$: RSB a=59 n0=1@24-3 n1=1@24-63 .ENTRY PURDY,^M MOVQ @4(AP),-(SP) BSBW PQMOD_R0 MOVAQ (SP),R4 MOVAQ PURDY_POLY,R5 MOVQ (R4),-(SP) PUSHL #n1 BSBB PQEXP_R3 MOVQ (R4),-(SP) PUSHL #n0-n1 BSBB PQEXP_R3 MOVQ (R5)+,-(SP) BSBW PQADD_R0 BSBW PQMUL_R2 MOVQ (R5)+,-(SP) MOVQ (R4),-(SP) BSBW PQMUL_R2 MOVQ (R5)+,-(SP) BSBW PQADD_R0 MOVQ (R4),-(SP) BSBB PQMUL_R2 MOVQ (R5)+,-(SP) BSBW PQADD_R0 MOVQ (R4),-(SP) BSBB PQMUL_R2 MOVQ (R5)+,-(SP) BSBW PQADD_R0 BSBW PQADD_R0 MOVQ (SP)+,@4(AP) MOVL #1,R0 RET PQEXP_R3: POPR #^M MOVQ #1,-(SP) MOVQ 8+4(SP),-(SP) TSTL 8+8(SP) BEQL 30$ 10$: BLBC 8+8(SP),20$ MOVQ (SP),-(SP) MOVQ 8+8(SP),-(SP) BSBB PQMUL_R2 MOVQ (SP)+,8(SP) CMPZV #1,#31,8+8(SP),#0 BEQL 30$ 20$: MOVQ (SP),-(SP) BSBB PQMUL_R2 EXTZV #1,#31,8+8(SP),8+8(SP) BRB 10$ 30$: MOVQ 8(SP),8+8+4(SP) MOVAQ 8+8+4(SP),SP JMP (R3) u=0 v=u+4 y=u+8 z=y+4 PQMOD_R0: POPR #^M CMPL v(SP),#-1 BLSSU 10$ CMPL u(SP),#-a BLSSU 10$ ADDL2 #a,u(SP) ADWC #0,v(SP) 10$: JMP (R0) PQMUL_R2: POPR #^M MOVL SP,R2 PUSHL z(R2) PUSHL v(R2) BSBB EMULQ BSBB PQMOD_R0 BSBB PQLSH_R0 PUSHL y(R2) PUSHL v(R2) BSBB EMULQ BSBB PQMOD_R0 PUSHL z(R2) PUSHL u(R2) BSBB EMULQ BSBB PQMOD_R0 BSBB PQADD_R0 BSBB PQADD_R0 BSBB PQLSH_R0 PUSHL y(R2) PUSHL u(R2) BSBB EMULQ BSBB PQMOD_R0 BSBB PQADD_R0 MOVQ (SP)+,Y(R2) MOVAQ Y(R2),SP JMP (R1) EMULQ: EMUL 4(SP),8(SP),#0,-(SP) CLRL -(SP) TSTL 4+8+4(SP) BGEQ 10$ ADDL2 4+8+8(SP),(SP) 10$: TSTL 4+8+8(SP) BGEQ 20$ ADDL2 4+8+4(SP),(SP) 20$: ADDL2 (SP)+,4(SP) MOVQ (SP)+,4(SP) RSB PQLSH_R0: .ENABLE LSB POPR #^M PUSHL v(SP) PUSHL #a BSBB EMULQ ASHQ #32,Y(SP),Y(SP) BRB 10$ PQADD_R0: POPR #^M 10$: ADDL2 u(SP),y(SP) ADWC v(SP),z(SP) BLSSU 20$ CMPL z(SP),#-1 BLSSU 30$ CMPL y(SP),#-a BLSSU 30$ 20$: ADDL2 #a,y(SP) ADWC #0,z(SP) 30$: MOVAQ Y(SP),SP JMP (R0) .END DOWNLOADED FROM P-80 SYSTEMS 304-744-2253 Downloaded From P-80 International Information Systems 304-744-2253 12yrs+