P T E R M Paragon Terminal Application for Windows NT Version 0.3b 6, July 1993 What's Going on Here? ===================== This is a beta of Paragon Terminal Application for Windows NT. What's so great about PTERM? Not much yet, but here is a list of features available so far: o Fully threaded, native Win32 application o Auto Zmodem download/upload o Prelminary TELNET support o Fast(er) ANSI (including color!) terminal emulation o Console based for speed The other valuable feature of PTERM is that as far as I know it is the only Zmodem capable native, multi-threaded Win32 terminal application available. It won't be for long, but for now... Some glaring omissions: o No Dialer!?!?! o No user interface driven configuration o No annoying "register me now, or die" messages and purposefully broken features. o Many other things which you will likely go insane without (but hang in there, it is my intention to evolve PTERM). Why Would You Do Such a Thing? ============================== Some months back, I was asked to beta test a C++ based, multi-platform communications library from a company called Lookout Mountain Software (William Herrera, owner/author, BBS 719-545-8572). The best method I could think of for testing this package was to use it as the core of a terminal application -- and PTERM was born. William has been extremely sensitive to the problems I found and enhancements I suggested, and should be commended for authoring such a stable and comprehensive communications library, which is available for DOS, Windows, OS/2 and Windows NT. All of the file transfer protocol code (Zmodem/Ymodem/Xmodem) is Williams, and he has done an excellent job implementing them (sealink and telink are also included in the communications library, but not in PTERM -- Xmodem and Ymodem will be available in the next release of PTERM). I Didn't Do It, I Swear! ======================== Every precaution has been taken to ensure the safe operation of PTERM in your Windows NT system. But, come'on, not only is this copy of PTERM in beta, but so is the current (March '93) build of NT, so obviously I cannot be held responsible for any damage done to your system (or psyche) by PTERM or any interaction involving PTERM. So all I can say is "I didn't do it!", so don't blame me! What Do I Need To Use Pterm? ============================ You need an Intel based (Intel Inside sticker optional) PC running Windows NT (March '93 build or later), with at least one serial port and a modem attached to that port. If someone would like to send me a MIPS R4400 or DEC Alpha AXP, I will be glad to get PTERM up on those platforms as well . Alright, What Do You Want For This? =================================== The list of things I want would be much too large to include in this document, so you will have to settle for what I need. What I need, from you, is the following: o Bug reports (there should be plenty) o Suggestions (again, no shortage expected here either) So, please, please inundate me with this information, I promise I will consider anything submitted (but nothing not submitted). Whereami? ========= I can be reached via internet mail at: roncox@indirect.com FIDOnet people, try netmail to UUCP at 1:1/31, first line of the message body: To: roncox@indirect.com I plan on finding a reliable FIDOnet system here in town, and when I do I will post my FIDOnet address for netmail. You can reach me on Compuserve at: 71722,3175 For slowest response, choose snail-mail: Ron Cox Paragon Consulting Group 4212 West Cactus STE 1110-229 Phoenix, AZ 85029 ATTN: PTERM If you are a Windows NT developer, and would like to contribute code to PTERM, let me know! Currently I have no plans to charge real money for PTERM, so you will get the same thing I do as far as that goes, nothing but recognition (we'll be lucky to get that!). Any code you contribute will become public domain, and may be distributed in the future (in source or object form) or used in other projects without restriction. Free? Something Must Be Broken! ================================ Actually, yes. I am releasing this with a several known bugs, and I'm sure many more unknown bugs. Many of the bugs which were present in the first beta release have been squashed (I hope!). I am of the belief that the bugs I know about will not significantly affect the operation. Some require bug fixes in NT or the SDK, others just require me to figure out what's going on. Here is a list of the bugs I know about: o The zmodem upload problem has been dramitically reduced, however you may still experience problems, especially after the last (or only) file of an upload. I am still working with the author of the communications library to get this solved. o Currently the getch() function in the Win32 SDK ignores CTRL-S. To get around this for now, I have made PTERM look for ALT-S, and when it gets that it will send a CTRL-S. Sorry about this, I will release an update as soon as it gets fixed in the SDK. o ANSI aupport may still be a bit weak. I have the ANSI document now, and am in the process of shoring this up. There has been improvement between this and the last beta however. If you experience problems (especially when calling a UNIX box), try setting the hosts emulation to ANSI, and if that is unsatisfactory, try VT100. Well, them's the major ones I have found so far. Please report others to me as necessary. Some 'features' to watch out for: o The TELNET support is VERY young, and several facets are still being worked out. Please report any problems you have with this! o Once PTERM quits, it closes the serial port, and if the modem on this serial port is currently connected to somewhere, that connection will be lost (since closing the serial port drops DTR). You may be able to prevent this by setting your modem to always hold DTR high, but this will have the side effect of preventing ALT-H from hanging up the connection. o If there exists, in your download path, a file with the same name as one you are attempting to download, the transfer will be aborted. I apologize for this, and I will have a reasonable automatic rename mechanism available in the next release. o PTERM has a lot of trouble with the ANSI codes coming from Maximus BBS software. I am working on getting this straight. It seems Maximus uses some strange (although probably completely legal) derivation of the cursor position command. I will get this fixed as soon as I can get a detailed document on ANSI emulation (and even better document on what the hell Maximus sends out for various sequences). I'm Getting Sleeeepy... ======================= I know, I know... Luckily, there are not many configurable options (yet) in PTERM, so this won't take much longer. Currently, I have PTERM read its configuration from the environment. The following environment variables are supported: PTPORT Set to the integer value of the port you wish PTERM to use. I have tested ports 1 and 2, but PTERM should be able to use any port available (please let me know if you have problems in this area). Example: set PTPORT=1 A feature new for this release is the addition of TELNET support. In order to start a TELNET session, you must have the appropriate service(s) running under NT. Then, set PTPORT to telnet, and start PTERM. Example: set PTPORT=telnet At this point, PTERM should operate as MS Terminal does in TELNET mode (albeit, faster!). PTWORD Set to the integer value of the data word size to open the port with. Typically 7 or 8 data bits. Example: set PTWORD=8 PTPARITY Set to the parity to open the port with. Supported values are: NONE EVEN ODD Example: set PTPARITY=NONE PTSTOP Number of stop bits to open the port with. Typically 1 or 2. Example: set PTSTOP=1 PTBAUD Set to the baud rate to open the port at. Supported speeds are 300, 1200, 2400, 4800, 9600, 19200, 38400, 57600, and 115200. PTERM locks the baud rate of the port at the speed given, as is necessary with most modern modems. Example: set PTBAUD=38400 PTDOWN Set to the path where downloaded files are to be placed. Example: set PTDOWN=e:\download PTAUTOZ If this variable is set to ANYTHING, auto Zmodem downloads and uploads will be enabled. To disable, make sure it is not in defined in the environment. Example(s): set PTAUTOZ=yes Enables auto Zmodem set PTAUTOZ= Disables auto Zmodem There you have it. Those are the available options for configuring PTERM. There is virtually no error checking done on configuration, so make sure you (as in your brain) go into debug mode if you have problems. Included in the archive with this document and PTERM.EXE, you should find a batch file called PT.CMD. This is the recommended method of starting PTERM. Place PT.CMD and PTERM.EXE in your path. Edit PT.CMD to set the environment variables appropriate to your system. To run PTERM, type PT from a command prompt. The batch file will set the environment variables, and start PTERM in its own console window. You can easily create multiple batch files to run multiple copies of PTERM each on a different port. This has worked for me with 2 ports simultaneously. The batch file approach has worked from NT's CMD.EXE as well as the 4DOS for NT command interpreter replacement. PTERM can be placed in (and executed from) the program manager, and even has its own embedded icon (just like a real Windows program!). To insure that the environment is set up correctly, start the control panel 'System' applet, and at the bottom use the two edit boxes to appropriately define the variables described above. You will have to log off and and back on for these to take effect, and will need to set them for each user account. When using a batch enabled file transfer protocol (Zmodem), the file selection dialog box which pops up for an upload will allow you to select multiple files from any one directory, using the standard Windows mouse/key combinations for multi-select lists (i.e. CTRL-CLICK adds a file to the list, and try SHIFT-CLICK to extend the list to the current point). For batch uploading, if PTERM finds a file called FILESTO.UPL in the current directory (the directory you were in when you started PTERM), it expects this file to contain a list of files (full paths) to upload, and will attempt to do so. If PTERM finds this file, it will go straight to uploading, bypassing the file selection dialog. I am sure I have forgotten a whole bunch of information here, but this should be enough to get you off the ground. Can I Run it Now? ================= Well... Ok... But, "Don't touch it, you'll break it..." (U.S. West T.V. ad) When you run PTERM, it will splat some information on the screen. First it will display the key bindings it recognizes, things like ALT-X to exit, PGDN to download, etc. Next, it displays the current settings as read in from the environment. Pressing F1 will display this information at anytime during the session. As mentioned, there is no dialer. This is too bad, I know, and will be one of the first things added. For those of you who have been spoiled all their lives by a dialer and do not know how to do it manually, if you have a modem which supports the Hayes command set (are there any which don't?) you can dial a number from PTERM like so: atdt555-5555 Tone dial 555-5555 atdp555-5555 Pulse dial 555-5555 (yuck!) On my USR Sportster, the command a/ executes the last command entered, and can be used to redial. Some Closing Thoughts ===================== Its not much, but there it is. Please let me know of any problems or suggestions for enhancements you have. On my 386-33, running the March '93 beta, I have experienced excellent transfer speeds using the Zmodem in PTERM. On text files, at 14.4K with .v42bis (57.6K maximum), I have seen 3800+ cps. The same setup on compressed files yields 1650 cps and higher. If you have 16550's, and access to a FIFO enabled serial driver (I rebuilt it using the March DDK), install it! The serial driver for NT is absolutely solid, and extremely efficient. Before installing the FIFO enabled serial driver, during a 1650 cps download I would experience around 1700 interrupts per second (approx. one for each character), and during heavy multi-tasking characters were easily dropped. However, after rebuilding the serial driver (the 16550 support is in there, just not enabled -- The final retail build when it arrives should add the ability to enable the FIFO's through the registry, with the same 1650 cps download I was getting around 200 interrupts per second. This makes sense since I built the serial driver to enable the FIFO's to fire off an interrupt when they queued 8 bytes, for an 8x decrease in interrupts. Subsequently, I have never seen PTERM drop a character in over 20 megs of file transfers. Another facet of the serial driver which operates well is RTS/CTS flow control. This is always turned on when PTERM runs, later it will be configurable. During heavy multi-tasking, RTS/CTS keeps the modem under control, holding it off when the serial buffer gets full. This has worked flawlessly. In closing, Windows NT has its problems, but all in all it is a solid, high performance operating system for the masses. I am confident in Microsofts (and mostly Dave Cutlers) ability to evolve NT, so much more is yet to come (MS: thanks sooo much for the console API, making all those UNIX utils that much easier to port!). Ron Cox Paragon Consulting Group Special Thanks ============== To the following, I give special thanks: Greg Kochaniak for his *extensive* help in getting PTERM's telnet support to be even as useful as it is! Dale Ross for being a sort of liason between Microsoft and myself to get some nasty bugs fixed. To ALL the other people (especially on internet) who emailed me with suggestions, and a couple even sent source code. Thanks! Technically Speaking ==================== Thought you were done, didn't you? Well, if you could care less about some of the technical details of PTERM, you are, else read on. As mentioned, PTERM makes full use of multi-threading. This stuff is a trip. It can make programming much more interesting (and in many cases greatly simplifies things!). There are 6 threads of execution, 5 secondary and the 1 main thread which all Win32 applications start with. The first two threads of interest are in Williams communications library. One is responsible for taking characters in from the serial port (actually, the serial driver) and placing them in a queue created by the library (the input queue) which is made visible to the user code. The second takes characters from another queue and writes them to the serial port (driver). The user code is responsible for placing characters to be sent out the port into this output queue, and does so through a set of 'Send' member functions. Also, both of Williams threads use overlapping (asynchronous) I/O, resulting in an even more efficient set up. Now for the threads I spawn. They are as follows: o Input thread Because I did not want to muck with the input queue being managed by the communications library, I created another layer. My input thread simply waits for characters to appear in the main input queue, and block copies them into a local queue used by the display thread (below). If there are no characters waiting in the main input queue, this thread sleeps until there is (in the discussions which follow, this thread will be referred to as 'my input thread', to differentiate it from the main input thread operating in the library). It may be that I can increase PTERM's efficiency slightly by removing this layer and having the display thread pull characters directly from the library managed input queue -- something I will consider. o Display thread This thread is solely responsible for removing characters from the local input queue (managed above) and deciding what to do with them. For instance, if the beginning of an ANSI escape sequence is detected this thread passes control to a function whose job it is to interpret the sequence (note: the ANSI code still executes within the display thread, no other thread has been created). One of the other jobs of the display thread is to optimize the output into the console window. This is done in a very simplistic manner. In the absence of ANSI escape sequences, the display thread will accumulate characters from the local input queue into a buffer -- up to 80. When the limit of 80 has been reached, or a special sequence is detected, the display thread blasts the buffer to the screen in a single call to WriteFile(). This is MUCH more efficient than calling something like putch() for each single character. However, at slow speeds (1200 and 2400 baud), it takes a noticeable amount of time to accumulate 80 characters, and this is the reason for the choppy display at these speeds. If you set the port to anything less than 9600 baud, PTERM reduces this 'blast' count to 8 characters, producing a smoother display. As if this wasn't enough, the display thread has one more job to perform. It watches for the tell-tale sequence of characters which signal the host is preparing for a Zmodem upload or download. If this sequence is found, it starts the appropriate code. If there are no characters in the local input queue, the display thread sleeps until there are. o User thread The user threads job is pretty easy. If the user hits a key, decide what to do with it. Typically the key will be sent right out the port. If it is a control key sequence (like ALT-X), then the user thread executes the code appropriate for that key. This thread blocks on the keyboard, and as such sleeps when there are no characters available. o Main thread The main thread is the first thread which executes (starting in main()). It initializes the port and other things, starts up the three threads above, and then goes to sleep waiting for all three of the threads above to terminate. At this point it wakes up and calls ExitProcess(), ending PTERM. Them's the threads, and a nice bunch of threads they are. However, if there is one thing I quickly learned from spawning threads all over the place, its that synchronization thingy. Here's the scenario: My input thread pulls characters from the main input queue to place in the local input queue. Fair enough. So, I start a Zmodem download. Boom! Of course, the Zmodem download code also wants to pull characters from the main input queue, so my thread fights with the Zmodem code. Remember, the Zmodem code is run as part of the display thread. So my input thread grabs some characters, then the Zmodem code [running in the display thread] grabs some characters, and so on. Zmodem will end up missing a bunch of characters it expected to see, and my input thread grab some characters from the transfer which the display thread will finally get and try to display. A mess... So, I need to find a way to synchronize the two threads. When Zmodem wants control of the main input queue, it needs to 'ask' for control from my input thread. Well, it really isn't as formal as all that. Just before my input thread grabs characters from the main input queue, it waits on a semaphore. Simplified, a semaphore is just an object which keeps count of how many threads have asked for control of it. When its count is 0, access is granted to the waiting thread. When a thread gets access to it, the semaphores count is incremented. Any other thread which waits on it goes to sleep until the semaphores count becomes 0 again. The count is decremented when a thread explicitly releases a semaphore, thus allowing another thread to gain access (this can get complicated when there are more than 2 threads). Ok, where were we? Oh, yes, my input thread waits on this semaphore gadget. Typically, it gets control instantly and drops into the main body of its code where it grabs characters from the main input queue and stuffs them in the local queue the display thread uses. Once it has transferred some characters, it releases the semaphore (and gives up the rest of its time slice). Its during the time between releasing the semaphore and waiting on it again that the file transfer code must act. The first line of code for a file transfer waits on the same semaphore. As soon as my input thread releases the semaphore, the file transfer code gets and holds access to it until the transfer is completed, thus preventing my input thread from mucking with the main input queue during the transfer. Whew! Being this is my first foray into threads, I am very interested to find out if my code breaks on a multi-processor machine. Does my code work now because it takes advantage of the synchronous behavior of a single processor? Interesting stuff, but I forgot to pick up my Sequent 16 processor monster-box at the grocery store the other day, so I suppose the answer will have to wait... I found one other interesting thing about Windows NT -- it seems to like to write data to the disk right away. Generally, this is a good thing, keeps your data safe. However, with Zmodem code writing 1K blocks to disk every .7 seconds or so (at 1650 cps), the disk access began to affect the performance of the download. The solution was easy, I just used a call to setvbuf() to create a memory buffer. The size I chose is 64K. So the system (C runtime) will accumulate 64K bytes from the fwrite()'s before writing to disk. Believe it or not, NT can actually write a 64K block to disk as quickly as a 1K block -- difference is, now Zmodem only hits the disk one a minute or so (at 1650 cps). Works great. Notice I said I made a 64K buffer. You 16 bit guys probably have (as I did) the signed integer maximum value memorized, 32K right? Well, under NT, a signed integer is 2^31, for a maximum [signed] value of 2 GB. So 64K was small compared to what I could make it. The file access areas of the transfer protocols are excellent candidates for overlapped I/O, probably doing away with the need for a write buffer. However, since Williams library is meant to be multi-platform, it is best to keep it as generic as possible. But, it is C++, and with proper inheritance the function responsible for writing data to disk could be overloaded (and William has isolated this operation for easy overloading!). A possible future enhancement to PTERM. One more technical note: PTERM does not change its base priority class, but it does manipulate the priority of several of its threads relative to the priority class it is started at. What? Ok, if you run PTERM from the command line like so: start pterm.exe then PTERM gets a base priority class of NORMAL. Using these commands: start /high pterm.exe start /realtime pterm.exe starts PTERM with a base priority class of HIGH or REALTIME respectively. Whats interesting is that when I started work on PTERM, I assumed I would need to have PTERM boost its base priority class to at least HIGH to make sure it could respond quick enough to bytes coming in over the port at high baud rates. This resulted in poor performance (dropping characters and such). So I assumed PTERM wasn't getting the priority necessary to keep up with bytes coming in the port. So, I pumped the base priority class up to REALTIME. The performance was even worse! Suddenly, it began to make sense. This is what I figure is happening: The serial driver itself probably runs at the upper end of NORMAL priority, or maybe even the lower end of HIGH priority. So I figure when I boosted the priority of PTERM to the same or above the priority of the serial driver, PTERM was stealing away time from the serial driver, causing the driver itself to drop characters! This is pure speculation, since I have no detailed knowledge of what priority the serial driver runs at. I assume there is a small part of the serial driver which is interrupt driven, and depending on the interrupt level it runs at, it can be preempted by other processes. I further assume that some other piece of the serial driver gets scheduled by the interrupt driven part above to remove characters from some small local buffer and place them in the buffer which ReadFile() uses. Again, this is pure speculation. At any rate, the interesting part is that when I changed PTERM to NOT change its base priority class, so that upon starting PTERM, it would default to the NORMAL priority class, performance was MUCH better! What does this tell me? The NT serial driver is incredibly efficient, yet can become sensitive to other high priority processes in the system (the behavior I described above was on my 386-33 -- this may not be an issue on faster machines, where higher priority processes might not have such adverse affects on the serial driver because the faster processor is able to juggle all the processes more efficiently). At any rate, what it means to you the user is that you are free to start PTERM at any of the 4 base priority classes, and your mileage may vary. Examples: start /idle pterm.exe start /normal pterm.exe start /high pterm.exe start /realtime pterm.exe I strongly recommend leaving it at NORMAL priority (by using the /normal switch, or not giving a priority at all). You are, of course, free to experiment. Well, the rest of PTERM is plain and boring (as if the preceding was not!). So not much else to say. Any specific questions? Feel free to contact me!