list 3 ********************************************************************* This article is being presented through the *StarBoard* Journal of the FlagShip/StarShip, SIGS (Special Interest Groups) on the Delphi and GEnie telecommunications networks. Permission is hereby granted to non-profit organizations only to reprint this article or pass it along electronically as long as proper credit is given to both the author and the *StarBoard* Journal. ********************************************************************* AMIGA TECH TIP: BBS FILE TRANSFERS by Thad Floryan (THADF on DELPHI) Most of you start to explore the world beyond your Amiga after purchasing a modem and a communication program. Many of you become participants in the forums and discussion groups on the BBS systems you reach while finding new "pen pals". Some of you attempt to build your software libraries from the vast collection of free, public-domain programs found on the BBS systems, all too often with disappointing results. The solution to your BBS woes is understanding how files and programs are stored and transferred, and knowing how and when to use a special "toolkit" of programs to fix problems. ---------------------------------------------------------------------- THE BASICS: TEXT FILE FORMATS Simple text files created with an Amiga editor are stored one character after another. Each line on an Amiga text file is ended by a line-feed whose ASCII code is 10 decimal (hex 0A). Other systems end their lines differently; for example, a Macintosh (tm) file has its lines ended by a carriage-return whose ASCII code is 13 decimal (hex 0D), whereas an IBM PC and many mainframes end their lines with both a carriage-return and a line-feed. The Amiga's file size will be the number of characters entered; there is no end of file character. On many other systems the presence of the character control-Z, whose ASCII code is 26 decimal (hex 1A), indicates the end of file. Amiga Notepad text files have their lines also ended by a line-feed, but there are more characters and binary information after the last line. The first extra character is a control-Z, and the remainder describe the pen and paper colors and the font and its style. The various word processors for the Amiga each have their own file formats, in which are stored tab settings, margin information, text rendition, and other specific data. Many of the word processors have a command to store text as a "simple" file. Source code statements for many of the programming languages available on the Amiga are also stored as "simple" ASCII files. For example, filenames ending with ".c" are usually C programs. The ".c" is the file's extension, and many other common source file extensions you may see on the Amiga include: extension Language .ASM Assembler .C C .FOR Fortran .MOD Modula-2 .PAS Pascal .SCR Forth screen file ---------------------------------------------------------------------- THE BASICS: PROGRAM FILE FORMATS Contrasted with text files, program files are binary and are not easily digested by editors or word processors. A program file is the translation of your source program into a machine executable form. As such, a program file is rigidly formatted into sections called "hunks" which contain code, data, and other information that control the loading of your program from disk into memory for execution. AmigaDOS insists on an exact file format for a program file, else it will display the dreaded "Unable to load program: file is not an object module". The most-often cause of this problem is extra data in the file beyond the "hunkend" section; several cures for this ailment are described later. For the technically-minded, the detailed description of the program file format is in the AmigaDOS Technical Reference Manual (CBM 327266-01) and reprinted in The AmigaDOS Manual (Bantam, ISBN 0-553-34294-0) which should be consulted for specifics. Program files typically have no extensions, but occasionally one may be found with the extension ".exe" which is short for "executable". An extension on a program file is a carryover from other systems and is not encouraged on the Amiga, especially for programs designed to be started from the Workbench. ---------------------------------------------------------------------- THE BASICS: DATA AND BINARY FILE FORMATS Data and binary files are, for the most part, formatted according to the whims of the creator. There is one notable exception: an Interchange File Format (IFF) data file for the exchange of pictures, text and sounds between paint programs, data base systems, information retrieval systems, and other creative and productivity application programs. Another exception is the ICON file which may be associated with a program file or a data file. The format of an IFF file is described in the documents available on the Amiga Freely Redistributable Softare Library Disk Number 16. The disks in this series are colloquially known as "Fish Disks" as tribute to Fred Fish who has contributed his expertise and time helping make the Amiga the success it deserves to be. "Fish Disks" are available from FAUG, many cooperating Amiga Dealers, and by direct subscription. A program file named MYPROG will usually have an accompanying ICON file named MYPROG.info. A paint program saving a picture as MYMASTERPIECE will create a MYMASTERPIECE.info ICON file . A word processor will create LETTER.info as a companion to a LETTER text data file. Another binary file type is the object file, which is an intermediate stage between a source program file and an executable program. Some software developers allow you to use the fruits of their labors with an object file while choosing not to reveal how they did it in a source program file. Object files have "hunk" sections and are meant to be linked with your other object files to become an executable program. The file extensions of object files are typically ".o", ".obj", ".sub" and ".lib". Data, object and binary files must be exchanged exactly like program files. Extra or spurious information in such a file will confuse the recipient or application program. Methods to assure exact transfers are discussed later. ---------------------------------------------------------------------- THE BASICS: PROTOCOLS Most BBS systems you reach will either be those operating a version of BBS-PC software on an Amiga, or those operating a FIDO node on an IBM PC or variant. All of these systems will offer a Files section for their validated members. The Files section on these systems will contain program and data files that you can use on your Amiga. Copying a file from a BBS system to your Amiga can be almost as easy as using the COPY command to duplicate files from one place to another on your system. However, your system and the BBS must speak the same "language" to transfer files, much as you need to converse in a common language to exchange thoughts with someone from another country. In the parlance of the BBS community, the common language is a "protocol". A protocol is an agreed-upon ritual of greetings, exchange, and farewells. The most common protocols you'll see have the names TEXT, XMODEM, XMODEM-CRC, and HVP/Smartcom. Without getting involved in a history lesson, suffice it to say that your communication program will understand one or more of these protocols. You must tell the BBS system which protocol you are using before you can exchange files successfully. The TEXT protocol is one in which the contents of the file are considered to be ASCII characters, and the text of the file is captured by your terminal program and stored verbatim into a file while you are reading it. But life is not always so simple. Some files may have been created on another type of computer system and may have different characters at the end of each line causing the text to overprint or display other strange characteristics. Don't despair, a solution to fix the files on your system is described later. The XMODEM protocol attempts to transfer the contents of any file to your system in chunks of 128 characters at a time. Each chunk is preceded and followed by control information which helps to assure exact transfer of the data. If the transfer of a chunk of data is inaccurate, the sender is instructed by the receiver to send it again. This "send and acknowledge" procedure is called handshaking. Files transferred using the XMODEM protocol will arrive at the destination exactly as sent, except the size of the resultant file will be a multiple of 128 characters or bytes. This "padding" of the received file is the source of problems transferring Amiga program files; remember, AmigaDOS expects a program file to adhere to an exact and specific format. Solutions to this dilemma are discussed later. A variation of the XMODEM protocol is one in which text files are processed differently from binary files. This variation will convert, say, a file stored with carriage-returns and line-feeds, or with carriage-returns only, to the line-feed only format expected on the Amiga. Consult the manual for your terminal program to see if this variation is supported by the program. If not, you may have to post-process the file after it is downloaded to your Amiga; more details later. The XMODEM-CRC protocol is similar to the XMODEM protocol, except that some additional information is appended to each chunk transferred that permits the receiver to correct any transmission errors. This additional "smarts" can save time when transferring files over a noisy telephone line, because the errors can be corrected locally (on your system) without requiring the sender to re-transmit the entire chunk that was in error. The same problems with 128-byte multiple file sizes apply here as with the standard XMODEM. The HVP/Smartcom protocol eliminates most of the problems occurring with XMODEM transfers in that the resultant file will be the exact size as the source file, and the transmitted chunks are 512 bytes between handshakes resulting in a faster file transfer. However, if the original file was uploaded to the BBS with XMODEM, you may still see the "file is not an object module" when you attempt to run a program downloaded with HVP/Smartcom. The same solutions for XMODEM transferred files can be applied to HVP/Smartcom transferred files if necessary. ---------------------------------------------------------------------- THE CURES: PREPARATION Many BBS systems provide a file directory listing showing a file's size and comments posted by the person who originally uploaded the file. Write down as much information about the file that you can find. One specific item to seek is any comment referring to a "chop" size; you may need to examine all E-MAIL postings to find this information. Knowing whether a file is text, a program, or data will also be helpful. ---------------------------------------------------------------------- THE CURES: SOFTWARE TOOLS You will need, as a minimum, copies of the "toolkit" programs listed below. I suggest you get copies of the following programs direct from your dealer, users' group, or a friend; some are available on the "Fish Disks". USQ Unsqueeze utilty. Its companion is SQ which is used if you're uploading files to a BBS. RemCr Removes carriage-returns from a text file. (Leaving, hopefully, line-feeds as a line terminator). AddCr Adds carriage-returns before every line-feed. Used before uploading a file to a BBS. 13to10 Converts all carriage-returns to line-feeds. Strip Removes control-characters and other binary "junk" from a text file. FixObj Adjusts a program file's size so "hunkend" is at the proper end of file. There is a new version: FixObj2. CHOP Fixes a file to a previously-known correct size. ARC Amiga archive utility. Groups collections of files, and splits them into correct-sized files. Additionally, the TYPE program (available at the CLI command level on your Amiga) can be a valuable aid to diagnosing problems with transferred files. ---------------------------------------------------------------------- PROGRAM TOOLS: TYPE The TYPE program/command is a starting point to help you determine which other tools you'll need to correct problems with downloaded files. With TYPE you can identify executable programs, IFF pictures, compressed files, text files, and other pertinent characteristics. For example, suppose you've downloaded a file named XYZ and you want to find out what it is. The following samples show how to interpret the hex dump from TYPE. Note that the "***BREAK" is a result of typing a control-C to interrupt TYPE; after a few lines of output, there is usually no need to see more. a)CLI> type XYZ opt h 0000: 000003F3 00000000 00000046 00000000 ...........F.... ***BREAK In this example, XYZ looks like an executable program. The "key" is the "000003F3" which is the code for a "hunkheader". In the absence of a file extension, it would be reasonable to assume this file is a program that could be "fixed" using FixObj or CHOP. b)CLI> type XYZ opt h 0000: 000003E7 00000002 73747269 70000000 ........strip... ***BREAK In this example, XYZ looks like an object file (the output of a language compiler); the clues are "000003E7" (code for a "hunkunit") and the name "strip". In fact, this file is the object output of the Lattice C compiler for the STRIP program. Size problems with this file could be fixed using FixObj or CHOP. c)CLI> type XYZ opt h 0000: 76FF3BF1 5665726D 6565722D 4769726C v.;.Vermeer-Girl 0010: 00000102 00010004 00030006 00050008 ................ 0020: 00070009 0000FF0B 000A000C 006FFFFF .............o.. 0030: FF0D000F 000E0011 00100013 00120015 ................ 0040: 00140017 00160019 0018001B 001A001D ................ 0050: 001C001F 001E0021 00200023 002200FC .......!. .#.".. ***BREAK This one is "tricky"; additional clues are necessary. From the BBS listing, this file was described as an IFF picture, yet it cannot be viewed with DeluxePaint (tm), Aegis Images (tm), GraphiCraft (tm), SeeILBM, ShowILBM or DPSlide, especially since the hex dump does NOT look like a typical IFF file dump per: CLI> type Vermeer-Girl opt h 0000: 464F524D 00009CDC 494C424D 424D4844 FORM....ILBMBMHD 0010: 00000014 014000C8 00000000 05000000 .....@.......... 0020: 00000A0B 014000C8 434D4150 00000060 .....@..CMAP...` 0030: F0F0F020 1030C050 50D05030 E06040F0 ... .0.PP.P0.`@. 0040: 8050F090 60F0A070 F09000F0 A000F0B0 .P..`..p........ 0050: 00F0C040 F0C080F0 B070E0A0 60E09050 ...@.....p..`..P ***BREAK Based on experience with other files, and because there is an item ("Vermeer-Girl") which could be a file name, this XYZ file then becomes a candidate for unsqueezing using the USQ program. As you use TYPE to examine other files, you'll gain experience identifying files by their contents. Another example using TYPE shows extraneous characters in a text file: CLI> type XYZ First line Second line Last line CLI> type XYZ opt h 0000: 46697273 74206C69 6E650D0A 5365636F First line..Seco 0010: 6E64206C 696E650D 0A4C6173 74206C69 nd line..Last li 0020: 6E650D0A ne.. Note the "0D0A" combinations which are carriage-returns followed by line-feeds. The carriage-returns can be removed using either the RemCr or STRIP programs. A final example shows another troublesome situation: CLI> type XYZ Last linene CLI> type XYZ opt h 0000: 46697273 74206C69 6E650D53 65636F6E First line.Secon 0010: 64206C69 6E650D4C 61737420 6C696E65 d line.Last line 0020: 0D . with lines ended by only a carriage-return (hex 0D). When typed to the screen, each line overprints the previous one resulting in "Last linene". This problem is fixed by using the 13to10 program, which converts all carriage-returns, whose ASCII code is 13 decimal, to line-feeds, whose ASCII code is 10 decimal. ---------------------------------------------------------------------- PROGRAM TOOLS: USQ and SQ The SQueeze and UNsQueeze utilities are gaining popularity on BBS systems for the storage and transfer of single files (see ARC for collections of files). The reason is obvious after examining the directory listing below: CLI> list Vermeer-Girl 40164 rwed Yesterday 23:22:52 Vermeer-Girl.qq 17293 rwed Yesterday 23:22:49 The file without the ".qq" extension is a normal IFF ILBM picture file created with a paint program. The file with the ".qq" is the same file after compression using SQ. Besides reducing disk space, a squeezed file will also download faster, saving you money and time. Additionally, a program, text or data file is ready for immediate use after unsqueezing! The squeezed file can be transferred using XMODEM without any worry over the effects of XMODEM padding since the unsqueezing process will create an exact image of the original file. The SQ and USQ programs are distributed in both source and executable form on Fish Disk Number 10. Their use is very straightforward: CLI> SQ file1 [ file2 ... filen ] squeezes file1 [ thru filen ] creating file1.qq [ thru filen.qq ]. Very often you'll see file extensions containing the letter "Q" which is a clue that SQ was used to create the file and that USQ should be used to unsqueeze the file. By convention, the following extensions are in common use: .QQ (none)for a squeezed IFF picture file .TQT (.TXT)for a squeezed text file .DQC (.DOC)for a squeezed documentation file .EQE (.EXE)for a squeezed executable program file Note also that the original file name is stored in the squeezed file as shown in example (c) of the TYPE program. CLI> USQ file.qq [ file2.qq ... filen.qq ] will sequentially expand file.qq [ thru filen.qq ] to their original sizes and names. ****** IMPORTANT NOTE ****** You should ALWAYS first display using the "TYPE file OPT H" command the first part of any file before using USQ. The reason is that often the original creater of a squeezed file was careless and may have squeezed a file using a directory path specification as part of the file name; USQ will attempt a restore to the same path specification on your system. Referring to example (c) of the TYPE program, the complete original file name is stored in the squeezed file; if the creator squeezed from, say, DF2:FILE, the contents of the squeezed file would be: CLI> type FILE.QQ opt h 0000: 76FF810B 58595A3A 46494C45 00110002 v...DF2:FILE.... 0010: 00010004 00030006 00050008 0007000A ................ 0020: 00090096 FF0B0091 FF9AFF93 FFDFFF0D ................ 0030: 00F2FF0F 000E000C 0010008B FF8CFF9E ................ 0040: FF90FFAC FF8DFF9B FFB3FFB9 FF9CFFF5 ................ ***BREAK and if your system doesn't have a DF2: the USQ will fail. Also, it is possible for USQ to overwrite another file on your system whose name is the same as that stored in the squeezed file. An ounce of prevention is worth a pound of cure; always CHECK before blindly using the USQ program. ---------------------------------------------------------------------- PROGRAM TOOLS: RemCr, AddCr, 13to10, STRIP This collection of four programs, and other similar programs, processes text files by converting, adding or deleting characters. The source code of these programs is readily available on many BBS systems and in public domain libraries; feel free to change the programs to suit your needs. Most utility programs of this nature are written in C, but you can write similar programs in any language available to you such as AmigaBasiC. RemCr simply removes all carriage-returns from a file. Usage: CLI> RemCr inputfile outputfile AddCr inserts a carriage-return ahead of every line-feed in a file and is used to prepare a file for sending to another type of computer. Usage: CLI> AddCr inputfile outputfile 13to10 converts all carriage-returns, whose ASCII code is 13 decimal, to line-feeds, whose ASCII code is 10 decimal. Usage: CLI> 13to10 inputfile outputfile STRIP removes most control characters, all NULLs, and other "garbage" from a text file, leaving only line-feeds, and all printable ASCII characters and symbols. A file processed by STRIP will be able to be edited using ED. Caution: this program also removes TAB characters, so if tabs are important to you, either use another program, or modify STRIP's source code to preserve the characters of interest to you. Usage: CLI> STRIP inputfile outputfile ---------------------------------------------------------------------- PROGRAM TOOLS: FixObj and FixObj2 The FixObj program will fix most problems caused by downloading an executable program file using XMODEM. FixObj2 is simply an updated version of FixObj. FixObj examines an input file for a final "hunkend" and removes extraneous data beyond that point in the output file. This is the program to use when you see "Unable to load program: file is not an object module". Usage: CLI> FixObj inputfile outputfile CAUTION: the inputfile cannot reside on a RAM: disk due to a problem with the "fseek()" file function in V1.1 AmigaDOS. The source code file and executable program file of FixObj is available on Fish Disk Number 10. The source code for FixObj2 is starting to appear on BBS systems around the country. ---------------------------------------------------------------------- PROGRAM TOOLS: CHOP This program corrects the size of any file whose size must be some exact quantity. Very often the "chop size" of a file will be listed on BBS systems either in the file directory descriptions or in E-MAIL posted by the person who uploaded the program. CHOP was the first available program on the Amiga that corrected XMODEM padding problems. Variations of CHOP are written in C and in Basic. All versions expect an input file name, an output file name, and a chop length. The "chop length" is the size you want the output file to be. For example: CLI> CHOP inputfile outputfile 9274 One version of CHOP written in ABasiC is: 50 input% = 1 60 output% = 2 65 count% = 0 100 input "Input file ";infile$ 110 input "Output file ";outfile$ 115 input "number of bytes to copy";total% 120 open "i", #input%, infile$ 130 open "o", #output%, outfile$ 200 if eof(input%) then goto 990 201 if (count% = total%) then goto 990 210 get #input%, char$ 220 print #output%, using "&"; char$; 221 count% = count% + 1 500 goto 200 990 close #input% 991 close #output% 999 end ---------------------------------------------------------------------- PROGRAM TOOLS: ARC ARC, also known as AmigArc, is gaining popularity as the "premier" tool for file storage on BBS systems and for use on one's own system. ARC collects and compresses groups of files together in a single file. A multi-file project can have all its files kept together. And, as with squeezed files, downloading time is reduced because the files are compressed. For example: CLI> list MEMPROJ.ARC 16640 rwed Yesterday 17:10:57 readme 2894 rwed Yesterday 17:46:40 mem.sch 12356 rwed Yesterday 17:46:29 clock.c 4047 rwed Yesterday 17:46:00 clockset.doc 1205 rwed Yesterday 17:46:07 mechanical 7107 rwed Yesterday 17:46:13 memtest.c 468 rwed Yesterday 17:46:31 parts.lst 1670 rwed Yesterday 17:46:35 clock.sch 3980 rwed Yesterday 17:46:04 mem.doc 5284 rwed yesterday 17:46:19 10 files - 129 blocks used The file named MEMPROJ.ARC is the single file which contains the other nine files, as shown by: CLI> ARC l MEMPROJ.ARC AmigArc v0.11 TEST version. Please do not use after August 1st,1986. Please refer any comments, BUGS, etc. to: Raymond S. Brand c/o The Lido Hotel (a Fido BBS) (203) 248-7324 or AMUSE (a Fido BBS) (212) 269-4879 Name Length Date ============ ======== ========= clock.c 4047 22 Jun 86 clock.sch 3980 22 Jun 86 clockset.doc 1205 22 Jun 86 mechanical 7107 22 Jun 86 mem.doc 5284 22 Jun 86 mem.sch 12356 22 Jun 86 memtest.c 468 22 Jun 86 parts.lst 1670 22 Jun 86 readme 2894 22 Jun 86 ==== ======== Total 9 39011 Extensive documentation accompanies ARC. The three most useful commands to ARC are L (to list the contents of the archive file), X (to extract files from the archive file), and A (to add files to the archive file). The simplest command forms are: CLI> ARC L archive.file CLI> ARC X archive.file CLI> ARC A archive.file * To avoid overwriting other files when extracting from an archive and to avoid including other files when building or adding to an archive, it is easiest to operate ARC while in a unique subdirectory or on a freshly formatted disk. ARC and its documentation is available from many users' groups and dealers. Updated and improved versions of ARC are available periodically; the current version is V0.11. ---------------------------------------------------------------------- PROGRAM TOOLS: Improved Terminal Programs Last but not least, smarter terminal programs are becoming available that will ease the burden of deciphering problems with XMODEM file transfers. Some programs include the capabilities of FixObj, and others combine the functions of RemCr, AddCr, 13to10 and STRIP while doing a file transfer. It is to your benefit to throroughly read the documentation that accompanies your terminal program, and request new features in updated versions. Many software producers do listen to their customer's suggestions with the result that everyone "wins" as products evolve to solve your requirements. ACTION> (Next, Down, Xm, List)