R:891027/0056z @:N6VV.#NOCAL.CA.USA.NA Pleasant Hill, CA #:26948 Z:94523 As of today Thursday October 26th , things have started to return to normal around the San Francisco Bay Area. We have learned a lot about the handling of emergency information and Health & Welfare traffic via packet radio. I thought I would try to get down my thoughts about the performance of packet radio and lessons learned from the recent emergency created by the earthquake while the impressions are still fresh in my mind. BACKGROUND: The Northern California Packet Association is comprised of and extensive network made up of hilltop node locations and over 30 BBS systems spread from Redding in the far northern part of the state to Kern County in the central part of the state. The BBS forwarding system employs a 220 MHz and 433 MHz backbone which utilizes 5 mountain top sites and 5 LAN Gateway BBSs. The theory is that each of the 5 LAN BBS systems serve a number of other BBSs on their LAN. The LANs are: NORTH BAY/ W6PW , SACRAMENTO VALLEY/WA6RDH, SANTA CRUZ/ N6IYA, EAST BAY/ N6VV, and a SOUTH BAY LAN with WB6ASR. The LAN of which I am part of, connects into the Central Valley Backbone which runs the length of the Great Central Valley of California from Southern California to the Oregon Backbone. As a LAN Gateway station, I am responsible for forwarding to the other LAN Gateway stations and to the BBSs on my LAN, which includes : W6FGC,WO6Y,KJ6FY,WB6V,K6RAU,N6OA, WB6ODZ,WB6MIF,N6ECP. In addition since my local node is on the Central valley backbone I also forward North to AL7IN in Oregon and south to K6IYK in Los Angeles. The Central Valley 220 Backbone is one of 2 trunks to the south. The other is down the Coast Route from WB6ASR and AA4RE. Recognizing the need for redundancy in the network, a number of backup routes have been created and we have had to use these many times in the past. For example last year, during the heavy fire season, a fire on Mt VACA took out the Sacramento Valley LAN Backbone node WA6RDH-11. To maintain communications Dennis, WA6RDH had added a 223.54 port on his BBS to be able to communicate directly with N6VV on the East Bay LAN frequency and to be able to pass Inter-LAN traffic without going thru the backbone. In addition, when selecting 2 meter nodes, considerable planning went into choosing frequencies to provide alternate paths in the event of the loss of the backbone. Most of the nodes in use in this area are triple or quad port nodes and include 2M access into the network. Strategic selection of 2M frequencies has given us the backup which was put to good use during the emergency. More on this later. N6VV BBS: The N6VV BBS is a 6 port system operating on the following frequencies: 144.99 223.54 433.41 441.50 14.109 21.097 I operate on 2 HF frequencies and provide in addition to N6IYA (14098), N6OA (14107), KB6IRS (14107), N6EEG (10149) the HF Gateway access to the Northern California area. The system is a 2M IBM PC/AT with 20Meg hard drive. On normal months I handle between 9 and 13,000 messages per month. BBS forwarding access into our area is therefore via one of the HF Gateway stations listed above or via the 2 southbound 220 links into Los Angeles and the single 220/433 link north to Oregon. TUESDAY the 17th of October - Like everyone else around here, I spent a considerable amount of time on that Tuesday trying to get tickets to the World Series but by 3:00 PM had decided that I was going to have to content myself with watching the series on television that evening. At about 4:30, me and most of the rest of the office started to migrate towards the exits in order to make it home in time for the TV coverage of the series which was scheduled to start at 5:00 PM local time. Incidentally, everyone around here is convinced that the World Series saved litteraly hundreds of lives at places like the Cypress Structure on Highway 880 since many thousands of people around the bay, were doing exactly what we were doing and trying to make it home before 5:00 PM to watch the game. I have driven the 880 stretch that collapsed many times and at 5:00 PM during the height of rush hour there would normally have been hundreds of cars jammed into that 12 block section which collapsed. I work in Concord only 3 miles from my home in Pleasant Hill so arrived home about 4:45 and walked out to the shack located about 100 feet behind my house, to "check the BBS", as is my usual habit when coming home. I was browsing thru the messages going and coming on the system, when suddenly shortly after 5:00 PM my chair started to bounce up and down. At first I thought it was just one of the many little earthquakes that any native Californian has grown accustomed to riding out with little concern. Suddenly the motion began to get much more violent and things began flying around the shack. A large bookcase to my left fell over and spilled hundreds of books and QSTs across the floor. Loud noises from stuff falling over in my storage area behind the shack convinced me it was time to get out of the building. I dashed for the door and stood just outside the shack looking up at my tower. My main concern at this moment was the tower immediately over my head. The Rohn 45 tower had a 6 element KLM 20M beam and a 4 element KLM 40M beam at 85 feet. The Phillistrand guy wires were alternately snapping and going slack and the beams were jiggling wildly. I decided it was probably prudent to try to get at least 85 feet away from the immediate area. Dashing thru my back yard to scoop up my 8 year old son we went to the front yard and watched the trees and telephone poles waving wildly in the air. I live on a small knoll and have an excellent view out across the Diablo Valley. For another 10 seconds we stood watching the eerie sight as the whole valley jiggled and shook. For about 30 seconds we stood there affraid to move. After checking with the rest of the family to see that they were all safe, I ran back to shack to see if it was OK. The power was off and sirens were already screeming all over the area. My neighbors were all out in the streets and some people had their car radios on listening to the first reports. The first serious damage reports were coming in and I heard that part of the Bay Bridge had collapsed. My first thought was power. I got out my generator and started setting up on the road next to my shack. My neighbors came over and helped me with the gasoline. I started the generator and let it run for a while to settle down. I got out my hand held and began to monitor the 2M repeater in our area. There was a lot of conversation on the air as the hams began to check in. This was a good sign since the repeater is co-located with the nodes that I use for forwarding. It is at a County site which has emergency power so I had assumed that it would be on the air. In a few minutes I had the VHF/UHF side of the BBS going. Another ham, WA6HAM Steve was bringing over another generator to get the HF station going. The first step was to find out what nodes and BBSs were on the air. I began working my way thru the network to find out who was there. To my horror, W6PW in San Francisco did not respond. In addition the Southbay LAN Backbone node on Crystal peak did not seem to be on. N6IYA the Santa Cruz LAN Gateway was not there. I started to get worried but was hopefull that it was just power that prevented them from being on the air. The SAC Valley stations including WA6RDH were up and the central valley guys and circuits all seemed to be OK. Without hearing any real news reports I began to assume that the worst areas were San Francsico and particularly the Santa Cruz and Gilroy areas. It turned out that I was right. I began working my way thru the Santa Cruz BBSs who I do not normally forward to because of our LAN Gateway structure when suddenly I got a connect with N6MPW. At least one BBS in that area was on. I started receiving messages from the other SysOps who were doing the same thing I was. We spent the first hour or two trying to put our network back togeather. WD6CMU from the Northbay LAN sent around a message saying he was taking over Gateway duties for the Northbay LAN. N6MPW took over gateway duties for the Santa Cruz LAN. That meant 4 of the 5 LANs were covered, but we still had no word from Roy AA4RE or Greg WB6ASR from the Southbay LAN. We later learned that the Crystal nodes had received severe damage since they were so close to the epicenter. I contacted Brad WA6AEO who is the control SysOp for most of our nodes in Contra Costa County and suggested that we put some of our emergency plans into place. We had already set up one node in the south part of the county on 144.93 which was WB6ASRs user port frequency frequency but I could not raise Greg on that port. We were able to move our Berkeley node on BALD to 144.99 which is the user port frequency of both W6PW and AA4RE. This would provide an alternate path for Roy into the network while his 220 access was down. The channel selection of 2 meter nodes and the frequency agility proved to be vital in reestablishing paths to these locations since Roy has returned to the air and at this time is still using the the BALD 144.99 node as his access to the network. That was the first lesson learned. HAVE SOME BACKUP PLANS for alternate paths when backbone nodes go away !! In our case it worked and all 5 LANs were back on the air within hours of the quake. I had started to compile a list of BBSs that were on and began to modify my forward files to get traffic to the right general area. Thankfully N6LDL was reachable on the 2M Norhtbay port of 144.97 and I set him up to recieve all San Jose, Los Gatos ZIPs. CMU was taking all the NORTHBAY LAN ZIPS. and N6MPW was taking Santa Cruz. One thing I learned during this rework of the forward files was that I would have been better off to have had independent files set up BY BBS instead of by LAN as it was a painstaking process rekeying all of the ZIP information. In that way I could have just moved my files around by BBS to reflect the crazy 2M forwarding patterns which evolved during the emergency. By this time the other generator had arrived and I was now up on 20 Meters on the 14109 NET. It was getting fairly late and I just had enough time to fire off a message to Dave W9ZRX that I was on before the band closed up. Sometime around this time K6IYK in Los Angeles connected to me and told me that the Coast route was gone but that the central valley 220 route had survived and to expect lots of NTS traffic. I gave him a brief rundown on the situation and told him that we were ready. By now traffic was starting to flow all around the network and Health and Welfare stuff was starting to come in from AL7IN in Oregon and K6IYK in Los Angeles. We sent out bulletins calling for a halt to ALL non essential bulletins on the network and I composed messages for the HF Networks requesting the same thing. Listening to the 2M and UHF repeaters into the Santa Cruz area it became apparent that the voice circuits were a zoo and that if any H&W traffic was going to make it into that area it was going to have to be on packet. The surviving repeaters were all tied up with tactical information and most of the net control stations did not want to hear about Health and Welfare traffic. Actual fights broke out on some of the repeaters over the handling of H&W traffic and I was very pleased when I started to hear people on the repeaters tell people "PUT IT ON PACKET". We had let them know that we had established routes into all the affected areas and we started to see traffic coming in from the outlying areas who seemed to be taking H&W traffic off of the 20M nets and dumping it onto the local BBS system. From my own experience I can tell you that phone service into the 408 area code was very spotty and some areas in the southbay were not available. HF SSB stations taking traffic for these areas had no way to deliver the traffic and most ended up just dumping the traffic into the local BBS system anyway. 415 was not bad after about 12 hours and was returning within 24 to many areas but I tried all nite to reach the Santa Cruz BBSs by phone and could not get thru. All during this period I was forwarding packet traffic for hours on end to N6MPW in Santa Cruz !! By 4AM in the morning I realized that I would have to also modify my outbound forward files and adopted the theory that I should just try to get it out of here to anybody so made heavy modifications to my forward files to get as much traffic to W9ZRX and W3IWI who are usually my best connects on 20 and 15 meters. As the sun came up and the HF bands started to come alive the traffic began to pore in. ZRX connected to me and had a 2 hour forwarding session. The inbound volumes were unbelievable. At some times I had W9ZRX on 20M, W3IWI on 15M, K6IYK from So. Cal on the 220 port, AL7IN from Oregon on the 433 port all forwarding to me at the same time. I only had 2 outbound ports to try to handle the 4 inbound ports which never stopped. I called for help and remote SysOps Brad WA6AEO and Dennis KA6FUB came over to the station and for the next 48 hours the BBS here was fully manned around the clock. We never went thru a normal forwarding cycle but had to use force forwards to try to stay up with the imbalance in inbound/outbound taffic. The number of messages on the board kept increasing since we could not keep up with it. I believe that the peak came on Wednesday nite with a total of about 800 active messages on the system. I had many hours before deleted all my bulletins so all of these messages were NTS. By the weekend we had handled over 5000 messages which had mostly been forwarded on into the Santa Cruz and Bay Areas. We had recruited a core of local hams to check in and handle the NTS destined for our area but even that was getting out of control. I was actually outside the seriously affected area and had power and phones back by the end of the first night, so I could imagine that the guys like MPW and PW were really starting to get jammed up. I started to receive messages from several SysOps that they wanted me to stop sending them traffic and one night got a call from Larry WB9LOZ, SysOp at W6PW that he already had 700 NTS messages backed up and could I just stop for a while. I explained to him that the message flow was like a freight train and that we could not just stop at certain points but would have to stop the whole flow. If I stopped forwarding out to the destination BBSs, I would have thousands of messages here and ultimately blow up. We would have to shut it off farther back stream or just shut down the gateways. We tried to contact KA6ETB the NCN Packet coordinator to see if we should send out a national bulletin asking everyone to please slow down the H&W and eventually one was originated. I knew that it would take several days for an @USA bulletin to make it around sufficently to have any effect so we had no alternative but to keep on forwarding. We need to think more about this flood problem and figure out a better solution. I am sure we will be discussing it heavily in the future but we did the best we could in the first 3 days after the quake. One important lesson learned here in the SF area is that packet turned out to be the most efficient means of delivering H&W traffic and while I have not heard the numbers from other gateway stations, I am now over 6000 pieces of traffic since the quake and I am sure the total will be in excess of 10,000 messages handled via packet. This is quite impressive since this is a tremendous increase in normal packet traffic and the software, systems, network and operators were able to respond wonderfully to this flood of traffic. I have heard horror stories from the guys who were handling H&W on the HF SSB Bands about jamming and poor organization. I am happy to tell you that other than the normal RTTY Jammers on 21097 (They don't like us there) we were able to move tremendous volumes of messages on HF Packet. LESSON LEARNED : 1. You need to have a backup Plan. I list this one first because we had a plan of sorts but had to make a lot of it up as we went. 2. Packet is wonderfully adapted to handling very high volumes of Health and Welfare traffic during emergencies, but probably shouldn't be used for tactical information handling. Voice circuits are better and the packet channels get jammed up with H&W anyway... 3. Because of networking possibilities, packet networks can respond better than any other form of Ham Communications to changes in the emergency situations. We completely rerouted traffic throughout the entire system in less than 3 hours... 4. We need a quiker way to handle network management bulletins. Maybe we need an emergency bulletin designator that EVERYONE supports, but is not used except under extreme emergency. 5. Every ham who uses a packet bulletin board should become familiar with packet NTS procedures. Under circumstances like this you have to get anyone with a TNC involved to handle the traffic. The vast majority of traffic handled locally at N6VV was NOT handled by our normal NTS liason people. We were recruiting people off the 2 meter repeaters to help. Fortunately we had a file called HOWTO.NTS in the file section that these people were able to download and read. Instant NTS handlers !! Many of the regular NTS people were working 24 hours a day in Red Cross facilities or emergency centers and never did check in to a BBS. Even with the thousands of hams in this area, we did not seem to have enough to go around. The emergency sites were recruiting hams from as far away as Sacramento to man sites in Santa Cruz. 6. A major problem we had on this end was the missadressed or non-addressed traffic that required manual intervention before it would flow thru the automated forwarding system. I only had 3 hours sleep by Friday night and the main reason was fixing this mail. Mail simply addressed NTSCA @ NTSCA with no further information was being received here. Each message like this required us to read them and in some cases to have a USPS ZIP code book lookup to get them on their way. When you are dealing with thousands of messages this can be a tedious job. I actually received personal mail addressed N6VV @ N6VV with a list of 25 friends they wanted me to contact. These BOOK type messages simply DO NOT WORK in the packet environment and require far too much manual intervention at the disaster site. All NTS traffic here and I thought elsewhere was ZIP code routed. personal mail is routed by BBS and ZIP code does not work for it but the NTS ZIP delivery system is well established and each NCPA BBS is provided with a complete list of zip codes for all of northern California and where they are supposed to be delivered. Messages received without ZIP code, die until manual intervention. These are just some of the lessons which we have learned. I am sure many more will be thought of in our afteraction meetings. In summary I think that the San Francisco Earthquake was the first great test for this new technology which we call packet and although I'm sure we will be able to find some faults, in general packet performed wondefully under what at times seemed like an impossible situation. My thanks to all the SysOps around the country who helped... Lew N6VV p.s. An interesting sidebar. All of these thousand of messages that were delivered were transmitted on 220.90 MHz. I can only hope that UPS will be able to utilize that frequency as effectively as we hams did during the Quake of '89