(PC developers specifically)
The Palm/USR conduit system is designed to allow a hotsync to occur with both native and foreign data. The designers assume that foreign data is in such a format that code will have to be written to implement a hotsync. Therefore, the conduit system assumes that each conduit will be different/unique. And that the data format for each conduit will be different/unique.
The designers of the HotSync system also provide a "backup conduit" which simply transfers raw data, in bulk.
I felt that these assumptions were a bit onerous. They assume that everybody who develops a PC app will be interested and capable of writing a DLL. I didn't think this was a reasonable assumption.
[Aside: To be a "real" Pilot programmer, you have to be conversent in 4 API's (Mac, Pilot, Win32 and ConduitSDK). That's 2 too many for me.]
[Aside: My assumption seems valid. Notice the really nice apps like PilotMoney and Hourz, which have taken the approach of "exporting" their data to a memo, and relying on the memopad conduit to move their data. Now, with a generic conduit, they don't have to do that.]
The Conduit Manager system uses a generic conduit, which can sync any Pilot database on a record-by-record basis. In order to do this, here are my assumptions:
This means that developers on the PC can work at a simpler level. There is no need to learn the entire Conduit management system. On the other hand, we (the PC developers) have to write the code to read and write MAC style numbers, and deal with fixed format records that are created by a foreign app. If you can live with this, climb abord.
In the coming weeks, I will bolster this package with documentation on the file formats, and sample code for the use of the GenCond.DLL and the Generic Conduit Manager.
Have a look at the sample code to follow this discussion
At the start of the viewer program, you can automatically select the .dat file that the HotSync/GenCond writes to. Simply use the default directory and append your specific directory and filename. Alternatively, you can let the user specify the filename by presenting the user with the FileOpen dialog.
Be aware that the files GenCond.DLL generates are almost identical to those created by the Backup conduit that ships with the Pilot. If the user is not using the GenCond.dll, you can direct your file open to the Backup directory, and the approriate .pdb file.
I recommend that you load the whole file into memory. Allocate a single file header (78 bytes) and read it in. Then allocate room for the file index, which is 8 bytes times the number of records. In the examples, I call this aRecEntry[ ].
(optional) If your database uses the "attributes" or "categories" blocks, you should allocate these blocks at this point and read them from the file.
Now you are ready to read the actual records. In my examples, I read them into an array aRec[ ].
Remember that some of the records you have loaded may be marked as deleted. These records must be preserved until the next hotsync, but should not be displayed.
What I usually do at this point is run through the aRecEntry[ ] and fill a list box with those records that are not deleted; the "deleted" flag is 0x80 in the aRecEntry[ ].attrib. I use the listbox .ItemData property to keep the appropriate index back into aRecEntry[ ].
This is the appropriate time to filter categories as well, or hide private/hidden records.
To sum up: at this point, you want to present your user with a subset of all the records.
You never know when the user will start a hotsync. If he does, we have to assume that he wants any changes saved to disk before the hotsync starts. To do this, watch out for the DDE message, and
At this point, reload the file from disk, since it may have new data in it.
The aRecEntry[ ] holds a lot of information. On disk, it is
long file pointer byte attributes (a nibble of status bits and a nibble of category 3byte ID (unique ID on the Pilot)
The file pointer is the offset from the beginning of the file. Once you have loaded the data into program memory, you don't really need this file pointer anymore, so if you wish you may change this 'long' into a 'void *' and use it as a pointer to the raw record data.
The attributes high bits are 'deleted', 'modified', 'archive' and 'hidden'. The 'deleted' bit means that this record is due to be deleted. When this is set, you can discard the raw record data, but do not delete the aRecEntry[ ] element; you need to keep is so that the next hotsync can tell the Pilot to delete the corresponding record. As a general rule, your viewer/editor app never fully deletes an entry, it just marks it for deletion. Only the conduit does the actual deletion.
The 'modified' bit tells the conduit that this record needs to be sent to the Pilot. New records should always be marked 'modified'.
The 'archive' bit is not used in this version of the conduit. Use the 'private/hidden' bit as you see fit.
The lower nibble of the attribute byte is the category number.
The next 3 bytes are the ID as assigned by the Pilot. Whenever you make a new record on the PC, always assigne the ID as 0x000000. Be aware that the Backup conduit that comes with the Pilot creates files which are identical to those generated by the GenCond.dll conduit, except that all the ID's are flushed to 0. If you allow your users to view/edit these backup files, the result is only appropriate for a PCtoPilot one-way sync (because all the ID's are 0).
One thing obviously missing from the aRecEntry structure is the length of the raw data record. When you are reading the data from the file, you have to deduce the record lengths by noticing where the next record starts. The length of the last record is determined by the end-of-file.
When you read the raw data into memory, you should either read it into a data object that is aware of its length (such as a VB string, or a std::cstring) or else modify the aRecEntry structure with extra data:
long file-pointer/void-pointer byte attribues 3byte ID long record-length
Remember that all 'numbers' in the file are stored in Mac format. You will almost certainly need a SwapShort() and SwapLong() routine.
Good Luck
Several people have asked me about grabbing data, or injecting data into the databases of the 4 main Pilot apps (address, datebook, todo & memopad). I would not recommend doing this. This is not what the generic conduit was designed for. The Pilot desktop apps have import and export facilities which should give you what you want.
The problem is twofold. First, the existing Pilot conduits significantly modify the data formats before saving the data to disk. Unless you know what these format are, you are risking data loss. If you over-ride the native conduits, your Pilot Desktop will not see the data.
Secondly, the entire Pilot sync mechanism is based on the assumption that one computer syncs with one Pilot. There are sub-mechanisms in place to deal with multi-computer, or multi-pilot, but there are not mechanisms in place to deal with a three-way-sync. What I mean by this is: if you decide to start syncing the Pilot address book to your down database, then you should kiss goodbye to the Pilot Desktop address book.
(hackers only)
In order to write a generic conduit, I had to figure out how to make sure that the conduit gets called by the HotSync system. Using HotSync 1.0, that is not simple, since the HotSync.exe (actually syncmgr.dll) enumerates the databases in the Pilot, and only calls the DLL's that are required to service this list. If the app that the user has requested does not make it onto this enumeration, the DLL will not get called.
To get around this, I install a proxy in the MemoPad conduit, by inserting an ApplicationX registry entry. During the startup of the gencond.dll, I reach back into the Component3 registry entry and execute the memcond.dll first, and then run through the ConduitManager entries.
For each one, the gencond.dll uses the registry information to locate a file and reads it into memory. It then tries to sync up, record by record and creates a new local file. If all went well, the old file is renamed and the new file written.
If you have an application that is running, it will get a couple of DDE messages. Before the file is opened by the conduit, you will get a "application/notify" message command with the string "sync start". You have 1.5 seconds to write the file to disk before the gencond.dll will try to read it.
After the file has been written, you will get a "application/notify" message command with the string "sync finished". You can then re-read the file into the app. [All records will be marked clean, and deleted records will have been purged.] [Look at the source for cbasVw, in Form_LinkExecute.]
In order to locate the files, each time the gencond.dll is executed, it puts an entry in the registry: HKCU/Software/Palm Computing/PilotDesktop/Preferences/LastUserDir ="name of user dir, without trailing backslash"
To locate the data files, simply concatenate a "\DirName" to this string. This should be the same DirName that the user put into the ConduitManager. Examples are "DinkyVw" or "cbasVw".
If you wish to use it, the GenCond.DLL exports a simple little routine to build up multiple backup files.
/*SDOC
Function: MakeBackupFiles
Params: sFileName the name of the target file nBackups the number of backups to keep
Action: The sFileName is used to locate a directory where the output files will go.
If the sFileName does not exist, the function exits with a 0
The current backup files are enumerated and the actual file called sFileName will be renamed to a backup file.
Files which exceed the backup limit are deleted.
Returns: 0 if no file activity took place; usually means that the given file does not exist
Comment: Style 0 backups proceed like
file.txt (original)
file.txt.1
file.txt.2
file.txt.3 ... (note: this destroys the Win95/NT association)
EDOC*/
C/C++ int MakeBackupFiles(const char* sFileName, int nBackups);
VB Declare Function MakeBackupFiles lib "gencond.dll"
alias _
"_MakeBackupFiles@8" (ByVal sFileName as String, _
ByVal nBackups as Long) as Long
All the .dat files saved on disk by the Generic Conduit (gencond.dll) follow a similar format. The file consists of a header, a record index (8 bytes per entry), an optional Attributes block, an optional Categories block, followed by the records themselves, concatenated.
The header is as follows:
struct tHeader {
char sTitle[32]; // description of this collection
DWORD dwVersion;
tTime tCreated;
tTime tCreated2; // in "seconds since Jan 1, 1970"
tTime tLastHS; // time of last hotsync
long dwRes0;
long ofsAttributes; //
long ofsCategories; //
DWORD dwType; // 'Data'
DWORD dwCreator; // 'dwDP'
DWORD dwRes1;
DWORD dwRes2; // reserved
WORD nRecs; // number of records
};
followed by an array[nRecs] of these:
struct tRecEntry {
union { DWORD ofs; // on disk, file offset to this record
byte* pRec; // in ram, this is a pointer to the compressed data
};
DWORD _id; // one byte with flags from kRec* above
// merged with 3 bytes of unique ID
};
The lower 3 bytes of _id is the Pilot unique number for this record. The upper byte is two nibbles:
and the lower nibble is the category ID
For the purpose of explaination, let's assume that ofsAttributes and ofsCategories are both 0.
Immediately following the array of tRecEntry are the data records. Each one starts at tRecEntry[i].ofs, and goes to tRecEntry[i+1].ofs (except the last one, which goes to EOF).
In VB, the structures are:
Private Type tHeader
sName As String * 32
dwUnknown1 As Long
dwTime1 As Long
dwTime2 As Long
dwTime3 As Long
dwLastSync As Long
ofsAppInfo As Long
ofsCategories As Long
dwType As Long
dwCreator As Long
dwUnknown2 As Long
dwUnknown As Long
wNumRecs As Integer
End Type
Const kOfsSort = &H34
Const kOfsCategories = &H38
Const kOfsCreator = &H3C
Const kOfsNumRecs = &H4C
Const kOfsEntries = &H4E
Private Type tRecEntry
ofs As Long
attrib As Long
End Type
Const DIRTY = &H40000000
Const DELETED = &H80000000
Const RE_PRIVATE = &H10000000
[See the VB source code included in the GCM distribution]