1 files changed, 529 insertions, 0 deletions

diff --git a/doc/ADATFS.txt b/doc/ADATFS.txt
new file mode 100644
index 0000000..e373690
--- /dev/null
+++ b/doc/ADATFS.txt
@@ -0,0 +1,529 @@
+HD24 ADAT file system format
+
+Reading ADAT HD24 disks
+-----------------------
+In the ADAT FST file system, the part of the file system
+that contains directory and song info is grouped in 
+32-bit words. When read on a regular PC, each group of 
+4 bytes will appear reversed. As such, when
+a sector contains a string 'TADATSF ', we should read this
+as 'ADAT FST'. In this document, we will assume that
+doing the byte swapping needed to properly read the sectors
+has already been done.
+
+It should be noted that the actual audio data has not
+undergone such byte swapping. As such, WAV data
+retrieved from the HD24 by FTP is stored in the same
+byte order as the data on disk.
+
+The sector size used on HD24 disks is 512 bytes.
+ADAT FST boasts disk support for disks up to 2 Terabyte.
+This is achieved by using 32-bit pointers that point to
+sector numbers. We will refer to these pointers as 
+sectorpointers.
+
+Writing ADAT HD24 disks
+-----------------------
+Writing HD24 disks is slightly tricky, because FST 
+filesystems contain backup copies of file system 
+sectors to help guarantee data consistency. 
+As a result, when we make an alteration to only the
+first copy of the file system structure on disk, the
+disk is considered inconsistent and the backup copy will
+be written back by the HD24 recorder at the next use.
+
+The backup data is stored at the end of the disk,
+in block-reverse order.
+The original order is something like this:
+
+block 0 
+        sector 0
+block 1 
+        sector 1
+block 2 
+        sectors 2,3,4
+block 3 
+        sectors 5,6,7,8,9,10,11,12,13,14,15,16,17,18,19
+block 4-102
+        1 sector of project info
+block 103-
+        song info block (2 sectors)
+        song allocation info block (5 sectors)
+
+The backup blocks are stored in reverse order,
+however each backup block maintains the original 
+sector ordering. That is, 
+
+block 0 has a backup at sector (end-0)
+        (where end=the last sector on disk)
+
+block 1 has a backup at sector (end-1)
+
+block 2, sectors 2,3,4 are backed up at sectors
+        end-4, end-3, end-2, respectively;
+
+block 3, sectors 5,6,7,8,9,10,11,12,13,14,15,16,17,18,19
+        are backed up at sectors
+        end-19,end-18,end-17,end-16,end-15,...end-5,
+        respectively.
+
+The validity of a block is indicated by a checksum which
+is stored in the last 4 bytes of a block (bytes 0x1FC,
+0x1FD, 0x1FE, 0x1FF of the last sector of a block).
+
+The value of the checksum is such that sum of all 32-bit 
+words in the block (which may span several sectors) is 0. 
+When writing a block, we must make sure that the resulting 
+checksum of the block adds up to 0 again.
+
+A second check refers to bytes 0x1f8-0x1fb of the last
+sector of a block. Bytes 0x1F8 and 0x1F9 contain the
+least significant 16 bit of the sector number at which
+the block starts. Thus, a project block which contains
+1 sector and starts on sector 0x14 will contain the 
+values 0x14, 0x00 on bytes 0x1f8 and 0x1f9; a song entry
+2 sectors long, and ending on sector 0x78 will contain
+the byte values 0x77, 0x00 on offset 0x1f8 and 0x1f9,
+respectively (because the song entry starts on sector
+0x77). Bytes 0x1FA and 0x1FB of the last sector of the
+block contain the bit-inverted value of 0x1F8 and 0x1F9**.
+
+** This paragraph assumes the sector is not byte-swapped
+as described in the first paragraph of the document.
+ 
+When a block is not equal to its backup, the checksum
+of both is validated. If one is valid and the other is
+not, the valid copy is written over the invalid copy.
+
+General ADAT FST layout
+-----------------------
+The general ADAT FST layout is as follows:
+
+-----------------------------------------------------------
+Sector 0 / Offset 0000:0000h
+Superblock, identifies drive as an ADAT FST drive.
+Size=512 bytes/0200h
+-----------------------------------------------------------
+Sector 1 / Offset 0000:0200h
+Drive info
+Size=512 bytes/0200h
+-----------------------------------------------------------
+Offset 0000:0400h
+3 sectors, Song Entry Usage Table
+Size=600h
+-----------------------------------------------------------
+Offset 0800h
+15 sectors, Drive Usage Table
+size=1e00h
+-----------------------------------------------------------
+         / Offset 0000:2800h
+Project info, 1 sector (512 bytes) per project entry
+Size: 99*200h=c600h
+-----------------------------------------------------------
+         / Offset 0000:ee00h
+Song info, 7 sectors (3584 bytes) per song entry 
+-----------------------------------------------------------
+        Undo info
+-----------------------------------------------------------
+        Audio data
+-----------------------------------------------------------
+        File system backup
+-----------------------------------------------------------
+
+
+Superblock layout
+-----------------
+Offset 0-0fh    Header line:
+                Offset 0        'ADAT FST'
+                Offset 8        Version number, e.g. '110 '
+                Offset 12       signature bytes: 55h, aah, cch, 33h
+Offset 10h-13h  Size of audio block     (0x480)
+Offset 14h-17h  Number of audio blocks per allocation cluster
+Offset 18h-1fh  0 ?
+Offset 20h-23h  0 ?
+Offset 24h-27h  0 ?
+Offset 28h-2bh  1 ?     sectors in superblock?
+Offset 2ch-2fh  1 ?     sectors in drive info block
+
+Offset 30h-33h  2 ?     sectornum of song entry usage table?
+Offset 34h-37h  3 ?     sectors in song entry usage table?
+
+Offset 38h-3bh  5 ?     sectornum containing allocation info? 
+Offset 3ch-3fh  15/0fh  - Number of sectors in allocation table?
+
+Offset 40h-43h  042fh=1071 ?? (anything to do with undo buffer?)
+                --
+                        On 1.2gb drive this formats as 0x20 00 00 00
+                --
+Offset 44h-47h  Number of free audio clusters on disk
+Offset 48h-4bh  SectorPointer to project info
+Offset 4ch-4fh  1
+Offset 50h-53h  Max allowed no. of projects?
+Offset 54h-57h  Max allowed no. of songs per project?
+Offset 58h-5bh  SectorPointer to first song info entry
+
+Offset 5ch-5fh  2       (sectors of song info entry w/o alloc info?)
+Offset 60h-63h  5       (sectors of alloc info per song entry?)
+Offset 64h-67h  7       (number of sectors that song info entries are apart?)
+Offset 68h-6bh          current number of songs on disk
+Offset 78h-7bh  041fh   (number of audio blocks reserved for undo buffer).
+                        1055 audio blocks=72 track minutes at 48 khz
+                Multiply by contents of offset 14h-17h for number of sectors
+                in undo buffer.
+
+Offset 7ch-7fh  Sector pointer to audio data block on disk (0x1397f6).
+Offset 80h-83h  32-bit word
+                number almost as big as number of sectors on disk
+
+                This number is related to that on offset 0084h and
+                always seems to have a difference of 0x14a46B to 
+                the end of the disk.
+
+                                0084h        0080h
+                40 gb disk:
+                                0x04c92d7f - 0x04B48914 = 0x14A46B
+                120gb disk:
+                                0x0df94baf - 0x0de4a744 = 0x14A46B
+
+                Perhaps the value is the number of allocatable sectors
+                on disk?
+                (1397f6 used in the beginning; an undo buffer is not
+                needed at the end of the drive)
+
+                0x10C75 used for actual filesystem.?
+                0x1397f6 (first data sector on drive) - (0x41f(size of undo buffer in audio blocks) * 0x480(size of audio block in sectors)) = 0x10C76
+                first data sector is not part of FS so this makes sense.
+                
+offset 84h-87h  32-bit word, highest sector number on disk. 
+                A backup of the first sector can be found here.
+offset 88h-1F7  (empty, zero - reserved for future use?)
+offset 1f8-1FF  Checksum information.
+                
+Drive info layout
+-----------------
+Sector 1
+Offset 0        8 byte drive name (for compatibility with
+                older ADAT FST versions), should be considered
+                obsolete. Replaced by name at offset 01bh.
+Offset 8-9      null bytes
+Offset 0ah-0bh  2 more bytes for drive name. Should be considered
+                obsolete
+Offset 10h      SectorPointer to last active project.
+offset 20h      List of 32-bit sector pointers to all projects.
+Offset 01b8h    64-byte drive name. Null-byte terminated
+                when shorter than 64 chars.
+
+Song Entry Usage Table
+----------------------
+FST 1.10 allows for 99 projects of 99 songs each, giving for
+9801 song entries. Each song entry is not necessarily pre-
+allocated to any given project; song entry 0 may belong to 
+project 0, while song entry 1 may belong to another project.
+The Song Entry Usage Table provides a quick way to look up
+the next free song entry without having to go through all 
+projects to see which song entries are in use.
+
+Each bit in the Song Entry Usage Table represents a song.
+When 0, the song entry is not in use; when 1, it is.
+
+Drive Usage Table
+-----------------
+Each bit in the Drive Usage Table represents a cluster. When 0, the
+cluster is free; when 1, the cluster is in use.
+32-bit words are filled one by one, least-significant bit first, 
+until the entire word has the value 0xFFFFFFFF. In other words, the
+table starting with the following words have the meaning as described:
+
+0x1 contains 1 bit on -> first cluster allocated
+0x3 contains 2 bits on -> first 2 clusters allocated
+0x1f contains 5 bits on -> first 5 clusters allocated
+0x1ff contains 9 bits on -> first 9 clusters allocated.
+
+At the end of the drive usage table, some space seems to be reserved,
+possibly representing the backup of the file system info, or something
+along those lines. The last 32-bit word of the Drive Usage Table is a
+checksum. When all 32-bit words in the entire Drive Usage Table are
+summed, the result must be 0.
+
+Project info
+----------------------
+The project info consists of (by default) 99 entries of 512 bytes each.
+The actual number of projects is contained by the superblock.
+
+100 project entries would be 0xc800 bytes.
+As projects are numbered starting at 01, a maximum of 99 projects
+is possible, giving for 0xc600 bytes.
+
+How to distinguish between an empty project and an unused project entry?
+Answer: Sector 1 (Drive info) contains the project list. 
+
+Projects not mentioned in the drive info project list are empty projects.
+
+Each project entry has the following layout:
+
+Offset 0        8-byte project name (obsolete)
+Offset 8        4-byte (2 bytes of which are partial project name too)
+offset 0ch-0fh  Number of songs in use in project
+offset 10h-19bh Song info table (a list of 32-bit pointers to the
+                initial sectors of song info). Length of table should be
+                396 bytes for 99 songs per project.
+                First entry in the table is the last accessed song.
+                Zero char means unused entry.
+Offset 01b8h    64-byte project name. Null-byte terminated if project
+                name is less than 64 bytes long.
+
+Song info table
+---------------
+The song info table is part of a project entry. It is a list of 32-bit
+pointers to sectors containing song info.
+
+Song info
+------------------------
+Song info starts at disk offset 0xee00h, or at
+sector 119 (the 120th sector on disk) (depending on
+info in superblock).
+
+Each song entry has a length of 2 sectors and is immediately
+followed by 5 sectors of file allocation info.
+
+Locate points/loops and track slip info is included in the song
+entries.
+
+Offset 00h-03h          32-bit number indicating no. of audio blocks in song
+Offset 04h-07h          ?0
+Offset 08h-0bh          32-bit number related to audio track interlacing.
+                        
+                        1/512th of the number of samples per interlace block
+                        before the audio switches to the next track.
+
+                        Value of this field equals Cluster size/(tracks*3).
+
+Offset 0ch-0fh          32-bit number related to audio track interlacing.
+
+                        number of sectors per interlace block
+                        before the audio switches to the next track.
+                        
+                        Value of this field equals audio block size
+                        /no. of tracks.
+                        Default audio block size is defined in superblock
+                        (0x480 sectors)
+
+                        An audio block size of 0x480 and 2 tracks gives a
+                        block size of 0x240h sectors or 0x48000 bytes
+                        (=294912 bytes).
+
+Offset 10h-13h          Status flags????????????????????
+                        Offset 0x12h bit 0x10h -> write protect?????
+
+Offset 28h-2Fh          First 8 bytes of song name.
+                        (Maintained for compatibility
+                        with FST before 1.10- 
+                        Obsoleted by long filenames
+                        at offset 3b8h)
+
+Offset 30h              ?
+
+Offset 31h              Number of physical tracks used by song.
+
+                        Note: 
+                        The HD24 has a high sample rate
+                        mode (88200 and 96000 samples per
+                        second, respectively). 
+
+                        In high sample rate mode, two
+                        physical tracks are used for each
+                        logical track.
+
+                        That is, a 12-track song in high
+                        sample rate mode will use 24 tracks
+                        on the machine. The number 24 is
+                        then what is indicated.
+
+                        This allows interlacing samples between
+                        tracks for compatibility with systems
+                        running at half the sample rate.
+                        In high sample rate mode, the track layout
+                        is:
+                        physical track 0-1 -> samples of logical track 1
+                        physical track 2-3 -> samples of logical track 2
+                        physical track 4-5 -> samples of logical track 3
+                        .. and so on. Even numbered samples (0,....) 
+                        are on even numbered tracks. 
+
+Offset 32h-33h          Two more filename characters for short filename,
+                        used in obsolete FST 1.0 standard.
+
+Offset 34h-36h          Sample rate in samples/second
+
+                        Typical values are:
+
+                        00h ach 44h -> 44100 Samples/sec
+                        00h bbh 80b -> 48000 Samples/sec
+                        01h 58h 88h -> 88200 Samoles/sec
+                        01h 77h 00h -> 96000 Samples/sec
+
+Offset 37h              Bit depth (18h or 24 by default)
+
+offset 38h-3bh          Unsigned 32-bit number containing 
+                        number of samples in song.
+                        
+                        For high sample rates, 2 tracks
+                        are used for each sample;
+                        therefore, a song that according
+                        to this 32-bit number contains 44100
+                        samples has the same duration at both
+                        44100 and 88200 kHz.
+
+offset 3ch              Write protect byte.
+
+                        This byte contains 8 bits (0-7, where
+                        7 is the most significant bit and 0 the
+                        least significant bit. In other words
+                        bit 7 has value 128, bit 0 has value 1)
+                        
+                        Bit 2 is on when song is write protected
+                        (value of the byte is 0x4 then), off when
+                        song is write enabled.
+
+offset 58h-B7h          Track slip info for track 1-24
+
+                        Track slip is measured in number of samples.
+                        E.g. for a song recorded at 48 khz, a value
+                        of 480 decimal (0x1e0) means 1/100 second
+                        (10 ms) of track slip.
+                        Track slip is stored as a signed 32-bit number.
+                        Valid values for the HD24 are -170..170 msec
+
+Offset 0b8h-1e3h        Locate point block
+
+Offset 3b8h-3e7h        Full song name, max 64 bytes; 
+                        terminated by null byte when less
+                        than 64 bytes
+
+Offset 400h-????        File allocation info (see below)
+
+Locate point block
+------------------
+The locate point block consists of 24 locate point entries.s
+
+Locate point entry
+------------------
+Each locate point entry has the following layout:
+Offset 0-3      32-bit time code
+                The time code represents the sample number
+                within the song.
+Offset 4-11     The 8-byte locate point name
+
+Locate point 0 indicates the relative time offset of the
+song. This is usually 0, but may be set to other values for
+tape transfers.
+
+The locate point is a 32-bit unsigned integer which contains the
+sample number within the song. In other words, in a 44.1 khz 
+song, the locate point 0:0:1:0 (hours/minutes/seconds/frames)
+is encoded as the number 44100. Thus, the locate point furthest
+in the song is 2^32=4294967296 samples. 
+This is enough for 44.1khz songs with a length of 27 hours.
+
+File allocation info
+====================
+The file allocation info consists of file allocation entries.
+Each file allocation entry has the following format:
+
+Offset 00h-03h  Starting sector number for audio block
+Offset 04h-07h  Number of audio blocks in allocation block
+
+In a valid song, the the 32-bit word on offset 0 of the song
+info equals the total sum of the number of audio blocks in the 
+file allocation info. 
+
+In my disk image, I see songs that had been previously deleted; 
+the song length in audio blocks is zero for these. This also
+goes for songs that have no audio data yet.
+
+Audio data layout
+=================
+The audio data of a FST file system is stored in a raw, uncompressed
+format. Audio data that consists of multiple channels is interlaced
+blockwise (as opposed to sample-by-sample interlacing).
+
+The data is stored as 24-bit signed little endian integer data.
+
+The typical audio block size as indicated in the superblock is
+0x480 sectors. In stereo audio, each subblock is then 1/2 audio blocks 
+or 0x240 sectors per channel.
+
+As the number of tracks increases, the block size per track decreases
+proportionally. 
+
+The default audio block size of 0x480 sectors lets itself be divided
+into 2,4,6,8,12,16,24 tracks (with a sample size of 24 bit). Each channel
+then takes up 0x240, 0x120, 0xc0, 0x90, 0x60, 0x48, 0x30 sectors, 
+respectively assuming 0x480 blocks per sector. In high sample rate
+modes, channels are paired to accomodate for the doubled sample rate. 
+This is why in high sample rate mode a max of 12 channels are possible,
+while the maximum song length is still over 27 hours on 88.2 kHz.
+
+
+Not yet accounted for in this document
+======================================
+
+How the undo buffer is used (however, this is not relevant because the
+undo buffer is cleared after each power off. It might as well exist
+in RAM only).
+
+How pitch info is stored in the songs
+auto play
+auto rtn
+auto rec
+
+HD24tools extensions to FST
+===========================
+An extension is proposed where a song on a HD24 drive is used as data
+area, to store non-audio information associated with a song or project.
+This would allow HD24 owners to store track notes, mix settings,
+protools projects and so on along with their songs.
+
+A degree of compatibility with the HD24 is desired; although the HD24
+will not be able to use the information, it should also not corrupt it.
+
+The HD24 does not offer unique file names; moreover, the position of a
+file is not constant within the FS. As a result, songs have no unique
+identifier that allows other data to be associated with them.
+
+It is proposed to use bytes 0x3a0-0x3bf of each song entry as unique
+identifier. Testing shows that the HD24 recorder does not touch these
+bytes, even after a scandisk operation. Moreover, when a song is copied
+to another drive, these bytes are copied over as well.
+
+A single HD24 song would provide ample storage space for most purposes.
+Additionally, by using a single song only, the negative impact to the
+end user is minimal as no users have ever been registered to actually
+create 99 project of 99 songs. To prevent the HD24 from corrupting the
+contents of the song, it can be created as 0-track song; previous
+experiments have shown that the track count is flexible. Having a song
+marked as 0 track prevents the HD24 from arming tracks or playing back
+audio from the song, and will give a user some feedback about the nature
+of the song, without causing the HD24 to claim that the song is corrupt.
+A track count of 0 survives a scandisk command, and it seems that the
+HD24 permits copying such songs to other drives.
+
+A possible way to store data in the song is to have the data song 
+contain a FAT file system. The advantage of the FAT file system is
+that it is well documented, and probably there will be library code
+available that will ease the implementation of an embedded FAT 
+system on HD24 drives.
+
+The root directory of the file system shall contain a 'songdata' 
+dir. This data directory shall contain a subdirectory named after the
+ID of each song. Of course the file names need to be long enough; if 
+this is not possible, a map file may be created to contain mappings 
+from song ID to songname subdirectory ID.
+
+Data placed in the appropriate songname directory is automatically 
+associated with the song. Should we so desire, we can also create a 
+'projdata' directory.
+
+By keeping the songdata directory structure flat, songs can be moved 
+across projects without losing their associated files.
+