====================
Larry Fish's Compass
====================

Survex can read Compass survey data - it supports survey data files
and project files (``.DAT`` and ``.MAK files``), closed data files (``.CLP``),
and processed survey data (``.PLT`` and ``.PLF`` files).  Survex 1.0.19 was
the first to support this but we currently recommend using 1.4.6 or newer if
you're working with Compass data as this version made significant improvements
to this support.  If you're using Compass ``.MAK`` files, then Survex 1.4.18
further improved parsing these.

--------------------
Compass .MAK support
--------------------

A Compass ``.MAK`` file defines a survey project to process, and
specifies one or more ``.DAT`` files to process, along with coordinate
systems and fixed points.  You can process a ``.MAK`` file with cavern
or aven as if it were a ``.svx`` file.

Survex understands most MAK file features.  Known current
limitations and assumptions:

- Survex handles any UTM zone and datum specified in the MAK file so
  long as the combination can be expressed as an EPSG code.  If there
  aren't any EPSG codes for a datum that suggests it's obsolete; lack of
  a code for a particular datum+zone combination suggests the zone is
  outside of the defined area of use of the datum.

  Some additional cases are also supported.  Example Compass files we've
  seen use "North American 1927" outside of its defined area of use,
  presumably because some users fail to change the datum from Compass'
  default.  To enable reading such files, if the datum is "North
  American 1927" or "North American 1983" and there isn't an EPSG code
  for the specified UTM zone in that datum, we generate a PROJ4 string
  of the form "+proj=utm ...".  Please let us know if support for
  additional cases which aren't currently supported would be useful to
  you.

- The ``@`` command which specifies a base location to calculate
  magnetic declinations at is handled, provided the datum and UTM
  zone are supported (see previous bullet point). The UTM
  convergence angle specified as part of this command is ignored
  as Survex knows how to calculate grid convergence.

- The ``%`` and ``*`` commands specify the UTM convergence angle
  (file-level and non-file-level respectively) and are ignored
  as Survex knows how to calculate grid convergence.

- Link stations are ignored. These have two uses in Compass. They
  were a way to allow processing large surveys on computers from
  last century which had limited memory. Survex can easily handle
  even the largest surveys on current hardware and ignoring them
  is not a problem in this case.

  The other use is they provide a way to process surveys together
  which use the same station names for different stations. In
  this case we recommend writing a ``.svx`` file to replace the MAK
  file which wraps the ``*include`` of each DAT file in
  ``*begin survey1``/``*end survey1``, etc so that stations
  have unique qualified names. Then the link stations can be
  implemented using e.g. ``*equate survey1.XX1 survey2.XX1``.
  This example from the Compass documentation:
  ::

     #FILE1.DAT;                /no links
     #FILE2.DAT,A22,A16;
     #FILE3.DAT,A16,B14;

  would look like this:
  ::

     *begin file1
     *include FILE1.DAT
     *end file1
     *begin file2
     *include FILE2.DAT
     *end file2
     *equate file1.A22 file2.A22
     *begin file3
     *include FILE3.DAT
     *end file3
     *equate file1.A16 file3.A16
     *equate file2.B14 file3.B14

  Note that the ``.svx`` version is able to more precisely represent
  what's actually required here - in the MAK version "you must
  carry ``A16`` into ``FILE2`` so you can then carry it into ``FILE3``
  even though ``FILE2`` doesn't need it for its own processing". If you want
  the exact analog of the MAK version you can change the ``A16`` equate to:
  ::

     *equate file1.A16 file2.A16 file3.A16

- The ``!`` command specifies project parameters, which are currently
  ignored.

- Compass seems to quietly ignore unknown commands, but Survex will issue a
  warning (since Survex 1.4.18 - older versions quietly ignored them).
  This warning will help identify any bugs in Survex's parsing of MAK files,
  and any new or undocumented MAK file commands which we don't support.
  It can also help detect typos in data entry.

- ``.MAK`` files can contain comments, which last from a ``/`` to the
  next ``/`` or end of the line.  Survex 1.4.18 added support for these
  (older Survex versions quietly ignored the ``/`` characters as unknown, but
  that meant they would still interpret any commented out ``.MAK`` file
  commands).

--------------------
Compass .DAT support
--------------------

A Compass ``.DAT`` file contains unprocessed survey data.  You can process a
``.DAT`` file with cavern or aven as if it were a ``.svx`` file.

You can even use ``*include compassfile.dat`` or ``*include
compassproject.mak`` in a ``.svx`` file and it'll work, which allows combining
separate cave survey projects maintained in Survex and Compass.

One point to note when doing so (this tripped us up!) is that
station names in Compass are case sensitive and so Survex reads
``.DAT`` files with the equivalent of ``*case preserve``. The default
in ``.svx`` files is ``*case lower`` so this won't work
::

   *fix CE1 0 0 0
   *include datfilewhichusesCE1.dat

because ``CE1`` in the ``*fix`` is actually interpreted as ``ce1``.  The
solution is to turn on preserving of case while you fix the point
like so:
::

   *begin
   *case preserve
   *fix CE1 0 0 0
   *end
   *include datfilewhichusesCE1.dat

If you want to be able to refer to the fixed point from Survex data too then
you can add in a ``*equate`` to a lower-case name to achieve that:
::

   *begin
   *case preserve
   *fix CE1 0 0 0
   *equate CE1 ce1
   *end
   *include datfilewhichusesCE1.dat
   *include svxfilewhichusesce1.svx

Or if you're just wanting to link a Compass survey to a Survex one, you can use
a ``*equate`` with ``*case preserve on``:
::

   *begin
   *case preserve
   *equate CE1 ce1
   *end
   *include datfilewhichusesCE1.dat
   *include svxfilewhichusesce1.svx

Survex understands most DAT file features.  Known current limitations and
assumptions:

- The cave name, survey short name, survey comment and survey
  team information are currently ignored (because this
  information isn't currently saved in the ``.3d`` file even for ``.svx``
  files).
- Survey date January 1st 1901 is treated as "no date specified",
  since this is the date Compass stores in this situation, and it
  seems very unlikely to occur in real data.
- Passage dimensions are currently ignored.
- Shot flag ``C`` in Compass causes flagged legs to not be subject to
  loop closure.  Survex currently sets the SDs of such legs to 1mm,
  so flagged legs can still move slightly during loop closure.
  (Since 1.4.16; earlier versions ignored this flag entirely.)
- Shot flag ``L`` is mapped to Survex's "duplicate" leg flag.
- Shot flag ``P`` is mapped to Survex's "surface" leg flag. The
  Compass documentation describes shot flag ``P`` as "Exclude this
  shot from plotting", but the suggested use for it is for
  surface data, and shots flagged ``P`` "[do] not support passage
  modeling". Even if it's actually being used for a different
  purpose, Survex programs don't show surface legs by default so
  the end effect is at least to not plot, which is what's intended.
- Shot flag ``S`` is mapped to Survex's "splay" leg flag.
- Surveys which indicate a depth gauge was used for azimuth
  readings are marked as ``STYLE_DIVING`` in the ``.3d`` file.
- Compass seems to quietly ignore a shot with the same "from" and "to"
  station. This seems likely to be a mistake in the data so Survex 1.4.12
  and later warn about this in a Compass DAT file (in native Survex data
  this is treated as an error, which is how older Survex versions treat
  it in Compass DAT files).

--------------------
Compass .CLP support
--------------------

A Compass .CLP file contains raw survey data after adjusting for
loop closure. The actual format is otherwise identical to a
Compass ``.DAT`` file, and Survex 1.4.6 and later support processing a
.CLP file with cavern or aven as if it were a ``.svx`` file (the extra
support is to recognise the ``.CLP`` extension, and to not apply the
instrument corrections a second time).

You can even use ``*include compassfile.clp`` in a ``.svx`` file
and it'll work, which allows combining separate cave survey
projects maintained in Survex and Compass.

Usually it is preferable to process the survey data without loop
closure adjustments (i.e. ``.DAT``) so that when new data is added
errors get distributed appropriately across old and new data, but
it might be useful to use the ``.CLP`` file if you want to keep
existing stations at the same adjusted positions, for example to
be able to draw extensions on an existing drawn-up survey which
was processed with Compass.

Another possible reason to use the data from a ``.CLP`` file is if
that's all you have because the original ``.DAT`` file has been lost!

-------------------------
Compass .PLF/.PLT support
-------------------------

A Compass ``.PLT`` file contains processed survey data.  The extension
``.PLF`` is also used for "special feature files" which have
essentially the same format.

Survex supports both reading and writing these files, each of which
are documented in separate sections below.

Reading Compass .PLF/.PLT
=========================

You can load these files with ``aven`` as if they were .3d files, and
similarly for other Survex tools which expect a .3d file such as
``survexport``, ``extend``, ``diffpos``, ``3dtopos`` and ``dump3d``.
(This support is actually provided by Survex's img library, so other
programs which use this library should also be able to read Compass
``.PLT`` files without much extra work.)

Survex understands most PLT file features.  Known current
limitations and assumptions:

- Survey date January 1st 1901 is treated as "no date specified",
  since this is the date Compass stores in this situation, and it
  seems very unlikely to occur in real data.

- Passage dimensions are translated to passage tubes, but Survex
  may interpret them differently from Compass.

- Shot flag ``C`` is ignored.  It only seems to be useful in unprocessed survey
  data.

- Shot flag ``L`` is mapped to Survex's "duplicate" leg flag.

- Shot flag ``P`` and plot command ``d`` are mapped to Survex's "surface"
  leg flag. The Compass documentation describes shot flag ``P`` as
  "Exclude this shot from plotting", but the suggested use for it
  is for surface data, and shots flagged ``P`` "[do] not support
  passage modeling". Even if it's actually being used for a
  different purpose, Survex programs don't show surface legs by
  default so the end effect is at least to not plot, which is what's
  intended.  Stations are flagged as surface and/or underground based
  on whether they are at the ends of legs flagged surface or
  non-surface (a station at the boundary can be flagged as both).

- Shot flag ``S`` is mapped to Survex's "splay" leg flag. A station
  at the far end of a shot flagged ``S`` gets the "station on wall"
  flag set since the Compass PLT format specification says: "The
  shot is a "splay" shot, which is a shot from a station to the
  wall to define the passage shape."

- Stations with "distance from entrance" of zero are flagged as
  entrances.

- Stations which are present in multiple surveys are flagged as
  exported (like when ``*infer exports on`` is in effect in ``.svx``
  files).

- Stations listed as fixed points are flagged as fixed points.

- If a PLT file only uses one datum and UTM zone combination and
  it is supported (the same combinations are supported as for MAK
  files) then they are converted to the appropriate EPSG code or
  PROJ4 string and this is reported as the coordinate system.
  Please let us know if support for additional cases which aren't
  currently supported would be useful to you. Files with multiple
  datums could be handled too, but we'd need to convert
  coordinates to a common coordinate system in the img library,
  which would need it to depend on PROJ. Please let us know if
  support for mixed datums would be useful to you.

Exporting Compass .PLT
======================

Survex can also create PLT files via ``aven``'s File->Export feature, and also
from the command line via ``survexport --plt``.

This export was originally added to allow importing data from Survex into
Carto.  The principal author of Carto has sadly died and it seems Carto is no
longer actively developed, but we've left this support in place in case it is
useful - the generated files can be used with Compass itself for example.
Here are some notes on this support:

- The whole Survex survey tree is currently exported as a single survey in
  the PLT file.

- Compass station names can't contain spaces, so any spaces (and also ASCII
  control characters) in Survex station names are replaced.

  Since Survex 1.4.19, the replacements are ``^`` followed by another
  character.  Byte value 0 becomes ``^@``, byte values 1 to 26 become ``^A`` to
  ``^Z``, byte values 27 to 31 become ``^a`` to ``^e``, space becomes ``^_``
  and ``^`` becomes ``^^`` (to ensure the mapping is reversible).  In real
  world use, only space and ``^`` are at all likely to occur.

  From Survex 1.2.1 to 1.4.18, a different system of replacements was used,
  based on that used in URLs: each replacement was ``%`` follow by two lowercase
  hex digits, e.g. space became ``%20``.  Additionally, ``%`` became ``%25``.
  The drawback of this encoding was that it needed 3 characters to represent
  each escaped one, which wass unhelpful as PLT format documents a 12 character
  limit on station names.

  Survex 1.2.0 and earlier didn't do any escaping.

- Survex supports anonymous survey stations.  However we can't write these
  as empty station names in Compass PLT (they aren't explicitly disallowed in
  the format documentation, but Compass' viewer gives an error if you ask it to
  read a PLT file with an empty station name).

  Since Survex 1.4.19, each anonymous station gets given a unique name of
  ``^`` followed by a number starting from one and incrementing for each
  anonymous station.  (Note that these invented names can't collide with actual
  station names because ``^`` is escaped to ``^^`` as documented above.)

  From Survex 1.4.10 to 1.4.18, the unique names were instead ``%:`` followed
  by the counter.

  Survex 1.4.6 first implemented support for exporting anonymous stations to
  PLT, but with names which typically exceeded the documented 12 character
  limit of the format.

  Survex 1.4.5 and earlier generated PLT files with empty station names, but
  as noted above, at least Compass' viewer wouldn't read these.

- The full Survex station name including survey prefixes is used - no attempt
  is currently made to shorten station names to fit within the 12 character
  limit documented for the Compass PLT format.  If you export a single survey
  the names should be short enough, but exporting the whole of a complex survey
  project will likely give names longer than 12 characters.

- Passage data is not included in the export (each exported leg has dummy LRUD
  readings of all ``-9`` which is needed to avoid a bug in some versions of
  Compass which don't cope with legs without LRUD).

- Survex's "surface" leg flag is mapped to Compass shot flag ``P``.
  The Compass documentation describes shot flag ``P`` as "Exclude this shot
  from plotting", but the suggested use for it is for surface data, and shots
  flagged ``P`` "[do] not support passage modeling".  Since Survex 1.4.10.

- Survex's "splay" leg flag is mapped to Compass shot flag ``S``.  Since Survex
  1.4.10.

- Survex's "duplicate" leg flag is mapped to Compass shot flag ``L``.  Since
  Survex 1.4.10.

- The Datum and UTM zone information is set if the Survex data has a coordinate
  system set which is expressed as an ``EPSG:nnnn`` code which corresponds to
  a UTM zone in a datum supported by Compass, or a PROJ4 string which matches
  those which Survex generates when processing Compass and Walls data which
  specifies a NAD27 or NAD83 UTM zone which doesn't have an EPSG code.  Since
  Survex 1.4.18.
