.. highlight:: rst

.. _scart:

#####
scart
#####

**Import/export MiniSEED data to/from SDS archives.**


Description
===========

The archive tool scart reads and writes :term:`SDS` archives and files
in miniSEED format and checks miniSEED archives or prints stream information.

* **Dump mode:** Create miniSEED files (multiplexed), e.g. for playbacks, from
  :term:`SDS` structured data (e.g. created by slarchive).
* **Dump mode:** Play back records directly out of an SDS structure.
* **Import mode:** Import multiplexed miniSEED files into a local SDS waveform
  archive.
* **Import mode:** Import data using the :ref:`global_recordstream` interface
  into a local SDS waveform archive.
* **Import mode:** Read data from any :ref:`global_recordstream` interface
  and dump it to file.
* **Check mode:** Check an archive of miniSEED files for out-of-order records in
  files.

.. warning::

   * When creating :term:`SDS` archives, scart simply appends the new records to
     existing ones. Multiple imports of the same data result in duplication.
   * Out-of-order imports of waveforms into a SDS archive result in out-of-order
     records which may not be processed. Clean your archive using :ref:`scmssort`.
   * **Before importing miniSEED data** into an SDS archive they must be sorted
     by time and duplicate records must be removed. Otherwise, the SDS archive
     may not be correctly readable by other modules. Therefore, combine scart
     with :ref:`scmssort` for multiplexing and removal of duplicates.

.. hint::

   In dump and import mode output streams may be filtered by

   * Time windows (:option:`-t`),
   * Network-station-location-channel (NSLC) lists (:option:`--nslc`) created,
     e.g., with :ref:`scinv` from inventories.

   In dump mode output streams may also be filtered by

   * Time window - stream lists (:option:`--list`, dump mode) generated by
     :ref:`scevtstreams` for particular events.

.. _scart-config:

Configuration
=============

scart can make use of :ref:`global_recordstream` implementations which are
provided by additional plugins. For loading additional plugins, e.g. the *xyz*
plugin create and configure :file:`scart.cfg`:

.. code-block:: sh

   plugins = xyz


Examples
========

.. hint::

   The usage of wildcards in place of network, station, location or channel code
   is allowed in many options (-n, -c, -l, --list, --nslc) and follows these rules:

   * Import mode: the wildcards are passed to the :ref:`global_recordstream` interface,
     that interprets them. Normally both "*" and "?" are supported by RecordStreams.
   * Dump mode: the wildcards are interpreted by scart command that supports "*" for
     network, station, location codes and "*", "?", "(", ")", "|" for channel code.

#. Extract data from the default :term:`SDS` archive in :file:`$SEISCOMP_ROOT/var/lib/archive`
   or from a local :term:`SDS` archive [SDS archive] into a miniSEED file :file:`file.mseed`
   and sort by end time of the records:

   .. code-block:: sh

      scart -dsvE -t '[start-time]~[end-time]' [SDS archive] > [file.mseed]
      scart -dsvE -t '[start-time]~[end-time]' > file.mseed
      scart -dsvE -t '[start-time]~[end-time]' -n '[NET1],[NET2]' > file.mseed
      scart -dsvE -t '[start-time]~[end-time]' -n '[NET]' -c '(E,H)H(1,2,3)' > file.mseed
      scart -dsvE -t '[start-time]~[end-time]' -n '[N1.S1.L1.C1],[N2.S2.L2.C2]' > file.mseed
      scart -dsvE -t '[start-time]~[end-time]' --nslc list.file > file.mseed
      scart -dsvE -t --list list.file > file.mseed


   It is possible to achieve the same result of the dump mode using a
   combination of the input mode and the :ref:`scmssort` command, which allows
   to read the input data from any supported :ref:`global_recordstream`,
   not only an SDS archive:

   .. code-block:: sh

      scart -I [record-stream] --list list.file --stdout | scmssort -u -E -v > file.mseed

   .. note::

      Sorting data is computational expensive but required for waveform playbacks.

#. Push miniSEED data from file :file:`file.mseed` or standard input
   (stdin) into a local :term:`SDS` archive or a file. Additionally, you may
   check if the records of archived files are correctly ordered, filter by time
   and/or with NSLC list and print the output streams:

   .. code-block:: sh

      scmssort -u -E [file.mseed] > sorted.mseed
      scart -I sorted.mseed --with-filecheck [SDS archive]

      scart -I [file.mseed] -t '[start-time]~[end-time]' --print-streams --nslc list.file -o [out.mseed]

      cat sorted.mseed | scart -I - [SDS archive]
      cat sorted.mseed | scart [SDS archive]

#. Collect data using the :ref:`global_recordstream` interface (e.g. FDSNWS server)
   and write to a miniSEED file or import it into a local :term:`SDS` archive. The
   data streams and the time spans can be defined in several ways. The data streams
   (:option:`--list`) can be automatically generated, e.g., by :ref:`scevtstreams`.

   .. code-block:: sh

      scart -I fdsnws://[server]:80 --list list.file [SDS archive]
      scart -I fdsnws://[server]:80 --list list.file -o file.mseed
      scart -I fdsnws://[server]:80 -t '[start-time]~[end-time]' --nslc list.file [SDS archive]
      scart -I fdsnws://[server]:80 -t '[start-time]~[end-time]' -n '[NET1],[NET2]' [SDS archive]
      scart -I fdsnws://[server]:80 -t '[start-time]~[end-time]' -n '[NET]' -c 'EH?' [SDS archive]
      scart -I fdsnws://[server]:80 -t '[start-time]~[end-time]' -n '[N1.S1.L1.C1],[N2.S2.L2.C2]' [SDS archive]

#. Check all files of an SDS archive or other directory structure for
   miniSEED files with out-of-order records:

   .. code-block:: sh

      scart --check [archive]

#. Print stream information from miniSEED files in archives or from
   :term:`RecordStream` without actually writing miniSEED data. In dump and
   import mode use the :option:`--test` if miniSEED data shall be read but not
   written.

   .. code-block:: sh

      scart --print-streams -I [miniSEED file] --test
      scart --print-streams -d -t [time span] --test [SDS archive]
      scart --print-streams --check [archive]

   The output looks like this:

   .. code-block:: sh

      # streamID       start                       end                         records samples samplingRate
      GE.RGN..BHZ      2022-12-08T15:34:41.895Z    2022-12-08T15:52:19.145Z    58 21145 20.0

   where the header and the text body are printed to stderr.



Command-Line Options
====================

.. program:: scart

:program:`scart [options] {archive-dir}`

The last option has to be the archive directory when dump mode is enabled.
When no archive directory is explicitly given,
\$SEISCOMP_ROOT\/var\/lib\/archive or the current directory
is used depending on whether \$SEISCOMP_ROOT has been set or not.
The default operation mode is import. That means that a multiplexed
MiniSEED file or another record source such as ArcLink is used to import
records into a SDS structure.


Verbosity
---------

.. option:: -v, --verbose

   Verbose mode.

.. option:: -h, --help

   Display a help message.


Mode
----

.. option:: --check

   Check mode. Check all files
   in the given directory for erroneous miniSEED records.
   All sub\-directories are included. If no directory is given,
   the default SDS archive is scanned. Checks are only complete
   for files containing exactly one stream. More complete checks
   are made with scmssort.

.. option:: -d, --dump

   Set export \(dump\) mode. Records are retrieved from an archive and
   written to standard output.

.. option:: -I

   Import mode \(default\): Specify the recordstream URL to read
   the data from for archiving. When using any other
   recordstream than file, a stream list file is needed.
   Specifying \- implies file:\/\/\- \(stdin\). If no mode is
   explicitly specified, \-I file:\/\/\- is assumed.


Processing
----------

.. option:: -c channels

   Channel filter to be applied to the data streams.
   Default for Dump: \"\(B\|E\|H\|M\|S\)\(D\|H\|L\|N\)\(E\|F\|N\|Z\|1\|2\|3\)\"
   Default for Import: \"\*\"

.. option:: -E

   Dump mode: sort records according to their end time.
   Default: start time.

.. option:: --files count

   Dump mode: Specify the number of file handles to cache.
   Default: 100.

.. option:: -i

   Ignore records without data samples.

.. option:: -l, --list file

   Import, dump mode: Use a stream list file with time windows instead
   of defined networks and channels \(\-n, \-c and \-t are ignored\).
   The list can be generated from events by scevtstreams. One
   line per stream. Line format: starttime;endtime;streamID
   The time format is the same as described in option '\-t'.
   
   Example:
   
   2019\-07\-17 02:00:00;2019\-07\-17 02:10:00;GR.CLL..BH?

.. option:: -m, --modify

   Dump mode: Modify the record time for real time playback.
   The first record time is NOW. The relative time of
   successive records to the first one are kept.

.. option:: -n networks

   Import, dump mode: Data stream selection as a comma separated list
   \"stream1,stream2,streamX\" where each stream can be NET or NET.STA
   or NET.STA.LOC or NET.STA.LOC.CHA.
   If CHA is omitted, it defaults to the value of \-c option.
   Default: \"\*\"

.. option:: --nslc file

   Import, dump mode: Stream list file to be used instead of
   defined networks and channels \(\-n and \-c are ignored\)
   for filtering the data by the given streams. Dump mode:
   Use in combination with \-t\! One line per stream, line
   format: NET.STA.LOC.CHA
   
   Example:
   
   GR.CLL..BH?

.. option:: --rename rule

   Import, dump mode: Rename stream data according to the provided
   rule\(s\). A rule is \"[match\-stream:]rename\-stream\" and match\-stream
   is optional. match\-stream and rename\-stream are in the
   \"NET.STA.LOC.CHA\" format. match\-stream supports special
   charactes \"?\" \"\*\" \"\|\" \"\(\" \"\)\". rename\-stream supports the
   special character \"\-\" that can be used in place of NET, STA,
   LOC, CHA codes with the meaning of not renaming those.
   \"\-\" can also be used as the last character in CHA code.
   Multiple rules can be provided as a comma separated list
   or by providing multiple \-\-rename options.

.. option:: -s, --sort

   Dump mode: Sort records by [start\-]time. To sort records by their
   endtime use \-E.

.. option:: --speed

   Dump mode: Specify the speed to dump the records. A value of 0 means
   no delay otherwise speed is a multiplier of the real time difference
   between the records. When feeding the records directly into the replay
   pipe a value of 1 \(real time\) is recommended.

.. option:: -t timeWindow

   Import, dump mode: Specify the time window \(as one properly
   quoted string\) to dump records for. Times are UTC and
   separated by a tilde \"\~\". To dump one hour of
   waveform data between 2008\/01\/01 00:00:00 and 2008\/01\/01
   01:00:00 use
   <\-t \"2008\-01\-01 00:00:00\~2008\-01\-01 01:00:00\">.


Output
------

.. option:: -o, --output

   Dump, Import mode: Write data to given file instead of creating
   a SDS archive. Deactivates \-\-stdout. Deactivated by \-\-test.

.. option:: --print-streams

   Print stream information only and exit. Works in import, dump
   and check mode. Output: NET.STA.LOC.CHA StartTime EndTime.

.. option:: --stdout

   Import mode: Write to stdout instead of creating a SDS archive.
   Deactivated by \-\-test and \-\-output.

.. option:: --test

   Test input only, deactivate all miniSEED output. This switch is
   useful for debugging and printing stream information with
   \-\-print\-streams.

.. option:: --with-filecheck

   Import mode: Check all accessed files. Unsorted or unreadable
   files are reported to stderr. Checks are only complete
   for files containing exactly one stream. More complete
   checks are made with scmssort.

.. option:: --with-filename

   Import mode: Print all accessed files to stderr after import.