.. highlight:: rst .. _caps2caps: ######### caps2caps ######### **caps2caps synchronizes CAPS servers in real-time** Description =========== :program:`caps2caps` can connect two |appname| server instances to synchronize waveforms and other data in real time. When one server 1 fails and the other one, server 2, continues to operate, the server 1 can back fill the data as soon as it becomes alive again. :program:`caps2caps` can also transfer the meta data of the input CAPS server, connected plugins and hardware information to the receiving server allowing them to be jointly displayed and analysed in CAPS server's web interface for all transferring :program:`caps2caps` instances. .. _fig-caps2caps: .. figure:: media/caps2caps.png :width: 18cm :align: center caps2caps instances connecting two |appname| servers transferring data from the remote into the local server. Here, caps2caps runs on the receiving server which can be exchanged with the input server. .. _sec-caps2caps-operation: Operation ========= *caps2caps* can run on either server to pull the data from the input server pushing them to the output server: .. _sec-caps2caps-sides: * **Run on the receiving output server** transferring data from a remote input CAPS server to the receiving CAPS server running locally. Advantages: * Receiving server controls the data request. * Configuration is easy to adjust, module is easy to maintain. * :program:`caps2caps` consumes only little hardware resources on the input but mostly on the output server which may be easier to access and control. Disadvantages: * For each input source a separate instance of :program:`caps2caps` must be set up and configured. * Each input source must be accessible from the receiving server which potentially requires the configuration of static IP addresses, VPN, firewall and access parameters of the input CAPS server. * Plugin and hardware meta data of the input server are not transferred and cannot be jointly visualised on the output server's CAPS web interface. * **Run on the input server** transferring data from the locally running input CAPS server to the receiving CAPS server running remotely. Advantages: * No static host address is required for the input server. * Access and firewall rules need to be configured only on the receiving server. * Plugin meta data such as channel or traffic information as well as hardware and SOH details such as CPU usage of the input server are transferred. They can be jointly visualised on the output server's CAPS web interface for all input servers. Disadvantages: * On each input server one instance of :program:`caps2caps` must be configured, maintained and operated which may require external access to this server. * :program:`caps2caps` consumes resources on the input server which may be limited. The time window of the requested data and their transfer to the output server depends on the configuration of time-window parameters such as :confval:`begin`, :confval:`end`, :confval:`maxDays`, :confval:`days` and :confval:`daysBefore`. These parameters are considered initially per stream unless a :ref:`journal exist for that stream `. Further limitations to particular streams is configurable by :confval:`streams` and :confval:`outOfOrder`. .. _sec-caps2caps-journaling: Journaling ========== In a journal file caps2caps keeps track of packages received and acknowledged by the receiving CAPS server. The journal file contains the per-stream information on the end time of the latest data package received by the output server. The journal is considered when establishing new connections to the input CAPS server in order to not request data again after case of a restart or in case of a loss of connection to a server. For each stream the begin time of the request time window is the journal time. Only if a journal is unavailable, the configured begin time (:confval:`begin` or :confval:`days`) is considered for defining request time-windows. .. warning:: The actual journal file is defined by :confval:`journal.file`. When operating multiple instances of :program:`caps2caps` on the same |scname| system, assure to configure the files uniquely avoiding confusion between the instances. .. _sec-caps2caps-situations: Situations ========== Data transfer may be affected by different situations of which * :ref:`Regular operation` * :ref:`On-demand data injection` * :ref:`Backfilling` (last-in/first out) are outlined below. The situations may establish in isolation or simultaneously. .. _sec-caps2caps-regular: Regular operation ----------------- During regular operation :ref:`CAPS plugins ` make the data automatically and in real time available on the input server. :program:`caps2caps` synchronizes the data automatically and in real time with the output CAPS server :ref:`running on either side `. The data is transferred in timely order in a first-in/first-out (fifo) fashion. The request time window results from configuration or :ref:`per-stream journals `. .. _fig-caps2caps-diagram1: .. figure:: media/diagram_backfilling_step1.png :width: 18cm :align: center Regualar operation. Here, caps2caps runs on the input server which can be exchanged with the receiving server. .. _sec-caps2caps-on-demand-injection: On-demand data injection ------------------------ Data may be injected into a CAPS server on demand by a :ref:`CAPS plugin ` such as :ref:`caps2caps` or :ref:`rs2caps`. This may create new past data in a CAPS archive. When synchronizing this CAPS archive with another output CAPS server during :ref:`regular operation `, **it is not guaranteed that the data is also transferred to the output CAPS server**. That has to do with the caps2caps plugin handling streams (section :ref:`sec-caps2caps-journaling`). Setting the request time window from the journal can lead to unintended behavior when past data is injected on demand into the input CAPS server and the start time of the data is before the current begin of the requested time window. In this case no data is transferred between the CAPS server instances. For injecting data on demand and synchronization to two CAPS servers we therefore recommend to send the data to both CAPS servers separately ensuring that both servers store the data. For :ref:`rs2caps` this works similar to: .. code-block:: bash rs2caps -I file.mseed --passthrough -O server1:18003 rs2caps -I file.mseed --passthrough -O server2:18003 In the future, we might have a better solution for this use case. .. _sec-caps2caps-backfilling: Backfilling ----------- Backfilling describes the transfer of data that were received on the input server with significant delay or after longer disconnection between the input and output servers but not as in regular operation. .. _fig-caps2caps-diagram2: .. figure:: media/diagram_backfilling_step2.png :width: 18cm :align: center Injection of data with significant delay or after longer disconnection between the input and output servers In such a situation, backfilling prioritizes the latest data over transferring the data in timely order. Backfilling therefore transfers data similar to a last-in/first-out manner which is essential to rapid-response and early warning systems. .. _fig-caps2caps-diagram3: .. figure:: media/diagram_backfilling_step3.png :width: 18cm :align: center Data transfer is re-established prioritizing most recent data until backfilling is complete when regular operation continues. The backfilling strategy is configurable and affects real time requests, that are requests with an open end time, only. If communications are restored after a significant outage, it may take some time to transfer all missing packages due to the limited bandwidth available. By default data is transmitted in order of record start time. However, there are use cases where it is desirable to prioritize the latest packages to get access to real time data as soon as possible. For this reason, the plugin supports two backfilling modes configurable by :confval:`backfilling.mode`: * **Pull**: During handshake all requests with no end time (open) are checked. For each request where the difference between system time and start time is greater than the value of :confval:`backfilling.maxRealTimeGap` the request will be split into two. * **Push**: Start times of outgoing packets for requests are checked. For each request where the difference between system time and start time is greater than the value of :confval:`backfilling.maxRealTimeGap` the request will be split into two. The backfilling mode can be set as follows: .. code-block:: bash backfilling.mode = PUSH The time window calculation in **Push** and **Pull** mode is the same and the split time is calculated as follows: .. code-block:: bash splitTime = systemTime - marginRealTimeGap Then there are two requests: * A new backfilling request with the time window [startTime~splitTime] * A real time request with the time window [spitTime~] The marginRealTimeGap value can be set in the configuration with .. code-block:: bash backfilling.marginRealTimeGap = 30 To enable the backfilling feature the option backfilling.maxRealTimeGap must be set like .. code-block:: bash backfilling.maxRealTimeGap = 60 which sets the maximum real-time data gap in seconds. .. note:: The configured output buffer size has an effect on the data prioritization as all buffered packages are forwarded to the network stack before they are buffered. That means the TCP implementation manages the packages. Let's assume the following scenario: * 1 MB buffer size, * A few records/samples of real time data and more than 1 MB historical data, * Network bandwidth 8 kB/s. In this scenario the outgoing buffer fills up immediately with historical data and it take some time to transfer the data due to the low bandwidth. This initial delay remains until all historical data has been transferred. Keep the buffer size low to avoid larger delays. The downside of this approach is that the general throughput is lower as fewer packages are in-flight and the plugin has to wait for server acknowledges. .. _sec-caps2caps-examples: Examples ======== * Run caps2caps as daemon module. #. Configure input and output hosts (:confval:`input.address`, :confval:`output.address`) in caps2caps module configuration, :file:`caps2caps.cfg`. #. Enable and start caps2caps .. code-block:: bash seiscomp enable caps2caps seiscomp start caps2caps * Run caps2caps on demand in a terminal with specific, explicitly specifying input and output hosts without encryption .. code-block:: bash caps2caps -I caps://inputServer:18002 -O caps://outputServer:18003 The same as above but with encrypted data transfer controlled by user name and password .. code-block:: bash caps2caps -I capss://user:password@inputServer:18002 -O capss://user:password@inputServer:output:18003 * Pull or push data depending on module configuration but ignore the journal file. This allows resending the data but it may cause a very big amount of data to be transferred. Ignoring the journal file must therefore be considered with care! .. code-block:: bash caps2caps -j "" Module Configuration ==================== | :file:`etc/defaults/global.cfg` | :file:`etc/defaults/caps2caps.cfg` | :file:`etc/global.cfg` | :file:`etc/caps2caps.cfg` | :file:`~/.seiscomp/global.cfg` | :file:`~/.seiscomp/caps2caps.cfg` caps2caps inherits :ref:`global options`. .. note:: Modules/plugins may require a license file. The default path to license files is :file:`@DATADIR@/licenses/` which can be overridden by global configuration of the parameter :confval:`gempa.licensePath`. Example: :: gempa.licensePath = @CONFIGDIR@/licenses .. confval:: streams Type: *string* Comma separated list of streams. Stream format: NET.STA.LOC.CHA. Streams may contain wildcards .. confval:: begin Type: *string* Begin of data time window. All SeisComP time formats are supported. Example: YYYY\-MM\-DDThh:mm:ss.sss . If omitted, the current UTC time is used. This value sets the initial start time for all requests. When data is received before and a journal is available, the begin time for each stream is taken from the journal each time the module is started and the configured value is ignored for the respective data stream. .. confval:: end Type: *string* End of data time window. All SeisComP time formats are supported. Example: YYYY\-MM\-DDThh:mm:ss.sss . .. confval:: maxDays Default: ``-1`` Unit: *day* Type: *int* Maximum number of days to acquire regardless if a time window is configured or read from journal. Values <\= 0 disable the check. The reference time is either the stream’s end time or, if no end time is specified, the current time in UTC. .. confval:: days Default: ``-1`` Unit: *day* Type: *int* The begin of the data time window as number of days before current time. .. confval:: daysBefore Default: ``-1`` Unit: *day* Type: *int* The end of the data time window as number of days before current time. .. confval:: timeWindowUpdateInterval Default: ``-1`` Unit: *s* Type: *int* The time interval at which the relative request time window defined by option days and\/or daysBefore is updated. Values <\= 0 disable the update. This feature is supported in archive mode only. A typical use case is when data has to be transmitted continuously with a time delay. .. confval:: realtime Default: ``true`` Type: *boolean* Only fetch real\-time but no archived data. .. confval:: outOfOrder Default: ``false`` Type: *boolean* Allow transfering out\-of\-order data which is not reveived in timely order. .. _input: .. note:: **input.\*** *Configuration of data input host.* .. confval:: input.address Type: *string* URL. Format: [[caps\|capss]:\/\/][user:pass\@]host[:port] . .. _output: .. confval:: output.address Default: ``localhost:18003`` Type: *string* Data output URL [[caps\|capss]:\/\/][user:pass\@]host[:port]. This parameter superseds the host and port parameter of previous versions and takes precedence. .. confval:: output.host Default: ``localhost`` Type: *string* Deprecated: Data output host. Use \"output.address\" instead. .. confval:: output.port Default: ``18003`` Type: *int* Deprecated: Data output port. Use \"output.address\" instead. .. confval:: output.timeout Default: ``60`` Unit: *s* Type: *int* Timeout when sending a packet. If the timeout expires, the connection will be closed and re\-established. .. confval:: output.maxFutureEndTime Default: ``120`` Unit: *s* Type: *int* Maximum allowed relative end time for packets. If the packet end time is greater than the current time plus this value, the packet will be discarded. .. confval:: output.bufferSize Default: ``1048576`` Unit: *bytes* Type: *uint* Size \(bytes\) of the packet buffer. .. confval:: output.backfillingBufferSize Default: ``0`` Unit: *s* Type: *int* Length of backfilling buffer. Whenever a gap is detected, records will be held in a buffer and not sent out. Records are flushed from front to back if the buffer size is exceeded. .. confval:: output.maxFutureEndTime Default: ``3600`` Unit: *s* Type: *int* CAPS maximum allowed packet end time. If a packet's end time is larger than current time plus this value the packet is discarded. .. _output.mseed: .. confval:: output.mseed.enable Default: ``false`` Type: *boolean* Enable on\-the\-fly miniSEED encoding. If the encoder does not support the input type of a packet, it will be forwarded. Re\-encoding of miniSEED packets is not supported. .. confval:: output.mseed.encoding Default: ``Steim2`` Type: *string* miniSEED encoding to use. \(Uncompressed, Steim1 or Steim2\) .. confval:: output.mseed.recordLength Default: ``9`` Type: *uint* miniSEED record length. The value represents the exponent in the expression 2\^n, e.g., 2\^9 \= 512 bytes .. _journal: .. confval:: journal.file Default: ``@ROOTDIR@/var/run/caps2caps/journal`` Type: *file* Journal file to store stream states. .. confval:: journal.flush Default: ``10`` Unit: *s* Type: *uint* Flush stream states to journal file every given seconds. .. confval:: journal.waitForAck Default: ``60`` Unit: *s* Type: *uint* Wait when a sync has been forced, up to the given seconds. .. confval:: journal.waitForLastAck Default: ``5`` Unit: *s* Type: *uint* Wait on shutdown to receive acknownledgement messages, up to the given seconds. .. _statusLog: .. confval:: statusLog.enable Default: ``false`` Type: *boolean* Log information status information e.g. max bytes buffered. .. confval:: statusLog.flush Default: ``10`` Type: *uint* Flush status every given seconds to disk. .. _host: .. confval:: host.storage Default: ``@LOGDIR@`` Type: *directory* Path to storage used to determine available disc space and capacity. .. confval:: host.os Type: *string* Set the operating system information, such as Ubuntu 24.04.1 LTS. If this information is not provided, the plugin will attempt to read the value from the PRETTY_NAME variable in the \/etc\/os\-release file. .. confval:: host.agent Type: *string* Set the agent string to send to the CAPS server. By default, the application name is used. .. confval:: host.group Type: *string* Set the group string to send to the CAPS server. .. _backfilling: .. note:: **backfilling.\*** *Controls backfilling parameters for open streams (with no end time).* .. confval:: backfilling.mode Default: ``PUSH`` Type: *string* Values: ``PULL,PUSH`` Set the backfilling mode: PULL: During handshake all requests with no end time\(open\) are checked. For each request where the difference between system time and start time is greater than the value of backfilling.maxRealTimeGap the request will be split into two requests. PUSH: Start times of outgoing packets for requests are checked. For each request where the difference between system time and start time is greater than the value of backfilling.maxRealTimeGap the request will be split into two. .. confval:: backfilling.maxRealTimeGap Default: ``-1`` Unit: *s* Type: *int* Sets the maximum real\-time data gap in seconds. This means, if the start time of the requested time window of a channel is before this value with respect to the current system time then the request is split into a real\-time request starting at system time \- marginRealTimeGap and a backfill request from requested start time to time \- marginRealTimeGap. That prioritizes real time data and backfills old data in parallel. .. confval:: backfilling.marginRealTimeGap Default: ``60`` Unit: *s* Type: *int* The time margin used to request real time data in combination with maxRealTimeGap with respect to system time. Command-Line Options ==================== .. _Generic: Generic ------- .. option:: -h, --help Show help message. .. option:: -V, --version Show version information. .. option:: --config-file file The alternative module configuration file. When this option is used, the module configuration is only read from the given file and no other configuration stage is considered. Therefore, all configuration including the definition of plugins must be contained in that file or given along with other command\-line options such as \-\-plugins. .. option:: --plugins arg Load given plugins. .. option:: -D, --daemon Run as daemon. This means the application will fork itself and doesn't need to be started with \&. .. _Verbosity: Verbosity --------- .. option:: --verbosity arg Verbosity level [0..4]. 0:quiet, 1:error, 2:warning, 3:info, 4:debug. .. option:: -v, --v Increase verbosity level \(may be repeated, e.g., \-vv\). .. option:: -q, --quiet Quiet mode: no logging output. .. option:: --print-component arg For each log entry print the component right after the log level. By default the component output is enabled for file output but disabled for console output. .. option:: --component arg Limit the logging to a certain component. This option can be given more than once. .. option:: -s, --syslog Use syslog logging backend. The output usually goes to \/var\/lib\/messages. .. option:: -l, --lockfile arg Path to lock file. .. option:: --console arg Send log output to stdout. .. option:: --debug Execute in debug mode. Equivalent to \-\-verbosity\=4 \-\-console\=1 . .. option:: --trace Execute in trace mode. Equivalent to \-\-verbosity\=4 \-\-console\=1 \-\-print\-component\=1 \-\-print\-context\=1 . .. option:: --log-file arg Use alternative log file. .. _Input: Input ----- .. option:: -I, --input arg Overrides configuration parameter :confval:`input.address`. URL of data input host. Format: [[caps\|capss]:\/\/][user:password\@]host[:port] . .. option:: --max-real-time-gap Maximum length of data gap after reconnecting. If exceeded, a real\-time stream and backfilling stream will be created in parallel. Setting this value will give highest priority to real\-time streams, e.g., for rapid response systems. .. _Streams: Streams ------- .. option:: -i, --inventory arg Inventory XML defining the streams to add. .. option:: -A, --add-stream arg List of streamIDs [NET.STA.LOC.CHA] to add. Wildcards are supported. Use comma\-separation without blanks for multiple IDs. .. option:: --begin arg Start time of data request. Applied only on streams not found in the journal. Format: 'YYYY\-MM\-DD hh:mm:ss.sss'. .. option:: --end arg End time of data request. Format: 'YYYY\-MM\-DD hh:mm:ss.sss'. .. option:: --max-days arg Unit: *day* Maximum number of days to acquire regardless if the time window is configured or read from journal. A value of 0 or less disables the check. .. option:: --days arg Unit: *day* Begin of data request time window given as days before current time. Applied only on streams not found in the journal. .. option:: --days-before arg Unit: *day* End of data request time window given as number of days before current time. .. _Mode: Mode ---- .. option:: --archive Disable real\-time mode. Only archived data is fetched and missing records are ignored. .. option:: --out-of-order Use to enable out\-of\-order mode. Allows transfering data which is not in timely order. .. _Output: Output ------ .. option:: -O, --output arg Overrides configuration parameter :confval:`output.address`. This is the CAPS server which shall receive the data. .. option:: -b, --buffer-size arg Unit: *bytes* Size \(bytes\) of the journal buffer. If the value ist exceeded, a synchronization of the journal is forced. .. option:: --backfilling arg Default: ``0`` Unit: *s* Buffer size in seconds for backfilling gaps. .. option:: --mseed Enable on\-the\-fly miniSEED encoding. If the encoder does not support the input type of a packet, it will be forwarded. Re\-encoding of miniSEED packets is not supported. .. option:: --encoding arg miniSEED encoding to use: Uncompressed, Steim1 or Steim2. .. option:: --rec-len arg miniSEED record length expressed as a power of 2. A 512 byte record would be 9. .. option:: --max-future-endtime arg Unit: *s* Maximum allowed relative end time for packets. If the packet end time is greater than the current time plus this value, the packet will be discarded. By default this value is set to 120 seconds. .. option:: --dump-packets Dump packets to stdout. .. option:: --test Disable socket communication. .. option:: --dump Dump all received data to stdout and don't use the input port. .. _Journal: Journal ------- .. option:: -j, --journal arg File to store stream states. Use an empty string to log to stdout. .. option:: -f, --flush arg Unit: *s* Flush stream states to disk every n seconds. .. option:: --wait-for-ack arg Unit: *s* Wait when a sync has been forced, up to n seconds. .. option:: -w, --wait-for-last-ack arg Unit: *s* Wait on shutdown to receive acknownledgement messages, up to the given number of seconds. .. _Status: Status ------ .. option:: --status-log Log information status information, e.g., max bytes buffered. .. option:: --status-flush arg Unit: *s* Flush status every n seconds to disk. .. _Host: Host ---- .. option:: --host-storage Path to storage used to determine available disc space and capacity. .. option:: --host-os Set the operating system information, such as Ubuntu 24.04.1 LTS to send to the CAPS server. .. option:: --host-agent Set the agent string to send to the CAPS server. .. option:: --host-group Set the group string to send to the CAPS server.