.. _tutorials_upgrade:

***************************
Updating/Upgrading SeisComP
***************************

You will ...

* Update/Upgrade a SeisComP system
* Migrate a SeisComP3 system to a newer SeisComP version

Pre-requisites for this tutorial:

* :ref:`Documentation of installation <installation>`
* Tutorial on :ref:`installation <tutorials_postinstall>` and |scname|
  previously installed

Afterwards/Results/Outcomes:

* Upgraded SeisComP

Time range estimate:

* 60 minutes

------------


Background
==========


Updating/upgrading |scname| to a higher
:ref:`release version <installation_versions>` is typically simple and the
steps described in :ref:`tutorials_upgrade-normal` can be applied.
**More actions** are required when

* Upgrading the major version of SeisComP as described in :ref:`tutorials_upgrade-normal`.
* Upgrading :ref:`from SeisComP3 to SeisComP in version 4.0.0. or higher <tutorials_upgrade_v4>`.
* Upgrading :ref:`from SeisComP3 Jakarta-2018.327 or older to Jakarta-2020.330 or
  SeisComP in version 4 or higher <tutorials_upgrade_seedlink>`.


Upgrade vs. update
------------------

Here we understand that *upgrades* increase the major release version number
while *updates* only increase the minor version or the patch number, see section
:ref:`installation_versions`.


Upgrading multiple machines
---------------------------

Applications can only connect to a messaging system that runs with a database
in an equal or lower data base schema version. However, upgrading |scname| to a
higher major release version typically increases the database scheme version.
Therefore, in distributed |scname| systems where one machine hosts the messaging
system and the database and all other machines are connected to this messaging,
the |scname| installation on the machine operating the messaging is always
upgraded last.

**Example:** A distributed system includes a processing system with the
messaging system and database and a GUI work station connected to the processing
system:

#. :ref:`Upgrade <tutorials_upgrade-normal>` the GUI work station
#. :ref:`Upgrade <tutorials_upgrade-normal>` the processing system, take actions
   to :ref:`upgrade the database version <tutorials_upgrade-db>`.

.. note::

   Always stop all SeisComP modules before upgrading:

   .. code-block:: sh

      seiscomp stop


.. _tutorials_upgrade_download:

Package download
----------------

Get and install the |scname| package in the any available version from
:cite:t:`gempa` or from the download website of :cite:t:`seiscomp` as described
in section :ref:`installation-packages`.


.. _tutorials_upgrade_changelog:

Documentation of changes
------------------------

The important novelties, optimizations and changes that are available after
upgrading are documented in the Changelog which is part of this documentation
and which can also be accessed through the *Docs* panel of :ref:`scconfig`.
The installed locally installed file is
:file:`$SEISCOMP_ROOT/share/doc/seiscomp/CHANGELOG`
The Changelog can also be read
`online <https://www.seiscomp.de/doc/base/changelog.html>`_ but care should be
take that the version number matches your.´

.. note::

   New major release with the features are regularly advertised and described in
   detail on the `News website of gempa GmbH <https://www.gempa.de/news/>`_ and
   on the :cite:t:`seiscomp-forum`.


.. _tutorials_upgrade-db:

Upgrade database schema version
-------------------------------

When upgrading |scname| to a higher major version number as set out in section
:ref:`tutorials_upgrade-normal`, upgrading the database schema is typically
required, too. The database version will be tested and the required actions will
be reported when executing:

.. code-block:: sh

   seiscomp update-config**Special case:**

or when pressing the Update Configuration button in :ref:`scconfig`.
An upgrade from version SeisComP3 jakarta-2017.334 to SeisComP in version 5.1.0
will give, e.g.:

.. code-block:: sh

   seiscomp update-config
   * starting kernel modules
   starting scmaster
   * configure kernel
   * configure scmaster
   INFO: checking DB schema version of queue: production
     * check database write access ... OK
     * database schema version is 0.10
     * last migration version is 0.12
     * migration to the current version is required. apply the following
       scripts in exactly the given order:
       * mysql -u sysop -p -D seiscomp -h localhost < /home/sysop/seiscomp/share/db/migrations/mysql/0_10_to_0_11.sql
       * mysql -u sysop -p -D seiscomp -h l
**Special case:**ocalhost < /home/sysop/seiscomp/share/db/migrations/mysql/0_11_to_0_12.sql
   error: updating configuration for scmaster failed

The shown migration scripts can be used directly as given and in the given
order but you need to first stop all modules writing to or deleting objects from
the database such as :ref:`scmaster`, :ref:`scdb`, :ref:`scardac` or
:ref:`scdbstrip` running on the same or any connected machine, e.g.

.. code-block:: sh

   seiscomp stop scmaster
**Special case:**
Thereafter, apply the migration according to your database:

* MySQL/MariaDB:

  .. code-block:: sh

     mysql -u sysop -p -D seiscomp -h localhost < /home/sysop/seiscomp/share/db/migrations/mysql/0_10_to_0_11.sql
     mysql -u sysop -p -D seiscomp -h localhost < /home/sysop/seiscomp/share/db/migrations/mysql/0_11_to_0_12.sql

* PostgreSQL:

  .. code-block:: sh

     psql -U sysop -d seiscomp -h localhost -W -f /home/sysop/seiscomp/share/db/migrations/postgresql/0_10_to_0_11.sql
     psql -U sysop -d seiscomp -h localhost -W -f /home/sysop/seiscomp/share/db/migrations/postgresql/0_11_to_0_12.sql

Using the migration scripts provides a more user friendly way than copying the
lines of MySQL code from the changelog which was practise in past SeisComP3
versions. In future versions we might add the option
to automatically run the migrations.

.. warning::

   Upgrading the database make take some time. Do no interrupt the process!
   During this time, the |scname| messaging system is unavailable causing a
   downtime of the system.

After applying the migration scripts the database should be at the correct version.
Test again with:

.. code-block:: sh

   seiscomp update-config

After successfully upgrading the database continue your previous upgrade
procedure.


.. _tutorials_upgrade-normal:

Normal Update/Upgrade
=====================

Normal updates/upgrades of the major version of |scname| takes only a few steps:

#. If you are tracking your |scname| installation using :program:`git`
   (recommended), ensure all previous changes are committed and the installation
   is clean.
#. Stop all SeisComP modules:

   .. code-block:: sh

      seiscomp stop

#. Download and install the |scname| package as described in section
   :ref:`installation-packages`.

#. Understand the Changelog of the installed |scname| as described in section
   :ref:`tutorials_upgrade_changelog`.

#. Infrequently configuration parameters require changes which
   are documented in the Changelog. You may also scan your configuration for
   deprecated or obsolete configuration parameters or values using
   `gempa-checkSCconfig.py <https://data.gempa.de/packages/Public/tools/>`_
   provided by :cite:t:`gempa`. Apply changes accordingly.

#. The software dependencies may have changed after upgrading. Install them as
   described in section :ref:`software_dependencies`.

#. Test the database schema version and update bindings

   .. code-block:: sh

      seiscomp update-config

   :ref:`Upgrade the database schema version <tutorials_upgrade-db>` if
   mismatches are reported. The mismatch reports will also give the upgrade
   instructions. Repeat *seiscomp update-config* after upgrading the database
   schema version.

   **Background:** When upgrading the major |scname| version you
   typically need to upgrade the database scheme version.



#. After a successful update/upgrade, start all modules again and observe the
   status:

   .. code-block:: sh

      seiscomp start
      seiscomp status started

#. If you are tracking your |scname| installation using :program:`git`
   (recommended), commit all changes.


.. _tutorials_upgrade_v4:

Upgrading from SeisComP3
========================

SeisComP in version >=4 has major differences to SeisComP3 which require
adjustments. The main differences are in the
:ref:`directories of the SeisComP installation <sec-tutorials_upgrading_path>`
and the :ref:`messaging system <sec-tutorials_upgrading_messaging>`.
The changes and the required actions are explained below. They must be considered
in addition to the steps set out in section :ref:`tutorials_upgrade-normal`.


.. _sec-tutorials_upgrading_path:

Files and directories
---------------------

With **SeisComP3** all the default installation typically required all modules
and configurations in the directories

* seiscomp3/ , typically $HOME/seiscomp3 or /opt/seiscomp3/
* $HOME/.seiscomp3/

As of **SeisComP in version 4** the directories are:

* seiscomp/ , typically $HOME/seiscomp/ or /opt/seiscomp/
* $HOME/.seiscomp/

**All configuration files** must be migrated to the new structures. This
includes:

* Configurations and inventory in seiscomp3/:

  * seiscomp3/etc/\*.cfg
  * seiscomp3/etc/inventory/
  * seiscomp3/etc/keys/

* Configurations in $HOME/.seiscomp3/
* Logs in $HOME/.seiscomp3/log (optional)
* All user-defined files and directories in seiscomp3/share/
* All user-defined :ref:`seedlink` and other templates in seiscomp3/share/templates/
* The waveform archive and other archives typically in seiscomp3/var/lib/
* User-defined files and directories in other places.

  .. warning::

     Some configuration default and description files have changed. Spread, arclink
     and arclinkproxy are not part of |scname| anymore. **Therefore, do not migrate:**

     * any default configuration, description and init files. Better enable the desired
       daemon modules again:

       .. code-block:: sh

          seiscomp/bin/seiscomp enable [module]

     *   any file related to spread or the arclink and arclinkproxy servers.

Configurations containing absolute paths, e.g. :file:`/home/sysop/seiscomp3/share/scautoloc/grid_custom.conf`,
must be adjusted. Better use :ref:`internal SeisComP variables <concepts_configuration_variables>`
such as *@DATADIR@* instead of *seiscomp3/share* or *seiscomp/share*.


Software dependencies
---------------------

The software dependencies may have changed.
:ref:`Install the missing ones <software_dependencies>`.


System variables
----------------

The system environment variables must be updated, e.g. in :file:`$HOME/.bashrc`.
Remove or uncomment the lines  :file:`$HOME/.bashrc` referring to the depreciated SeisComP3
version. Then execute

.. code-block:: sh

   seiscomp/bin/seiscomp print env >> $HOME/.bashrc
   source $HOME/.bashrc


Pipelines
---------

When using pipelines or alias modules, create and enable the alias module names again, e.g.

.. code-block:: sh

   seiscomp alias create [alias] [module]
   seiscomp enable [alias]

Migrate the module and bindings configurations of the alias modules including all related additional files which are referred to
in the configurations.


.. _sec-tutorials_upgrading_messaging:

Messaging system
----------------

One of the main changes SeisComP3 to SeisComP in version 4.0 is the :ref:`messaging system <concepts_messaging>`.
Spread does not exist anymore and only :ref:`scmaster` is started initially for
the messaging system. :ref:`scmaster` allows to operate several queues in parallel with
different databases. This flexibility comes with additional parameters which require
configuration. Migrate the legacy database parameters and configure the new one:


#. Remove or comment the obsolete *dbplugin* plugin manually from
   :file:`scmaster.cfg` and :file:`global.cfg` ::

   # plugins = dbplugin

#. Set up the messaging queues in the configuration of :ref:`scmaster` in
   :file:`scmaster.cfg`.

   * Add and configure a new queue or stay with the default ones.

     * *production* considers a database by default.
     * *playback* considers no database by default. Here, parameters can be
       exchanged through the messaging without storing in the database.

     In the following examples, the *production* queue shall be assumed.

     .. note::

        The *production* queue is used by default by all modules connected
        to the messaging system. When removing this queue and a database shall be
        used, another queue must exist
        and the queue name must be configured for all modules in the global
        :confval:`connection.server` parameter. See below for an example.

     * Add the required plugins per queue. Currently only *dbstore* is supported.
       Example for the *production* queue:

       .. code-block:: properties

          queues.production.plugins = dbstore

     * Add non-default message groups, e.g. *L1PICK* and *L1LOCATION* to the list
       of groups **in one of the ways**:

       * **Recommended:** Add groups per queues to defaults in
         :confval:`queues.$name.groups`, e.g. for the *production* group.
         This convenient configuration per queue
         considers the default groups in :confval:`defaultGroups` and simply adds
         new groups in the configuration of queues

         .. code-block:: properties

            queues.production.groups = ${defaultGroups}, L1PICK, L1LOCATION

       * **Special case:** Set groups per queue in :confval:`queues.$name.groups`,
         ignoring groups in :confval:`defaultGroups`

         .. code-block:: properties

            queues.production.groups = L1PICK, L1LOCATION, AMPLITUDE, PICK, LOCATION, MAGNITUDE, FOCMECH, EVENT, QC, PUBLICATION, GUI, INVENTORY, ROUTING, CONFIG, LOGGING, IMPORT_GROUP, SERVICE_REQUEST, SERVICE_PROVIDE

       * **Special case:** Set groups in :confval:`defaultGroups`

         .. code-block:: properties

            defaultGroups = L1PICK, L1LOCATION, AMPLITUDE, PICK, LOCATION, MAGNITUDE, FOCMECH, EVENT, QC, PUBLICATION, GUI, INVENTORY, ROUTING, CONFIG, LOGGING, IMPORT_GROUP, SERVICE_REQUEST, SERVICE_PROVIDE

       .. warning::

          When setting groups in the queues all groups configured in
          :confval:`defaultGroups` will be ignored unless `${defaultGroups}` is
          used. Add all groups from :confval:`defaultGroups` to the queues to
          keep the default groups.

     * Add the interface name, currently only *dbstore* is supported. Example for
       a queue names *production*

       .. code-block:: properties

          queues.production.processors.messages = dbstore

     * Add the database parameters which can be used from the legacy configuration

       .. code-block:: properties

          queues.production.processors.messages.dbstore.driver = mysql
          queues.production.processors.messages.dbstore.read = sysop:sysop@localhost/seiscomp3
          queues.production.processors.messages.dbstore.write = sysop:sysop@localhost/seiscomp3

       .. note::

          The name of the database can be freely chosen. The example assumes that
          the database named *seiscomp3* exists already and that it shall be continued
          to be used with the new SeisComP in version 4.x.x.

   * Add one or more of the queues to the :confval:`queues` parameter to register
     them by their names

     .. code-block:: properties

        queues = production, playback


#. Configure the connection parameters of all modules connecting to the messaging
   system in the global configuration, e.g. in :file:`global.cfg`.
   As in SeisComP3 the connection server is
   localhost. The queue name is added to the host by "/". The default queue
   is *production*, e.g.

   .. code-block:: properties

      connection.server = localhost/production

   .. note::

      If *production* shall be used, then no additional configuration is required.


Database
--------

After adjusting the structure, variables and configuration parameters, check if the
:ref:`database requires an upgrade <tutorials_upgrade-db>` as well.


Seedlink
--------

When upgrading from SeisComp3 in version Jakrata-2018.327 or older and using
:ref:`seedlink`, consider the section :ref:`tutorials_upgrade_2018.327`.


Automatic module check
----------------------

If applied, adjust the settings for automatic module status check, e.g. crontab entries.
For crontab use:

.. code-block:: sh

   crontab -e


System daemon
-------------

If |scname| is controlled by the system daemon, e.g. to start enabled |scname|
modules automatically during computer startup, then the startup script must be
adjusted.


.. _tutorials_upgrade_2018.327:

Upgrading From SeisComP3 <= Jakarta-2018.327
============================================


.. _tutorials_upgrade_seedlink:

SeedLink buffer
---------------

In SeisComP3 prior to Jakarta-2020.330 two stations with the same
station but different network code were mixed in one buffer directory.
As of  Jakarta-2020.330 and SeisComP in version 4 the buffer directories are now
unique!
Before upgrading :ref:`seedlink`, you should therefore rename the buffer directories
accordingly.

.. warning::

   You may discover data gaps if you do not rename the buffer directories.

**Example:**

#. Check the current situation:

   .. code-block:: bash

      sysop@host:~/seiscomp3/var/lib/seedlink/buffer$ ls
        PB02

#. Rename the directories properly:

   #. Stop seedlink:

      .. code-block:: sh

         sysop@host:seiscomp stop seedlink

   #. Upgrade to SeisComP3-jakarta-2020.330 or SeisComP in version 4 or higher.
   #. Rename all seedlink buffer directories to NET.STA, e.g.

      .. code-block:: bash

         sysop@host:~/seiscomp3/var/lib/seedlink/buffer$ mv PB02 CX.PB02
         sysop@host:~/seiscomp3/var/lib/seedlink/buffer$ ls
           CX.PB02

      .. note:

         The :ref:`script below <seedlink-buffer-script>` can be used for renaming the seedlink buffer directories.
   #. Update configuration:

      .. code-block:: bash

         sysop@host:seiscomp update-config

   #. Start SeedLink

      .. code-block:: bash

         sysop@host:seiscomp start seedlink

.. _seedlink-buffer-script:

Script for renaming the seedlink buffer directories:

.. code-block:: bash

   #!/bin/bash

   if [ -z ${SEISCOMP_ROOT+x} ]; then
           echo "Environment variable SEISCOMP_ROOT is not set."
           echo "Either use 'seiscomp exec [script]' or set SEISCOMP_ROOT to the installation "
        exit 1
        echo "path of your SeisComP installation."
   fi

   grep -A 2 ^station $SEISCOMP_ROOT/var/lib/seedlink/seedlink.ini | while read a b c; do
       if [ "$a" = station -a "$b" != .dummy ]; then
                id=$b
                sta=""
                net=""
                while read a b c; do
                        case $a in
                                --) break;;tutorials_upgrade_seedlink
                                name) eval sta=$c;;
                                network) eval net=$c;;
                        esac
                done
                if [ -z "$id" -o -z "$sta" -o -z "$net" ]; then
                        echo "Error parsing seedlink.ini"
                        break
                fi

                if [ "$id" != "$net.$sta" ]; then
                        mv -v "$SEISCOMP_ROOT/var/lib/seedlink/buffer/$id" "$SEISCOMP_ROOT/var/lib/seedlink/buffer/$net.$sta"
                else
                        echo "$id: No renaming required"
                fi
        fi
   done


.. _tutorials_proc_seedlink:

SeedLink stream processor
-------------------------

Since SeisComP3 in version Jakarta-2020.030 and SeisComP in version 4.0.0,
SeedLink stream processors (``proc`` parameter) can be attached to both, stations
and plugin instances. In order to distinguish between the two cases, either
``proc`` (attach to station) or ``sources.*.proc`` (attach to plugin instance)
parameter (or both) can be used in SeedLink bindings.


chain plugin
~~~~~~~~~~~~

In case of the :ref:`chain plugin <seedlink-sources-chain-label>` for
:ref:`seedlink`, there is
normally just one instance, so stream processors attached to this instance apply
to all stations. **This is normally not what we want.** Therefore the
chain plugin does not support the ``sources.*.proc`` option.

Before SeisComP3 in version Jakarta-2020.030 and SeisComP in version 4.0.0,
stream processors were always attached to stations, even when ``sources.*.proc``
was used. This means when upgrading:

#. ``sources.chain.proc`` must be renamed to ``proc``
#. streams\_\*.tpl templates must be moved one level up, from
   :file:`$SEISCOMP_ROOT/seiscomp/share/templates/seedlink/chain/` to
   :file:`$SEISCOMP_ROOT/seiscomp/share/templates/seedlink/`.

.. note::

   Using a stream processor with chain_plugin makes only sense when raw
   data is generated (:confval:`sources.chain.channels.unpack`).


Background
~~~~~~~~~~

A stream processor is an object defined in XML, which is used to create MiniSEED
from raw data and optionally downsample the data. What is the difference between
attaching a stream processor to station and plugin instance?

Let's take a look at the following stream processor definition in
:file:`$SEISCOMP_ROOT/share/templates/seedlink/streams_stream100.tpl`:

.. code-block:: XML

   <proc name="stream100">
     <tree>
       <input name="Z" channel="Z" location="" rate="100"/>
       <input name="N" channel="N" location="" rate="100"/>
       <input name="E" channel="E" location="" rate="100"/>
       <node filter="FS2D5" stream="BH">
         <node filter="F96C">
           <node filter="ULP" stream="LH">
             <node filter="VLP" stream="VH"/>
           </node>
         </node>
       </node>
     </tree>
   </proc>

This creates 20Hz BH\*, 1Hz LH\* and 0.1Hz VH\* streams from 100Hz Z, N, E raw
data. If one plugin instance is used for the station, it does not make a
difference whether this is attached to station or plugin instance. But suppose
the station is using two plugin instances—one for broad-band and the other for
strong-motion data—, both sending Z, N and E channels. Now if the stream processor
is attached to station, data from both plugin instances would mixed up. We must
attach a different stream processor to each plugin instance—one producing BH\*,
LH\* and VH\* and the other one producing BN\* and so on.