Installation¶

This page describes the installation procedure of the Web Interface and the GDS back-end.

Both, the back-end and the web interface, are integrated into the SeisComP environment. The following tables shows the location of the GDS bundle:

Location	Description
$SEISCOMP_ROOT/bin/gds	Binary of the back-end server
$SEISCOMP_ROOT/bin/quakelink	Binary of the QuakeLink server
$SEISCOMP_ROOT/share/gds/tools	Python framework for content filter and spooler scripts as well as sample implementations
$SEISCOMP_ROOT/share/gds/web	Base directory of the Web Interface

Requirements¶

The GDS relies on the following packages:

Dependency	Required by
Python >= 3.6	web interface
Python >= 2.7	back-end
Django = 3.2.*	web interface
Django-Bitfield	web interface
Django-Extensions = 3.*	web interface (optional)
Django-SimpleHistory = 3.0.*	web interface
MySQL or PostgreSQL server	back-end, web interface
Python3 MySQL (2.0.1) or PostgreSQL interface	web interface
psycopg2 2.5.4 (PostgreSQL only)	web interface
NGINX web server including Gunicorn	web interface (optional)
QuakeLink Server	back-end, web interface

The installation of a web server is optional since Django ships with a tiny build-in web server. This build-in server is sufficient for testing purposes and if the GDS should be configured respectively monitored only locally (same machine). To use the full potential of the web configuration approach, the integration into an NGINX web server is recommended.

Note

In addition to the dependencies listed above more packages are required by the SeisComP environment. Please refer to the SeisComP manual. Since SeisComP version Seattle dependencies may be installed via seiscomp install-deps. Issue seiscomp help install-deps for a list of available packages. Furthermore you might want to use the same database management system for the GDS database as used by the SeisComP messaging system to safe computational resources.

Ubuntu/Debian¶

Installation of Python3 and the package installer for Python (PIP).

sudo apt-get install python3 python3-pip

Database¶

Install one of the following database management systems (DBMS):

MariaDB
MySQL
PostgreSQL

For production environments it is also recommended to install the NGINX web server.

MariaDB¶

sudo apt-get install mariadb-server mariadb-client libmariadbclient-dev
python3 -m pip install --user mysqlclient

MySQL¶

sudo apt-get install mysql-server mysql-client libmysqlclient-dev
python3 -m pip install --user mysqlclient

PostgreSQL¶

apt-get install postgresql postgresql-client
python3 -m pip install --user psycopg2-binary

NGINX¶

sudo apt-get install nginx

Gunicorn¶

python3 -m pip install --user gunicorn

RHEL¶

Installation of the Python3 and the package installer for Python (PIP).

su -
dnf install python3 python3-pip
exit

Database¶

Install one of the following database management systems (DBMS):

MariaDB
PostgreSQL

For production environments it is also recommended to install the NGINX web server.

MariaDB¶

su -
dnf install epel-release
dnf install mariadb mariadb-server python3-mysqlclient

PostgreSQL¶

su -
dnf install postgresql postgresql-client
exit
python3 -m pip install --user psycopg2-binary

NGINX¶

su -
dnf install nginx
exit

Gunicorn¶

python3 -m pip install --user gunicorn

Django¶

The Web Interface requires the Django framework as well as the django-bitfield module. Although most distributions offer Django packages, it is highly recommended to use the particular Django version 3.2.

To supply the web interface with the required libraries, the Django framework and the bitfield module must be installed. The django-extension package is optional. It provides command line tools, e.g., for template debugging and secret key generation.

python3 -m pip install --user django==3.2.*
python3 -m pip install --user django-bitfield
python3 -m pip install --user django-extensions==3.*
python3 -m pip install --user django-simple-history==3.0.*

QuakeLink Server¶

As described in the Introduction the GDS back-end as well as the web front-end depend on the gempa QuakeLink server. Similar to the GDS, the QuakeLink server is integrated into the SeisComP environment and is started like every other SeisComP application.

Although the server may be launched without any configuration file it will be of no use unless an application is running which feeds the server with earthquake information from a data provider like SeisComP.

The following feeding applications are available:

sc2ql – Connects to a SeisComP system and listens for earthquake updates.
ql2ql – Connects to a different QuakeLink server and listens for earthquake updates.
qlpush – Reads SCML (SeisComP Markup Language) data files
fdsnws2ql – Reads event information from FDSNWS services

For further reading please refer to the QuakeLink documentation.

Web Interface¶

Database Initialization¶

The GDS database must be created and permissions have to be granted to a user.

MariaDB/MySQL¶

mysql -h localhost -u root -p
mysql> CREATE DATABASE gds CHARACTER SET utf8mb4;
mysql> GRANT USAGE ON gds.* TO sysop@localhost IDENTIFIED BY 'sysop';
mysql> GRANT ALL PRIVILEGES ON gds.* TO sysop@localhost;

If the MySQL server resides on a different machine, the localhost must be replaced by the server’s IP. Also the database name, user and password may need to be modified.

Django requires MySQL timezone information. Issue the following command to check if your MySQL database is timezone aware:

mysql> SELECT CONVERT_TZ(now(), 'UTC','Europe/Berlin');

If the result is NULL you need to populate your DBMS with timezone information:

mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -D mysql -u root -p
mysql -u root -p -e "FLUSH TABLES;" mysql

The GDS offers to log the full bulletin content in the database. Depending on the messages you plan to send and on the operating system you use the default maximum allowed database packet size might be to small. Use the following commands to review the current settings of your DBMS:

mysql -h localhost -u root -p
mysql> SHOW VARIABLES LIKE 'max_allowed_packet';

If you think you will hit this limit you need to increase the packet size value in your DB server configuration. E.g., for RHEL and MariaDB this is done in the /etc/my.cnf.d/server.cnf:

[mariadb]
max_allowed_packet = 16777216

PostgreSQL¶

sudo -u postgres -i
createuser -P -d sysop
   Enter password for new role:
   Enter it again:
   Shall the new role be a superuser? (y/n) n
   Shall the new role be allowed to create more new roles? (y/n) n
createdb -O sysop gds
exit

Create and Adjust Settings¶

The web interface ships with 2 example configuration files. Change to the front-end directory and create a copy of those files.

cd $SEISCOMP_ROOT/share/gds/web/gds
cp example-settings.py settings.py
cp example-settings-production.py settings-production.py

The $SEISCOMP_ROOT/share/gds/web/gds/settings.py holds the main configuration settings. Although the defaults in place will work for most installations it is a good practice to review them. Especially the connection settings for the database as well as the GDS and QuakeLink server should be checked.

The $SEISCOMP_ROOT/share/gds/web/gds/settings-production.py extends the default settings by options needed for a production environment using a reverse proxy, see Web Server Integration. Typically one may change the value of

FORCE_SCRIPT_NAME if you a are deploying the GDS on a sub-path other than /gds.
ALLOWED_HOSTS which must include all IPs and hostnames you will use to access the GDS web interface.

Note: The Django build-in web server is considered to be insecure and should not be exposed to the public. You may use it in production if your network is properly shielded.

Completing the Installation¶

The database must be initialized with the Django admin tables and the GDS model. Also a Django super user must be created to login to the web interface. The 3rd command (loaddata) is optional. It will create an example configuration containing an email service, a queue matching for all world-wide events and finally a subscription to this queue. The 4th command (collectstatic) is only needed for the production environment. It copies all static files into a single directory (STATIC_ROOT) which needs to be served by NGINX. The last command (generate_secret_key) generates a secret key used in production.

cd $SEISCOMP_ROOT/share/gds/web
python3 manage.py migrate
python3 manage.py createsuperuser
python3 manage.py loaddata sample-data.json
python3 manage.py generate_secret_key > gds/secretkey.txt

To test the configuration the Django build-in web server may be started:

cd $SEISCOMP_ROOT/share/gds/web
python3 manage.py runserver

By default the server is available under http://localhost:8000

Note: The Django build-in web server is considered to be insecure and should not be exposed to the public. You may use it in production if your network is properly shielded.

To run the Django build-in server with production settings and to serve on all network interfaces issue the following command:

cd $SEISCOMP_ROOT/share/gds/web
python3 manage.py runserver --settings=gds.settings-production 0.0.0.0:8000

Web Server Integration¶

If you verified that the build-in web server is running, you may stop it now (Ctrl+C) and continue with web server integration. We recommend a combination of NGINX and Gunicorn for this purpose.

NGINX¶

NGINX will be configured as a reverse proxy and relies on an external WSGI service, e.g., Gunicorn, to map HTTP requests to the Django web application. The communication between NGINX and Gunicorn will be realized through a unix socket, although TCP based deployments are possible, e.g, if both processes should run on different machines.

Copy the relevant parts of the server and upstream directive from $SEISCOMP_ROOT/share/gds/web/examples/nginx-gunicorn.conf to your
- /etc/nginx/nginx.conf or
- /etc/nginx/sites-available/YOUR-VHOST.conf in case you are using a virtual host setup
If you are deploying the GDS on a path other than /gds, adjust the location directives accordingly. Make sure the sub-path matches the FORCE_SCRIPT_NAME variable in your $SEISCOMP_ROOT/share/gds/web/gds/settings-production.py.
In addition to the WSGI service, NGINX will serve static files from STATIC_ROOT typically configured to /usr/share/nginx/html/gds/static in $SEISCOMP_ROOT/share/gds/web/gds/settings.py.

Create the static directory, change ownership to the Django user and collect the files:
```
sudo mkdir -p /usr/share/nginx/html/gds/static
sudo chown sysop:sysop /usr/share/nginx/html/gds/static
cd $SEISCOMP_ROOT/share/gds/web
python3 manage.py collectstatic
```
Test the NGINX configuration:
```
sudo nginx -t
```

Enable and start NGINX:

sudo systemctl enable nginx
sudo systemctl start nginx

Gunicorn¶

NGINX relies on an external WSGI service, e.g., Gunicorn, to map HTTP requests to the Django web application. The communication between NGINX and Gunicorn will be realized through a unix socket, although TCP based deployments are possible, e.g, if both processes should run on different machines.

Gunicorn will run under the same user under which Django is installed. The configuration examples use the default SeisComP user sysop for this purpose.

Copy the Gunicorn configuration file to the Django base directory:
```
cd $SEISCOMP_ROOT/share/gds/web/
cp examples/gunicorn/gunicorn.conf.py .
```
Adjust the configuration, e.g., logging level, path and format if necessary.
Copy the Gunicorn socket and service systemd configuration:
```
sudo cp $SEISCOMP_ROOT/share/gds/web/examples/gunicorn/gunicorn-gds.* /etc/systemd/system
```
Adjust the user, group, or WorkingDirectory variables if necessary

Enable and start the service:

sudo systemctl enable gunicorn-gds
sudo systemctl start gunicorn-gds

Trouble shooting¶

Check if the socket file /var/run/gunicorn-gds.sock was created and ensure that the NGINX user (nginx or www-data) has write access to this file

Make a request directly at the socket:

sudo -u nginx curl --unix-socket /var/run/gunicorn-gds.sock -H "SCRIPT_NAME: /gds" http

By default Gunicorn logs to $HOME/.seiscomp/log/gds-web/access,error.log
Reload gunicorn-gds service after modifications to your $SEISCOMP_ROOT/share/gds/web/gunicorn.conf.py or Django configuration file: sudo systemctl reload gunicorn-gds
Reload systemd configuration after modifications to systemd service and socket files: sudo systemctl daemon-reload
Reload the NGINX server configuration: sudo nginx -t && sudo systemctl reload nginx

Next Steps¶

Login using the administrator account you created
Create staff users with some/all privileges prefix by gds_
Create Services, Queues and Subscriptions
Configure (re)start the Back-end

Back-end¶

The GDS back-end server is integrated into the SeisComP environment and is started like every other SeisComP application. Before running the server, the database must be created and initialized. Also a $SEISCOMP_ROOT/etc/gds.cfg must be created with a proper Database Configuration and Service Configuration.

Database Configuration¶

Make sure that the correct database plugin is loaded. This can be done in $SEISCOMP_ROOT/etc/global.cfg or in SEISCOMP_ROOT/etc/gds.cfg if different from the DBMS used by SeisComP. Next the database URL must be specified as demonstrated in the following database specific sections.

MariaDB/MySQL¶

# List of plugins loaded at startup. Separate multiple names by comma. Add
# ${plugins} to consider all previously read values.
plugins = ${plugins}, dbmysql

# Define the database connection. If no database is configured (which is the
# default) and a messaging connection is available, the application will
# receive the parameters after the connection is established. Override these
# values only if you know what you are doing.
# The format is: service://user:pwd@host/database. "service" is the name of the
# database driver which can be queried with "--db-driver-list". IMPORTANT: This
# parameter replaces former "database.type" and "database.parameters" which
# have been removed.
database = mysql://sysop:sysop@localhost/gds

PostgreSQL¶

Note

If the postgresql_psycopg2 Django back-end is used an additional dbSequencePrefix must be configured.

# List of plugins loaded at startup. Separate multiple names by comma. Add
# ${plugins} to consider all previously read values.
plugins = ${plugins}, dbpostgresql

# Define the database connection. If no database is configured (which is the
# default) and a messaging connection is available, the application will
# receive the parameters after the connection is established. Override these
# values only if you know what you are doing.
# The format is: service://user:pwd@host/database. "service" is the name of the
# database driver which can be queried with "--db-driver-list". IMPORTANT: This
# parameter replaces former "database.type" and "database.parameters" which
# have been removed.
database = postgresql://sysop:sysop@localhost/gds

# By default a database sequence is named after the table, suffixed by "_seq":
# "[tablename]_seq". Sometimes the column name is used as well, e.g. when using
# postgresql_psycopg2 as Django database back-end. If set, the value of this
# parameter will be added to the sequence suffix:
# "[tablename][dbSequencePrefix]_seq"
dbSequencePrefix = _id

Service Configuration¶

For each Services, External Criteria and Filters a corresponding entry in the servers configuration file must exist. If you loaded the sample-data.json as described in ref:sec-inst-web-final then you may use the following configuration:

# Service directory (mandatory). For each service a folder must exist where
# ID.content and ID.address files are stored.
service.email.directory = @ROOTDIR@/var/lib/gds/email

# The full path to the filter application.
# Environment variables ('$NAME') and SeisComP path variables ('@NAME@') are
# resolved at time of application start. In addition the following event
# context variables ('@NAME@') are resolved at each command run: MATCH,
# REVISION, EVENTID, ORIGINTIME, LATITUDE, LONGITUDE, DEPTH, MAGNITUDE, RMS,
# PHASES, MAGNITUDECOUNT, EVALUATIONMODE, EVALUATIONSTATUS, REGION, EVENTTYPE
# and MAGTYPE.
# It is a good practice to use quotes for string variables as they may be empty
# or contain white spaces.
service.email.filter.primary.cmd = "@DATADIR@/gds/tools/filter_email.py -t 8"

# MIME type of the filter output. This parameter is used by the web front-end
# for content display.
service.email.filter.primary.mimetype = text/gds

# Timeout in seconds after which the command is aborted. Use a negative value
# to indicate an infinite timeout.
service.email.filter.primary.timeout = 10.0

# Spool command to execute.
service.email.spooler.cmd = "@DATADIR@/gds/tools/send_email.py -t 8"

# Polling interval in seconds the spooler will scan the service directory for
# new files.
service.email.spooler.interval = 2.0

# Timeout in seconds after which the command is aborted. Use a negative value
# to indicate an infinite timeout.
service.email.spooler.timeout = 10

Update Instructions¶

If you already operate a GDS version and you intent to install an update you should:

read the CHANGELOG file to get an overview of the changes
consider creating a database backup
extract the bundle (overriding existing files in the installation folder)
check for new or updated parameters in the $SEISCOMP_ROOT/share/gds/web/gds/example-settings.py and $SEISCOMP_ROOT/share/gds/web/gds/example-settings-production.py configuration files and apply them to your copies
apply database migrations if needed
remove and collect static files again
restart the GDS server and (optionally) the reverse proxy server

Migration to 2021.294 (Django 3.2, simple-history)¶

With version 2021.294 the GDS front-end switched from Django 3.1 to the LTS version 3.2. Also history support for the data model has been added allowing to review all changes to a particular configuration object and to revert back to a previous revision.

Python dependencies¶

Update Django and install the SimpleHistory django plugin:

python3 -m pip install --user django==3.2.*
python3 -m pip install --user django-simple-history==3.0.*

Update of configuration files¶

Compare your $SEISCOMP_ROOT/share/gds/web/gds/settings.py with the updated $SEISCOMP_ROOT/share/gds/web/gds/example-settings.py. Especially make sure to

add the line DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
add 'simple_history', to the INSTALLED_APPS list
add 'simple_history.middleware.HistoryRequestMiddleware', to the MIDDLEWARE list

Compare your $SEISCOMP_ROOT/share/gds/web/gds/settings-production.py with the updated $SEISCOMP_ROOT/share/gds/web/gds/example-settings-production.py. Especially make sure to

add 'simple_history', to the INSTALLED_APPS list

Database migrations¶

The data model changed. To complete the update run:

cd $SEISCOMP_ROOT/share/gds/web
python3 manage.py migrate
python3 manage.py populate_history --auto

Static files¶

The static files changed and need to be collected again:

cd $SEISCOMP_ROOT/share/gds/web
rm -rf static
python3 manage.py collectstatic

Migration to 2020.282 (Django 3.1)¶

With version 2020.282 the GDS front-end switched from Django 1.11 to 3.1 and requires a minimum Python version of 3.6. The back-end was updated to support Python 3 but continues to support Python 2 starting from version 2.7 in order to support SeisComP3 Jakarta 2018.327 and older.

Cleanup¶

Since this is a major update it is recommended to move your $SEISCOMP_ROOT/share/gds/web/ folder to a temporary location prior to the installation of the new GDS package and later on to restore your $SEISCOMP_ROOT/share/gds/web/gds/settings.py and $SEISCOMP_ROOT/share/gds/web/gds/settings-production.py while merging the new configuration options found in the $SEISCOMP_ROOT/share/gds/web/gds/example-settings* files . If you do not want to move your web folder or if you already extracted the GDS package then please remove at least the following files:

rm -rf $SEISCOMP_ROOT/share/gds/web/{django,django_extensions,bitfield}
rm $SEISCOMP_ROOT/share/gds/web/gds/templatetags/breadcrumbs.py

Additional Dependencies¶

The following new dependencies have been introduced:

Python >=3.6
django-3.1

Please refer to Django and Ubuntu/Debian respectively RHEL for installation instructions.

Back-end Configuration¶

In contrast to previous GDS versions the MIME type of a filter is no longer guessed but must be specified in the GDS back-end configuration, see

Common MIME types are

text/gds - GDS bulletin format used by most of the sample filter scripts shipped with GDS
text/x-python - Python dictionary displayed as key-value pairs
image/[subtype] - Images of any type
text/[subtype] - Generic text
application/pdf - PDF document

Filter and Spooler Library¶

The GDS Python library ($SEISCOMP_ROOT/share/gds/tools) uses the seiscomp-python shebang to determine the Python version the SeisComP Python bindings are compiled for. For SeisComP versions <= 2018.327 an alias pointing to either python2 or python3 needs to be created. When the GDS is installed via gsm this will be done automatically. It is recommended to use the same shebang in custom filter and spooler scripts.

Furhter noteable changes are:

renamed $SEISCOMP_ROOT/share/gds/tools/lib/xml.py to $SEISCOMP_ROOT/share/gds/tools/lib/scml.py to avoid naming conflicts
logging facility
- new messageLimit configuration parameter
- try to read log directory from SeisComP Environment, use $HOME/.seiscomp/log as fallback

Database Migrations after 2017.269¶

Starting with version 2017.269 the GDS front-end switched from Django 1.4 to 1.11.15. The new Django version provides build-in database migration mechanism.

To list applied and outstanding migrations issue:

cd $SEISCOMP_ROOT/share/gds/web
python3 manage.py showmigrations

To apply outstanding migrations run:

python3 manage.py migrate

Migration to 2017.269 (Django 1.11.15)¶

With version 2017.269 the GDS front-end switched from Django 1.4 to 1.11.15 special attention must be giving to this update.

Additional Dependencies¶

The following new dependencies have been introduced:

Python >=2.7
pytz
django-1.11.15
django-extensions (optional)
django-bitfield

Refer to Django and Ubuntu/Debian respectively RHEL for installation instructions.

Database¶

If you are updating from a GDS version older than 2017.079 make sure to run all appropriate database update scripts to be found in migrations/pre-2017.269

cd $SEISCOMP_ROOT/share/gds/web/gds/migrations/pre-2017.269/mysql
mysql -u sysop -p gds < 2016....
mysql -u sysop -p gds < 2017.079

Although the GDS datamodel was not changed between 2017.079 and 2017.269 you still need to be migrate your database to update the Django system tables. Since your system tables already have been initialized the initial migration step needs to be skipped:

cd $SEISCOMP_ROOT/share/gds/web
python3 manage.py migrate --fake-initial

New Directory Structure¶

all GDS related files are located under the gds subfolder
templates moved from contrib to templates folder
CSS and java script files moved from media to static folder, no need to serve media directory with NGINX anymore

Database Backup and Restore¶

It is a good idea create database backups on regular basis. Especially before upgrading to a newer GDS version a backup is recommended.

MariaDB/MySQL¶

Backup¶

Create a backup of the entire GDS database:

mysqldump -u sysop -p gds > /tmp/gds.sql

Create a backup of the GDS database but omit the rather large GDS log tables:

mysqldump -u sysop -p --no-data gds > /tmp/gds.sql
mysqldump -u sysop -p --no-create-info gds --ignore-table=gds.gds_log_event --ignore-table=gds.gds_log_service --ignore-table=gds.gds_log_recv >> /tmp/gds.sql

Restore¶

mysql -u sysop -p gds < /tmp/gds.sql

PostgreSQL¶

Backup¶

Create a backup of the entire GDS database:

pg_dump -U sysop -W -F c gds > /tmp/gds.sql

Create a backup of the GDS database but omit the rather large GDS log tables:

pg_dump -U sysop -W -F c --exclude-table-data=gds_log_* gds > /tmp/gds.sql

Restore¶

pg_restore -U sysop -W -c --if-exists -d gds /tmp/gds.sql