scdb

Populate a SQL database from XML files or messages.

Description

A major component of the SeisComP system is the database. Almost all applications have only read access to the database, but all the processing results and objects have to be written into the database. This was the task of scdb. In very first versions of SeisComP scdb was the only component that had write access to the database. Its task is to connect to scmaster and populate the database with all received notifier messages. Although it worked it introduced race conditions caused by the latency of the database backend since all other clients received the message at the same time. Accessing the database immediately at this point in time did not guarantee that the object was written already.

In consequence, the scmaster itself gained write access to the database and forwards messages to all clients after they are written to database.

scdb by definition does not check existing objects in the database. It only generates INSERT/UPDATE/DELETE statements based on the data used and sends these statements to the database. E.g. if scdb receives a message to insert a new object into the database and this object exists already, the database will raise an error because scdb hasn’t checked it.

Online mode

Now scdb can be used to maintain a backup or archive the database that is not part of the real time processing. When running scdb as database write daemon it can inform a client about the database connection to use. A client sends a DatabaseRequest message and scdb sends back a DatabaseResponse message containing the database connection parameters.

For that it connects to a messaging server and writes all received messages to a configured database, e.g. a backup database.

Note

The database connection received from the messaging server during the handshake is reported to clients requesting a database address. To overwrite the read-only database, just override the application’s database address (with the ‘-d’ option)

Offline mode

Another important task of scdb is to populate the database with any SeisComP data model content. In combination with scxmldump it can be used to copy events from one database to another.

For that it does not connect to a messaging server but reads data from XML files and writes it to the database. Offline mode will be used if the ‘–input/-i’ option is provided. Multiple input files can be specified by providing this option multiple times with separate filenames.

Warning

When reading XML files the output database address is not passed with -o but -d. The application’s database address is used.

Examples

1. Connect to a messaging server and write all messages to the output database seiscomp running on the host db-server:

   scdb -H [server] -o mysql://sysop:sysop@db-server/seiscomp
   

2. As above, but with the read-only database connection using the user sysop and the output database connection using the user writer:

   scdb -H [server] -d mysql://sysop:sysop@db-server/seiscomp \
                    -o mysql://writer:12345@db-server/seiscomp
   

3. Import data from the file data.xml and write it to the database seiscomp on the host db-server:

   scdb -i data.xml -d mysql://sysop:sysop@db-server/seiscomp
   

4. Import data from three files at once:

   scdb -i data1.xml -i data2.xml -i data3.xml \
        -d mysql://sysop:sysop@db-server/seiscomp
   

Module Configuration

etc/defaults/global.cfg

etc/defaults/scdb.cfg

etc/global.cfg

etc/scdb.cfg

~/.seiscomp/global.cfg

~/.seiscomp/scdb.cfg

scdb inherits global options.

connection.requestGroup

Type: string

Define the group on scmaster to subscribe for database requests.

connection.provideGroup

Type: string

Define the group on scmaster to send database response messages to.

output.type

Type: string

Define the output database connection type.

output.parameters

Type: string

Define the output database connection parameters.

Command-Line Options

scdb [options]

Generic

-h, --help

Show help message.

-V, --version

Show version information.

--config-file arg

Use alternative configuration file. When this option is used the loading of all stages is disabled. Only the given configuration file is parsed and used. To use another name for the configuration create a symbolic link of the application or copy it. Example: scautopick -> scautopick2.

--plugins arg

Load given plugins.

-D, --daemon

Run as daemon. This means the application will fork itself and doesn’t need to be started with &.

--auto-shutdown arg

Enable/disable self-shutdown because a master module shutdown. This only works when messaging is enabled and the master module sends a shutdown message (enabled with --start-stop-msg for the master module).

--shutdown-master-module arg

Set the name of the master-module used for auto-shutdown. This is the application name of the module actually started. If symlinks are used, then it is the name of the symlinked application.

--shutdown-master-username arg

Set the name of the master-username of the messaging used for auto-shutdown. If "shutdown-master-module" is given as well, this parameter is ignored.

Verbosity

--verbosity arg

Verbosity level [0..4]. 0:quiet, 1:error, 2:warning, 3:info, 4:debug.

-v, --v

Increase verbosity level (may be repeated, eg. -vv).

-q, --quiet

Quiet mode: no logging output.

--component arg

Limit the logging to a certain component. This option can be given more than once.

-s, --syslog

Use syslog logging backend. The output usually goes to /var/lib/messages.

-l, --lockfile arg

Path to lock file.

--console arg

Send log output to stdout.

--debug

Execute in debug mode. Equivalent to --verbosity=4 --console=1 .

--log-file arg

Use alternative log file.

Messaging

-u, --user arg

Overrides configuration parameter connection.username.

-H, --host arg

Overrides configuration parameter connection.server.

-t, --timeout arg

Overrides configuration parameter connection.timeout.

-g, --primary-group arg

Overrides configuration parameter connection.primaryGroup.

-S, --subscribe-group arg

A group to subscribe to. This option can be given more than once.

--content-type arg

Overrides configuration parameter connection.contentType.

--start-stop-msg arg

Set sending of a start and a stop message.

--m, --mode arg

scdb can either process a XML file and write it to the database or collect messages from scmaster. If connected to scmaster, the mode defines what objects are handled: none (no objects at all), notifier (notifier only) or all (all objects whereas non-notifier objects are INSERTED into the database).

Database

--db-driver-list

List all supported database drivers.

-d, --database arg

The database connection string, format: service://user:pwd@host/database. "service" is the name of the database driver which can be queried with "--db-driver-list".

--config-module arg

The config module to use.

--inventory-db arg

Load the inventory from the given database or file, format: [service://]location .

--config-db arg

Load the configuration from the given database or file, format: [service://]location .

--o, --output arg

If connected to scmaster, this flag defines the database connection to use for writing. The configured application database connection (as received from scmaster) is reported to clients as part of a database response messages.

Import

-i, --input arg

Define the import XML file to be written to database. Can be provided multiple times to import multiple files.

-b, --batchsize arg

Batch size of a database transactions [0..1000]. By default all INSERT/UPDATE/DELETE statements are executed in one single transaction.