scdbstrip

Database clean-up of processing results.

Description

SeisComP3s processing is continuously writing to the database. This causes the database to grow and to occupy much space on the harddisc. scdbstrip taggles this problem and removed processed objects from the database older than a configurable time span.

This clean-up procedure is based on events. scdbstrip will remove all events with an origin time older than specified. It will also remove all associated objects such as picks, origins, arrivals, amplitudes and so on.

scdbstrip does not run as a daemon. To remove old objects continuously scdbstrip should be added to the list of cronjobs running every e.g. 30 minutes. The more often it runs the less objects it has to remove and the faster it will unlock the database again.

Known issues

When running scdbstrip for the first time on a large database it can happen that it aborts in case of MYSQL with the following error message:

[  3%] Delete origin references of old events...08:48:22 [error]
execute("delete Object from Object, OriginReference, old_events where
Object._oid=OriginReference._oid and
OriginReference._parent_oid=old_events._oid") = 1206 (The total number
of locks exceeds the lock table size)

Exception: ERROR: command 'delete Object from Object, OriginReference,
old_events where Object._oid=OriginReference._oid and
OriginReference._parent_oid=old_events._oid' failed

That means your MYSQL server cannot hold enough data required for deletion. There are two solutions to this:

  1. Increase the memory pool used by MYSQL by changing the configuration to:

    innodb_buffer_pool_size = 64M

    The size of the new buffer depends on the size of the database that should be cleaned up.

  2. Run scdbstrip on smaller batches for the first time:

    $ scdbstrip -d seis:mypass@localhost/seiscomp3 --days 1000
$ scdbstrip -d seis:mypass@localhost/seiscomp3 --days 900
...
$ scdbstrip -d seis:mypass@localhost/seiscomp3 --days 100

Examples

  1. Keep the events of the last 30 days

    scdbstrip --days 30 -d mysql://sysop:sysop@localhost/seiscomp3

Configuration

etc/defaults/global.cfg
etc/defaults/scdbstrip.cfg
etc/global.cfg
etc/scdbstrip.cfg
~/.seiscomp3/global.cfg
~/.seiscomp3/scdbstrip.cfg

scdbstrip inherits global options.

database.cleanup.keep.days

Type: int

The number of days to preserve in the database. This value is just one part of the whole timespan. Hours and minutes are configured separately. Default is 30.

database.cleanup.keep.hours

Type: int

The number of hours to preserve in the database. This value is just one part of the whole timespan. Days and minutes are configured separately. Default is 0.

database.cleanup.keep.minutes

Type: int

The number of minutes to preserve in the database. This value is just one part of the whole timespan. Days and hours are configured separately. Default is 0.

Command-line

Generic

-h, --help

show help message.

-V, --version

show version information

--config-file arg

Use alternative configuration file. When this option is used the loading of all stages is disabled. Only the given configuration file is parsed and used. To use another name for the configuration create a symbolic link of the application or copy it, eg scautopick -> scautopick2.

--plugins arg

Load given plugins.

-D, --daemon

Run as daemon. This means the application will fork itself and doesn’t need to be started with &.

--first-new

Overrides configuration parameter firstNew.

Verbosity

--verbosity arg

Verbosity level [0..4]. 0:quiet, 1:error, 2:warning, 3:info, 4:debug

-v, --v

Increase verbosity level (may be repeated, eg. -vv)

-q, --quiet

Quiet mode: no logging output

--component arg

Limits the logging to a certain component. This option can be given more than once.

-s, --syslog

Use syslog logging back end. The output usually goes to /var/lib/messages.

-l, --lockfile arg

Path to lock file.

--console arg

Send log output to stdout.

--debug

Debug mode: –verbosity=4 –console=1

--log-file arg

Use alternative log file.

Database

--db-driver-list

List all supported database drivers.

-d, --database arg

The database connection string, format: service://user:pwd@host/database. “service” is the name of the database driver which can be queried with “–db-driver-list”.

--config-module arg

The configmodule to use.

--inventory-db arg

Load the inventory from the given database or file, format: [service://]location

--db-disable

Do not use the database at all

Settings

--days arg

Overrides configuration parameter database.cleanup.keep.days. The number of days to keep. Hours and minutes are also used the compute the whole time span.

--hours arg

Overrides configuration parameter database.cleanup.keep.hours. The number of hours to keep. Days and minutes are also used the compute the whole time span.

--minutes arg

Overrides configuration parameter database.cleanup.keep.minutes. The number of minutes to keep. Days and hours are also used the compute the whole time span.

--datetime arg

Replaces the days:hours:minutes timespan definition by an arbtrary absolute timestamp in UTC. The format is %Y-%m-%d %H:%M:%S.

-i, --invert

Delete all events after the specified time period and not before.

--keep-events

Event-IDs to keep in the database separated with comma.

Mode

--check
--clean-unused