scdbstrip¶
Database clean-up of processing results.
Description¶
SeisComP’s scmaster is continuously writing to the database. This causes the database to grow and to occupy much space on the harddisc. scdbstrip taggles this problem and removes processed objects from the database older than a configurable time span. The time comparison considers the object time, not the time of their creation.
This clean-up procedure is based on events. scdbstrip will remove all events with an origin time older than specified. It will also remove all associated objects such as picks, origins, arrivals, amplitudes and so on.
scdbstrip does not run as a daemon. To remove old objects continuously scdbstrip should be added to the list of cronjobs running every e.g. 30 minutes. The more often it runs the less objects it has to remove and the faster it will unlock the database again.
Known issues¶
When running scdbstrip for the first time on a large database it can happen that it aborts in case of MYSQL with the following error message:
[ 3%] Delete origin references of old events...08:48:22 [error]
execute("delete Object from Object, OriginReference, old_events where
Object._oid=OriginReference._oid and
OriginReference._parent_oid=old_events._oid") = 1206 (The total number
of locks exceeds the lock table size)
Exception: ERROR: command 'delete Object from Object, OriginReference,
old_events where Object._oid=OriginReference._oid and
OriginReference._parent_oid=old_events._oid' failed
That means your MYSQL server cannot hold enough data required for deletion. There are two solutions to this:
Increase the memory pool used by MYSQL by changing the configuration to:
innodb_buffer_pool_size = 64M
The size of the new buffer depends on the size of the database that should be cleaned up.
Run scdbstrip on smaller batches for the first time:
$ scdbstrip -d seis:mypass@localhost/seiscomp --days 1000 $ scdbstrip -d seis:mypass@localhost/seiscomp --days 900 ... $ scdbstrip -d seis:mypass@localhost/seiscomp --days 100
Examples¶
Keep the events of the last 30 days
scdbstrip --days 30 -d mysql://sysop:sysop@localhost/seiscomp
Configuration¶
etc/defaults/global.cfg
etc/defaults/scdbstrip.cfg
etc/global.cfg
etc/scdbstrip.cfg
~/.seiscomp/global.cfg
~/.seiscomp/scdbstrip.cfg
scdbstrip inherits global options.
Note
database.cleanup.keep.* Parameters controlling the time to keep objects in the database. The time comparison considers the object time, not the time of their creation.
-
database.cleanup.keep.days
¶ Type: int
The number of days to preserve in the database. This value is added to the whole timespan. Hours and minutes are configured separately. Default is
30
.
-
database.cleanup.keep.hours
¶ Type: int
The number of hours to preserve in the database. This value is added to the whole timespan. Days and minutes are configured separately. Default is
0
.
-
database.cleanup.keep.minutes
¶ Type: int
The number of minutes to preserve in the database. This value is added to the whole timespan. Days and hours are configured separately. Default is
0
.
Command-line¶
Generic¶
-
-h
,
--help
¶
show help message.
-
-V
,
--version
¶
show version information
-
--config-file
arg
¶ Use alternative configuration file. When this option is used the loading of all stages is disabled. Only the given configuration file is parsed and used. To use another name for the configuration create a symbolic link of the application or copy it, eg scautopick -> scautopick2.
-
--plugins
arg
¶ Load given plugins.
-
-D
,
--daemon
¶
Run as daemon. This means the application will fork itself and doesn’t need to be started with &.
Verbosity¶
-
--verbosity
arg
¶ Verbosity level [0..4]. 0:quiet, 1:error, 2:warning, 3:info, 4:debug
-
-v
,
--v
¶
Increase verbosity level (may be repeated, eg. -vv)
-
-q
,
--quiet
¶
Quiet mode: no logging output
-
--component
arg
¶ Limits the logging to a certain component. This option can be given more than once.
-
-s
,
--syslog
¶
Use syslog logging back end. The output usually goes to /var/lib/messages.
-
-l
,
--lockfile
arg
¶ Path to lock file.
-
--console
arg
¶ Send log output to stdout.
-
--debug
¶
Debug mode: –verbosity=4 –console=1
-
--log-file
arg
¶ Use alternative log file.
Database¶
-
--db-driver-list
¶
List all supported database drivers.
-
-d
,
--database
arg
¶ The database connection string, format: service://user:pwd@host/database. “service” is the name of the database driver which can be queried with “–db-driver-list”.
-
--config-module
arg
¶ The configmodule to use.
-
--inventory-db
arg
¶ Load the inventory from the given database or file, format: [service://]location
-
--db-disable
¶
Do not use the database at all
Settings¶
-
--days
arg
¶ Overrides configuration parameter
database.cleanup.keep.days
. The number of days to keep. Hours and minutes are also used the compute the whole time span.
-
--hours
arg
¶ Overrides configuration parameter
database.cleanup.keep.hours
. The number of hours to keep. Days and minutes are also used the compute the whole time span.
-
--minutes
arg
¶ Overrides configuration parameter
database.cleanup.keep.minutes
. The number of minutes to keep. Days and hours are also used the compute the whole time span.
-
--datetime
arg
¶ Replaces the days:hours:minutes timespan definition by an arbtrary absolute timestamp in UTC. The format is %Y-%m-%d %H:%M:%S.
-
-i
,
--invert
¶
Delete all events after the specified time period and not before.
-
--keep-events
¶
Event-IDs to keep in the database separated with comma.