gempa GmbH

Table Of Contents

This Page

sceval

Module to evaluate origins received from messaging

Description

sceval provides different evaluation methods to identify and to flag real and fake origins or to identify origins that have particular properties. Flagging is achieved by changing the origin status. Comments are set to document properties and the method. Origins from real and fake events are flagged as confirmed and rejected, respectively. Origins that cannot be flagged by one of the evaluation methods are kept unflagged.

The evaluation methods are (performed in the following order):

  1. Number of used phases
  2. Minimum depth
  3. Maximum depth
  4. Maximum azimuthal gap
  5. Extended gap criterion
  6. Station - Distance evaluation (main evaluation method)

The evaluation methods declare confirmation or rejection of an origin. If neither method allows confirmation or rejection, the origin status remains unchanged. The evaluation is finished when one method explicitely declares the status of an origin.

By default manually processed origins are excluded from evaluation. Manually processed origins can be explicitly evaluated by setting the origin.manual parameter.

For evaluation of origins by their arrivals only arrivals with weight > 0 are considered.

sceval also provides the plugin evsceval for scevent for evaluating origin comments and status.

QC parameters

During real-time application sceval accounts for relevant data conditions. sceval only considers arrivals of stations with a configurable latency and delay configured in qc.parameters and streams accepted by the module configured in setupName and included by streamMask. setupName is used to define the module which delivers the picks. In this way sceval can be used in different pipelines with different station setup.

The quality control paramters qc.parameters and streamMask determine if stations are included in the Station - Distance statistic. scqc must be running.

Evaluation methods

Minimum number of used phases

Declares: rejection

An origin is rejected if the number of used phases (arrivals) is less than minPhase. Unused phases, i.e. phases with zero weight, are not counted. minPhase may be over-ruled by distanceProfilesMinPhase.

Background

Setting minPhase allows locator tools such as scautoloc or scanloc to publish origins with only few arrivals. Those origins can be later safely rejected by sceval but are kept in the database for manual evaluation.

Minimum origin depth

Declares: rejection

An origin is rejected if the depth of the origin is less than minDepth.

Maximum origin depth

Declares: rejection

An origin is rejected if the depth of the origin is larger than maxDepth.

Background

Seismic networks sometimes detect teleseismic phases from large events. The location from such detections may result in very deep events. The same is true for networks monitoring mines or blasts. Those origins can be later safely rejected by sceval but are kept in the database for manual evaluation.

Maximum azimuthal gap

Declares: property - sets the comment maxGap.

Origins receive the comment "maxGap" if the maximum azimuthal gap exceeds maxGap. The comment is evaluated by scevent if the plugin evsceval loaded.

Background

The maximum azimuthal gap is the largest azimuthal distance between two neighboring stations with respect to the origin. The stations are sorted by their azimuth (Figure: Gap criterion).

_images/gapcriterion.png

Extended gap criterion

Declares: confirmation

Origins are confirmed if the number of used observed phases (arrivals) \ge: gapMinPhase AND the maximum azimuthal gap \le: maxGap. Setting gapMinPhase = 0 disables the extended gap criterion. Unused phases, i.e. phases with zero weight, are not counted.

Background

Origins meeting the gap criterion may be later rejected, e.g. by the Station - Distance evaluation. Rejection may even happen for real events when the epicentral distances to the closest stations are large, e.g. for remote earthquakes at mid-ocean ridges.

Origins of those events may be explicitely confirmed by the extended gap criterion.

Station - Distance evaluation

Declares: rejection or confirmation

The station - distance evaluation is only performed if distanceProfiles contains at least one valid profile and if the origin has a higher number of arrivals with non-zero weight than distanceProfilesMinPhase. sceval currently only evaluates stations that have P-wave arrivals.

The number of available stations within customizable distance intervals are compared with the number of stations that have picked arrivals of P waves associated to the tested origin. The distance intervals and the weights for the intervals are given by distanceProfile.$name.

Background

For real origins, thus real seismic events, the number of arrivals typically decreases with distance as compoared to the number of available stations (Figure: real event).

_images/sceval_real_event.png

Example of an origin for a real seismic event with number of available stations, picks (arrivals) and the distance-dependent weights (distanceProfile.$name.weights).

For fake origins, distant stations may have a proportionally higher number of arrivals as compared to close stations (Figure: fake event).

_images/sceval_fake_event.png

Example of a fake origin with number of available stations, picks (arrivals) and the distance-dependent weights (distanceProfile.$name.weights).

Based on the distance profile a mismatchScore is computed and compared against limits in order to origins confirm as real seismic events or to reject origins. For a mismatchScore lower or equal to mismatchScore.confirmed, the origin is flaged confirmed. For a mismatchScore lower or equal to mismatchScore.rejected, the origin is flaged rejected.

It is expected that closer stations have a higher chance to detect real earthquakes thus have a higher chance to have arrivals.

The mismatchScore is therefore calculated as a distance-dependent weighted mismatchScore comparing the number of available stations with the number of stations that have arrivals in a weighted scheme.

mismatchScore =\  &\frac{\sum_{i=1}^{10}weight_i*\frac{stationCount_i - arrivalCount_i}{stationCount_i}}{\sum_{i=1}^{10}weight_i}

where

  • i is the index over the intervals within the configurable distance range,
  • stationCount is the number of available stations,
  • arrivalCount is the number of associated arrivals and
  • weight is the configurable weight (distanceProfile.$name.weights)

within a defined distance interval.

Setup

Distance-dependent weight profiles for distances less than distanceProfile.$name.max can be defined and included by the profile name given using distanceProfiles.

The usage of the distance-dependent weight profile is controlled by the largest epicentral distance of stations with P phase picks for a considered event. The profile among all profiles is used that has the smallest distanceProfile.$name.max that is larger than the largest epicentral distance. If the largest epicentral distance is larger than the largest distanceProfile.$name.max from all available profiles, then the default profile is used as non-configurable fallback.

Note

Parameters of the default profile:

distanceProfile.$name.max = 180.0
distanceProfile.$name.weights = "1.0,0.75,0.5,0.25,0.01,0.01,0.01,0.01,0.01,0.01".

The weights (distanceProfile.$name.weights) are configured for equally-spaced distance intervals. The distance intervals are dynamically bound between 0 degree and the maximum epicentral distance for a considered event. The number of distance intervals is given by the number of weights (distanceProfile.$name.weights). Typically, 10 intervals are used. Higher weight should be given to lower distances.

Origin comments

The different algorithms write comment fields into the origin to identify the processing. When the origin status is changed by an algorithm, a comment indicating the algorithm is added to the origin parameters.

  1. scevalMethod - the evaluation algorithm that has changed the origin status
  2. mismatchScore - the mismatch score resulting from the Station - Distance evaluation
  3. maxGap - comment indicating that the station GAP exceeds the configured maxGap.

Note

Configuration example, e.g. for scolv.cfg or global.cfg

eventlist.customColumn = "mismatchScore"

# Define the default value if no comment is present
eventlist.customColumn.default = "-"

# Define the comment id to be used
eventlist.customColumn.originCommentID = mismatchScore

Plugin: evsceval

evsceval is a plugin provided by sceval for scevent.

Description

evsceval sets the type and the certainty of an event by evaluating:

  1. Comments check: The comments found for the preferred origin of an event. Such comment can be set, e.g. by sceval by evaluating maxGap.

    If a comment is recognized, the event type is set to the configured value. evsceval sets the event type in the following order:

    1. maxGap comment found
  2. Percentage check: The percentage of rejected origins of an event if the event has no manual origins.

    If the percentage exceeds a sceval.rejected the event certainty is set to suspected.

  3. Multiple-agency check: checking for multiple agencyIDs of all origins contained in the event.

    The plugin searches the agencyID of all origins related to an event with the agencies defined in sceval.multipleAgency.targetAgency. If the event contains origins with different agencyIDs, an agencyID with one of the defined names is found and the origin is neither manual nor the status is rejected, confirmed or final, the origin status is set to the value configured in sceval.multipleAgency.originStatus. The changed origin status can be used as an additional criterion for further processing such as publication on websites.

Note

Besides evsceval scevent can load other plugins such as evrc that comes with scevent. If other plugins also set the event type, then the event type is overwritten in the order of which the plugins are configured and loaded by scevent.

Configuration

The evsceval plugin is configured in the module configuration of scevent.

  1. Load the evsceval plugin by adding to the global configuration or to the global configuration of scevent.
  2. Configure sceval.maxGap defining the event type set if the preferred origin of an event has a maxGap comment. sceval sets the maxGap comment if the GAP criterion is met. Currently supported value are: "not existing", "not locatable","outside of network interest","earthquake", "induced earthquake","quarry blast","explosion","chemical explosion", "nuclear explosion","landslide","rockslide","snow avalanche","debris avalanche", "mine collapse","building collapse","volcanic eruption","meteor impact","plane crash", "sonic boom","duplicate","other".
  3. Configure sceval.rejected to set the event certainty.
  4. Configure the parameters related to the multiple-agency check.
plugins = ${plugins},evsceval

sceval.maxGap = "not locatable"

sceval.rejected = 25

sceval.multipleAgency.targetAgency = agency, agency1
sceval.multipleAgency.originStatus = reported

Note

scevent fails to set the event tpye if the evsceval plugin is loaded but sceval.maxGap is not defined.

Tuning: sceval-tune

sceval provides the auxiliary tool sceval-tune as a wrapper to run sceval in tuning mode. The tuning mode assists in providing the configuration parameters for a particular profile of the distanceProfiles in the Station - Distance evaluation. In tuning mode sceval considers manually evaluated events/origins to find the best-performing weight profile and threshold values for automatic confirmation or rejection of origins by sceval. The profile and threshold values can be used to configure sceval.

Known limitations

In the current version the tuning mode has limitations:

  • The extended GAP evaluation is not considered.
  • QC parameters are not considered.

Work Flow

Recommended work flow for tuning of sceval:

  1. Manually review and evaluate the origin status for many events. Choose "confirmed", "rejected" or leave the Origin status unflagged.

    Use scolv to manually evaluate the preferred origins of events by setting the origin status. During evaluation set the origin status to confirmed, rejected or unset (if flagging is not reasonable). For tuning of sceval do not add new picks on stations that were not picked and associated by the automatic system. Otherwise the origins with arrivals will not be representative of the automatic system. The considered time window should be long enough to contain a significant number of each rejected and confirmed origins, e.g. 50 rejected and 50 confirmed origins.

  2. Create an event XML file from the events that are manually reviewed.

    Generate the event XML file, e.g. eventlist.xml, from the evaluated events. Choose any file name. The file name is provided through parameters.

    Use scxmldump to dump the events into a XML file. Choose any file name. The file name is provided to sceval through parameters.

  3. Create a configuration XML file containing the system configuration.

    Use scxmldump to dump the system configuration into a XML file. Choose any file name. The file name is provided to sceval through parameters.

  4. Create an inventory XML file

    Use scxmldump to dump the staton inventory into a XML file. Choose any file name. The file name is provided to sceval through parameters.

    Note

    Example code for generating the XML files in the previous steps.

    #!/bin/bash
    
    # database
    db=mysql://sysop:sysop@localhost/seiscomp3
    
    # dump events
    evtList=`scevtls -d $db --begin '2016-05-22 12:00:00' --end '2016-06-30 23:59:59' |\
    awk '{for (i=1;i<=NF;i++)  printf("%s,", $i)}' | sed 's/,$//'`
    
    scxmldump -d $db -E $evtList -PAMf -o eventlist.xml
    # dump configuration
    scxmldump -d $db -C -o config.xml
    # dump inventory
    scxmldump -d $db -I -o inventory.xml
    
  5. Generate the list of weight profiles.

    Use sceval-tune to create a list of profiles considered for finding the best-performing weigth profiles. Choose a sufficient number of distance intervals, e.g. 5 - 10 intervals and a list of weight values. sceval-tune will find all meaningful permutations of weights for all distance intervals. Default value may be used. Save the profile listlist in a file. Review and adjust the profile list if desired.

    Required parameters for generating the list of profiles by sceval-tune:

    • -g

      generate the list of profiles

    • -f arg

      arg: name of the file containing the created profile list.

    Optional parameters:

    • -n arg

      arg: Maximum number of distance intervals. Uses default if -n is not set. Default: 10

    • -w arg

      arg: Weight profile to be tested. Uses default if -w is not set. Default: 1,0.75,0.5,0.25,0.1,0.01

    See the help sceval-tune -h for the relevant parameters to create the profile list.

    Note

    Help option and example to create the list of weight profiles:

    sceval-tune -h
    sceval-tune -f profiles.txt -g -n 6 -w 1,0.5,0.1
    
  6. Tune the configuration parameters for a particular profile of the distanceProfiles.

    Run sceval-tune in a command terminal to find the optimal configuration parameters for one of the distance profiles given in distanceProfiles. Repeat the tuning for all profiles individually. The tuning is relies on a sufficient number of manually confirmed and rejected events.

    For tuning provide the considered weight profiles in the profile list file. These weight profiles are evaluated identifying the best-performing weight profile and the associated threshold values mismatchScore.confirmed and mismatchScore.rejected. The best-performing weight profile and the threshold values are determined by optimizing the automatic wrt. the manual evaluation. As a measure, the mistfit is minimized.

    The configuration parameters and quality parameters are output to the command line. Based on the suggested configuraton parameters, the percentages indicate the number of events that would be confirmed or rejected by the automatic system wrt. the manual evaluation. The main plot figure is generated to review the performance of the tuning and the configuration parameters. The optional additional plot figure shows the results for the 9 best-performing weight profiles.

    Two possible values for mismatchScore.rejected are provided based on the confirmed or on the unflagged origins. Both values should be similar but the value for the confirmed origins should typically be chosen.

    If mismatchScore.confirmed and mismatchScore.rejected are largely separated, many origins may remain unflagged by sceval. Increasing mismatchScore.confirmed or decreasing mismatchScore.rejected will reduce the number of unflagged origins but will possibly result in wrong automatic flagging of real and fake events.

    Required parameters for tuning by sceval-tune:

    • -p arg

      arg: name of the distance profile that is to be tuned. The profile must be configured. The name must be contained in the configuration parameter distanceProfile.

    • -f arg

      arg: name of the file containing the profile list.

    • -ep arg

      arg: name of event XML file

    • --config-db arg

      arg: name of the configuration XML file.

    • --inventory-db arg

      arg: name of the inventory XML file.

    • --time-window arg

      arg: time window of the considered events.

    Optional parameters:

    • -m arg

      arg is the minimum number of phase arrivals in origin to consider the event. Equivalent to minPhase.

    • --addplots

      Show results from the 9 best-matching weight profiles.

    • --debug

    Note

    Example:

    sceval-tune -f profiles.txt --time-window "2016-01-01 01:01:01~2016-06-06 12:00:00" --inventory-db inventory.xml --config-db config.xml --ep events.xml -p 180 -m 6 --addplots --debug
    
  7. Configuration of sceval

    In sceval configure the tuned distance profile using the proposed configuration parameters.

Tuning results

sceval-tune provides the configuration parameters and the tuning statistics on the command terminal where it was executed. In addition, one main and one additional (--addplots) figure are created for visual inspection of the tuning results.

_images/sceval_tune_main.png

Main figure showing results for the best-performing weight profile. Solid line: mismatchScore values based on flagged events. Dashed line: mismatchScore values based on flagged and unflagged events.

_images/sceval_tune_add.png

Additional figure showing results for the 9 best-performing weight profiles. Generated when using --addplots. Solid line: mismatchScore values based on flagged events. Dashed line: mismatchScore values based on flagged and unflagged events.

References

sceval has been demonstrated, promoted and discussed with scientists and the SeisComP3 community at international science conferences, e.g.:

  1. D. Roessler, B. Weber, E. Ellguth, J. Spazier the team of gempa: Evaluierung der Qualität und Geschwindigkeit von Erdbebendetektionen in SeisComP3, 2017, Bad Breisig, Germany, AG Seismologie meeting
  2. D. Roessler, B. Weber, E. Ellguth, J. Spazier the team of gempa: EVALUATION OF EARTHQUAKE DETECTION PERFORMANCE IN TERMS OF QUALITY AND SPEED IN SEISCOMP3 USING NEW MODULES QCEVAL, NPEVAL AND SCEVAL , 2017, New Orleans, USA, AGU Fall Meeting, abstract S13B-0648.

Configuration

etc/defaults/global.cfg
etc/defaults/sceval.cfg
etc/global.cfg
etc/sceval.cfg
~/.seiscomp3/global.cfg
~/.seiscomp3/sceval.cfg

sceval inherits global options.

minPhase

Type: integer

Minimum number of phase arrivals (P or S) used for locating. Origins with fewer arrivals are rejected. Only consider arrivals with weight > 0. Default is 0.

minDepth

Type: double

Unit: km

minimum depth criterion: origins with depth less than minDepth are rejected. Default is -10.0.

maxDepth

Type: double

Unit: km

maximum depth criterion: origins with depth > maxDepth are rejected. Default is 745.0.

maxGap

Type: double

Unit: deg

Gap criterion: maximum allowed azimuthal gap between adjacent stations providing arrivals to one origin. Origins with a larger gap are rejected. Only consider arrivals with weight > 0. Default is 360.0.

gapMinPhase

Type: integer

Extended gap criterion: Origins with gap lower than maxGap and more than gapMinPhases arrivals are confirmed. gapMinPhase = 0: ignore check. Default is 0.

distanceProfiles

Type: list:string

Registration of distance profiles for station - distance evaluation. An empty list disables the station - distance evaluation.

distanceProfilesMinPhase

Type: integer

Minimum number of P-phase arrivals for applying the station - distance evaluation. Only consider arrivals with weight > 0. Default is 0.

qc.parameters

Type: list:string

Defines QC parameters to observe. Each parameter is associated with a value range. If any of the defined ranges is exceeded the corresponding station is disabled. Use '-Inf' resp. 'Inf' if no upper or lower bound should exist. Works with scqc only.

setupName

Type: string

Config setup name used for the initial setup of the active station list. By default the name is global. Default is scautopick.

origin.authorWhiteList

Type: string

Use to filter incoming origins by author. If omitted no filtering is applied.

origin.manual

Type: boolean

Enable/disable evaluation of origins whose evaluation mode is set to MANUAL. Default is disable. Check the box to enable the evaluation. Default is false.

mismatchScore.confirmed

Type: double

Mismatch between active stations and used stations. If the score is less than or equal to the evaluation status is set to CONFIRMED. Default is 0.5.

mismatchScore.rejected

Type: double

Mismatch between active stations and used stations. If the score is greater than or equal to the evaluation status is set to REJECTED. Default is 0.7.

Note

distanceProfile.$name.* $name is a placeholder for the name to be used and needs to be added to distancesProfiles to become active.

distancesProfiles = a,b
distanceProfile.a.value1 = ...
distanceProfile.b.value1 = ...
# c is not active because it has not been added
# to the list of distancesProfiles
distanceProfile.c.value1 = ...
distanceProfile.$name.max

Type: double

Unit: deg

Upper distance

distanceProfile.$name.weights

Type: list:double

Weights to use

Command-line

Generic

-h, --help

show help message.

-V, --version

show version information

--config-file arg

Use alternative configuration file. When this option is used the loading of all stages is disabled. Only the given configuration file is parsed and used. To use another name for the configuration create a symbolic link of the application or copy it, eg scautopick -> scautopick2.

--plugins arg

Load given plugins.

-D, --daemon

Run as daemon. This means the application will fork itself and doesn't need to be started with &.

Verbosity

--verbosity arg

Verbosity level [0..4]. 0:quiet, 1:error, 2:warning, 3:info, 4:debug

-v, --v

Increase verbosity level (may be repeated, eg. -vv)

-q, --quiet

Quiet mode: no logging output

--print-component arg

For each log entry print the component right after the log level. By default the component output is enabled for file output but disabled for console output.

--component arg

Limits the logging to a certain component. This option can be given more than once.

-s, --syslog

Use syslog logging back end. The output usually goes to /var/lib/messages.

-l, --lockfile arg

Path to lock file.

--console arg

Send log output to stdout.

--debug

Debug mode: --verbosity=4 --console=1

--trace

Trace mode: --verbosity=4 --console=1 --print-component=1 --print-context=1

--log-file arg

Use alternative log file.