# Change Log All notable changes to CAPS will be documented in this file. Please note that we have changed the date format from year-month-day to year.dayofyear to be in sync with `caps -V`. ## 2024.290 - Add Option to purge data via CAPS protocol API. Only users with the `purge` permission can delete data from archive. ## 2024.269 - Fixed crash on inserting data under some still unclear circumstances. ## 2024.253 - Add more robust checks to detect corrupted files caused by, e.g. faulty storages or hardware failures/crashes. Corrupt files could have caused segmentation faults of `caps`. ## 2024.215 - Fix webfrontend bug if `AS.http.fdsnws` is specified. This bug prevented the webfrontend to load. ## 2024.183 - Add record filter options to rifftool data dump mode ## 2024.151 - Improve logging for plugin port: add IP and port to disconnect messages and log disconnection requests from the plugin to INFO level. ## 2024.143 - Fix issue with merging raw records after a restart ## 2024.096 - Attempt to fix dashboard websocket standing connection counter ## 2024.094 - Fix errors when purging a datafile which is still active ## 2024.078 - Ignore records without start time and/or end time when rebuilding the index of a data file. ## 2024.066 - Ignore packets with invalid start and/or end time - Fix rifftool with respect to checking data files with check command: ignore invalid times. - Add corrupted record and chunk count to chunks command of rifftool. ## 2024.051 - Fix frontend storage time per second scale units - Fix frontend real time channel display update - Fix overview plot update when locking time time range ## 2024.047 - Update frontend ## 2024.024 - Update frontend ## 2024.022 - Add support for additional web applications to be integrated into the web frontend ## 2023.355 - Update web frontend - Close menu on channels page on mobile screens if clicked outside the menu ## 2023.354 - Update web frontend - Improve rendering on mobile devices ## 2023.353 - Update web frontend - Server statistics is now the default page - The plot layout sticks the time scale to the bottom - Bug fixes ## 2023.348 - Add support for `info server modified after [timestamp]` - Update web frontend ## 2023.347 - Some more internal optimizations ## 2023.346 - Fix bug in basic auth implementation that caused all clients to disconnect when the configuration was reloaded. ## 2023.331 - Correct system write time metrics ## 2023.328 - Extend notification measuring ## 2023.327 - Fix crash with `--read-only`. - Improve input rate performance with many connected clients. ## 2023.326 - Internal optimization: distribute notification handling across multiple CPUs to speed up handling many connections (> 500). - Add notification time to storage time plot ## 2023.325 - Internal optimization: compile client session decoupled from notification loop. ## 2023.321 - Decouple data disc storage from client notifications. This will increase performance if many real-time clients are connected. A new parameter has been added to control the size of the notification queue: `AS.filebase.params.q`. The default value is 1000. ## 2023.320 - Add file storage optmization which might be useful if dealing with a large amount of channels. In particular `AS.filebase.params.writeMetaOnClose` and `AS.filebase.params.alignIndexPages` have been added in order to reduce the I/O bandwidth. - Add write thread priority option. This requires the user who is running CAPS to be able to set rtprio, see limits.conf. ## 2023.312 - Do not block if inventory is being reloaded ## 2023.311 - Add average physical storage time metric ## 2023.299 - Fix storage time statistics in combination with client requests - Improve statistics plot in web frontend ## 2023.298 - Add storage time per package to statistics ## 2023.241 - Fix protocol orchestration for plugins in combination with authentication ## 2023.170 - Add section on retrieval of data availability to documentation ## 2023.151 - Fix crash in combination with invalid HTTP credential format ## 2023.093 - Add note to documentation that inventory should be enabled in combination with WWS for full support. ## 2023.062 - Add documentation of rifftool which is available through the separate package 'caps-tools'. ## 2023.055 - Internal cleanups ## 2023.024 - Fix crash if requested heli filter band is out of range - Improve request logging for heli requests ## 2023.011 ### Changed - Change favicon, add SVG and PNG variants ## 2023.011 ### Fixed - Client connection statistics ## 2023.010 ### Fixed - Crash in combination with websocket data connections ## 2023.004 ### Fixed - Reload operation with respect to access changes. Recent versions crashed under some circumstances. ## 2022.354 ### Added - Show connection statistics in the frontend ## 2022.349 ### Changed - Improved read/write schedular inside CAPS to optimize towards huge number of clients ## 2022.346 ### Fixed - Fixed statistics calculation in `--read-only` mode. ## 2022.342 ### Added - Read optional index files per archive directory during startup which allow to skip scanning the directory and only rely on the index information. This can be useful if mounted read-only directories should be served and skipped from possible scans to reduce archive scan time. ## 2022.341 ### Changed - Improve start-up logging with respect to archive scanning and setup. All information go to the notice level and will be logged irrespective of the set log level. - Add configuration option to define the path of the archive log file, `AS.filebase.logFile`. ## 2022.334 ### Fixed - Fixed bug which prevented forwarding of new channels in combination with wildcard requests. ## 2022.333 ### Changed - Improve websocket implementation ## 2022.332 ### Changed - Increase reload timeout from 10 to 60s ## 2022.327 ### Fixed - Fixed invalid websocket frames sent with CAPS client protocol - Fixed lag in frontend when a channel overview reload is triggered ## 2022.322 ### Added - Added system error message if a data file cannot be created. - Try to raise ulimit to at least cached files plus opened files and terminate if that was not successful. ## 2022.320 ### Fixed - Fixed storage of overlapping raw records which overlap with gaps in a data file. ## 2022.314 ### Fixed - Fixed trimming of raw records while storing them. If some samples were trimmed then sometimes raw records were merged although the do not share a common end and start time. ## 2022.307 ### Fixed - Fixed deadlock in combination with server info queries ## 2022.284 ### Fixed - Fixed segment resolution evaluation in frontend ## 2022.278 ### Fixed - Fixed memory leak in combination with some gap requests ## 2022.269 ### Fixed - Memory leak in combination with request logs. ### Changed - Removed user `FDSNWS` in order to allow consistent permissions with other protocols. The default anonymous access is authenticated as guest. Furthermore HTTP Basic Authentication can be used to authenticate an regular CAPS user although that is not part of the FDSNWS standard. This is an extension of CAPS. If you have set up special permission for the FDSNWS user then you have to revise them. The rationale behind this change is (as stated above) consistency. Furthermore the ability to configure access based on IP addresses drove that change. If CAPS authenticates any fdsnws request as user `FDSNWS` then IP rules are not taken into account. Only anonymous requests are subject to IP based access rules. We do not believe that the extra `FDSNWS` user added any additional security. ## 2022.265 ### Fixed - Crash in combination with MTIME requests. ## 2022.262 ### Added - Added modification time filter to stream requests. This allows to request data and segment which were available at a certain time. ## 2022-09-06 ### Improved - Improved frontend performance with many thousands of channels and high segmentation. ### Fixed - Fixed time window trimming of raw records which prevented data delivery under some very rare circumstances. ## 2022-09-02 ### Added - List RESOLUTION parameter in command list returned by HELP on client interface. ## 2022-08-25 ### Changed - Allow floating numbers for slist format written by capstool. ## 2022-08-25 ### Important - Serve WebSocket requests via the regular HTTP interface. The configuration variables `AS.WS.port` and `AS.WS.SSL.port` have been removed. If WebSocket access is not desired then the HTTP interface must be disabled. - Reworked the HTTP frontend which now provides display of channel segments, cumulative station and network views and a view with multiple traces. - In the reworked frontend, the server statistics are only available to users which are member of the admin group as defined by the access control file configured in `AS.auth.basic.users.passwd`. ## 2022-08-16 ### Added - Open client files read-only and only request write access if the index needs to be repaired or other maintenance operations must be performed. This makes CAPS work on a read-only mounted file system. ## 2022-07-12 ### Fixed - Fixed HELI request with respect to sampling rate return value. It returned the underlying stream sampling rate rather than 1/1. ## 2022-06-10 ### Fixed - Improve bad chunk detection in corrupt files. Although CAPS is pretty stable when it comes to corrupted files other tools might not. This improvement will trigger a file repair if a bad chunk has been detected. ## 2022-06-07 ### Fixed - Infinite loop if segments with resolution >= 1 were requested. ## 2022-05-30 ### Added - Add "info server" request to query internal server state. ## 2022-05-18 ### Fixed - Fix possible bug in combination with websocket requests. The issue exhibits as such as the connection does not respond anymore. Closing and reopening the connection would work. ## 2022-05-09 ### Added - Add gap/segment query. ## 2022-04-26 ### Important - With this release we have split the server and the tools - riffdump - riffsniff - rifftest - capstool into separate packages. We did this because for some use cases it make sense to install only these tools. The new package is called `caps-tools` and activated for all CAPS customers. ## 2022-03-28 ### Changed - Update command-line help for capstool. ## 2022-03-03 ### Added - Log plugin IP and port on accept. - Log plugin IP and port on package store error. ## 2021-12-20 ### Added - Explain record sorting in capstool documentation. ## 2021-11-09 ### Fixed - Fixed helicorder request in combination with filtering. The issue caused wrong helicorder min/max samples to be returned. ## 2021-10-26 ### Fixed - Fixed data extraction for the first record if it does not intersect with the requested time window. ## 2021-10-19 ### Changed - Update print-access help page entry - Print help page in case of unrecognized command line options ### Fixed - Do not print archive stats when the help page or version information is requested ## 2021-09-20 ### Fixed - Fixed crash if an FDSNWS request with an empty compiled channel list has been made ## 2021-09-17 ### Added - New config option `AS.filebase.purge.referenceTime` defining which reference time should be used while purge run. Available are: - EndTime: The purge run uses the end time per stream as reference time. - Now: The purge run uses the current time as reference time. By default the purge operation uses the stream end time as reference time. To switch to **Now** add the following entry to the caps configuration. ```config AS.filebase.purge.referenceTime = Now ``` ## 2021-05-03 ### Changed - Log login and logout attempts as well as blocked stream requests to request log. - Allow whitespaces in passwords. ## 2021-04-15 ### Fixed - Rework CAPS access rule evaluation. ### Changed - Comprehensive rework of CAPS authentication feature documentation. ## 2021-03-11 ### Important - Reworked data file format. An high performance index has been added to the data files which require an conversion of the data files. See CAPS documentation about upgrading. The conversion is done transparently in the background but could affect performance while the conversion is in progress. ## 2020-10-12 ### Added - Provide documentation of the yet2caps plugin. ## 2020-09-04 ### Fixed - Fixed gaps in helicorder request. ## 2020-07-01 ### Fixed - Don't modify stream start time if the assoicated data file couldn't deleted while purge run. This approach makes sure that stream start time and the data files are kept in sync. ## 2020-02-24 ### Added - Extended purge log. The extended purge log can be enabled with the configuration parameter `AS.logPurge`. This feature is not enabled by default. ### Changed - Log maximum number of days to keep data per stream at start. ## 2020-01-27 ### Fixed - Typo in command line output. ## 2019-11-26 ### Added - Added new command line option `configtest that runs a configuration file syntax check. It parses the configuration files and either reports Syntax OK or detailed information about the particular syntax error. - Added Websocket interface which accepts HTTP connections (e.g. from a web browser) and provides the CAPS protocol via Websockets. An additional configuration will be necessary: ```config AS.WS.port = 18006 # Provides the Websocket interface via secure sockets layer. # The certificate and key used will be read from # AS.SSL.certificate and AS.SSL.key. AS.WS.SSL.port = 18007 ``` ### Changed - Simplified the authorization configuration. Instead of using one login file for each CAPS interface we read the authentication information from a shadow file. The file contains one line per user where each line is of format "username:encrypted_pwd". To encrypt a password mkpasswd can be used. It is recommended to apply a strong algorithm such as sha-256 or sha-512. The command "user=sysop pw=`mkpasswd -m sha-512` && echo $user:$pw" would generate a line for e.g. user "sysop". The shadow file can be configured with the config option `AS.users.shadow`. Example: ```config # The username is equal to the password test:$6$jHt4SqxUerU$pFTb6Q9wDsEKN5yHisPN4g2PPlZlYnVjqKFl5aIR14lryuODLUgVdt6aJ.2NqaphlEz3ZXS/HD3NL8f2vdlmm0 user1:$6$mZM8gpmKdF9D$wqJo1HgGInLr1Tmk6kDrCCt1dY06Xr/luyQrlH0sXbXzSIVd63wglJqzX4nxHRTt/I6y9BjM5X4JJ.Tb7XY.d0 user2:$6$zE77VXo7CRLev9ly$F8kg.MC8eLz.DHR2IWREGrSwPyLaxObyfUgwpeJdQfasD8L/pBTgJhyGYtMjUR6IONL6E6lQN.2QLqZ5O5atO/ ``` In addition to user authentication user access control properties are defined in a passwd file. It can be configured with the config option `AS.users.passwd`. Each line of the file contains a user name or a group id and a list of properties in format "username:prop1,prop2,prop3". Those properties are used to grant access to certain functionalities. Currently the following properties are supported by CAPS: read, write.: "read and write.". By default a anonymous user with read and write permissions exists. Groups use the prefix **%** so that they are clearly different from users. Example: ```config user1: read,write %test: read ``` The group file maps users to different groups. Each line of the file maps a group id to a list of user names. It can be configured with the config option `AS.users.group`. Example: ```config test: user2 ``` With the reserved keyword **ALL** a rule will be applied to all users. Example: ```config STATIONS.DENY = all STATIONS.AM.ALLOW = user1 ``` - We no longer watch the status of the inventory and the access file with Inotify because it could be dangerous in case of an incomplete saved configuration. A reload of the configuration can be triggered by sending a SIGUSR1 signal to the CAPS process. Example: ```bash kill -SIGUSR1 ``` CAPS reloads the following files, if necessary: - shadow, - passwd, - access list, - inventory. ## 2019-10-15 ### Changed - Run archive clean up after start and every day at midnight(UTC). ## 2019-10-01 ### Changed - Increase shutdown timeout to 60 s. ## 2019-05-08 ### Fixed - Fixed potential deadlock in combination with inventory updates. ## 2019-04-23 ### Fixed - Improved plugin data scheduling which could have caused increased delays of data if one plugin transmits big amounts of data through a low latency network connection, e.g. localhost. ## 2019-04-08 ### Added - Added new config option `AS.filebase.purge.initIdleTime` that allows to postpone the initial purge process up to n seconds. Normally after a start the server tries to catch up all data which might be an IO intensive operation. In case of a huge archive the purge operation slow downs the read/write performance of the system too. To reduce the load at start it is a good idea to postpone this operation. ## 2019-03-29 ### Added - Added index file check during archive scan and rebuild them if corrupt. The lack of a check sometimes caused CAPS to freeze while starting up. ## 2018-12-11 ### Added - Added support for SC3 schema 0.11. ## 2018-10-18 ### Fixed - Spin up threads correctly in case of erroneous configuration during life reconfiguration. ## 2018-10-17 ### Fixed - Reinitalize server ports correctly after reloading the access list. This was not a functional bug, only a small memory leak. ## 2018-09-14 ### Fixed - High IO usage while data storage purge. In worst case the purge operation could slow down the complete system so that incoming packets could not be handled anymore. ## 2018-09-05 ### Added - Access rule changes do not require a restart of the server anymore. ## 2018-08-29 ### Changed - Assigned human readable descriptions to threads. Process information tools like top or htop can display this information. ## 2018-08-08 ### Changed - Reduced server load for real-time client connections. ## 2018-05-30 ### Fixed - Fixed unexpected closed SSL connections. ## 2018-05-25 ### Fixed - Fixed high load if many clients request many streams in real-time. ## 2018-05-18 ### Added - Add option to log anonymous IP addresses. ## 2018-04-17 ### Fixed - Improved handling of incoming packets to prevent packet loss to subscribed sessions in case of heavy load. ## 2018-03-08 ### Fixed - Fixed access list evaluator. Rather than replacing general rules with concrete rules they are now merged hierarchically. ## 2018-02-13 ### Added - Restrict plugin stream codes to [A-Z][a-z][0-9][-_] . ## 2018-01-31 ### Changed - CAPS archive log will be removed at startup and written at shutdown. With this approach we want to force a rescan of the complete archive in case of an unexpected server crash. ## 2018-01-30 ### Fixed - Fixed parameter name if HTTP SSL port, which should be `AS.http.SSL.port` but was `AS.SSL.http.port`. ## 2018-01-29 ### Fixed - Fixed caps protocol real time handler bug which caused gaps on client-side when retrieving real time data. ## 2018-01-26 ### Changed - Log requests per CAPS server instance. ### Fixed - Improved data scheduler to hopefully prevent clients from stalling the plugin input connections. ## 2018-01-02 ### Fixed - Fixed bug in combination with SSL connections that caused CAPS to not accept any incoming connections after some time. ## 2017-11-15 ### Added - Added option `AS.inventory` which lets CAPS read an SC3 inventory XML file to be used together with WWS requests to populate channel geo locations which will enable e.g. the map feature in Swarm. ## 2017-11-14 ### Fixed - Data store start time calculation in case of the first record start time is greater than the requested one. ## 2017-11-08 ### Fixed - WWS Heli request now returns correct timestamps for data with gaps. ## 2017-10-13 ### Fixed - FDSN request did not return the first record requested. ## 2017-08-30 ### Fixed - Segmentation fault caused by invalid FDSN request. - Timing bug in the CAPS WWS protocol implementation. ## 2017-06-15 ### Added - Add `AS.minDelay` which delays time window requests for the specified number of seconds. This parameter is only effective with FDSNWS and WWS. ## 2017-05-30 ### Feature - Add experimental Winston Wave Server(WWS) support. This feature is disabled by default. ## 2017-05-09 ### Feature - Add FDSNWS dataselect support for archives miniSEED records. This support is implicitely enabled if HTTP is activated. ## 2017-05-03 ### Feature - Support for SSL and authentication in AS, client and HTTP transport. ## 2017-03-24 ### Fixed - MSEED support. ## 2017-03-09 ### Changed - Moved log output that the index was reset and that an incoming record has not ignored to debug channel. ## 2016-06-14 ### Added - Added option `AS.clientBufferSize` to configure the buffer size for each client connection. The higher the buffer size the better the request performance. ## 2016-06-09 ### Added - Added out-of-order requests for clients. The rsas plugin with version >= 0.6.0 supports requesting out-of-order packets with parameter `ooo`, e.g. `caps://localhost?ooo`. - Improved record insertion speed with out-of-order records. ## 2016-03-09 ### Fixed - Low packet upload rate.