Command line tools¶

There are two different command line tools. One is used to help configuring the system in various ways from creating keys to testing out if a specific filter is behaving as expected. The other tool is for communicating with the server.

baltrad-exchange-config¶

Like the command indicates, it’s used for various configuration purposes. Most essential is the one for creating keys since it’s essential for a running system.

create_keys¶

This command will give you help creating the public/private keys used for the signature handling within the system. The default behaviour is to use “crypto” which is the internally created handling using PyCryptodome as library and the service currently supports RSA and DSA keys. The nodename is essential since it will be used to set information that will be used when identifying this node to other nodes. The command in itself is straight forward

You will get help on how to use the command by using the option ‘–help’

 %> baltrad-exchange-config create_keys --help

 Options:
-h, --help            show this help message and exit
-v, --verbose         be verbose
--type=TYPE           Library to use for generating key-pair to create.
                      Default: crypto (which is the internal key handling).
                      Other libraries can be keyczar
--encryption=ENCRYPTION
                      Encryption method to use. For internal use you can
                      either choose dsa or rsa. Default is: dsa
--destination=DESTINATION
                      Directory where the keys should be placed. Default is
                      current directory.
--nodename=NODENAME   Name of the node for this server.

%> baltrad-exchange-config create_keys --type=crypto --encryption=dsa --destination=/etc/baltrad/exchange/crypto-keys --nodename=myserver
Created:
  Private key: /etc/baltrad/exchange/crypto-keys/myserver.private
  Public  key: /etc/baltrad/exchange/crypto-keys/myserver.public
  Public json: /etc/baltrad/exchange/crypto-keys/myserver_public.json

The other type that is of interest is the keyczar support since that provides backward compatibility with already existing installations. The keyczar-project has been deprecated and is not supported any longer. However, to keep the exchange working, baltrad-exchange has got it’s own implementation that mimics the relevant parts that has been used in the keyczar-library. This means that it is possible to read the public/private keys that has been generated during installation of the baltrad software. It can also generate new keys compatible with the old installations. However, there is no support for setting up the DEX-subscription using this tool.

%> baltrad-exchange-config create_keys --type=keyczar --destination=/etc/baltrad/bltnode-keys --nodename=myserver
Created:
  Private key: /etc/baltrad/bltnode-keys/myserver.priv
  Public  key: /etc/baltrad/bltnode-keys/myserver.pub

create_publication¶

Creates a publication from the provided template and the provided property-file. If possible, the property file will be identified by checking standard installation path. An atempt to find the template will be used base on default location / name as well. The resulting json file should be possible to put in the local server config catalogue without modifications. If different sender protocols or different connection strategies should be used the configuration file needs to be modified manually.

Example: baltrad-exchange-config create_publication –desturi=https://remote.baltrad.node –name=”pub to remote node” –output=remote_node_publication.json

Options:

-h, --help: show this help message and exit
-v, --verbose: be verbose
--conf=CONF: Specified the property file to use to extract all relevant information to create the publication to specified host.
--template=TEMPLATE: The template to use for generating the publication / subscription.
--desturi=DESTURI: Specified the target of this publication
--name=PUBLICATION_NAME: The name of this publication. MANDATORY!
--output=OUTPUT: The output file name. If not specified, output will printed on stdout

create_subscription¶

Usage: baltrad-exchange-config create_subscription [OPTIONS]

Creates a subscription package from the provided template and the provided property-file. If possible, the property file will be identified by checking standard installation path. An atempt to find the template will be used base on default location / name as well. The output will be a tarball containing of one public key and one subscription.json file and mailed to the admin for the remote server.

Example: baltrad-exchange-config create_subscription –output=subscription_bundle.tar

Options:

-h, --help: show this help message and exit
-v, --verbose: be verbose
--conf=CONF: Specified the property file to use to extract all relevant information to create the publication to specified host.
--template=TEMPLATE: The template to use for generating the publication / subscription.
--output=OUTPUT: The output file name. Must contain .tar och .tgz or .tar.gz suffix. Will default to <nodename>.tar.gz

test_filter¶

Another useful tool is the test_filter which can be used to validate files against a filter to test how the subscription/publication filters should be defined. The matching uses the baltrad-db metadata querying and will generate a odim-source sqlite database so that the _bdb/source_name, _bdb/source:WMO and other bdb-specific meta attributes can be ued. This is in contrast with the naming that instead uses _baltrad/ as prefix for internal meta data usage.

A typical filter will have a structure will look something like.

{"filter":{
  "filter_type": "and_filter",
  "value": [
    {"filter_type": "attribute_filter",
      "name": "_bdb/source_name",
      "operation": "in",
      "value_type": "string",
      "value": ["sehem","seang", "sella"]
    },
    {"filter_type": "attribute_filter",
      "name": "/what/object",
      "operation": "in",
      "value_type": "string",
      "value": ["SCAN","PVOL"]
    }
  ]
}}

Since this section is about the test_filter command in the config-tool we will not explain more about the filter and refer to :ref: filters.rst.

The tool requires two mandatory options and the odim-h5 file to be matched against. If you for example have a test-filter in a json-file that you want match against you can also specify the –filter-path which will extract the filter definition from that path.

For example: –filter-path=/subscription/__test_filter

anders@host: ~ %> baltrad-exchange-config test_filter --help
Usage: baltrad-exchange-config COMMAND [ARGS]

where COMMAND can be one of:
  - create_keys
  - test_filter

to get more information about a specific command, write baltrad-exchange-config <COMMAND> --help <filename>

Options:
  -h, --help            show this help message and exit
  -v, --verbose         be verbose
  --odim-source=ODIM_SOURCE
                        The odim source file to use for identifying the source
                        of a file. This command will create a temporary source
                        in /tmp unless underwise specified.
  --dburi=DBURI         The location where odim sources can be found. Default
                        is to create a temporary db under /tmp.
  --filter=FILTER       Specifies a file containing a filter. Can be either a
                        subscription or publication cfg-file or else a
                        separate file containing toplevel 'filter'
  --filter-path=FILTER_PATH
                        Path within the json entry where the filter can be
                        found.

baltrad-exchange-client¶

This tool is used to communicate with the server in various ways. Examples of use can be to store files or to query for statistics and other server information. It is possible to get available commands by running

%> baltrad-exchange-client --help
Usage: baltrad-exchange-client COMMAND [ARGS]

where COMMAND can be one of:
  - batchtest
  - file_arrival
  - get_statistics
  - list_statistic_ids
  - post_message
  - server_info
  - store

to get more information about a specific command, write baltrad-exchange-client <COMMAND> --help

When communicating with the server it is in most cases required to provide security-related options and these options are common for all commands.

–conf=CONF Specifies the configuration file to use for extracting required option values. Default is to use the installations config file which usually is located under /etc/baltrad/exchange/etc/baltrad-exchange.properties
–noconf Disables the usage of property file and expects all attributes to be provided on command line. Can be useful in some circumstances
–url=SERVER_URL The location of the server. Usually http://localhost:8089 or https://localhost:8089.
-t TYPE, –type=TYPE Type of encryption to use. Either crypto, keyczar or noauth
-k KEY –key=KEY The path to the private key to sign messages with.
-n NAME –name=NAME The name to use as node-name.

When the baltrad-config-client binary is executed, url, type, key and name will be determined by reading the options from the property-file and these will be used as default values. Then, if the options are provided these will override the default values. This obviously will require that the user running the baltrad-exchange-client command is allowed to read the private key.

All other options are specific to the above commands. The help section for each of the commands will explain more about how to use each command.

batchtest¶

The batchtest uses a basefile that either is a scan or a pvol and uses that as a template and then updates the source and datetime before sending it to the exchange server. It is only the information for the swedish radars sekrn, sella, seosd, seoer, sehuv, selek, sehem, seatv, sevax, seang, sekaa and sebaa that will be set in what/source. The soure set will be in the format WMO:02666,RAD:SE51,PLC:Karlskrona,CMT:sekaa,NOD:sekaa.

Example:

%> baltrad-exchange-client batchtest --basefile=/data/incomming/seang_scan_202212150100_0_5.h5

file_arrival¶

Usage: baltrad-exchange-client file_arrival [OPTIONS]

Functionality for querying if a file from a specific source has arrived within a given time.

The returned data will be a json dictionary with status as name and either ERROR or OK in the value.

Example: baltrad-exchange-client file_arrival –source=sella –object_type=PVOL –limit=5 {“status: “OK”}

Note The property baltrad.exchange.server.statistics.incomming has to be True in order for this command to work.

get_statistics¶

Usage: baltrad-exchange-client get_statistics [OPTIONS] –spid=STAT_ID

Queries the exchange server for various statistics information. The spid is used to identify what statistics id that should be queried for. It is possible to query for all existing ids by executing the command list_statistic_ids.

Example: Using totals to know the total amount of entries for one source and spid

%> baltrad-exchange-client get_statistics --spid=server-incomming --totals

It is also possible to use filters and origins and object_type for identifying specific entries and get more detailed information.

Example: Identifying entries that arrived more than 5 minutes delay and processing time greater than 7 ms from test-node

%> baltrad-exchange-client get_statistics --spid=server-incomming --filter='datetime>=20240322080000&&entrytime>=20240322080500&&optime>=7' --origins=test-node
{'spid': 'server-incomming', 'origin': 'anders-nzxt', 'source': 'sebaa', 'hashid': '7b76c8342a796f3809d01eb07f91f0adb3065402',
 'entrytime': '2024-03-22T08:06:38.619946', 'optime': 7, 'optime_info': 'metadata', 'datetime': '2024-03-22T08:00:00', 'object_type': 'SCAN', 'elevation_angle': 0.5}

Currently the syntax for –filter is one entry follow by a number of entries separated by &&. Each entry should have the following syntax “name[OP]value” where OP can be <, >, <=, >= and =. Supported attributes are:

datetime - Nominal date time. In format YYYYmmddHHMMSS.
entrytime - When file was registered in the statistics database in UTC. Format YYYYmmddHHMMSS
optime - Time to process the file. Integer.

list_statistic_ids¶

Usage: baltrad-exchange-client list_statistic_ids [OPTIONS]

Queries the exchange server for the available statistics ids

%> baltrad-exchange-client list_statistic_ids

post_message¶

Usage: baltrad-exchange-client post_message [OPTIONS] MESSAGE

Posts a json message to the exchange server. Can be used to trigger for example a runnable job.

%> baltrad-exchange-client post_message '{"trigger":"trigger_4"}'

server_info

Usage: baltrad-exchange-client server_info [OPTIONS]

Provides some useful information about the server. Currently the following things can be queried for.

uptime How long the server has been running

nodename The name this server is identifying itself when sending files

publickey The public key that can be used to identify myself as

%> baltrad-exchange-client server_info uptime

supervise¶

Usage: baltrad-exchange-client supervise [OPTIONS]

This operation is using the statistics database to monitor that files are received timely. It can be seen like a call to get_statistics with –spid=server-incomming and instead of returning the entries only a message is returned indicating if criteria is meet or not.

Example: Check if any file from sebaa has arrived that has a nominal time within last 300 seconds

%> baltrad-exchange-client supervise --source=sebaa --limit=300
{'status': 'OK'}
OK

Example: Check if a PVOL has arrived for sebaa that has a nominal time within last 300 seconds

%> baltrad-exchange-client supervise --source=sebaa --object_type=PVOL --limit=300
{'status': 'ERROR'}
ERROR

Example: Check if any file arrived within last 300 seconds. Note, that entrylimit is used instead and limit is using default value

%> baltrad-exchange-client supervise --source=sebaa --entrylimit=300
{'status': 'OK'}
OK

store¶

Usage: baltrad-exchange-client store [OPTIONS] FILE [ FILE]

Posts a sequence of files to the exchange server.