 |
Frequently Asked Questions and User Guide
|
|
Frequently Asked Questions and User Guide
|
We've installed and configured a BALTRAD node, injected our own data into it, exchanged data with other nodes, generated products with all the data, and even created images for display. Now we want to distribute these products. Here is where we see how.
The node comes equipped with a Distribution route, accessed through the web-based user interface. You can use it to distribute data and/or products using local file copy, FTP, or SCP.
Note that if you use SCP, then you will need valid public and private keys generated by ssh-keygen and exchanged with the machine that will receive the distributed files. These keys are not part of the BALTRAD system, they are required to use the safe protocol. There are several pages on the web devoted to explaining how this works, e.g. this one hosted by Ubuntu where you can learn how to set this up.
We will give an example where we distribute the product we've configured and generated in Generating a composite to a remote computer using SCP. In the web-based interface, go to "Processing" --> "Routes" --> "Create distribution". The first four fields are easy to deal with, while the remaining ones require some attention.
First the "Destination" field which can have one of three different URI formats:
In the example below, we use FTP protocol for file distribution. Destination URI looks like the following:
ftp://rainbow:s3cret@172.31.85.101/opt/baltrad/odimH5/rb5input/BOR
The example URI using SCP would look like this:
scp://rainbow@172.31.85.101/opt/baltrad/odimH5/rb5input/BOR
Name template field supports naming the files. If this field is not empty, it will be used to generate the names of the uploaded files. ${} is used in the template to specify placeholders. The placeholder itself should be an ODIM_H5 metadata attribute provided with full path. Following is an example of a name template:
${/_bdb/source_name}_${/what/object}_${/what/date}T${/what/time}Z.h5
The template will produce the following file names:
ee_COMP_20120205T010000Z.h5 eehar_PVOL_20120205T010000Z.h5 eesur_PVOL_20120205T010000Z.h5 sease_SCAN_20120206T083200Z.h5
Besides standard ODIM_H5 attributes, the following placeholder variables are triggered with a special prefix /_bdb:
From version 2.1 of the node software we have started to introduce more advanced features when it comes to creating names in the distribution rules. For one, we have added more complex place holders that can provide names besides the information in the attributes. Secondly we have also added operations that can be applied on the place holder to give ability to modify the presentation.
The first namer we introduced was the possibility to specify your own date and time format from the what/date and what/time attributes. This attribute is defined as
For example, if you want to specify a date as 2014-12-27 11:12 then you write ${_beast/datetime:yyyy-MM-dd HH:mm}.
In some examples you can also use this feature for creating subdirectories, currently this is only available for the copy:/// destination. An example for this would be ${_beast/datetime:yyyy/MM/dd/HH/mm}. Hopefully we will be able to support this behaviour for both scp and ftp as well.
Since an ODIM H5 file can contain several different quantities it might be interesting to see what quantities that exists in a file by just looking at the filename. For this reason we have created something we call a hexcode namer. The behaviour of this is that you will get a hex code bit mask that defines what quantities that exists in the file.
If for example bit 1 is set for DBZH and bit 2 is set for TH and the file contains both DBZH and TH, then the result of this placeholder would be 0x3. The definitions of these fields can be found in odim_quantities.xml which is located in .../third_party/tomcat/webapps/BaltradDex/WEB-INF/classes/odim_quantities.xml. Note, that this file also exists as a duplicate in .../beast/etc/odim_quantities.xml but this file is not used in the node but it gives external applications the possibility to read the same information as is used in the tomcat app server.
In node version 2.1 we also introduces something that we are calling suboperations. These operations gives the possibility to even more refine the names that the placeholders gives. In some cases you might want to capitalize letters, only extract parts of a name and so on.
The suboperation is applied after the ending '}' for the placeholder, for example. ${what/source:CMT}.tolower().toupper(1) would ensure that only the first letter in the source would be capitalized. As you can see from example it is possible to chain the suboperations. Whenever a suboperation fails they will return the original string. E.g. "xyz".toupper(10) will result in "xyz" instead of null or some sort of exception.
Currently, the following suboperations are supported
The "Filter" is a generic ODIM filtering tool that allows you to define selection criteria based on metadata attributes in ODIM files. In our case we have defined a fairly simple set of two criteria, the /what/source attribute must contain the geographic area name. A full set of idntifiers is used here in order to match files against filter: WMO, RAD, PLC and CMT:
WMO:06194,RAD:DN44,PLC:Bornholm,CMT:dkbor
This ensures that in case if any of these identifiers match the file's /what/source attribute, the file will be distributed. The second selection criterion is that the /what/object attribute is PVOL, ie. a polar colume. This filtering functionality assumes you are conversant with ODIM_H5 and its metadata.
We are using sshj (https://github.com/hierynomus/sshj/downloads) to support file upload with scp and have noticed some things that might cause some head aches when reading the catalina.out log. You might see the following exception over and over again:
21 Dec 2014 14:18:05 WARN net.schmizz.sshj.SSHClient - Could not load keys due to: {}
java.io.FileNotFoundException: /home/baltrad/.ssh/id_dsa (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileReader.<init>(FileReader.java:72)
at net.schmizz.sshj.userauth.keyprovider.KeyProviderUtil.detectKeyFileFormat(KeyProviderUtil.java:40)
at net.schmizz.sshj.SSHClient.loadKeys(SSHClient.java:478)
This is really not any problem, it is just the SSHJ library saying that it couldn't find the .ssh/id_dsa file but it will keep on handling the file upload anyway. If you want to get rid of the exception from the log, you will just have to create the id_dsa file by typing:
%> ssh-keygen -t dsa
and follow the instructions.
1.9.1 for 