ROPO
Loading...
Searching...
No Matches
bRopo - a BALTRAD version of the FMI Anomaly detection and removal package ROPO

Introduction

bRopo is an adaption of the existing FMI software package ROPO. The bRopo is adapted to integrate with the RAVE framework and also to provide users with a Python interface.

Detectors

bRopo contains a number of detectors, most of them are accessible through the RAVE API but all detectors that are working on volumes, like ground clutter, are not supported at this time.

Speck

TBD by Markus

<MIN_DBZ> <max_a>  Threshold by min dBz, detect specks < A
-20dBz     5pix

SpeckNormOld

TBD by Markus

<MIN_DBZ> <max_a> <max_n>  Threshold by min dBz, then detect specks, size A_max_range <=> size N*A A
-20        5       16

Emitter

TBD by Markus

MIN_DBZ LENGTH   Filter unity-width emitter lines
10dbz   4

Emitter 2

TBD by Markus

MIN_DBZ LENGTH width     Filter emitter lines
-10dbz     4bins    2

Clutter

TBD by Markus

<MIN_DBZ> <max_incomp>   Remove specks under incompactness A
-5         5

Clutter 2

TBD by Markus

<MIN_DBZ> <max_smooth> Remove specks under smoothness
-5           60

Softcut

TBD by Markus

<max_dbz>     <r>     <r2>   Remove insect band
-10dbz    250km   100km

Biomet

TBD by Markus

<dbz_max> <dbz_delta> <alt_max> <alt_delta>   Remove insect band
-10dbz      5dBZ       5000m     1km

Ship

TBD by Markus

<min rel DBZ> <min A> Remove ships
50           20

Sun

TBD by Markus

<MIN_DBZ>  <min_length> <max_thickness>      Remove sun 
-20dBZ        100           3

Sun 2

TBD by Markus

<MIN_DBZ>  <min_length> <max_thickness> <azimuth> <elevation>   Remove sun 
-20dBZ        100          3             45        2

C API

To integrate cleanly with RAVE some wrappers have been implemented that wrap all ROPO functionality in RAVE Objects so that it is possible to send the objects around and also integrate properly with the RAVE Python APIs.

Two main objects have been defined, RaveFmiImage_t and RaveRopoGenerator_t. These contain most of the functionality you need when running the detectors. On top of this, you will need to be familiar with the different RAVE objects that you might run into. Especially, PolarScan_t and RaveField_t. You are also assumed to have fair knowledge about the RaveIO_t routines.

C API Usage

We have tried to keep the learning curve down as much as possible so mostly it should just be to get hold of a polar scan. Convert it into a RaveFmiImage_t that you pass into the RaveRopoGenerator. Then, when you have managed to do this, the real job comes to tuning the different detectors to your needs which might be a lot trickier.

static RaveFmiImage_t* getRestoredImage(const char* filename)
{
  RaveFmiImage_t* image = NULL;
  RaveRopoGenerator_t* generator = NULL;
  PolarScan_t* scan = NULL;
  RaveIO_t* raveio = NULL;
  RaveFmiImage_t* result = NULL;
  
  raveio = RaveIO_open("some_scan.h5");
  scan = (PolarScan_t*)RaveIO_getObject(raveio);
  image = RaveFmiImage_fromRave(scan, "DBZH");
  generator = RaveRopoGenerator_new(image);

  if (!RaveRopoGenerator_speck(generator, -20, 5) ||
      !RaveRopoGenerator_emitter(generator, -20, 6)) {
    goto fail;
  }

  result = RaveRopoGenerator_restore(50);
fail:
  RAVE_OBJECT_RELEASE(generator);
  RAVE_OBJECT_RELEASE(image);
  RAVE_OBJECT_RELEASE(scan);
  RAVE_OBJECT_RELEASE(raveio);
  return result;
}

As you can see from the above example, it is pretty straight forward. Of course you can perform more complex operations, like getting classifications, updating quality information in a polar object etc. Let's assume you have got hold of a classification field, you can easily add it to a polar scan as a quality field by doing the following.

 ...
 RaveFmiImage_t* classification = RaveRopoGenerator_getClassification(generator);
 RaveField_t* quality = RaveFmiImate_toRaveField(classification);
 PolarScan_addQuality(scan, quality);
 RAVE_OBJECT_RELEASE(quality);
 RAVE_OBJECT_RELEASE(classification);
 ...

There are a lot more variants on how to work with the API so please refer to the header file for more information.

Python API

In contrast to the C API, the Python API is not very chatty, and it is very easy to implement compact, easy to understand, detector sequences with a lot of variants for getting a specific result.

Firstly, all detectors return the generator itself which means that you can easily chain a number of detectors on one line.

 b = _ropogenerator.new(image)
 b.speck(-20,5).emitter(-20,5).ship(15,8)
 restored = b.restore(100)

Easy enough! Let's assume that you want to run a detector, then update the image itself and then run another detector. This is also a simple operation by using restoreSelf.

 b.speck(-20,5).restoreSelf().ship(15.8)
 restored = b.getImage() # Will return the image after speck.
 restored2 = b.restore() # Returns the image with both speck and ship executed.

Let's assume that you want to get a number of different restored images without having to re-initialize the generator. It can be good to know that the result from all detectors are stacked which means that if you want to get individual classification fields from each detector you will have to run an operation called declassify. Fortunately this is also quite obvious when you have seen it.

 b = _ropogenerator.new(image)
 speckprob = b.speck(-20,5).classification
 speckemitterprob = b.emitter(-20, 5).classification  #speck and emitter classification
 emitterprob = b.declassify().emitter(-20,5).classification # Only emitter classification

And as usual, there is another way to do the above.

 b = _ropogenerator.new(image)
 b.speck(-20,5).emitter(-20,5)
 speckprob = b.getProbabilityField(0)
 emitterprob = b.getProbabilityField(0)
 speckemitterprob = b.classification

All detectors will store some meta information in the FmiImage so that you are able to identify what has been done on the image. The two arguments how/task and how/task_args contain this information. You can get hold of it by doing the following

 b = _ropogenerator.new(image)
 c = b.speck(-20,5).emitter(-20,5).classification
 print "TASK: %s"%c.getAttribute("how/task") # prints TASK: fi.fmi.ropo.detector.classification
 print "ARGS: %s"%c.getAttribute("how/task_args") # prints ARGS: SPECK: -20,5; EMITTER: -20,5

That was a brief explanation on how to use the Python API, what values you should provide to the different detectors is up to the user and will not be covered here.

To give you a head start on how to use the RAVE support to run the different detectors and store the result. The following example will process all DBZH parameters in the volume with the three detectors: speck, emitter and ship. Then it will store the classification (probability field) and the restored image as quality parameters in the scans. Finally it will store the file with the name "myprocessedvolume.h5".

 volume = _raveio.open("myvolume.h5").object
 for i in range(volume.getNumberOfScans()):
   scan = volume.getScan(0)
   fmiimg = _fmiimage.fromRave(scan, "DBZH")
   generator = _ropogenerator.new(fmiimg)
   classification = generator.speck(-20, 5).emitter(-20, 6).ship(20,30).classification
   restored = generator.restored(50)
   scan.getParameter("DBZH").addQuality(classification.toRaveField())
   scan.getParameter("DBZH").addQuality(restored.toRaveField())
 
 output = _raveio.new()
 output.object = volume
 output.filename = "myprocessedvolume.h5"
 output.save()

Real-time bRopo

We have included a plugin for use with the RAVE Product Generation Framework (PGF). This plugin is called rave_pgf_ropo_plugin.py and the PGF is the XML-RPC server presented and described in the RAVE documentation. Basically, this plugin allows you to register bRopo as a product generator that can be run in real time by the PGF server.

Registering this bRopo with RAVE can be done by using the pgf_registry command with the following options and arguments:

 % pgf_registry -a -H http://<host>:<port>/RAVE --name=fi.fmi.ropo -m rave_pgf_ropo_plugin -f generate -d 'Detects and removes non-precipitation echoes.'

Note that this plugin's generate function does not take any arguments except the files list, and this list must contain only one single file containing either a polar scan or volume. The reason for this is that bRopo has so many bells and whistles that, at least for the purposes of real-time operation, it makes more sense to organize all arguments in a separate registry.

The bRopo options/arguments is an XML file designed to be read, understood, and edited easily by human beings. There is no functionality to verify the integrity of the XML after you have edited the registry, but on a development system you can install it and check by loading it:

 (shell)$ python
 >>> import ropo_realtime

If the ropo_realtime module is loaded correctly, it will do so silently. If the Python interpreter throws an error that looks like this: "xml.parsers.expat.ExpatError: not well-formed (invalid token):", then the XML is invalid and you need to fix it. There is no functionality to validate the integrity of the bRopo arguments themselves, ie. whether the argument values given are correct in number and within reasonable bounds.

The format of an XML entry is the following:

 <lvrix threshold="COLD" parameters="DBZH" highest-elev="2.0" restore-thresh="80" restore-fill="True" softcut="5,170,180" speckNormOld="-20,24,8" emitter2="-30,3,2" ship="20,8" speck="-30,12" />

The options threshold, parameters, highest-elev, and restore-thresh are all mandatory, and so is either restore or restore-fill. The other options are the anomaly detectors, and they can be used or not depending on your preferences. Note that for real-time purposes only those anomaly detectors given above in the example are the ones supported. This is because these are the ones that are battle-tested after 10 years of real-time use.

The example values above are almost the same as the default values, but they have been tuned for use with data from the Latvian radar at Riga aiport. Additional radars that need special tuning of bRopo's options can be added as separate entries. The important thing to remember is that each radar is identified using its NOD identifier that should be included in the /what/source metadata attribute in ODIM_H5.

The values for the threshold option are looked up according to a dictionary found at the top of the ropo_realtime.py module. The item for each entry in the THRESHOLDS dictionary contains a 12-tuple of dBZ thresholds, one for each month of the year starting with January. The idea is to be able to define reflectivity thresholds that are either "flat", ie. static throughout the year, or a bit more dynamic depending on their location. The functinality in bRopo looks up the threshold of the month based on the /what/date attribute in ODIM_H5. You are free to define your own thresholds according to this scheme, but keep in mind that the tuple must always have 12 values.

All numerical arguments are given as strings, and they are converted to integers or floats during runtime.

Data from radars without a specific entry will be processed according to a set of options/arguments called default.

If you edit the registry, you must restart the PGF server for the contents to be reloaded. If you're running the command-line ropo tool, you don't need to reload anything. If you want to look up the options/arguments automatically instead of writing them all out manually when using the command-line tool, just use ropo's --lookup option.

Generated by doxygen 1.11.0