Date: 13 Feb 2017

Author: Daniel Michelson and Anders Henja

Version: jenkins-Rave-577

Copyright: © 2014 by the Swedish Meteorological and Hydrological Institute (SMHI), Norrköping, Sweden

Legals: RAVE is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

: RAVE is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

: You should have received a copy of the GNU Lesser General Public License along with RAVE. If not, see http://www.gnu.org/licenses/. By obtaining, using, and/or copying this software and/or its associated documentation, you agree that you have read, understood, and will comply with the following terms and conditions:

Introduction

RAVE was conceived in 1996 by Håkan Bolin who built it as his MSc at Umeå University, doing his work at SMHI for Daniel Michelson. It was used for a short time as an R&D tool until 1998 when Anders Henja and Daniel Michelson re-wrote it as part of an EU Framework IV project called PEP in BALTEX. This version was used to establish the BALTEX Radar Data Centre at SMHI in 1999. It was also integrated into the second generation operational NORDRAD system, where it was (is still) used in real time for several years. A second rewrite was started in 2006, focussed on facilitating management of data, but it was frozen due to lack of available resources.

So much for history. The BALTRAD project, part-funded by the Baltic Sea Region, has given us the ability to perform a much-needed overhaul. The underlying data structures have been completely replaced. The new ones are built with speed, ease of use, and a small memory footprint as priorities. Instead of being largely a Python-based system, RAVE is now largely C-based with a thin Python API. This is valid for the underlying data structures that are read to/from ODIM_H5 files, the system's native format. It is also valid most of the algorithms that process data in various ways. However, there is an XML-RPC server included that is designed to interface with the so-called baltrad-node, and this server is written entirely in Python. See a quick overview below.

The V in RAVE stands for visualization. A couple of prototypes have been created, and a 3-D visualizer using PyVTK was created as another MSc thesis by Aron Ernvik at Linköping University, but the stock version of RAVE has never, and may never, contain its own front end.

API Overview

RAVE processing functionality has been developed as a C library that provides the necessary functions for working with radar data in various ways. It's base is a small framework for manipulating objects by reference counting. It also contains some functions for supporting memory tracking and debugging. On top of this framework, several different objects have been created in order to support navigation and data processing. Most of the APIs have been developed for radar processing, but there is nothing that says that it could not be used for other types of processing of raster spatial data.

In order to support Python, a similar Python API has been developed on top of the C API; there are several reasons for this, but the most obvious one is that it is much easier to work with Python instead of C. The system becomes a lot easier to configure and manage at a high level of abstraction, without taking a performance hit.

Before we dig down deep, we should first take a look at the fundamentals in Introduction to the RAVE C APIs.

RAVE XML-RPC Server

This Python-based server is designed to interact with the BALTRAD system, the so-called "baltrad-node". This node exchanges data and manages them locally, but doesn't actually process them except by interacting with external services like this XML-RPC server. The baltrad-node is the spider in the web; it has the knowledge of which data are available and what needs to be done with them. If baltrad-node contains a rule to send a given kind of data to RAVE for processing, then it's up to the rule to ensure that the correct criteria are defined. When the criteria are fulfilled, baltrad-node will perform a remote procedure call on RAVE, and RAVE will perform the task according to the call and then inject the resulting ODIM_H5 file into the baltrad-node via the node's single point of entry managed by the DEX (Distribution and Exchange System).

This server is simple and light-weight, but it contains features like:

Fully daemonized, but can also be run in the foreground (good for debugging).
Data processing functionality can be added to the server, updated, and removed during runtime.
If the server is stopped before the job queue is empty, jobs that have not been run will be saved and then run when the server restarts.
Logging, thus far to text file.

Learn more about how the server works and how messages to it must be formatted at RAVE XML-RPC Server.

Interactive RAVE

One of the advantages of Python is that it can be used interactively in a terminal, more-or-less like interactive scientific toolboxes like MATLAB and IDL, but free. RAVE can be used in this way as well, and this is intended to facilitate research and development and also rapid prototyping of new functionality.

Interactive RAVE contains an introduction to how interactive use of RAVE is achieved.