File objects

To accomodate for a more general way of handling file I/O a new concept of file objects has been introduced that will be able to support different structures without creating separate data objects just to handle a specific data file.

These objects are not intended for use when working with the different data structures but should be seen as a layer that can be used between the I/O layer and the data objects used within rave.

Note, these objects does not care about conforming to a specific file format. Instead that is up to the actual file writing to enable / disable. Currently the file object will allow almost anything and can create ODIM H5 file that are not compatible with the ODIM H5 specification and as such is up to the user to ensure that the file object that is saved follows the requirements.

Reason for a general file object

The idea behind the file objects is that it should be possible to map the actual file I/O like ODIM H5 or FM-301 into a file object that will be mapped into the correct data object. The file object in turn will be mapped into a general structure that is internal for rave so that we don't need to adjust the internals just due to a different version of a file format. This mapping could then be done using a simple text-based file like json or something similar.

However, this is work in progress and for now the file objects will just be containers for reading/writing ODIM H5 files.

I/O handling using file objects

The first thing to do when creating a HDF5 file from scratch is to define some data to be written. The syntax is quite simple

and will not require a lot of work to understand. The saving is done like always when using the _raveio api so that hasn't changed. The object to use is _fileobject .

  import _raveio, _fileobject, _rave

  fileobj = _fileobject.new()
  fileobj.attributes["Conventions"]="ODIM_H5/V2_2"
  fileobj["/what"].attributes["source"]="NOD:setst"
  fileobj["/what"].attributes["date"]="20200414"
  fileobj["/what"].attributes["time"]="160000"
  fileobj["/what"].attributes["object"]="SCAN"
  fileobj["/what"].attributes["version"]="H5rad 2.2"
  fileobj["/where"].attributes["height"]=99
  fileobj["/where"].attributes["lat"]=61.1
  fileobj["/where"].attributes["lon"]=10.2
  fileobj["/dataset1/what"].attributes["product"]="SCAN"
  fileobj["/dataset1/where"].attributes["elangle"]=0.5
  fileobj["/dataset1/where"].attributes["nbins"]=480
  fileobj["/dataset1/where"].attributes["nrays"]=360
  fileobj["/dataset1/where"].attributes["rstart"]=0.0
  fileobj["/dataset1/where"].attributes["rscale"]=500.0
  fileobj["/dataset1/data1/what"].attributes["quantity"]="DBZH"
  fileobj["/dataset1/data1/what"].attributes["gain"]=0.5
  fileobj["/dataset1/data1/what"].attributes["nodata"]=255
  fileobj["/dataset1/data1/what"].attributes["offset"]=-32.0
  fileobj["/dataset1/data1/what"].attributes["undetect"]=0.0
  fileobj["/dataset1/data1/data"].data = numpy.zeros((360,480), numpy.uint8)

  rio = _raveio.new()
  rio.object = fileobj
  rio.save("test.h5")

It is also possible to load a ODIM H5 file into the general file object format regardless if it is a known type like SCAN, PVOL, .. or not.

>>> obj = _raveio.openFileObject("test.h5").object

The simplest way to display the content of the file object is to use the built in string representation.

>>> print(str(obj))
GROUP '/' {
  Conventions = ODIM_H5/V2_2
  GROUP 'dataset1' {
    GROUP 'data1' {
      DATASET 'data' {
        DIMENSIONS = 480 x 360
        DATATYPE = UCHAR
        LOADED
      }
      GROUP 'what' {
        undetect = 0
        nodata = 255
        gain = 0.5
        offset = -32
        quantity = DBZH
      }
    }
    GROUP 'what' {
      product = SCAN
    }
    GROUP 'where' {
      nbins = 480
      elangle = 0.5
      rscale = 500
      rstart = 0
      nrays = 360
    }
  }
  GROUP 'what' {
    time = 160000
    date = 20200414
    object = SCAN
    version = H5rad 2.2
    source = NOD:setst
  }
  GROUP 'where' {
    lat = 61.1
    lon = 10.2
    height = 99
  }
}