StructureDS and enhancements

[rschmunk]

I'm trying to work with a satellite data product that includes some information in structures. The goal is to be able to make 2D color-contour plots of any specified member of a 2D array of a structure. But it seems that when I obtain a StructureDS, there is no coordinate system attached. This even though the structure metadata does specify a grid_mapping attribute, per the following CDL:

Structure {
  float r;
    :_FillValue = NaNf; // float
  float i;
    :_FillValue = NaNf; // float
} HH(yCoordinates=24879, xCoordinates=9090);
:description = "Focused SLC image (HH)";
:mean_real_value = 0.0; // double
:max_real_value = 0.0; // double
:sample_stddev_real = 0.0; // double
:min_imag_value = 0.0; // double
:mean_imag_value = 0.0; // double
:max_imag_value = 0.0; // double
:sample_stddev_imag = 0.0; // double
:units = "1";
:grid_mapping = "projection";
:long_name = "geocoded single-look complex image HH";
:min_real_value = 0.0; // double

A variable in the same group as the HH structure does report the projected grid's coordinate system, and I am able to plot that pretty easily.

Optimally, I'd like to be able to select("r") or select("i") from the HH StructureDS and obtain a VariableDS with 2D content of the specified dimensions, but as things stand in v5.6, the select(String) method that StructureDS inherits from Structure only provides a plain Variable. I started looking at whether it was possible to promote the Variable to a VariableDS, but implementing a VariableDS.Builder looks like a convoluted mess and even then I still need to get the StructureDS's coordinate system.

Any ideas, suggestion, other? Is any of what I'm talking sound like a deficiency in netCDF-Java for which a new feature enhancement or three would be a plus?

Edit: Crud, looking like the select(String) method in Structure and StructureDS may not do what I thought it does.

[rschmunk]

I've been diving deeper into accessing how to retrieve Structure member content, and it looks like my hopes for a convenient way of doing so have crashed on the rocks. AFAICT, I'll have to use StructureDataIterators to get StructureData, and then extract StructureMember info from there.

Unfortunately, all my current code for retrieving and plotting data is reliant on being able to work with VariableDS objects. So that means a major refactoring if I pursue this further.

All that aside, it would seem to remain a useful enhancement if StructureDS objects could be enhanced to include coordinate system info as are VariableDS objects.

[tdrwenski]

Hi @rschmunk! I definitely agree that the Structures part of the library is not very user friendly and may be lacking some needed features.

Have you tried to use the Structure::findVariable method? If you have a StructureDS I would expect the member variables to be returned by that to be VariableDSs (but maybe this is not the case in your situation?). As you already noted, I don't think the select method is the correct way to just get the members. I am thinking something like this:

    try (NetcdfDataset dataset = NetcdfDatasets.openDataset("/path/to/file.nc")) {
      StructureDS structureDS = (StructureDS) dataset.findVariable("HH");
      VariableDS variableDS = (VariableDS) structureDS.findVariable("r");
    }

I also see a StructureDS::getCoordinateSystems method. Does that work for you for getting the coordinate system info directly from the StructureDS?

[rschmunk]

@trdwenski, I'll doublecheck the findVariable method in a bit when I'm at a computer with a better display set-up.

But StructureDS::getCoordinateSystems provides no information at all. As best I could tell from reviewing the NJ library source, the coordinate system enhancement just never gets applied to a StructureDS. As in, I'm not sure I even see any code to do so.

[rschmunk]

@trdwenski, I took a look at the Structure.findVariable method and it seems to be a bust. If I try

VariableDS variableDS = (VariableDS) structureDS.findVariable("r");

from the structure shown in the sample CDL above, it returns a VariableDS that is scalar valued rather than one claiming dimensions of [24879][9090].

[rschmunk]

Finally taking a first look at NJ6 source, after seeing a deprecated notice too many days ago in NJ5's VariableDS.Builder about coordinate systems. And what ho? Is that a setCoordinateSystems method for adding coordinate system info in NJ6's StructureDS?

That would seem a start at resolving the coding problems I currently confront.

[tdrwenski]

That sounds like incorrect behavior-- both that you are getting a scalar VariableDS and that the getCoordinateSystems is not returning anything. I am not familiar with the intended API changes in version 6.x, but Unidata is no longer developing versions 6+ which are located in this repo. So therefore some deprecations in version 5.x need to be revisited. But in any case the deprecated code should still function correctly in version 5.x.

I am not sure we will be able to take a look right away, but maybe you can make an issue for the problems you have found in the 5.x API?

[rschmunk]

@tdrwenski , I've been trying to modify StructureDS in my copy of 5.6.+ to include code for reporting a CoordinateSystem (if there should be one), but not getting anywhere quickly as I try to work on three other things at the same time.

[rschmunk]

FWIW: After hacking at things for a while, I managed to create a version of netCDF-Java 5.10 (snapshot) that will report coordinate systems if any for a StructureDS. But I jumped through some really weird hoops dealing with VarProcess in CoordSystemBuilder, and then at the very end had to copy a deprecated but apparently still used method from VariableDS over to StructureDS.

It's all a bit of a mess, and I'll to take a second look at a lot of what I did to see whether some of was necessary at the end.

[lesserwhirls]

The code shared by the old vs new APIs really make things complicated. The project needs to make a push to version 6 so that we can remove the old APIs to have a clean target for fixes.

[rschmunk]

I had been meaning to ask, what's up with the version 6 fork? It's how old now?