Compile with HDF4 driver

[meggart]

I think this has come up on discourse https://discourse.julialang.org/t/reading-hdf4-files/29470/3 and also recently for my own work I wished our GDAL would support HDF4 for reading MODIS data. I already familiarized myself a bit with BinaryBuilder and would tackle this, but it looks like we might want to wait for the new artifact system and until JuliaPackaging/BinaryBuilder.jl#441 is merged.

@visr I saw you commenting on the PR, are you working on updating this build script as well? If yes, let me know so that we don't do any conflicting changes.

[visr]

Yeah since the GDAL build is quite complicated, and the new Artifacts system will be much simpler for working with dependencies, I would favor switching to this and requiring at least julia 1.3 for the GDAL related packages. What do you think?

Are the TRMM and MODIS data not also available in newer formats? It's not on the list at #65 right now. It probably won't be the easiest driver to add. Though I see conda-forge builds it as well.

I'm right now primarily interested in JuliaGeo/GDALBuilder#10, which will unblock yeesian/ArchGDAL.jl#76. And then JuliaGeo/GDALBuilder#2. The first step is made now with https://github.com/JuliaBinaryWrappers/GEOS_jll.jl.

If you have time available, it would be great to have some help. I think once the foundation is there and we use the Artifacts, adding more drivers should become easier.

[meggart]

Yeah since the GDAL build is quite complicated, and the new Artifacts system will be much simpler for working with dependencies, I would favor switching to this and requiring at least julia 1.3 for the GDAL related packages. What do you think?

Completely agree, lets not care too much about backporting, especially since old package versions won't break.

Are the TRMM and MODIS data not also available in newer formats? It's not on the list at #65 right now. It probably won't be the easiest driver to add. Though I see conda-forge builds it as well.

If you have time available, it would be great to have some help. I think once the foundation is there and we use the Artifacts, adding more drivers should become easier.

Yes, I would like to help here. Just let me know when you have some specific tasks in mind. In the meantime I will work towards making a HDF4 artifact. Yesterday I had a first successful build script using the old BinaryBuilder. However, for the new branch it is not yet clear to me how to use other artifacts as dependencies (I need libjpeg and libz, which are both available on https://github.com/JuliaBinaryWrappers but I don't know how to access these during the build process)

[visr]

Awesome. Elliot's blog post, though not yet published, is a good reference. It also says how to add dependencies in build_tarballs.jl (dependencies = ["c_simple_jll"]).

As you probably saw I just made a PR for GEOS at JuliaGeo/LibGEOS.jl#65.
We need to do the same for PROJ (which requires https://github.com/JuliaBinaryWrappers/SQLite_jll.jl). If we have that we should have enough to attempt a build of GDAL.

By the way for JPEG there was also discussion in #64, which resulted in https://github.com/cbdavis/OpenJPEGBuilder (not yet in the GDAL build). I'm not sure about the differences between that and https://github.com/meggart/LibJPEGBuilder, and if we need both?

[visr]

(I think I will hold off a bit for now, since I have limited time available and currently cannot run BinaryBuilder locally, and don't want to move it all to a VM, but I'd be happy to support where I can.)

[meggart]

Thanks for the link to the blog post. I don't think we will need LibJPEGBuilder, it was more or less an experiment of mine to test BinaryBuilder. In production use I would try to refer to use https://github.com/JuliaBinaryWrappers/JpegTurbo_jll.jl which is already available.

[kongdd]

So, is this problem solved?

[visr]

No, we don't currently support HDF4, HDF5 or netCDF.

[Rapsodia86]

Any plans in the near future?

[visr]

Are you interested in HDF4 or HDF5 / netCDF? For HDF4 I don't know, nobody is working on it to my knowledge. For HDF5 / netCDF many people are interested in getting good builds, and a few are working towards it. But I can't say when it would be ready. See also #134.

[Rapsodia86]

I do work with remote sensing data and so I am interested in all. HDF4 is mostly MODIS (at least for me). I process it using python gdal. I wish that at some point all the processing would be done using only julia, instead of jumping between julia/python.
Thanks for the link to the netCDF topic.

[juliohm]

@Rapsodia86 I don't know your requirements, but NCDatasets.jl is a really nice package for reading/writing NetCDF: https://github.com/Alexander-Barth/NCDatasets.jl

[Rapsodia86]

@juliohm thanks for pointing that package. I am aware about it, but never had a chance to use it. Surely, will have to give it a try!

[lazarusA]

I see that this JuliaPackaging/Yggdrasil#7465 got merged. Would that mean that now support for HDF4 is possible here?

[visr]

I think so, yes. I had some hesitation on adding a dependency to HDF4 since HDF5 and NetCDF support is already there, but I see that:

It is still being developed: https://github.com/HDFGroup/hdf4/
The HDF4_jll is being maintained: JuliaPackaging/Yggdrasil#8590
The binaries are not that large: https://github.com/JuliaBinaryWrappers/HDF4_jll.jl/releases

So if there is still a lot of HDF4 data out there, we should add it, similarly to for instance JuliaPackaging/Yggdrasil#5703.

[lazarusA]

that would be great! I would like to remove my dependencies on calling python just for a read function for hdf4 files 😄 .