Support GDAL /vsicurl and /vsizip prefixes to DEM paths

[Ryanf55]

Desired behavior

Allow users to supply DEM data to Gazebo that is zipped, or DEM data to Gazebo that is remotely hosted. I would like all of the following able to be supplied to the Load() function:

• https://gdal.org/user/virtual_file_systems.html

Alternatives considered

Preventing users from using anything except local raw DEM files and stuff in Fuel.

Implementation suggestion

• GDAL can load zipped DEM if you add the /vsizip prefix to the path
• GDAL can load remotely hosted DEM if you add /vsicurl/<your_server_address> to the path

Use case:

• As a drone user that relies on ArduPilot and grid_map_geo, I want Gazebo to use the same terrain data as my drone's flight controller and path planner.

Example files - these are all zipped DEM hosted on a server

• https://terrain.ardupilot.org/SRTM1/
• https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip

Additional context

You can add the following code block to Dem_TEST.cc and observe gazebo fail to load the data.

1. Add the code block to the Dem_TEST.cc file:

   TEST_F(DemTest, LargeVRT)
   {
     // Load a large VRT DEM (used by ArduPilot)
     common::Dem dem;
     EXPECT_EQ(dem.Load("/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt"), 0);
   }

2. Rebuild the repo
3. Observe the failure

ryan@B650-970:~/Dev/gz_harmonic_ws/src/gz-common$ ./build/gz-common5/bin/UNIT_Dem_TEST 
Running main() from /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest_main.cc
[==========] Running 13 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 13 tests from DemTest
<<<<<<<<<<<<<<<< OMITTED FOR BREVITY >>>>>>>>>>>>>>>>>>>>
[ RUN      ] DemTest.LargeVRT
[Err] [SystemPaths.cc:425] Unable to find file with URI [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt]
[Err] [SystemPaths.cc:525] Could not resolve file [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt]
[Err] [SystemPaths.cc:425] Unable to find file with URI [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt]
[Err] [SystemPaths.cc:525] Could not resolve file [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt]
[Err] [Dem.cc:115] Unable to find DEM file[/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt].
/home/ryan/Dev/gz_harmonic_ws/src/gz-common/geospatial/src/Dem_TEST.cc:291: Failure
Expected equality of these values:
  dem.Load("/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt")
    Which is: -1
  0
[  FAILED  ] DemTest.LargeVRT (0 ms)
[----------] 13 tests from DemTest (48 ms total)

[----------] Global test environment tear-down
[==========] 13 tests from 1 test suite ran. (48 ms total)
[  PASSED  ] 12 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] DemTest.LargeVRT

 1 FAILED TEST
```

That said, GDAL can load the path `/vsizip/home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/data/ap_srtm1.zip` fine:
```
$ gdalinfo /vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt
<<<<<<<<<<<<<<<< OMITTED FOR BREVITY >>>>>>>>>>>>>>>>>>>>
       /vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/S84W179.hgt.zip/S84W179.hgt
       /vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/S84W180.hgt.zip/S84W180.hgt
Size is 1296001, 604801
Coordinate System is:
GEOGCRS["WGS 84",
    DATUM["World Geodetic System 1984",
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433]],
        AXIS["geodetic longitude (Lon)",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433]],
    ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (-180.000138888888898,84.000138888888884)
Pixel Size = (0.000277777777778,-0.000277777777778)
Corner Coordinates:
Upper Left  (-180.0001389,  84.0001389) (180d 0' 0.50"W, 84d 0' 0.50"N)
Lower Left  (-180.0001389, -84.0001389) (180d 0' 0.50"W, 84d 0' 0.50"S)
Upper Right ( 180.0001389,  84.0001389) (180d 0' 0.50"E, 84d 0' 0.50"N)
Lower Right ( 180.0001389, -84.0001389) (180d 0' 0.50"E, 84d 0' 0.50"S)
Center      (   0.0000000,  -0.0000000) (  0d 0' 0.00"E,  0d 0' 0.00"S)
Band 1 Block=128x128 Type=Int16, ColorInterp=Undefined
  NoData Value=-32768
```

Here's more info on the zip file:
```
$ unzip -l /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/data/ap_srtm1.zip
Archive:  /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/data/ap_srtm1.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
 11758320  2024-01-09 21:34   ap_srtm1.vrt
---------                     -------
 11758320                     1 file
 ```

## Long term

I want Gazebo to automatically load terrain wherever my drone flies and expose the necessary services to load more "tiles", move the origin, and not run out of memory. ArduPilot's ground station software of MAVProxy, QGroundControl, and Mission Planner all can do this today. When using Gazebo, the ROS aerial industry currently has to hard code a single DEM tile, and if the drone flies out of bounds of the tile, there is no capability to load new tiles in Gazebo. Being able to just give Gazebo a link to a terrain server and call it good would be awesome.

[Ryanf55]

Note - with a small modification to the library, it can load these kinds of files. For the very large ArduPilot terrain data, the call to RasterIO to load data results in a segmentation fault.

[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from DemTest
[ RUN      ] DemTest.LargeVRT
[Err] [SystemPaths.cc:425] Unable to find file with URI [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt]
[Err] [SystemPaths.cc:525] Could not resolve file [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt]
[Err] [SystemPaths.cc:425] Unable to find file with URI [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt]
[Err] [SystemPaths.cc:525] Could not resolve file [/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt]
[New Thread 0x7fffec2f9640 (LWP 364071)]
[Thread 0x7fffec2f9640 (LWP 364071) exited]

Thread 1 "UNIT_Dem_TEST" received signal SIGSEGV, Segmentation fault.
0x00007ffff62aefe6 in GDALCopyWords64 () from /lib/libgdal.so.30
(gdb) bt
#0  0x00007ffff62aefe6 in GDALCopyWords64 () from /lib/libgdal.so.30
#1  0x00007ffff6160631 in VRTSourcedRasterBand::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*) () from /lib/libgdal.so.30
#2  0x00007ffff627c0b3 in GDALRasterBand::RasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*) () from /lib/libgdal.so.30
#3  0x00007ffff7faccc7 in gz::common::Dem::LoadData (this=0x7fffffffcdb0) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/geospatial/src/Dem.cc:508
#4  0x00007ffff7faac41 in gz::common::Dem::Load (this=0x7fffffffcdb0, _filename="/vsizip/vsicurl/https://terrain.ardupilot.org/SRTM1/ap_srtm1.zip/ap_srtm1.vrt") at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/geospatial/src/Dem.cc:202
#5  0x000055555557d6a9 in DemTest_LargeVRT_Test::TestBody (this=0x5555556832e0) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/geospatial/src/Dem_TEST.cc:290
#6  0x00005555555c7279 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x5555556832e0, method=&virtual testing::Test::TestBody(), location=0x5555555df94b "the test body")
    at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:2599
#7  0x00005555555bf6e5 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x5555556832e0, method=&virtual testing::Test::TestBody(), location=0x5555555df94b "the test body")
    at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:2635
#8  0x000055555559a7f8 in testing::Test::Run (this=0x5555556832e0) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:2674
#9  0x000055555559b2cb in testing::TestInfo::Run (this=0x555555689760) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:2853
#10 0x000055555559bc09 in testing::TestSuite::Run (this=0x555555652290) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:3012
#11 0x00005555555abe3b in testing::internal::UnitTestImpl::RunAllTests (this=0x555555688230) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:5870
#12 0x00005555555c8300 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x555555688230, 
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x5555555aba20 <testing::internal::UnitTestImpl::RunAllTests()>, 
    location=0x5555555e03c0 "auxiliary test code (environments or event listeners)") at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:2599
#13 0x00005555555c08fd in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x555555688230, 
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x5555555aba20 <testing::internal::UnitTestImpl::RunAllTests()>, 
    location=0x5555555e03c0 "auxiliary test code (environments or event listeners)") at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:2635
--Type <RET> for more, q to quit, c to continue without paging--
#14 0x00005555555aa45d in testing::UnitTest::Run (this=0x55555560f660 <testing::UnitTest::GetInstance()::instance>) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest.cc:5444
#15 0x00005555555dbf1f in RUN_ALL_TESTS () at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/include/gtest/gtest.h:2293
#16 0x00005555555dbe98 in main (argc=1, argv=0x7fffffffd398) at /home/ryan/Dev/gz_harmonic_ws/src/gz-common/test/gtest_vendor/src/gtest_main.cc:51

We can take the approach I did in grid_map_geo and allow users to configure the maximum DEM size to load (in pixel-counts). Unfortunately, attempting to load a raster too big results in a segmentation fault, which is an existing vulnerability of this code. I haven't found any way to do robust runtime protections in a platform-portable way. Protecting against this is a separate problem.

Interestingly, the std::vector::resize call does not throw, and instead shows this for the vector:

Size is 3725520624 x 4 bytes per float = 14902082496 bytes = 14.902082496 Gigabytes. Technically, my computer is letting me allocate that much. You can see here in the process monitor, the process actually has that memory, but it's not contiguous.