Differing GDAL versions

[jakimowb]

Describe the bug

The latest FORCE docker images make use of recent (3.10+) GDAL versions. This is great, since it allows to use several improvements that have been added to GDAL over the past years.
However, the GDAL version that is available in the docker image python3 is still heavily outdated.

Expected behavior
The GDAL version used by force should be used by the standard python3 as well

How to reproduce

1. Open a shell in a force container:

docker run \
--rm \
-it \
-v /data:/data \
davidfrantz/force:3.10.04 bash  

2. Check the GDAL version that is available on CLI and used by FORCE:

ubuntu@a8692154b7ce:~$ gdal --version
GDAL 3.11.3 "Eganville", released 2025/07/12

Great! the GDAL is up-to-date

3. Check GDAL version used by standard python:

ubuntu@a8692154b7ce:~$ python3
Python 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from osgeo import gdal
>>> print(gdal.VersionInfo(''))
GDAL 3.8.4, released 2024/02/08

Arrgs, GDAL is almost two years old :-(

[pjonsson]

I noticed this before Christmas (guessing it's caused by https://github.com/davidfrantz/base_image/blob/e3109be8837feca11fc7111d3d7eafa578545b55/Dockerfile#L61), but don't want to flood the repository with a billion PRs that will end up causing merge conflicts.

[davidfrantz]

Hi,

I was under the impression that the line mentioned by @pjonsson takes care of exactly that.

I also was a bit surprised that the GDAL image does not contain this already - the same for libgdal, libproj etc.

Not sure how to fix this, though, I am not a Python programmer.

[jakimowb]

In the base image docker file (https://github.com/davidfrantz/base_image )
the problem is introduced by apt-get -y install libgdal-dev.
Interestingly this affects gdal-config, not the new gdal API (gdal --version) :

gdal-config --version 
3.13.0
gdal --version
GDAL 3.13.0dev-a388cb2469d775a3a282f5bc161bea405c04b36f, released 2026/01/07

... 

apt-get -y install libgdal-dev 

...

gdal-config --version
3.8.4
gdal --version
GDAL 3.13.0dev-a388cb2469d775a3a282f5bc161bea405c04b36f, released 2026/01/07
``

[pjonsson]

problem is introduced by apt-get -y install libgdal-dev. Interestingly this affects gdal-config, not the new gdal API (gdal --version) :

Oh, yeah, that installs some old libgdal-dev that ends up being first in the PATH. The new GDAL API wasn't a part of 3.8.x, so that is why gdal --version works.

[pjonsson]

I think the reason the base image installs libgdal-dev is because it's difficult to build the sf R package otherwise. It looks like sf does the right thing when building, but it still fails, so I just filed a bug on GDAL about some broken meta-data for package configuration.

I wasn't really planning on fighting R today, but since I've started I'll continue with it for a bit more next week.

[pjonsson]

The GDAL base image already contains the Python package, so I've made a PR that stops installing the Python package from PyPi which should solve the immediate problem reported here.

Unfortunately this issue uncovers that there are multiple GDAL libraries installed in the base image, so there's a risk that Force links to the older version of GDAL. Not installing the older GDAL library makes the sf module for R fail installation.

I assume sf is something important, so I'll continue looking at making that work without installing the older GDAL library tomorrow afternoon.

[davidfrantz]

Thanks Peter, this solves the python part at least.

But yes, sf is essential. It needs quite a bunch of packages, which is why libgdal, libproj etc are installed. It took me a while to notice this, because Rscript -e 'install.packages(.. does not fail on error. I was quite surprised that those are not available in the image already - or they are, and R just does not find them.

Now that I see this, FORCE is likely built against the old version...

[pjonsson]

I was quite surprised that those [libraries] are not available in the image already - or they are, and R just does not find them.

GDAL uses an internal PROJ that has its symbols renamed to something non-public, so users are apparently meant to install libproj from the distribution packages and use that version of PROJ.

I'm still not sure what the root cause for sf is, I've fixed some things in the GDAL images and I've updated PROJ to have all the symbols renamed. There are a few things in sf that look quite suspicious to me, so I will try to fix those next, but I'm not sure that will solve the real problem.

Even if I had magically fixed everything, the fixes aren't available in a release yet, so FORCE can't use them for the moment.

Now that I see this, FORCE is likely built against the old version...

I've thought about this for quite a bit, and I'm not sure what the best way to move forward is. We could build FORCE before installing the libraries required to build sf and its dependencies. This would slow down the build of the image in this repository which isn't ideal, but it would ensure that things are built as intended, and that feels pretty important to me. Once GDAL 3.13.0 and everything else I've had to fix is released we can move back to the current setup where sf is a part of the base image.

[davidfrantz]

Thanks so much for looking into this!

I've thought about this for quite a bit, and I'm not sure what the best way to move forward is. We could build FORCE before installing the libraries required to build sf and its dependencies. This would slow down the build of the image in this repository which isn't ideal, but it would ensure that things are built as intended, and that feels pretty important to me. Once GDAL 3.13.0 and everything else I've had to fix is released we can move back to the current setup where sf is a part of the base image.

Hmm, this would require quite some refactoring since we do not have a rooted image at this point, no? And the R-package installation is very time consuming and would be part of the FORCE image, which is built rather frequently.

Another option would be to temporarily disable force-map-accuracy. Or to translate to C entirely, which would also decrease the build time of the base image significantly. I am leaning towards the latter, though I need to find time to implement this properly.

What do you think?

[pjonsson]

Hmm, this would require quite some refactoring since we do not have a rooted image at this point, no? And the R-package installation is very time consuming and would be part of the FORCE image, which is built rather frequently.

The dependencies for the R packages aren't entirely clear to me, but my impression is that we could leave a fair chunk of the R package installation in base (it's just sf and anything that depends on sf that needs to be moved, anything sf depends on can still be installed in base).

With the UID/GID-renumbering, we're leaving base as root, so unless there is a USER ubuntu in the FORCE image now we're probably back to compiling things as root. But even if things were like before the renumbering, there's nothing preventing the FORCE image to do USER root if it needs to, so it can build FORCE as a user and then switch to root to install some R packages.

Is your concern for build time mainly about the CI, or your local builds? I haven't tried, but I'm guessing that it would be possible to get the R package installation to hit the Docker cache, so then that part would be more or less instant in your local builds.

Having said that, I would still not advice you to bet the house on the Docker cache functioning as we want without testing first, it frequently surprises me even today and I've been keeping an eye on it for the past 3 years.

Speaking of speeding things up, I made a pull request that added support for using ccache to the build system. That doesn't speed up the installation of R packages, but it does speed up the build of FORCE, so that would speed up the build of the image.

Another option would be to temporarily disable force-map-accuracy. Or to translate to C entirely, which would also decrease the build time of the base image significantly.

Sorry, I'm not intimately familiar with FORCE, so what you are suggesting isn't clear to me. I did a git grep force-map and it's just a single line in the history documentation, can you elaborate a bit?

I am leaning towards the latter, though I need to find time to implement this properly.

FORCE is high on my list of priorities right now, so I can do the work with shuffling things around in the Dockerfile(s).

[davidfrantz]

Is your concern for build time mainly about the CI, or your local builds?

Primarily, I am concerned about the CI build time. Right now, it is pretty fast (~1min on GH). Locally, I usually compile and test everything on the bare metal machine, unless I want to test version-specific functionality.

Speaking of speeding things up, I made a pull request that added support for using ccache to the build system. That doesn't speed up the installation of R packages, but it does speed up the build of FORCE, so that would speed up the build of the image.

Yes, I have seen this. I didn't have time to look into it yet, but I will.

Sorry, I'm not intimately familiar with FORCE, so what you are suggesting isn't clear to me. I did a git grep force-map and it's just a single line in the history documentation, can you elaborate a bit?

rstats/force-map-accuracy.r is the only tool that relies on the sf package. It is used to compute an area-adjusted map accuracy for classifications. I am thinking about either disabling it for the moment (until the sf issue is resolved), simply breaking it by not installing sf (until the sf issue is resolved), or translating it from R to C to get rid of sf entirely.

[pjonsson]

rstats/force-map-accuracy.r is the only tool that relies on the sf package. It is used to compute an area-adjusted map accuracy for classifications. I am thinking about either disabling it for the moment (until the sf issue is resolved), simply breaking it by not installing sf (until the sf issue is resolved), or translating it from R to C to get rid of sf entirely.

I don't know how important that script is for the FORCE users, but my gut feeling is that you should spend your time on other things than re-implementing existing features unless there is a really good reason. I think we have other options that don't take your time available right now, so I would suggest we start with those.

Installing the R packages is annoyingly slow right now, but I think I can speed it up without too much effort, so I will spend the afternoon on that.

[davidfrantz]

I will think about it. Let's simply destroy rstats/force-map-accuracy.r for now. We remove sf and all its dependencies.

It can still be executed by using an older FORCE image. So that's the most simple and efficient way.