Changes required for installation under Ubuntu 24.04 LTS

[michaelkubina]

We are currently exploring the installation of eScriptorium (as a first step just for internal use only) with the help of your magnificent documentation (https://github.com/UB-Mannheim/eScriptorium_Dokumentation/blob/main/Lokale_Installation_eScriptorium.md). I was able to get it running under Ubuntu 24.04 LTS (Noble Numbat) in a virtual machine, but had some initial struggles that i would like to share. I believe some changes should be made to the description of the installation. Either by providing a seperate section that describes changes specifically for Ubuntu 24.04 LTS in the affected steps or as a seperate page? Not quite sure which way you would prefer... maybe its even general enough to work from Ubuntu 20.04 upwards ... but i have no knowledge about WSL, so have not tested it there.

Also i cant really tell at the moment, if actually everything works as it should and if the vulnerabilites mentioned by NPM are actually harmful or not - the eScriptorium GIT progressed a lot and the interface changed a lot as well. I was able to segment, recognize and train a new model based upon an imported model (german_print/german_handwriting). So from this perspective everything so far worked out as it should.

Important: It is set that our servers will run with nothing less then the current stable LTS release, which is 24.04. We won't use older versions on new servers because of our policy.

Installation issues

Ubuntu 24.04 LTS ships with Python 3.12.x which causes the installation to fail, as we can't install and use necessary dependencies. This is because the virtual python environment will use the distros version, if not specifically provided another one.

e.g. Kraken (https://pypi.org/project/kraken/)
installation fails because --> Python <=3.11.99, >=3.8

So we need to install at least Python 3.8.x - the eScriptorium Wiki luckily mentioned this as well (https://gitlab.com/scripta/escriptorium/-/wikis/full-install). I had tested for Python 3.11.x to fullfil krakens dependency, but the installation failed at a later point, when checking via manage.py ... so even this version is to high.

So we would need (at least if not specifically) to install Python 3.8.x under Ubuntu 24.04 LTS. From what i have seen, only two steps would need to be changed then:

1. Installation relevanter Programme

This affects step 1, so the call would actually be:

# add custom ppa
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install git postgresql postgresql-contrib libpq-dev redis-server gettext netcat-traditional jpegoptim pngcrush libvips build-essential python3.8 python3.8-dev python3.8-venv npm

Not sure if we actually require python3.8-pip as well? It's just mentioned in the Scripta-Documentation but not in yours ... so just as a mention.

7. Virtuelle Umgebung

Also we need to create a virtual python environment under this step with the now appropriate version:

python3.8 -m venv env
source env/bin/activate
pip install -U pip setuptools wheel

We can check the python version in our virtual environment simply by typing python --version ... in our case: Python 3.8.19

Everything else went through without any problems...

[stweil]

We use Python 3.11 / Debian stable in our installations.

The latest code of our test system is here: https://gitlab.com/stweil/escriptorium/-/commits/latest. It uses some newer packages, supports non-root installation, supports Opensearch as alternative to Elasticsearch and shows how to add settings and modifications for a local instance.

[stweil]

python3-pip is not required, python3-venv is sufficient because virtual Python environments already provide the necessary pip command.

[stweil]

The installation of Python 3.11 and related modules is the only special step which should be required for very recent Linux distributions.

I wouldn't base a new installation on Python 3.8 because that version is rather old with a higher risk that it does not provide newer versions for all Python modules which are used by eScriptorium.

[stweil]

Maybe for server installations this documentation is a better starting point.

Some advanced instructions are still missing, especially installation, configuration and operation of a search index (Elasticsearch or OpenSearch) and support for alignment.

[stweil]

I had tested for Python 3.11.x to fullfil krakens dependency, but the installation failed at a later point, when checking via manage.py ... so even this version is to high.

As we use 3.11.2, the installation should not fail. Maybe the failure was caused by an old package requirement which is already fixed in our branch (and which should be backported to the official code).

[michaelkubina]

As we use 3.11.2, the installation should not fail. Maybe the failure was caused by an old package requirement which is already fixed in our branch (and which should be backported to the official code).

Yes, this is it! I tried the installation again with Python 3.11.9 and the check failed for psycopg2, which has a dependency that uses a version lower to 2.9. I see in your latest test code that you have omited this limitation (https://gitlab.com/stweil/escriptorium/-/commit/e179812253459cd6a7bdae2b49feee764304672c) and i have changed the requirements.txt as well. Now the installation worked without a problem! I see also, that you have updated some NPM related dependencies as well...will look into it after my vacation.

Thank you for both the installation quide for a FQDN server and your latest code of your test system. Both are very valuable!

Not sure what to do with this ticket here now...it would certainly be helpful showing a path on how to use an older python version then the one shipped with ubuntu 24.04. But on the other hand, maybe kraken will update its dependencies soon as well, so that there are no actions required in downgrading the Python version for the virtual enviroment. And i believe you said your changes for the dependencies will be pulled into the main repository as well at some point???

My initial proposal for python3.8 is not useful anymore and instead of python3.8 it should be python3.11. Also the psycopg2-binary dependency in the app/requirements.txt needs to be resolved before the installation succeeds. But apart from that all the steps still worked for ubuntu24.04

[stweil]

It's not only Kraken which still has problems with Python 3.12. I estimate that we'll have to stick to 3.11 at least for this year.

My latest merge request for the requirements was merged today, so I can send a new one with the updates for Python 3.11.

[git0some]

With python3.8 -m venv env
or python3 -m venv env

Step 7

pip install -r app/requirements.txt -r app/requirements-dev.txt

ERROR: Could not find a version that satisfies the requirement dfine-kraken~=0.4.1 (from versions: none)
ERROR: No matching distribution found for dfine-kraken~=0.4.1

[stweil]

Python 3.8 is a very old release and does not support all newer software. Use Python 3.12 or 3.13 for the installation.

See https://github.com/mittagessen/dfine_kraken/blob/main/pyproject.toml#L23.

[git0some]

I'll try 3.12 now.
Downgrade to Python 3.12 because of missing imghdr in 3.13.

[git0some]

This steps helped to install it

sudo apt install software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt install python3.12 python3.12-venv python3.12-dev

7. Virtuelle Umgebung

python3.12 -m venv env
source env/bin/activate
pip install -U pip setuptools wheel

pip install -r app/requirements.txt -r app/requirements-dev.txt

11. Installation überprüfen

cd ~/escriptorium/app
python3.12 manage.py check --settings escriptorium.local_settings

12. SQL-Tabellen

python3.12 manage.py migrate --settings escriptorium.local_settings

13. Übersetzungen aktualisieren

python3.12 manage.py makemessages --all --settings escriptorium.local_settings
python3.12 manage.py compilemessages --settings escriptorium.local_settings

14. OPTIONAL

python3.12 manage.py createsuperuser

16. Start des Servers

python3.12 manage.py runserver --settings escriptorium.local_settings

[stweil]

The installation will support all recent Python releases up to 3.13 as soon as the merge request https://gitlab.com/scripta/escriptorium/-/merge_requests/525 was merged.