Test failures in 4.0.2

[apteryks]

Hi,

We've recently upgraded afdko to 4.0.2 in Guix, but it seems there are now test failures (it's unclear why I didn't catch those before upgrading the package -- potentially they were caused by some other update, or are non-deterministic):

=================================== FAILURES ===================================
_ test_type1_supported_hint[/tmp/guix-build-python-afdko-4.0.2.drv-0/afdko-4.0.2/tests/otfautohint_data/input/dummy/font.ps] _
[gw2] linux -- Python 3.11.11 /gnu/store/v76lvm9985mj5h3kwrkif7gnqv3q30ck-python-wrapper-3.11.11/bin/python

path = '/tmp/guix-build-python-afdko-4.0.2.drv-0/afdko-4.0.2/tests/otfautohint_data/input/dummy/font.ps'

    @pytest.mark.parametrize("path", glob.glob(get_input_path("dummy/font.p*")))
    def test_type1_supported_hint(path):
        out_path = get_temp_file_path()
        options = OTFOptions(path, out_path)
        if os.path.basename(path) == 'font.ps':
            options.ignoreFontinfo = True
        hintFiles(options)
    
        path_dump = get_temp_file_path()
        out_dump = get_temp_file_path()
        subprocess.check_call(["tx", "-dump", "-6", path, path_dump])
        subprocess.check_call(["tx", "-dump", "-6", out_path, out_dump])
>       assert differ([path_dump, out_dump])
E       AssertionError: assert False
E        +  where False = differ(['/tmp/guix-build-python-afdko-4.0.2.drv-0/tmp70nhe84s', '/tmp/guix-build-python-afdko-4.0.2.drv-0/tmp1hi9zbq4'])

tests/otfautohint_test.py:577: AssertionError
----------------------------- Captured stdout call -----------------------------
--- /tmp/guix-build-python-afdko-4.0.2.drv-0/tmp70nhe84s
+++ /tmp/guix-build-python-afdko-4.0.2.drv-0/tmp1hi9zbq4
@@ -3459,2 +3459,3 @@
   718 topedge
+  780 841 hstem
   53 140 vstem
@@ -9054,2 +9055,3 @@
   714 topedge
+  752 802 hstem
   81 161 vstem
@@ -9064,2 +9066,3 @@
   714 topedge
+  752 802 hstem
   81 161 vstem
@@ -9072,2 +9075,3 @@
   714 topedge
+  752 802 hstem
   81 161 vstem
@@ -20932,2 +20936,3 @@
   714 topedge
+  752 802 hstem
   60 145 vstem
@@ -20942,2 +20947,3 @@
   714 topedge
+  752 802 hstem
   60 145 vstem
@@ -20950,2 +20956,3 @@
   714 topedge
+  752 802 hstem
   60 145 vstem
@@ -20959,2 +20966,3 @@
   714 topedge
+  752 802 hstem
   60 145 vstem
______________________________ test_decimals_ufo _______________________________
[gw2] linux -- Python 3.11.11 /gnu/store/v76lvm9985mj5h3kwrkif7gnqv3q30ck-python-wrapper-3.11.11/bin/python

    def test_decimals_ufo():
        path = get_input_path("dummy/decimals.ufo")
        out_path = get_temp_dir_path()
        options = OTFOptions(path, out_path)
        options.roundCoords = False
    
        hintFiles(options)
    
>       assert differ([path, out_path])
E       AssertionError: assert False
E        +  where False = differ(['/tmp/guix-build-python-afdko-4.0.2.drv-0/afdko-4.0.2/tests/otfautohint_data/input/dummy/decimals.ufo', '/tmp/guix-build-python-afdko-4.0.2.drv-0/tmp46cibv5p'])

tests/otfautohint_test.py:821: AssertionError
----------------------------- Captured stdout call -----------------------------
--- /tmp/guix-build-python-afdko-4.0.2.drv-0/afdko-4.0.2/tests/otfautohint_data/input/dummy/decimals.ufo/glyphs.com.adobe.type.processedglyphs/braceleft.glif
+++ /tmp/guix-build-python-afdko-4.0.2.drv-0/tmp46cibv5p/glyphs.com.adobe.type.processedglyphs/braceleft.glif
@@ -14,6 +14,6 @@
       <point x="142.4" y="-11.2" type="curve"/>
-      <point x="142.4" y="37.6"/>
+      <point x="142.4" y="37.6" name="hintRef0001"/>
       <point x="147.2" y="79.2"/>
       <point x="147.2" y="133.6" type="curve"/>
-      <point x="147.2" y="184.8" name="hintRef0001"/>
+      <point x="147.2" y="184.8" name="hintRef0002"/>
       <point x="133.6" y="211.2"/>
@@ -24,6 +24,6 @@
       <point x="147.2" y="311.2" type="curve"/>
-      <point x="147.2" y="365.6" name="hintRef0002"/>
+      <point x="147.2" y="365.6"/>
       <point x="142.4" y="407.2"/>
       <point x="142.4" y="456" type="curve"/>
-      <point x="142.4" y="508"/>
+      <point x="142.4" y="508" name="hintRef0003"/>
       <point x="153.6" y="528.8"/>
@@ -36,13 +36,13 @@
       <point x="88.8" y="460" type="curve"/>
-      <point x="88.8" y="401.6"/>
+      <point x="88.8" y="401.6" name="hintRef0004"/>
       <point x="96" y="359.2"/>
       <point x="96" y="304.8" type="curve"/>
-      <point x="96" y="274.4" name="hintRef0003"/>
+      <point x="96" y="274.4"/>
       <point x="81.6" y="244"/>
       <point x="27.2" y="243.2" type="curve"/>
-      <point x="27.2" y="201.6" type="line"/>
-      <point x="81.6" y="200.8"/>
+      <point x="27.2" y="201.6" type="line" name="hintRef0005"/>
+      <point x="81.6" y="200.8" name="hintRef0006"/>
       <point x="96" y="170.4"/>
       <point x="96" y="139.2" type="curve"/>
-      <point x="96" y="85.6" name="hintRef0004"/>
+      <point x="96" y="85.6"/>
       <point x="88.8" y="43.2"/>
@@ -78,3 +78,3 @@
               <string>hstem 528.8 37.6</string>
-              <string>vstem 27.2 72</string>
+              <string>vstem 96 51.2</string>
             </array>
@@ -89,3 +89,3 @@
               <string>hstem 528.8 37.6</string>
-              <string>vstem 88.8 53.6</string>
+              <string>vstem 96 51.2</string>
             </array>
@@ -100,3 +100,3 @@
               <string>hstem 528.8 37.6</string>
-              <string>vstem 27.2 72</string>
+              <string>vstem 88.8 53.6</string>
             </array>
@@ -111,3 +111,25 @@
               <string>hstem 528.8 37.6</string>
-              <string>vstem 88.8 53.6</string>
+              <string>vstem 96 51.2</string>
+            </array>
+          </dict>
+          <dict>
+            <key>pointTag</key>
+            <string>hintRef0005</string>
+            <key>stems</key>
+            <array>
+              <string>hstem -121.6 37.6</string>
+              <string>hstem 201.6 41.6</string>
+              <string>hstem 528.8 37.6</string>
+              <string>vstem 27.2 72</string>
+            </array>
+          </dict>
+          <dict>
+            <key>pointTag</key>
+            <string>hintRef0006</string>
+            <key>stems</key>
+            <array>
+              <string>hstem -121.6 37.6</string>
+              <string>hstem 201.6 41.6</string>
+              <string>hstem 528.8 37.6</string>
+              <string>vstem 96 51.2</string>
             </array>
=============================== warnings summary ===============================
../../../gnu/store/bbfi42m2k72hknq8gi3i6ly76q50rpzg-python-fs-2.4.16/lib/python3.11/site-packages/fs/__init__.py:4: 32 warnings
  /gnu/store/bbfi42m2k72hknq8gi3i6ly76q50rpzg-python-fs-2.4.16/lib/python3.11/site-packages/fs/__init__.py:4: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
    __import__("pkg_resources").declare_namespace(__name__)  # type: ignore

../../../gnu/store/bbfi42m2k72hknq8gi3i6ly76q50rpzg-python-fs-2.4.16/lib/python3.11/site-packages/fs/__init__.py:4: 32 warnings
  /gnu/store/bbfi42m2k72hknq8gi3i6ly76q50rpzg-python-fs-2.4.16/lib/python3.11/site-packages/fs/__init__.py:4: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('fs')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    __import__("pkg_resources").declare_namespace(__name__)  # type: ignore

../../../gnu/store/bbfi42m2k72hknq8gi3i6ly76q50rpzg-python-fs-2.4.16/lib/python3.11/site-packages/fs/opener/__init__.py:6: 32 warnings
  /gnu/store/bbfi42m2k72hknq8gi3i6ly76q50rpzg-python-fs-2.4.16/lib/python3.11/site-packages/fs/opener/__init__.py:6: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('fs.opener')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    __import__("pkg_resources").declare_namespace(__name__)  # type: ignore

../../../gnu/store/ym4hd0kk40y4fq6m8balv2bamfc2kib0-python-setuptools-80.9.0/lib/python3.11/site-packages/pkg_resources/__init__.py:2558: 32 warnings
  /gnu/store/ym4hd0kk40y4fq6m8balv2bamfc2kib0-python-setuptools-80.9.0/lib/python3.11/site-packages/pkg_resources/__init__.py:2558: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('fs')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(parent)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/otfautohint_test.py::test_type1_supported_hint[/tmp/guix-build-python-afdko-4.0.2.drv-0/afdko-4.0.2/tests/otfautohint_data/input/dummy/font.ps] - AssertionError: assert False
 +  where False = differ(['/tmp/guix-build-python-afdko-4.0.2.drv-0/tmp70nhe84s', '/tmp/guix-build-python-afdko-4.0.2.drv-0/tmp1hi9zbq4'])
FAILED tests/otfautohint_test.py::test_decimals_ufo - AssertionError: assert False
 +  where False = differ(['/tmp/guix-build-python-afdko-4.0.2.drv-0/afdko-4.0.2/tests/otfautohint_data/input/dummy/decimals.ufo', '/tmp/guix-build-python-afdko-4.0.2.drv-0/tmp46cibv5p'])
===== 2 failed, 1523 passed, 8 skipped, 8 xfailed, 128 warnings in 26.22s ======

Would you have an idea of a reason these may fail like this?

The inputs used are:

antlr4@4.10.1 bash-minimal@5.2.37 java-antlr4-runtime-cpp@4.10.1 libxml2@2.14.6 ninja@1.13.1
+ openjdk@24.0.1 pkg-config@0.29.2 psautohint@2.4.0 python-booleanoperations@0.9.0 python-defcon@0.12.2
+ python-fontmath@0.9.4 python-fonttools@4.59.2 python-lxml@4.9.4 python-pytest-xdist@3.8.0 python-pytest@8.4.1
+ python-scikit-build@0.18.1 python-setuptools-scm@8.3.1 python-tqdm@4.67.1 python-ufonormalizer@0.6.2
+ python-ufoprocessor@1.9.0 python-wheel@0.46.1 util-linux@2.40.4

[apteryks]

Also happens with ufoprocessor updated to 1.14.0 and ufonormalizer updated to 0.6.3. I'll disable these tests in Guix for now.

[apteryks]

We've recently updated libxml2 to 2.14.6 (to fix CVEs).

[skef]

The reference fontTools version for 4.0.2 is 4.43.0, as it should be listed in the requrements.txt file. We most often see this sort of test divergence when the fonttools version is different, although it can also be caused by other things.

From eyeballing the test failures you're seeing are probably not signs of a real problem. Subtle differences in glyph outline data representation are causing subtle differences in the output of some tools.

[apteryks]

OK, so we're using a newer fonttools (4.59.2). That's probably it. Thanks for the test failures interpretations. I'll leave them disabled and revisit in the future, perhaps. Perhaps adobe-type-tools could update to newest fonttools, since we now know it works, with some minor adaptations to the expected test data :-).

[skef]

Fixed in AFDKO 5.0.0