skale ssl upload fails intermittently, often

[kucharskim]

OS: Ubuntu 22.04 LTS

Fix worked on via:

• Fix check_ssl_connection() function #866

# skale version
SKALE Node CLI version: 2.6.2

[2025-06-17 11:34:38,717 DEBUG] __main__:117 - MainThread - cmd: skale ssl upload -f -k /etc/letsencrypt/live/andromeda03.skale.prod.chorus1.net/privkey.pem -c /etc/letsencrypt/live/andromeda03.skale.prod.chorus1.net/fullchain.pem, v.2.6.2
[2025-06-17 11:34:38,718 INFO] node_cli.core.ssl.check:110 - MainThread - Staring healthcheck server on port 4536 ...
[2025-06-17 11:34:38,718 DEBUG] node_cli.core.ssl.utils:50 - MainThread - Starting detached subprocess: ['openssl', 's_server', '-cert', '/etc/letsencrypt/live/andromeda03.skale.prod.chorus1.net/fullchain.pem', '-cert_chain', '/etc/letsencrypt/live/andromeda03.skale.prod.chorus1.net/fullchain.pem', '-key', '/etc/letsencrypt/live/andromeda03.skale.prod.chorus1.net/privkey.pem', '-WWW', '-accept', '127.0.0.1:4536', '-verify_return_error', '-verify', '1']
[2025-06-17 11:34:39,719 INFO] node_cli.core.ssl.check:87 - MainThread - Server successfully started
[2025-06-17 11:34:39,720 INFO] node_cli.core.ssl.check:188 - MainThread - Checking healthcheck endpoint ...
[2025-06-17 11:34:39,720 INFO] node_cli.core.ssl.check:196 - MainThread - Connecting to public ssl endpoint 127.0.0.1:4536 ...
[2025-06-17 11:34:39,720 DEBUG] node_cli.core.ssl.utils:50 - MainThread - Starting detached subprocess: ['openssl', 's_client', '-connect', '127.0.0.1:4536', '-verify_return_error', '-verify', '2']
[2025-06-17 11:34:40,721 ERROR] node_cli.core.ssl.check:207 - MainThread - Healthcheck connection failed
[2025-06-17 11:34:40,721 DEBUG] node_cli.core.ssl.utils:63 - MainThread - Detached process ['openssl', 's_client', '-connect', '127.0.0.1:4536', '-verify_return_error', '-verify', '2'] output:
verify depth is 2
Can't use SSL_get_servername
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = E5
verify return:1
depth=0 CN = andromeda03.skale.prod.chorus1.net
verify return:1
CONNECTED(00000003)
---
Certificate chain
 0 s:CN = andromeda03.skale.prod.chorus1.net
   i:C = US, O = Let's Encrypt, CN = E5
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: Jun  1 09:02:09 2025 GMT; NotAfter: Aug 30 09:02:08 2025 GMT
 1 s:CN = andromeda03.skale.prod.chorus1.net
   i:C = US, O = Let's Encrypt, CN = E5
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: Jun  1 09:02:09 2025 GMT; NotAfter: Aug 30 09:02:08 2025 GMT
 2 s:C = US, O = Let's Encrypt, CN = E5
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDsTCCAzegAwIBAgISBiRcwBdCvpdtgFs13lgvLhPAMAoGCCqGSM49BAMDMDIx
CzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQDEwJF
NTAeFw0yNTA2MDEwOTAyMDlaFw0yNTA4MzAwOTAyMDhaMC0xKzApBgNVBAMTImFu
ZHJvbWVkYTAzLnNrYWxlLnByb2QuY2hvcnVzMS5uZXQwWTATBgcqhkjOPQIBBggq
hkjOPQMBBwNCAATKxaTFt2sHAaOWmYDfZ8d1yBNLrSCQ45x2nHa0ZNwUKZQ3ZOkP
k5JuZ3cJ7l5TVqzV+cXFKk3CC5N+dRkgcqaSo4ICMDCCAiwwDgYDVR0PAQH/BAQD
AgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAA
MB0GA1UdDgQWBBQdFZybIYsV/MB3zenDUHw6u4hzwTAfBgNVHSMEGDAWgBSfK1/P
PCFPnQS37SssxMZwi9LXDTAyBggrBgEFBQcBAQQmMCQwIgYIKwYBBQUHMAKGFmh0
dHA6Ly9lNS5pLmxlbmNyLm9yZy8wLQYDVR0RBCYwJIIiYW5kcm9tZWRhMDMuc2th
bGUucHJvZC5jaG9ydXMxLm5ldDATBgNVHSAEDDAKMAgGBmeBDAECATAtBgNVHR8E
JjAkMCKgIKAehhxodHRwOi8vZTUuYy5sZW5jci5vcmcvNjUuY3JsMIIBBAYKKwYB
BAHWeQIEAgSB9QSB8gDwAHUAzPsPaoVxCWX+lZtTzumyfCLphVwNl422qX5UwP5M
DbAAAAGXKu+sFQAABAMARjBEAiBBAFasICRh4gM7oz3js+yD0W8A0U8qe05/Angl
oRNKVQIgT3AQtbD5Tgh+sVvQFlWW4zInBBiu/VLg29UXBHfFWgwAdwAN4fIwK9MN
wUBiEgnqVS78R3R8sdfpMO8OQh60fk6qNAAAAZcq76wAAAAEAwBIMEYCIQDkqqCR
qzF2ne9Cl1FFN/0xpWftmZ69Hq9ubJGhi66mtAIhANhxoAPmDN9VMtnRi6/CqGuC
9m7nwulUcHFRnfLxnFbIMAoGCCqGSM49BAMDA2gAMGUCMQCMbgO7YDT08oPGlx/v
IhosyRDWndxiKH13QUjw2ndMUxBZZxDAbIHBIoCVLwehUMoCMHiu99cp88XkC714
T5o+ZwFfkuVsvWjjx5FWfr5VdTbftzA5elPUOG+BsCOYMuNwLg==
-----END CERTIFICATE-----
subject=CN = andromeda03.skale.prod.chorus1.net
issuer=C = US, O = Let's Encrypt, CN = E5
---
No client certificate CA names sent
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3461 bytes and written 407 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE

[2025-06-17 11:34:40,722 DEBUG] node_cli.core.ssl.utils:63 - MainThread - Detached process ['openssl', 's_server', '-cert', '/etc/letsencrypt/live/andromeda03.skale.prod.chorus1.net/fullchain.pem', '-cert_chain', '/etc/letsencrypt/live/andromeda03.skale.prod.chorus1.net/fullchain.pem', '-key', '/etc/letsencrypt/live/andromeda03.skale.prod.chorus1.net/privkey.pem', '-WWW', '-accept', '127.0.0.1:4536', '-verify_return_error', '-verify', '1'] output:
verify depth is 1
Using default temp DH parameters
ACCEPT
40374AB225780000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:322:

[2025-06-17 11:34:40,722 ERROR] node_cli.core.ssl.upload:36 - MainThread - Certificate/key pair is incorrect
Traceback (most recent call last):
  File "node_cli/core/ssl/upload.py", line 34, in upload_cert
  File "node_cli/core/ssl/check.py", line 95, in check_cert_openssl
  File "node_cli/core/ssl/check.py", line 208, in check_ssl_connection
node_cli.core.ssl.check.SSLHealthcheckError: OpenSSL connection verification failed
[2025-06-17 11:36:02,641 DEBUG] __main__:117 - MainThread - cmd: /usr/local/bin/skale lvmpy heal --yes, v.2.6.2
[2025-06-17 11:39:03,042 DEBUG] __main__:117 - MainThread - cmd: /usr/local/bin/skale lvmpy heal --yes, v.2.6.2

[kucharskim]

I've added debugging code:

     ]
     expose_output = not silent
     with detached_subprocess(ssl_check_cmd, expose_output=expose_output) as dp:
         time.sleep(1)
         code = dp.poll()
+        print(f"This is a TEST and code is {code}", file=sys.stderr, flush=True)
         if code is not None:
             logger.error('Healthcheck connection failed')
             raise SSLHealthcheckError('OpenSSL connection verification failed')

[kucharskim]

This is the result:

Jun 17 17:23:12 python[231920]: This is a TEST and code is 0
Jun 17 17:23:12 python[231920]: Command failed with following errors:
Jun 17 17:23:12 python[231920]: --------------------------------------------------
Jun 17 17:23:12 python[231920]: Certificate check failed. OpenSSL connection verification failed
Jun 17 17:23:12 python[231920]: --------------------------------------------------
Jun 17 17:23:12 python[231920]: You can find more info in /root/.skale/.skale-cli-log/debug-node-cli.log
Jun 17 17:23:12 systemd[1]: mikolaj-is-testing-skale-ssl-upload.service: Main process exited, code=exited, status=1/FAILURE

[kucharskim]

Script succeeds when code is None and anything else, the script fails. However I am not sure should it fail when code is 0 (zero). The code is None when openssl is still running, so I think that is also not good. I think there should be a loop there when dp.poll() is None it should wait for an int code, as proper exit code.

[oleksandrSydorenkoJ]

Verified 2.6.3-beta.3
#826