Fixes #39645 - Replace Fog::SSH/SCP with system SSH

[lzap]

Rubygem net-ssh does not provide all features for PQC and overall, we would like to simplify crypto stack in Foreman. This patch refactors the SSH service class so it shells out to ssh and sshpass instead.

The previous workflow was also quite complex, the main goal of the SSH finish script is not super-resilient SSH scheduling, but to bootstrap other components like Remote Execution or Ansible and for this, there is no need to establish a long-lasting connection via sockets.

Furthermore, the old solution copied the script using scp which is also unnecessary. Instead, there are simply two connections made using ssh command:

• The initial wait for the VM is done via ping method, previously this was named estabilish_connection! but this would be confusing. I made the method explicit so constructor does not perform this - this was confusing behavior.
• The final call is then done via deploy! method, instead of scp the script is simply passed via STDIN - no need of temporary files.

Other major changes to further simplify the code:

• For sshkey authentication, the temporary file with the private key is no longer created in the constructor relying on the cleanup method call. Instead, it is created just for the minimum possible time in the run_ssh method and cleaned immediately.
• Logging was moved to run_ssh method (DRY).
• Unit test was written by Claude, a bit heavy to my taste I will happily drop some cases.

I thought for a moment that we could even get rid of the ping and simply try to execute the template directly in a loop, but I discarded the idea. It is safer to be explicit.

I tested this via rails command in the proper SELinux policy with a small change (see the linked PR):

systemd-run --pipe --wait --service-type=oneshot \
  -p SELinuxContext=system_u:system_r:foreman_rails_t:s0 \
  -p User=foreman \
  -p WorkingDirectory=/usr/share/foreman \
  -p Environment=RAILS_ENV=production \
  -p PrivateTmp=no \
  /usr/share/foreman/bin/rails runner /tmp/test_ssh_provision.rb

This is a followup for previous work in #10518 and #10615 which was reverted.

TODO

• Create PR to SELinux policy with the changes.
• Add ssh and sshpass dependencies to foreman package.
• Drop ssh-ssh/scp packages from our repos. (After we drop the library from ReX)

Relevant PRs:

• Add openssh-clients dependency to foreman foreman-packaging#14041
• Add openssh-client and sshpass to foreman Depends foreman-packaging#14042

[ogajduse]

Packaging ACK, as I see that you have the packaging changes on your radar.

Reading from the PR description:

Drop ssh-ssh/scp packages from our repos.

[lzap]

Packaging ACK, as I see that you have the packaging changes on your radar.

For the record, the packages can be only removed after ReX also drops the dependency, it still needs it for relatively trivial reason and there is a ticket for that. I created https://redhat.atlassian.net/browse/SAT-49367 and I will move it to the ReX epic if core is merged earlier.

[stejskalleos]

Few comments from the first look, after the fix I will test the workflow

[stejskalleos]

From Ruby 3.0.z docs:

This method is not recommended and exists mostly for backward compatibility. Please use Tempfile.create instead, which avoids the cost of delegation, does not rely on a finalizer, and also unlinks the file when given a block.

[stejskalleos]

Comment from AI:

ssh_options missing -oBatchMode=yes

Not setting -oBatchMode=yes means SSH could prompt for a passphrase interactively if key auth is configured but the key is encrypted (or if sshpass fails for password auth). Since these are machine-generated ephemeral keys in a provisioning context this is low-risk, but worth noting.

[lzap]

Thanks for the review, all remarks done except not sure what you meant by the 0600 comment. I just added the test there for extra coverage - it is hardcoded in (both) functions).

[stejskalleos]

With the new changes, provisioning from an image in Libvirt no longer works.

The foreman will timeout eventually:

2026-09-07T13:18:56 [W|app|2cf3c822] Warning: Permanently added '192.168.190.94' (ED25519) to the list of known hosts.
2026-09-07T13:18:56 [W|app|2cf3c822] root@192.168.190.94: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive,hostbased).
2026-09-07T13:18:56 [D|app|2cf3c822] SSH connection to 192.168.190.94 failed, retrying...
.
.
.
2026-09-07T13:18:59 [W|app|2cf3c822] Warning: Permanently added '192.168.190.94' (ED25519) to the list of known hosts.
2026-09-07T13:18:59 [W|app|2cf3c822] root@192.168.190.94: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive,hostbased).
2026-09-07T13:18:59 [W|app|2cf3c822] Failed to login via SSH to saul-scowden.dhcp-libvirt.lan: ERF42-1034 [Foreman::Exception]: Failed to connect to 192.168.190.94 after 120 seconds
2026-09-07T13:18:59 [I|app|2cf3c822] Backtrace for 'Failed to login via SSH to saul-scowden.dhcp-libvirt.lan: ERF42-1034 [Foreman::Exception]: Failed to connect to 192.168.190.94 after 120 seconds' error (Foreman::Exception): ERF42-1034 [Foreman::Exception]: Failed to connect to 192.168.190.94 after 120 seconds

but the ssh locally works:

ssh root@192.168.190.94
Web console: https://saul-scowden.dhcp-libvirt.lan:9090/ or https://192.168.190.94:9090/

Last login: Mon Sep  7 11:18:27 2026 from ...

I used the https://cloud.centos.org/centos/10-stream/x86_64/images/CentOS-Stream-GenericCloud-10-latest.x86_64.qcow2 image with following sshd_config:

Port 22
AddressFamily any
ListenAddress 0.0.0.0
ListenAddress ::

# Permit Full Root & Anonymous Access
PermitRootLogin yes
PermitEmptyPasswords yes
PermitUserEnvironment yes

# Enable All Authentication Methods
PubkeyAuthentication yes
PasswordAuthentication yes
KbdInteractiveAuthentication yes
HostbasedAuthentication yes
GSSAPIAuthentication yes

# Disable Security Controls & Strict Checks
StrictModes no
UsePAM yes
IgnoreRhosts no
IgnoreUserKnownHosts yes

# Network & Forwarding Features (Fully Open)
AllowTcpForwarding yes
AllowAgentForwarding yes
X11Forwarding yes
X11UseLocalhost no
GatewayPorts yes
PermitTTY yes
PermitTunnel yes

# Logging
SyslogFacility AUTH
LogLevel INFO

# Subsystem Definition
Subsystem sftp /usr/libexec/openssh/sftp-server

[lzap]

Libvirt does not support SSH keys, can you either try with EC2/RHOS, or simply fake this by editing the image via Rails Console and setting SSH KeyPair there? SSH orchestration does not care and if SSH key pair is set, it will give it a priority.