[JENKINS-64106] Switch SSH client from Trilead to Apache Mina SSH

[jenkins-infra-bot]

In order to avoid continued maintenance of a forked and dead SSH library (Trilead), we should migrate the code in SSH Build Agents to use Mina SSH. This will allow us to keep up to date with any cryptographic or security updates published upstream as well as support more SSH server configurations.

---

Originally reported by jvz, imported from: Switch SSH client from Trilead to Apache Mina SSH

• assignee: ifernandezcalvo
• status: Open
• priority: Minor
• component(s): ssh-slaves-plugin
• resolution: Unresolved
• votes: 2
• watchers: 6
• imported: 20251212-090250

Raw content of original issue

In order to avoid continued maintenance of a forked and dead SSH library (Trilead), we should migrate the code in SSH Build Agents to use Mina SSH. This will allow us to keep up to date with any cryptographic or security updates published upstream as well as support more SSH server configurations.

[jenkins-infra-bot]

ifernandezcalvo:

• Original comment link
• Raw content of original comment:

  First functional SSH client implemented with  Apache Mina https://github.com/kuisathaverat/ssh-slaves-plugin/tree/apache_mina there is a new SSH Launcher on the launcher selector "Launch agents via SSH (Apache Mina SSHD)" this uses Apache Mina SSHD library for the connections.
  

First functional SSH client implemented with Apache Mina https://github.com/kuisathaverat/ssh-slaves-plugin/tree/apache_mina there is a new SSH Launcher on the launcher selector "Launch agents via SSH (Apache Mina SSHD)" this uses Apache Mina SSHD library for the connections.

[jenkins-infra-bot]

allan_burdajewicz:

• Original comment link
• Raw content of original comment:

  ifernandezcalvo I was wondering if maybe this should be relying on https://issues.jenkins.io/browse/JENKINS-64105 ?
  

ifernandezcalvo I was wondering if maybe this should be relying on https://issue-redirect.jenkins.io/issue/64105 ?

[jenkins-infra-bot]

ifernandezcalvo:

• Original comment link
• Raw content of original comment:

  In the new implementation for Apache Mina SSH, I did not rely on the SSH Credentials plugin to verify or authenticate the SSH keys the correct implementation used for Trilead is unnecessary complex and does not add any value (5 lines of code VS ~100), it is something already implemented in the library.
  However, when I finish the work here I will stop maintaining Trilead SSH and at some point, the old Trilead SSH launcher should disappear. 
  

In the new implementation for Apache Mina SSH, I did not rely on the SSH Credentials plugin to verify or authenticate the SSH keys the correct implementation used for Trilead is unnecessary complex and does not add any value (5 lines of code VS ~100), it is something already implemented in the library.
However, when I finish the work here I will stop maintaining Trilead SSH and at some point, the old Trilead SSH launcher should disappear.

[jenkins-infra-bot]

viceice:

• Original comment link
• Raw content of original comment:

  Do we really need a new launcher method? can't we just replace the implementation of the existing? 
  

Do we really need a new launcher method? can't we just replace the implementation of the existing?

[jenkins-infra-bot]

ifernandezcalvo:

• Original comment link
• Raw content of original comment:

  I made a replacement of the library at https://github.com/kuisathaverat/ssh-slaves-plugin/tree/apache_mina I tested it an works, but there is some weird disconnections if there is no activity on the channel. I have peddling to review it and try new versions of the library.
  

I made a replacement of the library at https://github.com/kuisathaverat/ssh-slaves-plugin/tree/apache_mina I tested it an works, but there is some weird disconnections if there is no activity on the channel. I have peddling to review it and try new versions of the library.

[jenkins-infra-bot]

viceice:

• Original comment link
• Raw content of original comment:

  Oh, your branch is very old, it still uses/ compiles java v8. So it should be rebased. Maybe switching to java v17 will solve the issues.
  

Oh, your branch is very old, it still uses/ compiles java v8. So it should be rebased. Maybe switching to java v17 will solve the issues.

[jenkins-infra-bot]

ifernandezcalvo:

• Original comment link
• Raw content of original comment:

  I just rebase my changes to the latest core and Apache Mina API plugin that uses 2.15.0 Apache Mina ssh. After fixing some changes and refactors on packages, I made the same test I did three years ago. The new version can keep a stable connection for more than 10 minutes, and the library reports more info about the connection, so it is easy to debug errors. I will work in the following weeks to finish the launcher and all the stuff related to it. If everything goes fine, I will release a beta launcher that will replace the older one as soon as we are confident with the result.
  
  feat: new launcher using Apache Mina sshd library #570
  

I just rebase my changes to the latest core and Apache Mina API plugin that uses 2.15.0 Apache Mina ssh. After fixing some changes and refactors on packages, I made the same test I did three years ago. The new version can keep a stable connection for more than 10 minutes, and the library reports more info about the connection, so it is easy to debug errors. I will work in the following weeks to finish the launcher and all the stuff related to it. If everything goes fine, I will release a beta launcher that will replace the older one as soon as we are confident with the result.

#570

[jenkins-infra-bot]

felfert:

• Original comment link
• Raw content of original comment:

  I just tested this in my dev environment (I am the maintainer if the jclouds plugin which uses SSHLauncher) and this might solve a long standing problem where there is a resource leak with the trilead based launcher. The reason for that is, that the trilead ssh2 does not have a full implementation of client-side keepalive on the application level. They only provide a ping() method and leave the implementation of the actual logic to the user (which would be the Channel implementation in jenkins). In jclouds, destroying a node, with most cloud providers just result in a forced disposal of the agent VM, leaving the sshd on the VM no chance to send a proper disconnect. So for years now, whenever this happens, a reader thread of the trilead ssh keeps readin forever and never realizes that the remote host has gone. With active keepalive on the client side in mina this should be easy. Browsing thru your code, I noticed, that you already are setting the the various HEARTBEAT parameters in your ConnectionImpl so this should solve that problem. Perhaps it would be nice, to have these configurable. I also noticed a small problem in the config GUI: When testing, I switched between old SSHLauncher and SSHApacheMinaLauncher and whenever i changed the launcher, all the parameters (hostname, credentialsId etc) were reset. I believe, this is due to having two config.jelly each for the corresponding launcher. So this should probably changed into a single common config.jelly, providing common config for both launchers and then having extended parameters (like these hearbeat setting) in some separate jelly. Maybe I can offer some help (I just don't know how to mess with another user's PR on github. Maybe I can create PR's against your personal branch? Never done that before)
  
   
  
  Anyway: GREAT work 
  
  Cheers
  
   -Fritz
  

I just tested this in my dev environment (I am the maintainer if the jclouds plugin which uses SSHLauncher) and this might solve a long standing problem where there is a resource leak with the trilead based launcher. The reason for that is, that the trilead ssh2 does not have a full implementation of client-side keepalive on the application level. They only provide a ping() method and leave the implementation of the actual logic to the user (which would be the Channel implementation in jenkins). In jclouds, destroying a node, with most cloud providers just result in a forced disposal of the agent VM, leaving the sshd on the VM no chance to send a proper disconnect. So for years now, whenever this happens, a reader thread of the trilead ssh keeps readin forever and never realizes that the remote host has gone. With active keepalive on the client side in mina this should be easy. Browsing thru your code, I noticed, that you already are setting the the various HEARTBEAT parameters in your ConnectionImpl so this should solve that problem. Perhaps it would be nice, to have these configurable. I also noticed a small problem in the config GUI: When testing, I switched between old SSHLauncher and SSHApacheMinaLauncher and whenever i changed the launcher, all the parameters (hostname, credentialsId etc) were reset. I believe, this is due to having two config.jelly each for the corresponding launcher. So this should probably changed into a single common config.jelly, providing common config for both launchers and then having extended parameters (like these hearbeat setting) in some separate jelly. Maybe I can offer some help (I just don't know how to mess with another user's PR on github. Maybe I can create PR's against your personal branch? Never done that before)

 

Anyway: GREAT work

Cheers

 -Fritz

[jenkins-infra-bot]

felfert:

• Original comment link
• Raw content of original comment:

  Just a short followup:
  
  Have tested here with this branch of the jclouds-plugin . Just like expected: Works like a charm! Before: Spawning 10 VMs, then destroying them, left 10 sockets open on the controller host and 10 trilead reader threads in the JVM. With your ApacheMina launcher, after a minute the sockets are gone 
  

Just a short followup:

Have tested here with this branch of the jclouds-plugin . Just like expected: Works like a charm! Before: Spawning 10 VMs, then destroying them, left 10 sockets open on the controller host and 10 trilead reader threads in the JVM. With your ApacheMina launcher, after a minute the sockets are gone

[jenkins-infra-bot]

ifernandezcalvo:

• Original comment link
• Raw content of original comment:

  Really good news that you could integrate it with jclouds plugin. The plan is to fix the Spotbugs stuff and merge a first version, then we will add the key verifiers and the features that are not supported yet. About the keepalive settings, it is something we will put in the advanced configuration for sure, along with other now fixed settings.
  

Really good news that you could integrate it with jclouds plugin. The plan is to fix the Spotbugs stuff and merge a first version, then we will add the key verifiers and the features that are not supported yet. About the keepalive settings, it is something we will put in the advanced configuration for sure, along with other now fixed settings.

[jenkins-infra-bot]

[Original depends_on from Jira: JENKINS-64105]