Skip to content

Conversation

@aldbr
Copy link
Contributor

@aldbr aldbr commented Oct 31, 2025

In LHCb, we would like to submit a very large number of jobs in parallel via the SSHCE, and we are generally limited to about 300 jobs per cycle because the stamps are generated by the SSHCE and sent through SSH to a batch system. Problem: the line sent is way too long and cannot be executed.

Note: I have not added too many checks for failures for now, I keep it simple and will give it a try in LHCb to see how it works.

BEGINRELEASENOTES
*Resources
CHANGE: SSHCE export inputs/import outputs as a JSON file instead of using the stdout/stderr
ENDRELEASENOTES

@chaen
Copy link
Contributor

chaen commented Nov 10, 2025

Monitor jobs by batch of 1000 (or more) instead of 100

@aldbr
Copy link
Contributor Author

aldbr commented Nov 13, 2025

Going through this I discovered a few old problems with the SSHCE, as well as a minor security issue that I will solve in another PR, but these changes are working.
I think it's also a good opportunity to finish the scripts I am making for testing the CEs, and finish #7703

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants