Looper slow to execute with large PEP

[donaldcampbelljr]

Discovered this when troubleshooting #543 . This is related to #476.

PEP from issue 543 is 289 samples and executing a looper run on it is quite slow, causing looper to hang for a few seconds before submitting the jobs.

[donaldcampbelljr]

Actually the subsample table is 1439 samples in size!

[nleroy917]

I had a 10,000 sample table recently and it took about a minute to start submitting, and then 7 minutes total to submit all my (batched) jobs

[donaldcampbelljr]

• Find out out where/why its slow. Is it Looper or is it in Peppy.

[donaldcampbelljr]

Is it due to submission script writing to local disk?

[donaldcampbelljr]

We looked at the a .prof file using snake viz this morning and started digging into the results.

Some areas to investigate for improvement:

looper/looper/project.py

Lines 690 to 700 in 450ef7d

	for sample in self.samples:
	if piface_key in sample and sample[piface_key]:
	piface_srcs = sample[piface_key]
	if isinstance(piface_srcs, str):
	piface_srcs = [piface_srcs]
	for source in piface_srcs:
	source = self._resolve_path_with_cfg(source)
	try:
	PipelineInterface(
	source, pipeline_type=PipelineLevel.SAMPLE.value
	)

Iterating over each sample and the subsequent pipeline interfaces to build a PipelineInterface via loading a .yaml from disk could be improved by caching the pipeline interface .yaml in memory and pulling it from a hashmap instead.

Other areas for improvement were found in Peppy during attribute_merging:
https://github.com/pepkit/peppy/blob/60523c51cc7243680797f681a9d49a1ecec3c6ca/peppy/project.py#L774-L790

There are a few nested loops in this function.

Performance for running large PEP:

14.5 seconds with _samples_by_piface taking 5.6 seconds and attr_merge from Peppy taking 3.6 seconds to complete

[nleroy917]

attr_merge from Peppy taking 3.6 seconds to complete

attr_merge is quite algorithmically inefficient with up to 4 nested loops. Good opportunity to use pandas syntax to speed that one up or some up with a more clever algorithm

[donaldcampbelljr]

First commit using precache did not do much. In fact seemed a little slower by 0.5 seconds. However, the second commit where I used a preloaded yaml during validation seemed to shave off ~ 5 seconds.

[donaldcampbelljr]

I refactored to properly use the correct input schema and have that be pre-loaded before creating all the pifaces. Unfortunately this actually decreased last week's performance so we are only using a 2 sec gain (out of 14.5 seconds). So, I'm not sure this is the solution we thought it would be to significantly help performance.

[donaldcampbelljr]

Maybe try using this rust implementation of json-schema validation:
https://github.com/Stranger6667/jsonschema/tree/master/crates/jsonschema-py