Long time needed to find CNEs across entire chromosomes

[tbusschau]

Hi @lorrainea ,

I'm using CNEFinder to identify CNEs across the genomes of snakes. I'm running one chromosome at a time. Previously, the analyses finished quickly for fragments of the genome but running the entire CHR1 (±300mb) is taking very long. The analyses has been stuck at the following step for 4 days now.
-Extending 1841772 merged matches of minimum length 75, with an additional extension threshold of 0.05

I'm running CNEFinder with 64 cores and 180gb ram. I just started a new run using 128 cores and 400gb ram and hoping for some improvement.

Is this normal or do you have any suggestions to speed up the analyses?

Best regards,
Theo

[lorrainea]

Hi, for the length of the sequences given, it seems normal. You should try and alter the parameters if possible.
For example --mem-length can be increased from the default value of 18. Also if you can increase --sim-threshold that would also speed up the computation. Also try and use multiple threads if possible.

[tbusschau]

Hi Lorraine,

Thank you. I've tried using between 8 and 128 threads already.
The latest run I tried after your advice is as follows.
Computing CNEs with minimum length 50, maximum length 2000 and similarity threshold 98.00%
-Identifying maximal exact matches of minimum length 25
-Merging 3139692 maximal exact matches
-Extending 3139692 merged matches of minimum length 25, with an additional extension threshold of 0.05

Using a minimum length of 50 and similarity of 0.98 autimatically increased the maximal exact matches length to 25.
In this run and other runs I've tried the merged matches increased quite a bit. Is the goal to reduce the number of merge matches to reduce the computation time?

Alternatively, would it make sense to search smaller overlapping fragments of the query sequence at a time and merge the results?

Best,
Theo

[lorrainea]

Could you possibly send over your files so I can have a play around with some parameters? Increasing Q should have decreased the number of maximal exact matches but it seems here to have unusually increased.