add transfer modules

[hy395]

Description of your changes

• add hound_transfer.py:

  • main script for running transfer;

• add transfer_helper.py:

  • helper functions to insert adapter layers to existing architecture.
  • helper functions to merge adapters.

• layers.py:

  • added IA3, IA3_ff, Lora, Locon, AdapterHoulsby class definitions;
  • add seqlen_train argument to allow model to take any input seq length;
  • add additional arguments to SqueezeExcite class;

• blocks.py:

  • add options to specify adapter in json file,
  • add additional arguments to SqueezeExcite.

• seqnn.py:

  • add methods to insert adapter;

• hound_eval.py:

  • optionally use fl16;

• hound_eval_spec.py:

  • optionally use fl16;

• trainer.py:

  • add option to do loss scaling;
  • add function to print GPU memory usage when training;

evaluation code:

• add borzoi_test_genes.py
• add pygene.py to support borzoi_test_genes.py

test:

• code to test inserting ia3; see if added trainable variables match expectation.

Issue ticket number and link

Type of change

• Bug fix
• New feature
  • Backwards Incompatible?
• Refactoring / code clean-up
• Documentation add / update
• Automated Test
• Other (please specify)

(If applicable) How has this been tested?

see if trainable variables match expectation.

[davek44]

for big borzoi, this is too low; I want to exclude all of the ChIP groups.

[davek44]

It doesn't look like the code parses this for locon.

[hy395]

you are correct. i removed the option to change adapter_latent in locon4, leaving it to default.