open benchmark leaderboard

[pengzhangzhi]

Hi,

Great work!
I wonder if you have open benchmark results for all the models? Not just the mutation prediction tasks.

[tyang816]

Hi,
You can find more detailed results in our arXiv paper (https://arxiv.org/pdf/2503.15438).
Thx!

[pengzhangzhi]

thank you! Would you like to elucidate the training hyperparameters in the paper? Would love to reproduce the results : )

[tyang816]

Please see Section 5.1.😊

[pengzhangzhi]

Hey, thank you for the response : ) I tried to run the benchmarks following the hyperparams, and it seems hard to exactly replicate the number in the paper. I wonder if you can open-source the scripts that implement all the experiments? Would love to follow that : )

[tyang816]

Hey, thank you for the response : ) I tried to run the benchmarks following the hyperparams, and it seems hard to exactly replicate the number in the paper. I wonder if you can open-source the scripts that implement all the experiments? Would love to follow that : )

You can find the scripts here.

We didn't adjust the hyperparameters too much. It could be because of the PyTorch or GPU version that causes some differences. Could you please give us your experimental situation so that we can reproduce it?
Thank you!

[pengzhangzhi]

Thanks. You are saying that the following hyper-params should be used for all benchmarks & models? It doesn't seem complete/align with the paper as in the paper, you use #epochs 100. Also other configs like early stop. I generally believe It's reproducible, but just worry about these details. I got weridly bad results on the mutation tasks, will post it later.

# train with batch token
dataset=DeepLocBinary
pdb_type=AlphaFold2
plm_model=esm2_t30_150M_UR50D
lr=5e-4
python src/train.py \
    --plm_model facebook/$plm_model \
    --dataset_config data/$dataset/"$dataset"_"$pdb_type"_HF.json \
    --learning_rate $lr \
    --gradient_accumulation_steps 8 \
    --batch_token 12000 \
    --output_dir debug/$dataset/$plm_model \
    --output_model_name af2_lr"$lr"_bt12k_ga8.pt

[pengzhangzhi]

FYI, I'm using the following scripts to launch the training for all the tasks, let me know if u can spot anything sus

COMMON_ARGS="--training_method plm-lora \
--pooling_method mean \
--learning_rate 0.0005 \
--num_epochs 100 \
--max_seq_len 1024 \
--gradient_accumulation_steps 2 \
--warmup_steps 0 \
--patience 10 \
--num_workers 4 \
--batch_token 48000 \
"

# List of dataset configs without AlphaFold2 or ESMFold
declare -A DATASETS=(
    ["Demo_Solubility"]="data/Demo/Demo_Solubility_HF.json"
    ["DeepET_Topt"]="data/DeepET_Topt/DeepET_Topt_HF.json"
    ["DeepLoc2Multi"]="data/DeepLoc2Multi/DeepLoc2Multi_HF.json"
    ["DeepLocBinary"]="data/DeepLocBinary/DeepLocBinary_HF.json"
    ["DeepLocMulti"]="data/DeepLocMulti/DeepLocMulti_HF.json"
    ["DeepSol"]="data/DeepSol/DeepSol_HF.json"
    ["DeepSoluE"]="data/DeepSoluE/DeepSoluE_HF.json"
    ["EC"]="data/EC/EC_HF.json"
    ["eSOL"]="data/eSOL/eSOL_HF.json"
    ["FLIP_AAV_des-mut"]="data/FLIP_AAV/FLIP_AAV_des-mut_HF.json"
    ["FLIP_AAV_low-vs-high"]="data/FLIP_AAV/FLIP_AAV_low-vs-high_HF.json"
    ["FLIP_AAV_mut-des"]="data/FLIP_AAV/FLIP_AAV_mut-des_HF.json"
    ["FLIP_AAV_one-vs-rest"]="data/FLIP_AAV/FLIP_AAV_one-vs-rest_HF.json"
    ["FLIP_AAV_sampled"]="data/FLIP_AAV/FLIP_AAV_sampled_HF.json"
    ["FLIP_AAV_seven-vs-rest"]="data/FLIP_AAV/FLIP_AAV_seven-vs-rest_HF.json"
    ["FLIP_AAV_two-vs-rest"]="data/FLIP_AAV/FLIP_AAV_two-vs-rest_HF.json"
    ["FLIP_GB1_low-vs-high"]="data/FLIP_GB1/FLIP_GB1_low-vs-high_HF.json"
    ["FLIP_GB1_one-vs-rest"]="data/FLIP_GB1/FLIP_GB1_one-vs-rest_HF.json"
    ["FLIP_GB1_sampled"]="data/FLIP_GB1/FLIP_GB1_sampled_HF.json"
    ["FLIP_GB1_three-vs-rest"]="data/FLIP_GB1/FLIP_GB1_three-vs-rest_HF.json"
    ["FLIP_GB1_two-vs-rest"]="data/FLIP_GB1/FLIP_GB1_two-vs-rest_HF.json"
    ["GO_BP"]="data/GO_BP/GO_BP_HF.json"
    ["GO_MF"]="data/GO_MF/GO_MF_HF.json"
    ["GO_CC"]="data/GO_CC/GO_CC_HF.json"
    ["MetalIonBinding"]="data/MetalIonBinding/MetalIonBinding_HF.json"
    ["PaCRISPR"]="data/PaCRISPR/PaCRISPR_HF.json"
    ["PETA_CHS_Sol"]="data/PETA_CHS_Sol/PETA_CHS_Sol_HF.json"
    ["PETA_LGK_Sol"]="data/PETA_LGK_Sol/PETA_LGK_Sol_HF.json"
    ["PETA_TEM_Sol"]="data/PETA_TEM_Sol/PETA_TEM_Sol_HF.json"
    ["ProtSolM"]="data/ProtSolM/ProtSolM_HF.json"
    ["SortingSignal"]="data/SortingSignal/SortingSignal_HF.json"
    ["TAPE_Fluorescence"]="data/TAPE_Fluorescence/TAPE_Fluorescence_HF.json"
    ["TAPE_Stability"]="data/TAPE_Stability/TAPE_Stability_HF.json"
    ["Thermostability"]="data/Thermostability/Thermostability_HF.json"
)

[tyang816]

Thanks. You are saying that the following hyper-params should be used for all benchmarks & models? It doesn't seem complete/align with the paper as in the paper, you use #epochs 100. Also other configs like early stop. I generally believe It's reproducible, but just worry about these details. I got weridly bad results on the mutation tasks, will post it later.

# train with batch token
dataset=DeepLocBinary
pdb_type=AlphaFold2
plm_model=esm2_t30_150M_UR50D
lr=5e-4
python src/train.py \
    --plm_model facebook/$plm_model \
    --dataset_config data/$dataset/"$dataset"_"$pdb_type"_HF.json \
    --learning_rate $lr \
    --gradient_accumulation_steps 8 \
    --batch_token 12000 \
    --output_dir debug/$dataset/$plm_model \
    --output_model_name af2_lr"$lr"_bt12k_ga8.pt

Other configs are default in the args.py, so I didn't write it in the training scrips.

[tyang816]

FYI, I'm using the following scripts to launch the training for all the tasks, let me know if u can spot anything sus

COMMON_ARGS="--training_method plm-lora \
--pooling_method mean \
--learning_rate 0.0005 \
--num_epochs 100 \
--max_seq_len 1024 \
--gradient_accumulation_steps 2 \
--warmup_steps 0 \
--patience 10 \
--num_workers 4 \
--batch_token 48000 \
"

# List of dataset configs without AlphaFold2 or ESMFold
declare -A DATASETS=(
    ["Demo_Solubility"]="data/Demo/Demo_Solubility_HF.json"
    ["DeepET_Topt"]="data/DeepET_Topt/DeepET_Topt_HF.json"
    ["DeepLoc2Multi"]="data/DeepLoc2Multi/DeepLoc2Multi_HF.json"
    ["DeepLocBinary"]="data/DeepLocBinary/DeepLocBinary_HF.json"
    ["DeepLocMulti"]="data/DeepLocMulti/DeepLocMulti_HF.json"
    ["DeepSol"]="data/DeepSol/DeepSol_HF.json"
    ["DeepSoluE"]="data/DeepSoluE/DeepSoluE_HF.json"
    ["EC"]="data/EC/EC_HF.json"
    ["eSOL"]="data/eSOL/eSOL_HF.json"
    ["FLIP_AAV_des-mut"]="data/FLIP_AAV/FLIP_AAV_des-mut_HF.json"
    ["FLIP_AAV_low-vs-high"]="data/FLIP_AAV/FLIP_AAV_low-vs-high_HF.json"
    ["FLIP_AAV_mut-des"]="data/FLIP_AAV/FLIP_AAV_mut-des_HF.json"
    ["FLIP_AAV_one-vs-rest"]="data/FLIP_AAV/FLIP_AAV_one-vs-rest_HF.json"
    ["FLIP_AAV_sampled"]="data/FLIP_AAV/FLIP_AAV_sampled_HF.json"
    ["FLIP_AAV_seven-vs-rest"]="data/FLIP_AAV/FLIP_AAV_seven-vs-rest_HF.json"
    ["FLIP_AAV_two-vs-rest"]="data/FLIP_AAV/FLIP_AAV_two-vs-rest_HF.json"
    ["FLIP_GB1_low-vs-high"]="data/FLIP_GB1/FLIP_GB1_low-vs-high_HF.json"
    ["FLIP_GB1_one-vs-rest"]="data/FLIP_GB1/FLIP_GB1_one-vs-rest_HF.json"
    ["FLIP_GB1_sampled"]="data/FLIP_GB1/FLIP_GB1_sampled_HF.json"
    ["FLIP_GB1_three-vs-rest"]="data/FLIP_GB1/FLIP_GB1_three-vs-rest_HF.json"
    ["FLIP_GB1_two-vs-rest"]="data/FLIP_GB1/FLIP_GB1_two-vs-rest_HF.json"
    ["GO_BP"]="data/GO_BP/GO_BP_HF.json"
    ["GO_MF"]="data/GO_MF/GO_MF_HF.json"
    ["GO_CC"]="data/GO_CC/GO_CC_HF.json"
    ["MetalIonBinding"]="data/MetalIonBinding/MetalIonBinding_HF.json"
    ["PaCRISPR"]="data/PaCRISPR/PaCRISPR_HF.json"
    ["PETA_CHS_Sol"]="data/PETA_CHS_Sol/PETA_CHS_Sol_HF.json"
    ["PETA_LGK_Sol"]="data/PETA_LGK_Sol/PETA_LGK_Sol_HF.json"
    ["PETA_TEM_Sol"]="data/PETA_TEM_Sol/PETA_TEM_Sol_HF.json"
    ["ProtSolM"]="data/ProtSolM/ProtSolM_HF.json"
    ["SortingSignal"]="data/SortingSignal/SortingSignal_HF.json"
    ["TAPE_Fluorescence"]="data/TAPE_Fluorescence/TAPE_Fluorescence_HF.json"
    ["TAPE_Stability"]="data/TAPE_Stability/TAPE_Stability_HF.json"
    ["Thermostability"]="data/Thermostability/Thermostability_HF.json"
)

May I know which task performance weridly? Are all the results not aligned with our paper?
Emmm... I don't have a GPU with such large CUDA memory, hh. It's hard for me to reproduce the batch_tokens=48000.

[tyang816]

--max_seq_len 1024 \

Oh, for the bacth_token mod, we didn't set the max_seq_len, maybe remove this will make results better?

[pengzhangzhi]

lol no worries! will keep u posted later. on the flight to Singapore