Question about CASP14/CASP15 reconstruction results vs *Flow Autoencoders are Effective Protein Tokenizers*

[auroua]

Hello,

Thank you very much for making this repository and code available.

While reading the paper “Flow Autoencoders are Effective Protein Tokenizers” (https://arxiv.org/abs/2510.00351), I noticed that it also reports reconstruction performance on CASP14 and CASP15. However, the reported results are quite different from those obtained in this repository.

• The first figure below is taken from Flow Autoencoders are Effective Protein Tokenizers:

  

• The second figure shows the corresponding reconstruction results from this repository:

  

The values appear to differ substantially between the two. I was wondering if you could help clarify what might be causing this discrepancy. Any insight into the source of these differences would be greatly appreciated.

[mahdip72]

Thanks @auroua for the thoughtful question and for digging into the results.

We have been investigating the same discrepancy as well. For ESM-3 VQVAE, we implemented our own evaluation scripts because the full reference scripts aren’t publicly available. Since we weren’t certain our reimplementation exactly matches the authors’ setup, we asked the ESM team to confirm our implementation here.
For FoldToken 4, we re-implemented an efficient version based on this repo. We requested verification in this issue and also opened a PR for review (So far we haven’t received a response):

• Issue: Standalone PDB ⇄ token inference scripts A4Bio/FoldToken_open#5
• PR: Add standalone PDB ↔ token inference scripts A4Bio/FoldToken_open#6

We’ve documented these differences and our current understanding in the README. Independent validation of our ESM-3 VQVAE and FoldToken 4 evaluation scripts would be very welcome.

As for the table in Flow Autoencoders are Effective Protein Tokenizers, my impression is that they reported metrics from the original papers rather than re-running all methods under a single evaluation pipeline, but I could be wrong. If you find otherwise, we would really appreciate an update. If the reference authors share evaluation scripts or additional details, we will happily re-run and update the table.

Thanks again for raising this!

[auroua]

Hi @mahdip72,

Thank you very much for your clear and detailed reply.

I will adopt this structural tokenizer in my own model and evaluate its performance. Once I have updated results, I’ll share them here for reference.

Thanks again for open-sourcing and maintaining such a great project for the community!

[mahdip72]

@auroua I have news! After reiterating on the evaluation scripts, I realized that the reconstruction issue comes from the missing coordinates and missing amino acids in experimental PDBs.
It turns out that many methods handle those noisy regions differently, which causes different indexing which breaks the alignment and consequently TM-score and rmsd calculations.
I have resolved them one by one and reran the evaluation pipeline and updated the manuscript. Also, new methods have been added there.

By that, we have a full fledged and fair comparison with all relevant open source methods.

Hope that helps your work!