Discrepancy in eval_end2end_ace2005e

[InfiniH]

Hi,
I'm encountering a puzzling problem while running the eval_end2end_ace2005e.py with the DEGREE_e2e_ace05e.mdl pretrained model. The evaluation results I'm obtaining are significantly different from the results reported in the associated paper, even though I have not made any modifications to the model itself.
Has anyone else encountered a similar problem when running the eval_end2end_ace2005e with pretrained model?
Could you kindly provide insights into potential areas that might lead to such discrepancies? Are there any data processing intricacies that could be causing this?
Any assistance in troubleshooting this matter would be greatly appreciated.

[ihungalexhsu]

It would be easier for us to identify your issue if you could save the models generated text and share some samples in the thread.

[InfiniH]

The generative model I'm utilizing was acquired from pre-trained models (DEGREE_e2e_ace05e.mdl) within the code, and it wasn't trained by me personally. Now I can furnish you with the log files for your examination.

{ "pred text": [ "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is <Trigger>", "Event trigger is Former \n Howard Davies stopped working for some people or some organization at somewhere.", "Event trigger is <Trigger>", "Event trigger is begins \n Callum McCarthy got new job and was hired by some people or some organization in London.", "Event trigger is <Trigger>", "Event trigger is <Trigger>" ], "pred triggers": [ [ 0, 1, "Personnel:End-Position" ] ], "gold triggers": [ [ 0, 1, "Personnel:End-Position" ], [ 5, 6, "Personnel:Start-Position" ] ], "pred roles": [ [ [ 0, 1, "Personnel:Start-Position" ], [ 15, 16, "Place" ] ] ], "gold roles": [ [ [ 0, 1, "Personnel:End-Position" ], [ 3, 5, "Person" ] ], [ [ 5, 6, "Personnel:Start-Position" ], [ 3, 5, "Person" ] ] ] },

[InfiniH]

In order to better address the problem, I would appreciate it if you could provide me with your email address. I'll be able to send you the log files to resolve this issue more thoroughly.

[ej0cl6]

Notice that sometimes package version matters as well. So please check if you packages match ours. Especially for transformer.