BUG: Empty nodes are not handled properly

[ijindal]

Issue: For some the languages\corpuses, a issues with the token counting is observed. This issue is related to empty nodes in UD enhanced dependency. Arguments in UP data counts empty nodes as one of the token id.

Impact: So far, we know some of the repositories are affected.

• Access the impact. List the repositories which are impacted?

Fix:

• Need to programmatically identify the empty nodes in each sentence and adjust the respective token ids in arguments.

[michalu]

The list of affected repositories/files:

UD_Ukrainian-IU/uk_iu-up-test.conllup
UD_Ukrainian-IU/uk_iu-up-train.conllup
UD_Ukrainian-IU/uk_iu-up-dev.conllup

UD_Dutch-LassySmall/nl_lassysmall-up-dev.conllup
UD_Dutch-LassySmall/nl_lassysmall-up-test.conllup
UD_Dutch-LassySmall/nl_lassysmall-up-train.conllup

UD_Russian-SynTagRus/ru_syntagrus-up-test.conllup
UD_Russian-SynTagRus/ru_syntagrus-up-train.conllup
UD_Russian-SynTagRus/ru_syntagrus-up-dev.conllup

UD_Czech-PDT/cs_pdt-up-train.conllup
UD_Czech-PDT/cs_pdt-up-dev.conllup
UD_Czech-PDT/cs_pdt-up-test.conllup

UD_Spanish-AnCora/es_ancora-up-dev.conllup
UD_Spanish-AnCora/es_ancora-up-train.conllup
UD_Spanish-AnCora/es_ancora-up-test.conllup

UD_Finnish-TDT/fi_tdt-up-train.conllup
UD_Finnish-TDT/fi_tdt-up-dev.conllup
UD_Finnish-TDT/fi_tdt-up-test.conllup

UD_Czech-FicTree/cs_fictree-up-train.conllup
UD_Czech-FicTree/cs_fictree-up-dev.conllup
UD_Czech-FicTree/cs_fictree-up-test.conllup

UD_Czech-CAC/cs_cac-up-train.conllup
UD_Czech-CAC/cs_cac-up-test.conllup

UD_Dutch-Alpino/nl_alpino-up-train.conllup
UD_Dutch-Alpino/nl_alpino-up-dev.conllup
UD_Dutch-Alpino/nl_alpino-up-test.conllup