OCR results in analysis results

[lyndonnixon]

In the TKK video chapter, Nelleke van der Krogt is not detected but her name is visible on screen - seems that OCR results are not included in the current analysis results?

[jthomsen]

yes, this is true; OCR is not included in the normal analysis; DanielS did extra OCR analysis for the UMONS scenario, but as far as I remember from the Potsdam meeting in January, he considers OCR too problematic for being integrated into the normal process

[lyndonnixon]

OK. Text visible in a frame could of course be used by an editor to determine the label of an annotation.

[lyndonnixon]

DanielS, e-mail on 24 July 2014:

OCR is not part of the integrated workflow but something that I provide on demand for certain tasks (some videos for UMons, some experiments for speaker database, MediaEval).
there is no EXMARaLDA integration yet
the script is erroneous in a sense that it creates broken xml because of an internal bug. I have some semi-automatic scripts that fix this, but this is nothing that can be integrated as-is in existing pipelines without prior bug-fixing where I don't know how long this takes because this is not my code.

Benefits of OCR is mainly:

recognition of entities that are not spoken (and thus do not appear in the subtitles)
recognition of entities that are spoken but out-of-vocabulary and thus out of reach for speech recognition

[lyndonnixon]

Since RBBs curation of its video material [1] seems to indicate OCR could provide useful labels to offer to editors for annotation of persons in the news item would it be feasible (for testing) to add OCR into the analysis of the current scenario video [2]?

[1] http://www.linkedtv.eu/wiki/index.php/File:Manual_curation_rbb_aktuell.xlsx
[2] http://api.linkedtv.eu/mediaresource/adb65e0a-642b-432f-aa86-c296dab0375a

[Daniel-Stein]

I can run video OCR on the file offline if this is needed. A fully functional integration into the REST service and the such is not feasible(*).

(*) for the time being and with my current time resources.

[rtroncy]

Regarding the broken XML output of the OCR tool of @Daniel-Stein, I would like to remind the work of UMons in fixing it, see:

• OCRCleaner: Perl script to clean up the broken XML file from Daniel S. See source code and usage at https://github.com/Zarfab/LinkedTV
• OCR2SRT: a tool to convert OCR results into an .srt file, developed by Fabien Grisard. See source code, executable and usage at https://github.com/Zarfab/LinkedTV
  I had already updated http://www.linkedtv.eu/wiki/index.php/Resources#WP1:_OCR_Toolkit some time ago

[Daniel-Stein]

fyi, I started processing of the video you mentioned. I can fix the xml beforehand using the scripts I made for mediaeval. ETA tomorrow. Note though that we have not established any porting into EXMARaLDA, so indeed considering the OCR2SRT tool might be an option (I have not worked with it so far).

[Daniel-Stein]

file processed, fixed and uploaded. You can find the xml in the ftp, at /Data/RBB/OCR

[rtroncy]

Thanks @Daniel-Stein ! Who is doing what now? This is not part of the EXB format, so this is not part of the workflow. We might want to run OCR2SRT for this specific file, and then NERD-ifying it to see what it gives. I believe the WP1 and WP5 leaders should step in to coordinate the work if there is something to do.
Further, who is working on matching OCR results with FaceDetection results?

[lyndonnixon]

• someone needs to convert the XML to SRT using the OCR2SRT tool ( @Daniel-Stein : since your OCR output atm is not immediately re-useable in LinkedTV at all does it not make sense to extend your LinkedTV instance of the OCR software to incorporate OCR2SRT - if possible - and add the SRT output directly to the Platform associated to the media resource - if possible - ask @jthomsen )
• then @rtroncy can NERDify the SRT to see what it gives (and - maybe manually - incorporate the annotation to the rest of the RDF annotation in the Platform -just a side comment that we would like to see both the original extracted text as well as any entities extracted for that text in the RDF since many names in RBB video probably do not have a DBPedia URI!)

[lyndonnixon]

Nothing has happened since and clearly it may not be possible to rely on @Daniel-Stein so how do we proceed on OCR in LinkedTV:
Option 1: CERTH or others take some other OCR tool esp one that may be able to be integrated into the WP1 REST service and put results directly into EXB
Option 2: IAIS has to run all TKK and RBB video through its OCR tool offline and provide results to the platform. @jthomsen maybe can integrate OCR2SRT so that results are converted to srt, @rtroncy TV2RDF is configured to check for both srt files (transcript AND optimal OCR results)
Option 3: lets not bother with OCR even though it seems useful for video annotation
I dont mind which option we take (number 3 is somehow default through inaction) but would like that WP1 makes an official statement after discussion with IAIS, then we close or continue the issue as appropriate.